High performance & throughput, asynchronous execution?

Oct 5, 2013 at 9:54 PM
I need to execute as many scripts as possible (thousands of different user scripts, not easily cachable always). Is there anything to be gained from using TPL to asynchronously execute scripts? Since the work is CPU-bound, im not seeing much performance gain from this strategy. In my scenario I am processing web requests with the V8Runtime. I do not want to block the threads because this can quickly consume the ASP.NET thread pool. Any recommendations on squeezing the highest throughput from Clearscript in this type of scenario? Currently im seeing around 40 executions/second on a single core due to how expensive it is to recreate the v8 context. Thanks!
Coordinator
Oct 7, 2013 at 3:11 PM
Hello, joshrmt!

You appear to have answered part of your question. If blocking ASP.NET request threads is undesirable, then you must use TPL, async/await, or some other technique to process your requests asynchronously. The current thinking is that asynchrony is the way to achieve scale on the server. It's at the core of Node.js, for example.

Of course, asynchrony can't be your final answer if your request processing is CPU-bound. A thread stuck in a loop is just as unavailable as one waiting for I/O, just as likely to force crippling expansion of the thread pool, and particularly devastating on a server with a small core count. If your request processing requires heavy CPU activity, the only way to keep your web server responsive is to offload that processing, probably to a farm of application servers. That'll increase your minimum latency and your costs, but as far as we can tell it's the only way to improve your throughput.

Bringing this back to ClearScript, what do you mean by "recreate the V8 context"? What kind of setup does your application require for script execution?
Oct 7, 2013 at 9:41 PM
Hey there, thanks for the response. I have been creating a platform similar to Node.js. The main problem I was having is due to how expensive it is to instantiate/dispose the V8 contexts. So now I have started reusing contexts rather than disposing them. However, I am running untrusted user code in these contexts, and was looking for a clean way to 'recycle' the state of the context without having to actually dispose of it and create a new one.

I see that Engine.Script seems to hold all of the top level objects, so instead of disposing the context, I am iterating through the Engine.Script properties and deleting them. I believe this should create a 'clean slate' for the next execution. My concern is that I may not be cleaning up everything properly. Do you have any advice on cleaning up the contexts so that other script evaluations do not have 'leftovers'? Alternately, is there a higher performance technique to instantiate/dispose contexts? The 20 millisecond overhead is far too expensive to perform on a 'per-request' basis.
Coordinator
Oct 8, 2013 at 3:43 AM
I see that Engine.Script seems to hold all of the top level objects, so instead of disposing the context, I am iterating through the Engine.Script properties and deleting them.

Can you provide sample code that demonstrates your technique for iterating and deleting the global properties?
Oct 8, 2013 at 11:55 AM
Edited Oct 8, 2013 at 11:55 AM
Sure, this is what im doing:
private static void Cleanup(V8ScriptEngine engine) {
            var data = engine.Script as DynamicObject;
            var cleanup = new StringBuilder();

            foreach (var item in data.GetDynamicMemberNames()) {
                if (item != "EngineInternal") {
                    cleanup.Append("delete " + item + ";");
                }
            }

            engine.Execute(cleanup.ToString());
            engine.CollectGarbage(true);
}
Coordinator
Oct 8, 2013 at 4:26 PM
What you're doing is fast; the only thing that might make it faster is to do the cleanup via compiled script rather than generating delete statements that have to be compiled every time. Also, why force garbage collection?

We've run some tests, and your technique is much faster than instantiating V8ScriptEngine or invoking V8Runtime.CreateScriptEngine(); the latter is significantly faster than the former, but not nearly as fast as reusing the context.

However, your approach doesn't guarantee a clean context. The global object is not the only place where a script may have hung its laundry. For example, it may have added data and methods to a built-in prototype.

We'd also prefer it if ClearScript users never had to worry about EngineInternal. ClearScript really should provide a method for resetting a context or a faster way to construct a new one. We'll look into it; thanks for pointing it out!
Oct 8, 2013 at 5:07 PM
Edited Oct 8, 2013 at 5:16 PM
Ahh, im not so much worried about the speed at this point as I am about guaranteeing a clean context. You make a great point about overriding the built-in prototypes. I enjoyed the performance gain with V8Runtime.CreateScriptEngine(), but it is not viable because the resource constraints are not isolated per context. Since I am using multi-threaded access to many contexts at once, the resource constraints being exceeded will effect all of the contexts that share the same V8Runtime.

It is absolutely mandatory that I find a fast way to get a clean context as quickly as possible, here are some ideas:
  1. Is there a way I can purge all of the built-in objects and then re-build them? Is there even a way I can 'get at' the built-in objects, or are they part of V8? For example, the Math object.
  2. Is there a way I can intercept calls to access the built-in objects? Perhaps I could override them by adding my own objects in their place.
  3. My last option (it will never come to this!) is to use something like 'js.js' (https://github.com/jterrace/js.js/) to create a Javascript interpreter within the context. Although the execution would obviously be slower (200 times slower), it would still be very fast because my bottleneck is in the v8 context creation, not in the script execution.
I was forcing garbage collection because I am worried that future users of the context may trigger the resource constraints if everything is not cleaned up each iteration.

Thank you for all your help so far! The level of support is amazing.
Coordinator
Oct 8, 2013 at 9:01 PM
Hi joshrmt,

I enjoyed the performance gain with V8Runtime.CreateScriptEngine(), but it is not viable because the resource constraints are not isolated per context. Since I am using multi-threaded access to many contexts at once, the resource constraints being exceeded will effect all of the contexts that share the same V8Runtime.

We're not suggesting that you share a single runtime among all your contexts; that would disable multithreaded access completely. What we're saying is that, as an alternative to cleaning up a context for reuse, you could dispose it and create a new context within the same runtime. Doing so is (currently) much slower, but (a) it guarantees you a clean context, and (b) it's faster than invoking the V8ScriptEngine constructor, which creates both a context and a runtime.

Is there a way I can purge all of the built-in objects and then re-build them?

Unfortunately V8 does not seem to provide a way to "reset" a context in this manner. Creating a new context seems to be the only way, and this is where ClearScript could probably do a bit better than it does today. We're looking into it.

Is there even a way I can 'get at' the built-in objects, or are they part of V8? For example, the Math object.

The built-ins are not enumerable, but you can still access them via global properties. For example, engine.Script.Math.

Is there a way I can intercept calls to access the built-in objects? Perhaps I could override them by adding my own objects in their place.

Yes, you can delete or overwrite the built-ins with your own objects. However, some things might be difficult to duplicate exactly.

​I was forcing garbage collection because I am worried that future users of the context may trigger the resource constraints if everything is not cleaned up each iteration.

That shouldn't be necessary, as V8 automatically triggers garbage collection when it encounters memory pressure.
Oct 8, 2013 at 9:51 PM
Ahh, I didn't realize that V8ScriptEngine constructor worked like that. I hadn't thought to create a bunch of runtimes and reuse them to create engine contexts. I am going to do some benchmarks on this, thanks! Also, thanks for the advice on the garbage collection. Perhaps with these adjustments things will be fast enough. I will let you know.