Highly concurrent server design

Mar 20, 2014 at 6:33 PM
Edited Mar 20, 2014 at 9:31 PM
What do you think about this setup for a highly concurrent server scenario.

To give the discussion some context, let us suppose that this server lets people browse a hierarchical filesystem. Let us also suppose that this server lets administrators setup javascript to calculate properties on-demand, so as users click on files or directories, in addition to the normal properties they also see the output of each configured javascript property.
  1. One V8Runtime instance
  2. Each script is compiled and stored in an immutable hashtable. If the script associated to a property is modified, then a new script is compiled and a new mutated hashtable is installed with Interlocked.CompareExchange. The next request to come in will thus see the new list of script definitions to choose from.
  3. When a request comes in on a worker thread that has no existing V8ScriptEngine instance assigned, a new instance will be created from the V8Runtime.CreateScriptEngine method. The script to be executed is located in the hashtable and executed. The V8ScriptEngine instance will be kept around, associated to that thread.
a) Are there any issues with this setup? The data being passed in to the individual scripts is always immutable, so there is no problem with shared state there. The goal is to be completely lock-free if possible and to support high concurrency.

b) The V8Runtime may eventually end up with a lot of junk old scripts presumably, but assuming the scripts don't change very often that shouldn't be significant. Is it possible to eventually allow the compiled script to be collected if nothing has a reference to it anymore?

c) The scripts themselves are created by trusted administrators, so if they want to junk up the environment by smashing all the prototypes and replacing Math with an object that always vends NaNs then that's their fault :) Presumably if we needed a higher safety mode we could dump the V8ScriptEngine and create a new instance, correct?

d) A big question is whether the compiled script can be re-used among V8ScriptEngine instances simultaneously, or if there are thread issues or locks involved. Could someone mutate the compiled script and introduce threading issues?

e) Another question relates to resource usage... are the limits enforced on the entire runtime or on each V8ScriptEngine instance?

f) Under what scenario would you even want multiple V8Runtime instances?



All opinions welcome! I think it would be awesome (and cut down on questions) to have some scenarios like this laid out and on the documentation page here on the site; then people could choose a design that matches their goals.
Coordinator
Mar 21, 2014 at 5:10 AM
Edited Mar 21, 2014 at 5:37 PM
Greetings, xenadu!

Your application sounds very interesting!

One V8Runtime instance [...]
Under what scenario would you even want multiple V8Runtime instances?

Actually, without multiple V8 runtime instances there can be no concurrency. V8 runtimes are not thread-safe and explicitly block multithreaded access.

When a request comes in on a worker thread that has no existing V8ScriptEngine instance assigned, a new instance will be created from the V8Runtime.CreateScriptEngine method.

That sounds good, but again, in order to support concurrent script execution, your worker threads must not share V8 runtimes. This also means that they can't share compiled scripts.

Is it possible to eventually allow the compiled script to be collected if nothing has a reference to it anymore?

Sure. A V8 compiled script will eventually be garbage-collected if it's no longer referenced. At that time it'll release its unmanaged counterpart, making it available to V8's garbage collector.

Presumably if we needed a higher safety mode we could dump the V8ScriptEngine and create a new instance, correct?

Yes, and this is why it sometimes makes sense to instantiate V8Runtime and call V8Runtime.CreateScriptEngine() even if you don't plan to share the runtime. This way, if you need a clean execution environment, you can spin up a new engine instance efficiently and reuse previously compiled scripts.

A big question is whether the compiled script can be re-used among V8ScriptEngine instances simultaneously, or if there are thread issues or locks involved. Could someone mutate the compiled script and introduce threading issues

A V8 compiled script is bound to a single V8 runtime instance. It is therefore incapable of concurrent execution. It is also immutable. Its advantage over plain-text JavaScript is that it can be re-executed more efficiently and reused across engines that share the runtime.

Another question relates to resource usage... are the limits enforced on the entire runtime or on each V8ScriptEngine instance

V8 resource constraints are applied at the runtime level, but nothing prevents you from creating a separate runtime for a given engine instance. In fact, that's exactly what happens when you invoke a V8ScriptEngine constructor.

Thanks for your questions, and good luck!
Apr 26, 2014 at 7:36 PM
After prototyping and testing ClearScript, it turns out not to be a good fit for us. Thinking about it, I realize it should have been obvious to me that the goals of the project are far different from our scenario. Neither is good or bad, just different.


Just to document this for others: If you don't pass in any parameters nor make any calls back to .Net from JS, ClearScript using V8 is much faster while in JS land, twice as fast as IronJS in my simple tests. I'm sure it would be even faster if you were using even moderately complicated JS code. This is a great scenario for using ClearScript!

However our object model deals with very large datasets with very complex behavior. It is entirely impractical to "lob objects" across the V8 boundary to JS. That means we end up with a lot of calls being proxied back and forth between .Net and V8. That would be bad enough but the behavior of dynamic is also slow and the primary use for me is getting a delegate out that will invoke some JS when called, meaning every invocation goes through dynamic; in some cases I can be making this transition millions of times, and the average JS script itself probably makes 20-50 calls back into .Net, so the total boundary transitions are a quarter of a billion in the worst case. (Basically I tell my customers: "this is what your JS script looks like: function theScript(param1, param2, param3, param4, param5) { }; fill in the body and return me an [Array, string, number, bool]", then I invoke theScript() where required, except there may be dozens or hundreds of these for various scenarios.)

I tried various schemes and benchmarked against IronJS and IronJS was just way faster - by an order of magnitude or even multiple orders of magnitude in some cases. I tried pre-registering my object type and using attributes to control the proxying behavior (to avoid the reflection hit on every call) but it just couldn't get close to closing the gap. The more .Net objects I pass in to the JS function, the worse the gap. The more calls JS makes back to .Net, the worse the gap.

To use V8 and do what I want to do would require a different approach, one that allowed fast function calls between the two worlds, and that allowed me to pre-register all types during setup so there is no reflection and no use of dynamic. Obviously that is far different from the goals of the ClearScript project!


I do want to say thank-you to whoever the anonymous Microsoft-ians are who work on this project. It is much appreciated, even though it isn't useful for my scenario.


Side note: I wish Microsoft would put some resources into making JS a first-class citizen for these scenarios. If I want to host a JS runtime and compile JS into "native" methods for that platform, on Java I have Rhino/Nashorn that "compile" JS into JVM byte code (Nashorn is very much like IronJS actually). On iOS/Mac, I have JavaScriptCore, though Objective-C is native already. But for .Net environments, there aren't any good options and none supported, even in the form of open-source funding. Unfortunately the IronJS project is dead since Fredrik hasn't been making commits in over a year; can't blame him, working for free after all! I may have to block out some time to learn F# and work on it.
Coordinator
Apr 28, 2014 at 2:50 AM
Hello xenadu,

It does sound like ClearScript isn't quite what you're looking for, as it heavily favors interop simplicity over performance. If reflection is a deal breaker, then ClearScript is definitely not for you.

Thanks for giving ClearScript a shot, and good luck!
May 9, 2014 at 9:39 AM
Hello xenadu,

I have similar scenario like you except the scripts are not 'safe' in my case. Performance is also important in my case so I did some profiling and I created some performance improvements of the ClearScript engine. These improvements have very good performance impact on my application and I think, they can help also in your case. I've uploaded them to the 'PerformanceImprovements' fork on this server. I don't know if some of the improvements are merged or rewritten to the main trunk (they were commited on the 24th of Feb.).
It will be very helpfull, if you have some time and you will execute your tests also on the improved version. Would be nice to have results from independent source.