Benchmarks

Jan 31, 2013 at 4:05 PM
I'd love to see some benchmarks pitting this against Javascript.NET, Jurassic, etc. For reference http://jurassic.codeplex.com/wikipage?title=Benchmarks&referringTitle=Home

These type of silly figures tend to be very useful when it comes to convincing a team to use a product :)
Coordinator
Jan 31, 2013 at 4:22 PM
Edited Aug 3, 2013 at 4:21 PM
If you use ClearScript's V8ScriptEngine class, you get V8 underneath, and since SunSpider tests the core JavaScript language only, you should get Chrome-like results. The results for JScriptEngine should be very different. Of course, this is all theory until someone tries it :)
Coordinator
Feb 1, 2013 at 7:41 PM
Edited Aug 3, 2013 at 4:21 PM
And again we learn that theory must always go hand-in-hand with practice :)

We created a small ClearScript-based application that downloads and runs the SunSpider benchmark suite, mocking out just enough of the HTML DOM to let it run to completion.

We found that ClearScript performs significantly worse than Chrome running on the same PC. Part of the reason is that ClearScript instructs V8 to call back to the host every time the script accesses a global object. This has significant negative impact on the performance of some SunSpider benchmarks, particularly bitops-bitwise-and, which performs no less than 600,000 global accesses per run.

ClearScript uses this callback mechanism to implement the HostItemFlags.GlobalMembers behavior, but we believe that a compatible fix is possible. Stay tuned!
Coordinator
Feb 2, 2013 at 6:37 PM
Edited Feb 3, 2013 at 11:19 AM
We've pushed out an update that gives hosts the ability to disable support for HostItemFlags.GlobalMembers, and includes a new benchmark application and additional performance improvements.

Here are some SunSpider results for ClearScript and Chrome on a Core i7 920 PC:

ClearScript/JScript: 4394.3ms +/- 0.1% [details]
ClearScript/V8 (default): 1076.2ms +/- 0.3% [details]
ClearScript/V8 (no GlobalMembers support): 259.5ms +/- 1.3% [details]
Chrome 24: 188.1ms +/- 1.5% [details]

Notes:
  • These results are for 32-bit versions of all script engines.
  • The JScript results are for "classic" JScript; ClearScript does not support Chakra.
  • 64-bit JScript crashes on the "controlflow-recursive" test.
Jan 6, 2014 at 3:36 PM
Edited Jan 6, 2014 at 3:38 PM
What is GlobalMembers support?

When I ran the test below, I only see a small performance increase after disabling global members. Also, I compared it with Javascript.NET and the differences is huge. But if I compiled the script before running it, it is almost 2~3X of Javascript.NET. (Our application needs to update hundreds if not thousands of items when a property is modified, so measuring performance like this is very important)

Script is "Math.random();"

Clearscript in 942
Clearscript with DisableGlobalMembers in 930
Javascript.NET in 21

//After compiled
Clearscript in 57
Clearscript with DisableGlobalMembers in 53
Javascript.NET in 27
public static void TestClear(string script, bool disableGlobal)
{
    var engine = new V8ScriptEngine(disableGlobal ? V8ScriptEngineFlags.DisableGlobalMembers : V8ScriptEngineFlags.None);
    
    var sw = new Stopwatch();
    sw.Start();

    //var compiled = engine.Compile(script);

    for (int i = 0; i < 10000; i++)
    {
        //engine.Evaluate(compiled);
        engine.Evaluate(script);
    }

    sw.Stop();
    Console.WriteLine("Clearscript" + (disableGlobal ? " with DisableGlobalMembers" : "") + " in " + sw.ElapsedMilliseconds);
}
Jan 6, 2014 at 5:41 PM
I notice the bottleneck is here
var uniqueName = documentNameManager.GetUniqueName(documentName, "Script Document");
It is generating a new name even the same script is being executed twice. If I changed it to
uniqueName = code.GetHashCode().ToString() // Just an example to create a unique code based on the script
If the same script is being executed twice, it will run extremely fast and efficient (Should have the same performance as compiling the script).

I think Javascript.NET has similar optimization because the first code below runs almost 8~10x slower than the second one
for (int i = 0; i < 10000; i++)
{
      engine.Run("Math.random() + " + i);
}

for (int i = 0; i < 10000; i++)
{
      engine.Run("Math.random()");
}
Coordinator
Jan 6, 2014 at 5:49 PM
Hi ravetam!

GlobalMembers is an old Windows script engine feature that we added to V8ScriptEngine for API compatibility. It allows you to expose a host object so that its members (rather than the object itself) appear as global properties within the script environment. Support for this feature is expensive on V8 even when it remains unused, so the only way to avoid the performance hit is to disable it completely.

Now, as for your performance numbers...

V8 executes "Math.random()" extremely quickly; if you move the loop into script code, you'll see that it can run 10,000 iterations in just a couple of milliseconds. In fact, compiling such a script probably makes little or no difference in terms of execution speed. What you're really measuring there is the overhead of invoking V8 via ClearScript vs. JavaScript.NET.

We took a look and were surprised to find that generating a unique script name represents most of ClearScript's overhead. When you compile the script you do this once as opposed to 10,000 times, and that explains the improvement over raw script execution. We'll definitely take a closer look at this.

The remaining difference - approximately twice the invocation overhead for ClearScript vs. JavaScript.NET - is likely due to differences in library design. JavaScript.NET is definitely lighter, and its single-assembly design is an advantage.

Thanks for bringing this to our attention!
Jan 7, 2014 at 2:54 PM
Thanks for the explanation on GlobalMembers.

I just realize that the script still works even no unique name is generated (Using a same document name). Is there any pitfall or anything that I should be aware of if i plan on using that?
Coordinator
Jan 7, 2014 at 3:26 PM
Heh, it looks like we were investigating simultaneously :)

It is generating a new name even the same script is being executed twice.

Right. When you call a method that takes the code as a string, ClearScript considers it a new script. It doesn't cache the code for subsequent lookup, deferring to the underlying script engine for that sort of optimization. It might benefit from doing what you suggest, but the code cache could also explode under some usage patterns.

By the way, the unique script name is just metadata that shows up in script debuggers; it doesn't affect script execution. JavaScript.NET doesn't bother with script names at all, probably because it doesn't support script debugging.

If the same script is being executed twice, it will run extremely fast and efficient (Should have the same performance as compiling the script).

If that's how it works, then V8 does it all under the covers. Neither ClearScript nor JavaScript.NET look up previously compiled scripts. When given a code string to execute, they both call V8's compilation method and then its execution method.

I think Javascript.NET has similar optimization because the first code below runs almost 8~10x slower than the second one

It's possible that V8 does what you suggest behind the scenes, but it could also be that V8 optimizes "Math.random()" completely away when it detects that your script doesn't use the return value.

Cheers!
Coordinator
Jan 7, 2014 at 3:32 PM
I just realize that the script still works even no unique name is generated (Using a same document name). Is there any pitfall or anything that I should be aware of if i plan on using that?

Only that you might confuse script debuggers. Come to think of it, we've never tested debugging support without unique script names.
Jan 7, 2014 at 3:41 PM
Edited Jan 7, 2014 at 3:41 PM
Heh, it looks like we were investigating simultaneously :)
Yeah, love the extension method and ability to add host type (enum, static class) in Clearscript. Just trying to make it work.
Only that you might confuse script debuggers. Come to think of it, we've never tested debugging support without unique script names.
Good point but maybe you can consider only turning on unique script name generation when this flag is present? V8ScriptEngineFlags.EnableDebugging
Coordinator
Jan 8, 2014 at 1:46 PM
Good point but maybe you can consider only turning on unique script name generation when this flag is present? V8ScriptEngineFlags.EnableDebugging

We'd prefer to keep the unique names for now, as they also show up in script error stack traces, which V8 makes available even when debugging is disabled. With some improvements we're making in the next release, generating a unique name takes approximately 600 nanoseconds on an old Core i7 920 PC; this is unlikely to affect real-world script execution times significantly. Besides, if you're executing a script many times, you can compile it to eliminate the overhead.