Google adds Octane to its benchmark suite

A new browser benchmark suite is cruising down Main Street: Google Octane, which updates the V8 benchmark by adding more tests.

Chrome 21's score on Google Octane v1, which replaces the V8 benchmark.

(Credit: Screenshot by Seth Rosenblatt/CNET)

Although speed is a major factor in browser choice, many people don't want to know why their favorite browser is fast -- they just care that it is. But the standards and tests used to determine how we measure a browser's speed can be varied, so Google has set out to further define the playing field.

This doesn't appear to be a case of the tech giant stomping all over open standards, though. Octane v1 is a revamp of Google's V8 benchmark and adds five tests to the eight current ones, the company said today in a blog post announcing Octane.

"The V8 benchmark suite has served us well for the past three plus years, but something we noticed is that the Web is advancing much faster than imagined," Stefano Cazzulani, Google's project manager for V8, said on the phone with CNET from Germany. V8 is Google's custom JavaScript engine that it has bolted to the Chrome browsers' HTML-rendering WebKit engine.

Octane is the evolution of V8, said Cazzulani. "V8 will still be there, still interesting, but the new [tests] are where we are focusing for today's users. We'd like everybody [who benchmarks browsers] to use that."

Octane's new JavaScript tests are open-source -- so anybody can access them -- and look at multiple JavaScript characteristics, including math and bit operations, support for future computer language features, emulation, JavaScript parsing and compilation, floating point math, properties containing doubles, and accessor properties.

The five new tests in Octane are pdf.js, Firefox maker Mozilla's PDF reader in JavaScript that measures decoding and interpretation time; Mandreel, which runs the 3D Bullet physics engine ported from C++; GB Emulator, which emulates the GameBoy architecture and runs a 3D simulation; Code Loading to measure how quickly a JavaScript engine starts running code after loading a large JavaScript program like a social widget; and the widely used 2D physics game engine Box2DWeb.

As with the V8 benchmark, the higher the score, the better the result. My informal testing put Chrome 21 at an Octane version 1 score of 4,945 and a V8 version 7 score of 4,221. Firefox 14 hit 3,090 on Octane, and 2,404 on V8.

Because of the JavaScript code for typed arrays, a mechanism for accessing raw binary data, and how some browsers can't handle it, Octane won't work in all browsers. Cazzulani said that Internet Explorer 9 and Safari will have problems with pdf.js and Mandreel, both of which use typed arrays. Internet Explorer ought to work fine, he said, as will many modern mobile browsers.

Cazzulani also noted that older mobile devices will struggle with the test, although in casual tests I noticed that to be true with more than just mobile devices. Both Firefox 15 beta for Android and Opera 12 for Android struggled to finish the test, while Chrome for Android cruised through it. There are several reasons why this could be, including that the version of the test I was running still had unfixed bugs, or that there are still problems with the test implementation itself. Since the test is new, it's hard to say.

Cazzulani was honest about the limitations of the test, too. He said that Octane itself has revealed areas for improvement for the V8 team, although he refused to say more about such areas. "We've seen that the other guys are doing a great job. Of course, we optimize for the V8 every day, but we need something to guide us in the future. We have work to do."

Updated 12:25 p.m. PT: Google has released a chart showing browser compatibility with Octane. For desktops, Safari v5.1.7 is on the list, but Internet Explorer 9 is not. On mobile, both Chrome and Safari on iOS fail because they share the restrictions associated with the older version of WebKit.

CNET Top 5
Companies Apple could buy with their billions
Apple's sitting on a massive pile of cash. Here are five interesting ways they could spend it.
Play Video
 

Member Comments