Mozilla, Google and Apple are building a new browser benchmark

Mozilla, Google and Apple have announced that they are working together to build a new browser benchmark. The service will be called Speedometer 3.

You may have heard of Speedometer, which is hosted on Browserbench.org, it runs a series of simulated tests to measure the responsiveness of web apps. It's worth noting that these are computational tasks, and results often differ based on the platform (operating system and browser) that you use. The upcoming version of the benchmark is designed to measure real-world browser performance on the web.

Mozilla, Google and Apple are building a new browser benchmark

The Speedometer 3.0 project has 3 participants: Blink/V8, Gecko/SpiderMonkey, and WebKit/JavaScriptCore, i.e. the underlying cores of the browser made by Google, Mozilla, and Apple. Interestingly, Microsoft is not a part of this cross-industry collaborative effort, but given that its Edge browser relies on Chromium's Blink and V8 engines, perhaps the company's absence at the table won't matter much.

Engadget points out that the Speedometer's Joint Governance Policy will require a nod of approval from at least two of the participating projects, for any non-trivial change made to the benchmark's code. The current iteration of the tool was developed by Apple's WebKit team. The problem with this is, it wasn't designed to be a universal tool. As noted above, the major browsers in the market have different rendering engines, and script engines. So, it is quite difficult to benchmark their performance accurately.

We have seen browser makers occasionally run a benchmark and claim that their program is superior to the others. But, what is this based on? That's the real problem, these numbers don't truly reflect the actual performance, and merely act as a marketing ploy to convince people their browser is ahead of the rest.

That's why the next version of the benchmark, Speedometer 3.0, is being developed co-developed by the three major players, to ensure a certain level of fairness, or a standard if you will. The browser makers will share the code for the benchmark, and non-trivial changes made to it will include anything that impacts the result such as test cases, test runners, bug fixes, and the appearance of the tool.

Trivial changes such as modifications to the documentation on the other hand will not require the approval of the other parties. All major changes that are made to the project will require a consensus, i.e., everyone must agree to the decision.

The internet is an ever-evolving space, older technologies are replaced by new standards, which are faster, consume less resources, etc. Benchmarks need to catch up with those, relying on an old version (based on old standards) will deliver inconsistent, and inaccurate results.

This collab project will allow developers to understand how to optimize the performance of their browser, for example, the upcoming benchmark will represent real-world workloads such as various JavaScript frameworks. This in turn could help improve the cross-browser and cross-platform performance, and in a sense, have a greater impact on the internet itself.

Source