webkit / motionmark Goto Github PK

MotionMark Browser Benchmark

License: Other

HTML 19.04% CSS 5.49% JavaScript 75.47%

motionmark's Introduction

WebKit

WebKit is a cross-platform web browser engine. On iOS and macOS, it powers Safari, Mail, Apple Books, and many other applications. For more information about WebKit, see the WebKit project website.

Trying the Latest

On macOS, download Safari Technology Preview to test the latest version of WebKit. On Linux, download Epiphany Technology Preview. On Windows, you'll have to build it yourself.

Reporting Bugs

Search WebKit Bugzilla to see if there is an existing report for the bug you've encountered.
Create a Bugzilla account to report bugs (and comment on them) if you haven't done so already.
File a bug in accordance with our guidelines.

Once your bug is filed, you will receive email when it is updated at each stage in the bug life cycle. After the bug is considered fixed, you may be asked to download the latest nightly and confirm that the fix works for you.

Getting the Code

Run the following command to clone WebKit's Git repository:

git clone https://github.com/WebKit/WebKit.git WebKit

You can enable git fsmonitor to make many git commands faster (such as git status) with git config core.fsmonitor true

Building WebKit

Building for Apple platforms

Install Xcode and its command line tools if you haven't done so already:

Install Xcode Get Xcode from https://developer.apple.com/downloads. To build WebKit for OS X, Xcode 5.1.1 or later is required. To build WebKit for iOS Simulator, Xcode 7 or later is required.
Install the Xcode Command Line Tools In Terminal, run the command: xcode-select --install

Run the following command to build a macOS debug build with debugging symbols and assertions:

Tools/Scripts/build-webkit --debug

For performance testing, and other purposes, use --release instead.

Embedded Builds

To build for an embedded platform like iOS, tvOS, or watchOS, pass a platform argument to build-webkit.

For example, to build a debug build with debugging symbols and assertions for embedded simulators:

Tools/Scripts/build-webkit --debug --<platform>-simulator

or embedded devices:

Tools/Scripts/build-webkit --debug --<platform>-device

where platform is ios, tvos or watchos.

Using Xcode

You can open WebKit.xcworkspace to build and debug WebKit within Xcode. Select the "Everything up to WebKit + Tools" scheme to build the entire project.

If you don't use a custom build location in Xcode preferences, you have to update the workspace settings to use WebKitBuild directory. In menu bar, choose File > Workspace Settings, then click the Advanced button, select "Custom", "Relative to Workspace", and enter WebKitBuild for both Products and Intermediates.

Building the GTK Port

For production builds:

cmake -DPORT=GTK -DCMAKE_BUILD_TYPE=RelWithDebInfo -GNinja
ninja
sudo ninja install

For development builds:

Tools/gtk/install-dependencies
Tools/Scripts/update-webkitgtk-libs
Tools/Scripts/build-webkit --gtk --debug

For more information on building WebKitGTK, see the wiki page.

Building the WPE Port

For production builds:

cmake -DPORT=WPE -DCMAKE_BUILD_TYPE=RelWithDebInfo -GNinja
ninja
sudo ninja install

For development builds:

Tools/wpe/install-dependencies
Tools/Scripts/update-webkitwpe-libs
Tools/Scripts/build-webkit --wpe --debug

Building Windows Port

For building WebKit on Windows, see the WebKit on Windows page.

Running WebKit

With Safari and Other macOS Applications

Run the following command to launch Safari with your local build of WebKit:

Tools/Scripts/run-safari --debug

The run-safari script sets the DYLD_FRAMEWORK_PATH environment variable to point to your build products, and then launches /Applications/Safari.app. DYLD_FRAMEWORK_PATH tells the system loader to prefer your build products over the frameworks installed in /System/Library/Frameworks.

To run other applications with your local build of WebKit, run the following command:

Tools/Scripts/run-webkit-app <application-path>

iOS Simulator

Run the following command to launch iOS simulator with your local build of WebKit:

run-safari --debug --ios-simulator

In both cases, if you have built release builds instead, use --release instead of --debug.

Linux Ports

If you have a development build, you can use the run-minibrowser script, e.g.:

run-minibrowser --debug --wpe

Pass one of --gtk, --jsc-only, or --wpe to indicate the port to use.

Contribute

Congratulations! You’re up and running. Now you can begin coding in WebKit and contribute your fixes and new features to the project. For details on submitting your code to the project, read Contributing Code.

motionmark's People

Contributors

Stargazers

Watchers

Forkers

jrmuizel mozilla bgrins yiyix sky2 cxslucyfer smfr yudshj vmiura servetus zachernuk kaidokert dynamis shallawa camillobruni seanpm2001 laszlogombos

motionmark's Issues

"Images" test seems to not do the same thing in Safari as other browsers

Here's a screenshot with Firefox on left, Safari on right - loading https://browserbench.org/MotionMark1.2/developer.html?warmup-length=2000&warmup-frame-count=30&first-frame-minimum-length=0&test-interval=30&display=minimal&tiles=big&controller=ramp&frame-rate=50&time-measurement=performance&suite-name=MotionMark&test-name=Images

Firefox has some kind of fade out / dithering effect for the images when they disappear that does not appear to happen in Safari. Chrome's behavior matches Firefox.

images-recording-compressed.mov

Remove the "tiles" option

The "tiles" option (big tiles or 512x512 tiles) is a WebKit-specific thing that should be removed.

The 'flat' regression profile overestimates results

On the Multiply test on Chrome running with default settings we sometimes see unusually high scores. The score displays somewhat binary behavior, typically scoring ~3700 but sometimes scoring ~5800 or more. I believe the ~5800 scores are bogus.

Looking into the cause of this variation, I believe the 'Strings.json.profiles.flat' profile gives a poor regression fit for noisy samples.

Flat profile:

Slope profile:

I didn't dig thoroughly into the regression code, but I believe the 'flat' mode calculates the 'error' in a way that can't be directly compared to that of 'slope' profile, and in some cases the 'flat' is picked as a better fit despite not being so.

Forcing 'Strings.json.profiles.slope' profile seems to stabilize our results at the ~3700 range which I believe is more correct.

Refactor MotionMark code

The benchmark workflow and the score calculation are very tightly coupled. We need to make the calculation of the score separate such that experimenting and hacking it is easier.

The regression calculation has some bugs. We need to fix these bugs and we need to follow the algorithm of the two segments regression since its correctness is proved.

The variance of the samples is filtered in many ways but still the end result is not very satisfying for all the browsers. We need to remove the cause of the variance rather than filtering unwanted samples.

Frame rate detection on MotionMark occasionally sees older devices as 90 fps instead of 60

The incremental frame rate average calculation gives much more weight to the last frame. So sometimes it can be wildly inaccurate. And this makes it pick the wrong predefined frame rate.

Rename the "master" directory to "core"

Avoid using "master" because it's not inclusive.

tests/core/focus.html is unused

We removed the focus test from the core suite. This file can be removed.

Developer mode runs in a small portion of the screen

Compare:
https://browserbench.org/MotionMark1.3/developer.html?warmup-length=2000&warmup-frame-count=30&first-frame-minimum-length=0&test-interval=60&display=progress-bar&tiles=big&controller=ramp&system-frame-rate=60&frame-rate=50&time-measurement=performance&suite-name=MotionMark&test-name=Paths&complexity=5093
vs the 1.2 version
https://browserbench.org/MotionMark1.2/developer.html?warmup-length=2000&warmup-frame-count=30&first-frame-minimum-length=0&test-interval=60&display=progress-bar&tiles=big&controller=ramp&system-frame-rate=60&frame-rate=50&time-measurement=performance&suite-name=MotionMark&test-name=Paths&complexity=5093

The 'Paths' test runs an inconsistent workload.

Points in the 'Paths' test have an 'isSplit' property. Initially 5% of points have isSplit = true. While the test animates, the 'isSplit' property on points is randomly toggled, eventually reaching a state where 50% of points have isSplit = true. This creates a gradual increase in the complexity of the test, as can be noticed in this screenshot running Paths on Safari Tech Preview 181 with fixed complexity 70000.

Add an index.html to the root with a redirect to MotionMark1.2/

It would be helpful to have an index.html file at the root in order to host the repo using a static file server. Rather than needing to create some kind of home page, could we add an empty html file with something that redirects to the latest version like <meta http-equiv="Refresh" content="0; URL=./MotionMark1.2/" />

There are two files called "motionmark.js" so rename one of them

There exists runner/motionmark.js and debug-runner/motionmark.js and it's really hard to keep track of which is which when editing files.

I propose renaming debug-runner/motionmark.js to debug-runner/debug-runner.js

MotionMark's multiply suite reaches max complexity on faster devices

This is basically the same as https://bugs.webkit.org/show_bug.cgi?id=240605 . More specifically, faster machines may be able to handle max complexity of the multiply suite. This can lead to high variance in the score. Additionally, on120hz displays when requestanimationframe is serviced at 120hz (as happens in chrome) a score of 0 may be given.

Remove "focus.html" from the core directory

The focus test was removed from the core suite in https://commits.webkit.org/237656@main, so remove the unused file.

A version of this test still exists as dom/focus.html.

Runs from developer.html use a targetFrameRate of 50fps, but normal runs use 60fps

options["frame-rate"] is 50 when run from developer.html, but runs from index.html use 60fps.

Warn if the user is in dark mode when running the test

Light vs. dark mode could possibly impact scores. We should probably advise people to run the benchmark in light mode.

Tests ran via the "link" feature have incorrect stage width & height

When running tests via the links generated in developer mode, the Stage element's width & height are too small. This makes running from URLs, e.g. for automated testing, behave differently than running the benchmark.

Example URL:
https://browserbench.org/MotionMark/developer.html?warmup-length=2000&warmup-frame-count=30&first-frame-minimum-length=0&test-interval=30&display=minimal&tiles=big&controller=fixed&system-frame-rate=60&frame-rate=60&time-measurement=performance&suite-name=MotionMark&test-name=CanvasArcs&complexity=1000

Example output:

Expand the grid size of the Multiply test automatically when needed

This test has been requiring enlarging the grid size manually every time a fast browser maxes out the maximum grid size.

We should be to do this automatically by enlarging the grid size and reducing the tile size when a larger complexity is needed to lower the frame rate.

Need to increase Multiply max complexity again

Some devices with smaller screens are hitting Multiply max complexity again. As a stop-gap measure before a larger redesign, let's bump the number of rows again.

Factors affecting test results

Hi, Is the final test result related to network factors? Why do the test results fluctuate so much in several tests?

Move Random methods out of Stage class

Utilities.createSubclass() can create one level of class inheritance. Utilities.extendObject() does not recursively extend the prototype of all the ancestors methods. The solution is to use ES6 classes. A step towards making Stage and its subclasses be ES6 classes is to move the Random methods out of Stage class.