Giter Club home page Giter Club logo

speedracer's Introduction

๐Ÿ‘‹ Speedracer was a great experiment but it's not maintained anymore. Let me know if you know a good alternative that I can redirect to.

speedracer

๐ŸŽ๏ธ๐Ÿ’ฅ ............ ๐Ÿš—๐Ÿš•๐Ÿš™๐Ÿš“๐Ÿš’๐Ÿš๐Ÿšš๐ŸšŒ๐Ÿš›๐Ÿšœ

Speed Racer

Collect performance metrics for your library/application.

Build Status Build Status Say hi!


Speed Racer is a performance runner, like a test runner, but for performance ๐ŸŽ๏ธ. It runs scripts (races) in Chrome (headlessly if possible) and produces detailed traces and reports on scripting, rendering and painting.



See what's new in ๐Ÿ“ฆ 0.2.0 or what's being cooked for ๐Ÿ“ฆ 0.3.0

Installation

npm install -g speedracer

Speed Racer needs Google Chrome to run your files. It will run it headlessly if it finds a proper intallation of Canary (Mac OS X only for now).

Usage

Speed Racer comes with two commands right now:

  • run: collect performance metrics and save them.
  • display: display a summary of generated reports.

Create races

A race can be seen as a unit test. It contains a piece of code that will be profiled by Speed Racer. Under the hood, it uses Chrome DevTools protocol to drive Chrome and collect traces. Races can import es6 / commonjs modules and use most of es6 features, depending on your version of Google Chrome: es6 support

Here is an example of a file containing a race:

import race from 'speedracer'

race('my first race', () => {
  // ... stuff to profile
})

You can define as many races as you want per file, Speed Racer will collect and run them sequentially.

You can also define asynchronous races like so:

import race from 'speedracer'

race('my first async race', () =>
new Promise(resolve => {
  // ... stuff to profile
  resolve()
}))

Run races

Then you need to collect metrics!

For each race, Speed Racer will produce two artifacts:

  • a trace: a raw dump of Google Chrome tracing events, it contains a lot of detailed metrics about your race.
  • a report: a report created by Speed Racer from those events, it summarizes important metrics.

Those artifacts will be saved in the .speedracer directory by default.

To run races, type speedracer run perf/*.js or simply speedracer perf/*.js. Note that it will run all .js files in the perf directory by default, so you can omit perf/*.js if you are using this directory.

For more details, type speedracer run --help. You can browse examples here.

Display reports

Once the artifacts have been created, you can quickly display a summary report for each run. Type speedracer display to see all the reports or speedracer display .speedracer/a-file-name/* to see the reports of a specific file.

For more details, type speedracer display --help.

Go further

Speed Racer is still a baby so it does not provide advanced analysis yet, just a basic summary. But it has several goals:

  • regression testing: compare runs over time and report how it's faster/slower.
  • benchmarking: compare several races to see which is the best.
  • analysis: give precise insights of what is slow and why.
  • auditing: give advices on how to improve performance.

If you want to use Speed Racer for one of this use cases, you can leverage it and analyze the traces and reports it produces. I would be glad to receive your feedback and ideas on the subject!

Traces

Traces are json files with the .trace.gz extension. They are basically huge arrays of events produced by Google Chrome. Those events give tons of informations about the overall performance of race. Here is the detail format.

Traces can be pretty big, so they are saved gziped.

You can analyze them the way you want or load them in the Timeline/Performance tab of Chrome like so:

  1. Locate and decompress your trace:
# first you need to locate and decompress the trace

$ cd .speedracer
$ ls
text-fixtures-multiple
$ cd text-fixtures-multiple
$ ls
render-60-frames.speedracer
render-60-frames.trace.gz
search-10e4-first-primes-very-long.speedracer
search-10e4-first-primes-very-long.trace.gz
$ gunzip render-60-frames.trace.gz
  1. Load it in Chrome Devtools or use DevTools Timeline Viewer and enjoy ๐ŸŽ‰

Reports

Reports are json files with the .speedracer extension. They provide a performance summary for a given race.

Here is the format:

{
  "meta": {
    "title": "render 60 frames",
    "group": "test-fixtures-multiple",
    "id": "render-60-frames"
  },
  "profiling": {
    "categories": {
      "scripting": 13.217000007629395,
      "rendering": 11.370999991893768,
      "painting": 9.248999938368797
    },
    "events": {
      "Animation Frame Fired": 7.994999974966049,
      "Composite Layers": 7.0119999796152115,
      "Update Layer Tree": 6.503000020980835,
      "JS Frame": 5.1060000360012054,
      "Recalculate Style": 4.867999970912933,
      "Paint": 2.236999958753586,
      "Run Microtasks": 0.11599999666213989
    },
    "functions": {
      "FireAnimationFrame": 7.994999974966049,
      "CompositeLayers": 7.0119999796152115,
      "UpdateLayerTree": 6.503000020980835,
      "UpdateLayoutTree": 4.867999970912933,
      "f:render@24": 2.32600000500679,
      "Paint": 2.236999958753586,
      "f:requestAnimationFrame@": 2.1010000109672546,
      "f:ws.onmessage@24": 0.1940000057220459,
      "f:finishRace@24": 0.15600000321865082,
      "f:@": 0.1300000101327896,
      "RunMicrotasks": 0.11599999666213989,
      "f:Promise@": 0.10099999606609344,
      "f:startRace@24": 0.09800000488758087
    }
  },
  "rendering": {
    "firstPaint": 0.00805,
    "fps": {
      "mean": 60.98,
      "variance": 3.9,
      "sd": 1.97,
      "lo": 56.92,
      "hi": 63.47
    }
  }
}

You can display, analyze or compare them depending on your needs.


๐ŸŽ๏ธ๐Ÿ’ฅ ............ ๐Ÿš—๐Ÿš•๐Ÿš™๐Ÿš“๐Ÿš’๐Ÿš๐Ÿšš๐ŸšŒ๐Ÿš›๐Ÿšœ

speedracer's People

Contributors

danielruf avatar greenkeeper[bot] avatar ngryman avatar orlandohohmeier avatar wain-pc avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

speedracer's Issues

Doesn't run races

Windows 10.
Code is the high cpu example from the fixtures directory.

Speedracer finds the file in /perf and appears to be running, but then:

0 races
faster n/a
slowest n/a
No races found in perf/index.js

Thanks

unable to import other files in race files

I'm trying to import a JS file I want to test, but when I do and I run speedracer run it always says 0 races No races found in perf/DateTimePicker.perf.js.. When I remove the additional import it says 1 race.

My code:

import race from 'speedracer'
import DateTimePicker, { validateDate } from './DateTimePicker'

race('my first race', () => {
  console.log('RACEEE')
})

puppeteer

We can also use puppeteer instead of downloading Chromium manually.

Chrome Debugger could not be reached

chrome: 58
os: macOS 10.12.4

trace:

๐ŸŽ  Speed Racer v0.1.2

 โฏ Starting
   โœ– Launching headless browser
     โ†’ Chrome Debugger could not be reached.
     Starting tracing server
   Tracing
   Reporting
Error: Chrome Debugger could not be reached.
    at isDebuggerReady.then.isReady (/usr/local/lib/node_modules/speedracer/lib/launch-chrome.js:74:27)
    at process._tickCallback (internal/process/next_tick.js:109:7)

Let me know if you need more info.

Debugger connection check fails to work

Hello!
I'm running speedracer under Linux, and for some reason Chrome fails to open port 9222 for debugging if I don't use sudo. The problem is that speedracer actually outputs the issue:
chrome-launcher chrome error: Error: connect ECONNREFUSED 127.0.0.1:9222 +4ms
(at least with debug output enabled) but doesn't stop after that. Here's the sample output:

DEBUG=* speedracer run race.js

  chrome-launcher locate chrome +0ms
  chrome-launcher trying: /Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary +0ms
  chrome-launcher auto locate chrome +1ms
  chrome-launcher chrome found: /usr/bin/google-chrome +8ms
  chrome-launcher spawn chrome with flags: --headless,--remote-debugging-port=9222,--disable-gpu,--disable-translate,--disable-extensions,--disable-background-networking,--safebrowsing-disable-auto-update,--disable-sync,--metrics-recording-only,--disable-default-apps,--no-first-run,--user-data-dir=/tmp,--js-flags="--expose-gc" +1ms
  chrome-launcher check debugger ready: 1 +2ms
  chrome-launcher chrome error: Error: connect ECONNREFUSED 127.0.0.1:9222 +4ms
  chrome-launcher check debugger ready: 2 +501ms
  chrome-launcher chrome ready +1ms
  server start server +0ms
  server listening on port: 3000 +3ms
  server serving directory: /home/speedracer +0ms
  runner-server listening on port: 3001 +0ms
  driver load file race.js +38ms
  director file loaded: race.js +114ms
  director no races found: race.js +3s

I assume stopping at failures like that and outputting them in non-debug mode could potentially save some time for other users. Thanks in advance!

[RFC] Grabbing race artifacts programmatically

Currently, race creates two artifacts (a trace and a report) as files, but it's not possible to associate the race and the artifacts programmatically.

In order to automate perf testing (mostly based on reports), I'd love that the artifacts are passed as a result of the race function (in a Promise). Something like:

describe('Prime number finder', () => {
  it('should run under 10s for a 6 digits prime', () => {
    return race('prime race', () => prime(6))
        .then((report, trace) => {
            assert(report.profiling.functions.prime < 10000)
        })
  });
}) 

Note: In this use case, the first argument of the race() function is not useful, so my 2c would be to move it as second argument, and make it optional. Also, in this use case, I don't need the artifacts as files.

What do you think of this use case / syntax?

Roadmap

My ultimate goal for Speed Racer is to build a perf runner that will achieve different purposes:

  • Keep track of the overall performance of your project.
  • Quickly identify slow pieces of code.
  • Compare several implementations.

Here is the associated roadmap (subject to change):

  • Allow multiple runs per file (like a test runner): 3f8beb6
  • show command, that displays reports like 0.1.x version.
  • run command
  • test command 30e9611
    • Snapshot mechanism (much like Jest but for reports).
    • Pass if all runs are equal or faster than the previous ones.
  • compare command, compare several reports between them.
  • More metrics.

Feedback welcomed ๐ŸŽ‰


/cc @paulirish @samccone

Missing ms module

Just found this gem via npm- tried installing and running but it seems like ms is missing from package.json

Silent fail if server port is in use (EADDRINUSE)

Hi! Cool project ๐Ÿ˜Ž

When i first started to play with this, couldn't get it to work. I just got "no races found in ...".
Then i checked with debug logging, and the issue was that the default port 3000 was in use.

What do you think of defaulting to a random port using get-port ?

Are there any reason you'd want to know the port?

If not, it should at least throw an error and quit if the port is in use ๐Ÿ‘

โฏ speedracer run
  โœ”   0 races

  ๐Ÿ   fastest  n/a
      slowest  n/a

  โš    No races found in perf/index.js.

โœจ  Done in 4.02s.

With DEBUG=*

...
  server start server +1ms
  server server error: Error: listen EADDRINUSE :::3000 +3ms
  runner-server listening on port: 3001 +1ms
  driver load file perf/index.js +40ms
  director file loaded: perf/index.js +13ms
  director no races found: perf/index.js +3s

โœจ  Done in 4.06s.

I can make a pr when you know how you want to solve this :)

Documentation needed on what each metric in the report means

Hi ,

Is there any documentation you can help me with please where i can get the meaning and units of each metric that we get in the following report?

{ "meta": { "title": "render 60 frames", "group": "test-fixtures-multiple", "id": "render-60-frames" }, "profiling": { "categories": { "scripting": 13.217000007629395, "rendering": 11.370999991893768, "painting": 9.248999938368797 }, "events": { "Animation Frame Fired": 7.994999974966049, "Composite Layers": 7.0119999796152115, "Update Layer Tree": 6.503000020980835, "JS Frame": 5.1060000360012054, "Recalculate Style": 4.867999970912933, "Paint": 2.236999958753586, "Run Microtasks": 0.11599999666213989 }, "functions": { "FireAnimationFrame": 7.994999974966049, "CompositeLayers": 7.0119999796152115, "UpdateLayerTree": 6.503000020980835, "UpdateLayoutTree": 4.867999970912933, "f:render@24": 2.32600000500679, "Paint": 2.236999958753586, "f:requestAnimationFrame@": 2.1010000109672546, "f:ws.onmessage@24": 0.1940000057220459, "f:finishRace@24": 0.15600000321865082, "f:@": 0.1300000101327896, "RunMicrotasks": 0.11599999666213989, "f:Promise@": 0.10099999606609344, "f:startRace@24": 0.09800000488758087 } }, "rendering": { "firstPaint": 0.00805, "fps": { "mean": 60.98, "variance": 3.9, "sd": 1.97, "lo": 56.92, "hi": 63.47 } } }

Speedracer fails to run races

I've tried all of the provided examples but speedracer always fails to run races with a message similar to the following:

  โœ”   0 races

  ๐Ÿ   fastest  n/a
       slowest  n/a

  โš    No races found in high-cpu.js.

Performance runner

0.3.0 will offer a way to integrate SR in a classic development workflow. The aim of this version is to provide an easy way to track any performance regression.

Snapshots

A new command will be introduced: test. It will act like the run command but for a different purpose: see if all races are as fast or faster than before.

In order to achieve this, we'll introduce the concept of snapshots for SR. Like test runners offering a snapshot feature, SR will create and save a report snapshot for each race on the first run. Subsequent runs will compare new reports with their snapshot and display appropriate results.

If the new report is as fast or faster, the test is considered as passed. If not, a diff will be displayed and the test command will return a non-zero error code.

Visual diff

The visual diff should be clear enough to highlight the parts where the run has been slower. Because reports are a DSL, we'll use an in-house diffing/display mechanism instead of a general purpose one (i.e. jest-snapshot).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.