[RFC] Grabbing race artifacts programmatically about speedracer HOT 5 OPEN

Interesting thought I like that, may be useful for quite a bit of use cases. However, this syntax is invalid since promise fulfillment callback can receive only one argument, so maybe it should receive result object with report and trace as properties.

from speedracer.

Yep, my bad. So it should be:

describe('Prime number finder', () => {
  it('should run under 10s for a 6 digits prime', () => {
    return race('prime race', () => prime(6))
        .then(({ report, trace }) => {
            assert(report.profiling.functions.prime < 10000)
        })
  });
}) 

from speedracer.

@fzaninotto What you are asking (perf testing) is what's coming next:

• #9
• https://github.com/ngryman/speedracer/projects/2.

Speed Racer will create snapshots of your reports, compare them and tell you what's slower/same/better. Do you think it would fulfil your needs?

To answer your question, for now SR is aimed to be a CLI tool, but 0.4.0 will expose an API so you can customize things the way you want :)

from speedracer.

I'm not a fan of snapshot testing. It's like saying: "I don't know precisely why, but it seemed to work in the past". I prefer giving precise conditions (like "it should run under 10s in my example"), that are also meaningful when dumped as error. Besides, snapshot testing forces the developer to commit large data files (the result of previous tests) in the code repository, just for tests... Not my cup of tea.

from speedracer.

@fzaninotto Well that's a point of vue for general purpose snapshot testing.

In SR context, the way I see it is more like a reference than a source of truth: "it should be around this value" (aka. report.profiling.functions.prime < 10000). Perhaps the term snapshot is misleading...

I think it makes sense in perf testing as a perf test is like a regression test: you want to make sure new features / updates do not alter perf in a bad way. And the way to test it is ... always the same. So instead of having to write imperative code again and again, you take a snapshot as reference and let SR test it for you.

That said, if you want to go further, you could do it in 2 phases:

• run SR and output reports in a temporary directory.
• use your favorite test runner to load reports and test them with an assertion library.

0.4.0 will offer an API that let you run races from anywhere, so it will be even simpler and near what you are proposing.

from speedracer.