mcollina / autocannon Goto Github PK

as titled

We're nearing a stage where this is close to a 1.0.0 release, we should do some indepth docs for new users, along with more samples.

I ask because if you do something like autocannon google.com it doesn't recognise the actual address. Simple enough to fix, just check url.indexOf('http') === 0 and if not, prepend 'http://' before it's parsed by node's internal url module.

This check would also pass for https, hence why I don't check for http://

Similar to #65.

I'm not 100% sure that it's needed.

I think it might be a good idea to avoid regressions, as this can be used a CLI tool.

A fairly common use case involves changing node versions and then profiling, it would be cool if you didn't have to reinstall

The name (or title) switch should be emitted in the reaults. We may even add the time at which it finished. This makes the json results easier to debug.

In a microservice architecture it is nice to have constant load on the systems to track p99 latencies.

This talk gives a nice real world example done by Uber. https://www.youtube.com/watch?v=kb-m2fasdDY

Hence it would be cool to have:

const instance = autocannon({
  url: 'https://url.url',
  connections: 30, 
  pipelining: 3,
  duration: 0 // for safe infinite
}, console.log)

instance.on('report', sendToAnalytics)

I installed autocannon as local module (it had some issues with global installation),
and getting js error while trying to run simple test:

$ autocannon -c 1 -d 1s http://n...
/home/alexi/autocannon/node_modules/autocannon/node_modules/progress/lib/node-progress.js:51
    if ('number' != typeof options.total) throw new Error('total required');
                                          ^

Error: total required
    at new ProgressBar (/home/alexi/autocannon/node_modules/autocannon/node_modules/progress/lib/node-progress.js:51:49)
    at track (/home/alexi/autocannon/node_modules/autocannon/lib/progressTracker.js:28:25)
    at start (/home/alexi/autocannon/node_modules/autocannon/autocannon.js:92:5)
    at Object.<anonymous> (/home/alexi/autocannon/node_modules/autocannon/autocannon.js:101:3)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Function.Module.runMain (module.js:441:10)
    at startup (node.js:139:18)

Thank you.

PS. Linux, [email protected], [email protected]

I'm building an electron app, I'm trying to use this library but I got the following error when trying to import it.

ERROR in ./app/~/autocannon/autocannon.js
         Module parse failed: /Users/ndelvalle/workspaces/muzen/app/node_modules/autocannon/autocannon.js Unexpected character '#' (1:0)
         You may need an appropriate loader to handle this file type.
         SyntaxError: Unexpected character '#' (1:0)
             at Parser.pp$4.raise (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:2221:15)
             at Parser.pp$7.getTokenFromCode (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:2756:10)
             at Parser.pp$7.readToken (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:2477:17)
             at Parser.pp$7.nextToken (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:2468:15)
             at Parser.parse (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:515:10)
             at Object.parse (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:3098:39)
             at Parser.parse (/Users/ndelvalle/workspaces/muzen/node_modules/webpack/lib/Parser.js:902:15)
             at DependenciesBlock.<anonymous> (/Users/ndelvalle/workspaces/muzen/node_modules/webpack/lib/NormalModule.js:104:16)
             at DependenciesBlock.onModuleBuild (/Users/ndelvalle/workspaces/muzen/node_modules/webpack-core/lib/NormalModuleMixin.js:310:10)
             at nextLoader (/Users/ndelvalle/workspaces/muzen/node_modules/webpack-core/lib/NormalModuleMixin.js:275:25)
             at /Users/ndelvalle/workspaces/muzen/node_modules/webpack-core/lib/NormalModuleMixin.js:259:5
             at Storage.finished (/Users/ndelvalle/workspaces/muzen/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:38:16)
             at /Users/ndelvalle/workspaces/muzen/node_modules/graceful-fs/graceful-fs.js:78:16
             at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:445:3)
          @ ./~/babel-loader!./~/vue-loader/lib/selector.js?type=script&index=0!./app/src/components/main/harass/Harass.vue 8:17-38

Removing the shebang line didn't solve it, It creates lots of warnings like:

WARNING in ./app/~/native-hdr-histogram/histogram.js
         Critical dependencies:
         6:16-37 the request of a dependency is an expression
          @ ./app/~/native-hdr-histogram/histogram.js 6:16-37

         WARNING in ./app/~/native-hdr-histogram/LICENSE
         Module parse failed: /Users/ndelvalle/workspaces/muzen/app/node_modules/native-hdr-histogram/LICENSE Unexpected token (1:21)
         You may need an appropriate loader to handle this file type.
         SyntaxError: Unexpected token (1:21)
             at Parser.pp$4.raise (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:2221:15)
             at Parser.pp.unexpected (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:603:10)
             at Parser.pp.semicolon (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:581:61)
             at Parser.pp$1.parseExpressionStatement (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:966:10)
             at Parser.pp$1.parseStatement (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:730:24)
             at Parser.pp$1.parseTopLevel (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:638:25)
             at Parser.parse (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:516:17)
             at Object.parse (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:3098:39)
             at Parser.parse (/Users/ndelvalle/workspaces/muzen/node_modules/webpack/lib/Parser.js:902:15)
             at DependenciesBlock.<anonymous> (/Users/ndelvalle/workspaces/muzen/node_modules/webpack/lib/NormalModule.js:104:16)
             at DependenciesBlock.onModuleBuild (/Users/ndelvalle/workspaces/muzen/node_modules/webpack-core/lib/NormalModuleMixin.js:310:10)
             at nextLoader (/Users/ndelvalle/workspaces/muzen/node_modules/webpack-core/lib/NormalModuleMixin.js:275:25)
             at /Users/ndelvalle/workspaces/muzen/node_modules/webpack-core/lib/NormalModuleMixin.js:259:5
             at Storage.finished (/Users/ndelvalle/workspaces/muzen/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:38:16)
             at /Users/ndelvalle/workspaces/muzen/node_modules/graceful-fs/graceful-fs.js:78:16
             at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:445:3)
          @ ./app/~/native-hdr-histogram ^\.\/.*$

         WARNING in ./app/~/native-hdr-histogram/Makefile
         Module parse failed: /Users/ndelvalle/workspaces/muzen/app/node_modules/native-hdr-histogram/Makefile Unexpected character '#' (1:0)
         You may need an appropriate loader to handle this file type.
         SyntaxError: Unexpected character '#' (1:0)
             at Parser.pp$4.raise (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:2221:15)
             at Parser.pp$7.getTokenFromCode (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:2756:10)
             at Parser.pp$7.readToken (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:2477:17)
             at Parser.pp$7.nextToken (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:2468:15)
             at Parser.parse (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:515:10)
             at Object.parse (/Users/ndelvalle/workspaces/muzen/node_modules/acorn/dist/acorn.js:3098:39)
             at Parser.parse (/Users/ndelvalle/workspaces/muzen/node_modules/webpack/lib/Parser.js:902:15)
             at DependenciesBlock.<anonymous> (/Users/ndelvalle/workspaces/muzen/node_modules/webpack/lib/NormalModule.js:104:16)
             at DependenciesBlock.onModuleBuild (/Users/ndelvalle/workspaces/muzen/node_modules/webpack-core/lib/NormalModuleMixin.js:310:10)
             at nextLoader (/Users/ndelvalle/workspaces/muzen/node_modules/webpack-core/lib/NormalModuleMixin.js:275:25)
             at /Users/ndelvalle/workspaces/muzen/node_modules/webpack-core/lib/NormalModuleMixin.js:259:5
             at Storage.finished (/Users/ndelvalle/workspaces/muzen/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:38:16)
             at /Users/ndelvalle/workspaces/muzen/node_modules/graceful-fs/graceful-fs.js:78:16
             at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:445:3)
          @ ./app/~/native-hdr-histogram ^\.\/.*$

Any ideas that could help me out using the lib in this environment?

Maybe we should use node's cluster, or node-webworker-threads. Or is a single event loop enough to drive this?

When pipelining is activated, the latency measurement is wrong, because we only maintain one startTime (https://github.com/mcollina/autocannon/blob/master/lib/myhttp.js#L109) and not one for each pipelined request.

There should be a synchronous API to change the payload if we want to, something like

const autocannon = require('autocannon')

const instance = autocannon({
    url: 'http://localhost:3000',
    connections: 1000,
    duration: 20
}, console.log)

instance.on('response', function (client, statusCode, returnBytes, responseTime) {
  client.setHeaders({ .. })
  client.body = ...
}) 

Just hangs,

• Hangs when installing via npm (any version)
• Hangs when installing locally to link (master branch)

I'm trying to get it working with Node 4 now via local, I'll ping when updated. (Although it looks to be hanging too)

Maybe we can pass in a function that returns an object, or a function that accepts a client.

What do you think @thekemkid?

Currently, you can supply the request headers and body when creating an autocannon instance. In future, I would like to see the ability to supply an array of objects with headers, body, method and path attributes. This might look like so:

autocannon({
  url: ...,
  requests: [
    {
      method: 'POST',
      headers: {Login form data}
      path: '/api/login'
    },
    {
      method: 'GET',
      path: '/api/user/me/sensitiveData',
      headers: {auth headers}
    },
    ......
  ]
}, cb)

This would allow autocannon to pre-generate the http request data, and then create connections, pass in the requests, and each connection can run through the requests array repeatedly until completion.

Because of the nature of this being a load testing tool, you can not use data returned from one request in a subsequent request. This is because requests are prebuilt and cached/passed around. This would mean you must pregenerate any needed auth tokens or w/e ahead of time and embed in the testing tool.

Hi!
I've installed autocannon but during the installation I get this:

delvedor ~ $ sudo npm i autocannon -g
/usr/local/bin/autocannon -> /usr/local/lib/node_modules/autocannon/autocannon.js

> [email protected] install /usr/local/lib/node_modules/autocannon/node_modules/native-hdr-histogram
> node-gyp rebuild

  CC(target) Release/obj.target/zlib/zlib/adler32.o
  CC(target) Release/obj.target/zlib/zlib/compress.o
  CC(target) Release/obj.target/zlib/zlib/crc32.o
  CC(target) Release/obj.target/zlib/zlib/deflate.o
  CC(target) Release/obj.target/zlib/zlib/gzclose.o
  CC(target) Release/obj.target/zlib/zlib/gzlib.o
  CC(target) Release/obj.target/zlib/zlib/gzread.o
  CC(target) Release/obj.target/zlib/zlib/gzwrite.o
  CC(target) Release/obj.target/zlib/zlib/infback.o
  CC(target) Release/obj.target/zlib/zlib/inffast.o
  CC(target) Release/obj.target/zlib/zlib/inflate.o
../zlib/inflate.c:1507:61: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    if (strm == Z_NULL || strm->state == Z_NULL) return -1L << 16;
                                                        ~~~ ^
1 warning generated.
  CC(target) Release/obj.target/zlib/zlib/inftrees.o
  CC(target) Release/obj.target/zlib/zlib/trees.o
  CC(target) Release/obj.target/zlib/zlib/uncompr.o
  CC(target) Release/obj.target/zlib/zlib/zutil.o
  LIBTOOL-STATIC Release/zlib.a
  CC(target) Release/obj.target/histogram/src/hdr_encoding.o
  CC(target) Release/obj.target/histogram/src/hdr_histogram.o
  CC(target) Release/obj.target/histogram/src/hdr_histogram_log.o
  CXX(target) Release/obj.target/histogram/hdr_histogram_wrap.o
  CXX(target) Release/obj.target/histogram/histogram.o
  SOLINK_MODULE(target) Release/histogram.node
/usr/local/lib
└─┬ [email protected] 
  ├─┬ [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├─┬ [email protected] 
  │ │ └── [email protected] 
  │ ├── [email protected] 
  │ └── [email protected] 
  ├── [email protected] 
  ├── [email protected] 
  ├─┬ [email protected] 
  │ ├── [email protected] 
  │ └── [email protected] 
  ├─┬ [email protected] 
  │ └── [email protected] 
  ├── [email protected] 
  ├── [email protected] 
  └─┬ [email protected] 
    ├── [email protected] 
    ├── [email protected] 
    ├── [email protected] 
    ├─┬ [email protected] 
    │ ├── [email protected] 
    │ └── [email protected] 
    ├── [email protected] 
    └── [email protected] 

As you can see the installation terminates without errors.

If I use autocannon by requiring it I have no problem, but if I use it via command line I get:
env: node\r: No such file or directory.

I've tried both via local and global installation and I'm using autocannon 0.3.0

This is the server:

const http = require('http');
var len = 1024 * 1024 * 1024;
var chunk = Buffer.alloc(len, 'x');

var server = http.createServer(function(req, res) {
  res.write(chunk);
  res.end()
});

server.listen(1234, function() {
});

And I am using this line: ./autocannon.js -d 10 -j localhost:1234

Running 10s test @ http://localhost:1234
10 connections

Stat         Avg    Stdev   Max
Latency (ms) 7344   1129.22 9497
Req/Sec      1      1.85    6
Bytes/Sec    1.1 GB 2.02 GB -1.88 GB

10 requests in 10s, 10.74 GB read

I think the problem lies in the int32 conversion:

https://github.com/mcollina/native-hdr-histogram/blob/master/hdr_histogram_wrap.cc#L91-L107

Damn, I remember thinking I will never hit that bug when I wrote those lines.....

cc @thekemkid

If running a test against a server which dies, maybe you want your test to stop?

In this case, it might be useful to allow a user to specify a threshold of errors within a specific timeframe to cause a bailout. This would surely damage perf though A better way of managing this is to just exit after the error threshold is hit (not needing an error timeframe).

Examples:

Say you have a server you are benchmarking for half an hour. The server dies after 10 minutes. To detect this, we could either:
Detect if there was 1000 errors from the server within the last second OR we could detect if there was 10000 errors overall.

The goal is to let users build their own tool if they want to, embedding autocannon.

Currently, the documentation doesn't show how to customise the headers on the request.

For the duration option, browsing the code, it is required, but never checked, @mcollina, should we put a check in for the duration and test it, or default it to some value, eg. 10? Obviously, this will need a unit test and documentation work, too.

Supported HTTP methods should be documented.

Show the default value for the http pipelining value.

Update the .track doc, I left outputStream listed as a param name when I should have listed opts.

How are these handled and detected?

As titled, currently -n just removes the progress bar.

currently all response types are counted, this throws averages off (e.g. a 502 response could be much faster than 200)

just count 2xx responses

I think we can build another tool that takes autocannon json and produces an html page, possibly with comparisons with a previous run. We may want to add some metadata to the CLI to add a name to the given bench.

Also, we can print the latency using a histogram and so on.. :D.

currently defaults to 80 instead of 443

Needed or not? This tool is obviously meant to be used for systems in development, so not supporting https makes perfect sense, but we need to document it.

As titled, we might probably want to override some stuff in chalk.

Chalk is disabled because of a check in https://github.com/chalk/supports-color/blob/master/index.js.

See https://github.com/chalk/supports-color/blob/master/index.js#L41-L43.

We can either internalize that check, or sending them a PR to expose that as a function that accepts a stream. Or maybe it's just a new module.

And expose it :).

cc @thekemkid

as titled

I saw both average and mean in the json output, a they used as aliases or mean is different from the arithmetic mean here?

When benchmarking this node.js script with:
autocannon -d 1 localhost:1234

Result:

Running 1s test @ http://localhost:1234
10 connections

/usr/local/lib/node_modules/autocannon/node_modules/pretty-bytes/index.js:6
        throw new TypeError('Expected a number, got ' + typeof num);
        ^

TypeError: Expected a number, got number
    at module.exports (/usr/local/lib/node_modules/autocannon/node_modules/pretty-bytes/index.js:6:9)
    at asBytes (/usr/local/lib/node_modules/autocannon/lib/progressTracker.js:170:20)
    at EventEmitter.instance.on (/usr/local/lib/node_modules/autocannon/lib/progressTracker.js:59:38)
    at emitOne (events.js:101:20)
    at EventEmitter.emit (events.js:188:7)
    at Timeout.setInterval (/usr/local/lib/node_modules/autocannon/lib/run.js:181:15)
    at Timeout.wrapper (timers.js:425:11)
    at tryOnTimeout (timers.js:232:11)
    at Timer.listOnTimeout (timers.js:202:5)

This does not happen, when -j is used, which produces:

{"url":"http://localhost:1234","requests":{"average":1680,"mean":1680,"stddev":0,"min":1680,"max":1680,"total":1680,"sent":1690},"latency":{"average":5.54,"mean":5.54,"stddev":2.36,"min":3,"max":19,"p50":4,"p75":7,"p90":9,"p99":14,"p999":17,"p9999":19,"p99999":19},"throughput":{"average":null,"mean":null,"stddev":null,"min":-1,"max":0,"total":1761814320},"errors":0,"timeouts":0,"duration":1,"start":"2016-08-08T10:58:21.518Z","finish":"2016-08-08T10:58:22.540Z","connections":10,"pipelining":1,"non2xx":0,"1xx":0,"2xx":1680,"3xx":0,"4xx":0,"5xx":0}

Under Linux:
autocannon -d -j example.com > stdout 2> stderr

Result: JSON is stored in stdout. But default output (i.e. one with progress, results table etc.) is stored in stderr. This does not happen neither when output is not piped to file, nor when -j is not used.

se should remember to add the following:

process.once('SIGINT', () => {
  instance.stop()
})

to correctly handle CTRL-C

instance.on('response', handleResonse)

Missing a "p" in handleResponse

Under Linux, with node 6.3.0. Also happens on master:

autocannon -d 1 example.com > out 2>&1

Result - error in node_modules/progress/lib/node-progress.js:177:

TypeError: this.stream.clearLine is not a function

Happens only if both stdout and stderr are redirected.

We should be able to force a dns lookup each X requests to support AWS ELBs:

Single Client Tests
In single client tests, a load generator will issue requests to the server using the same client. In most cases, the client does not re-resolve the DNS after the test is initiated. A good example of this type of test is Apache Bench (ab). If you are using this type of testing tool, there are two options:

• Write discrete tests at a given sample size and concurrency then start a new set of tests and invoke these tests separately so that the client will re-resolve the DNS (possibly even forcing a re-resolution depending on the operating system and other factors). It is important to save detailed request logs so you can aggregate the logs for a single view of the request details.
• Launch multiple client instances and initiate the tests on the different instances at approximately the same time. AWS CloudFormation can help you do this by creating your load test script and launching multiple clients, then simply issuing remote SSH commands to initiate the tests. You may want to bring up more load generating clients based on Auto Scaling rules (for example, when the average CPU of other load clients reaches a certain level, add more load generators).

However, depending on the load you are trying to generate, your test may be constrained by the client's ability to issue requests. In this case, you need to consider distributed tests.

cc @thekemkid

As per @davidmarkclements comment in #54, we should review the API (and do a general code cleanup).

The amount of options a user now has available to them has grown greatly over the last week or so, and with that growth came a lot more code, and less API thought/considerations.

As shown by @AndreasMadsen in https://github.com/nodejs/node/pull/8140/files#r75463752, autocannon does not match wrk in some areas.

Wrong space when writing to file:

$ autocannon -c 100 http://localhost:3000
Running 10s test @ http://localhost:3000
100 connections

Stat         Avg      Stdev    Max
Latency (ms) 5.11     3.04     96
Req/Sec      17832.41 2170.06  20671
Bytes/Sec    1.99 MB  241.6 kB 2.36 MB

178k requests in 10s, 19.79 MB read

vs

$ autocannon -c 100 -j http://localhost:3000 > out.json

Running 10s test @ http://localhost:3000
100 connections

Stat         Avg     Stdev     Max
Latency (ms) 4.53    2.34      96
Req/Sec      19916.8 1427.08   21599
Bytes/Sec    2.22 MB 160.66 kB 2.49 MB

199k requests in 10s, 22.11 MB read

overrides duration, set amount of requests to make per conn, and finish when all requests are complete

for instance, "I want to see how long it take for 100 connection to concurrently make one request"

as well as loading body from file, allow a --data flag for body to be set inline
dependant on #1

Many benchmarking tools allow users to specify the rate of requests per second. It would be nice to support this. If the rate cannot be met, we should just do as many requests as we can. If no rate is specified, we should do as many requests as possible.

@mcollina We'll need to chat about this, as I'm not sure how the code for this would look or work. Rate limiting on a huge rate that cannot be met will add an unnecessary overhead.

Issue seems to stem from hdr histogram. Testing as appropriate, will report findings after.

Output:

Glens-MacBook-Pro-2:autocannon glen$ npm run test

> [email protected] test /Users/glen/work/nearform/os/autocannon
> standard && tap test/*.test.js

test/format.test.js ................................... 4/4
test/myhttp.test.js ................................. 29/29
test/run.test.js .................................... 47/48
  run
  not ok throughput.stddev exists
    at:
      line: 34
      column: 7
      file: test/run.test.js
    stack: |
      test/run.test.js:34:7
      Timeout.setInterval (lib/run.js:100:7)
    source: |
      t.ok(result.throughput.stddev, 'throughput.stddev exists')

total ............................................... 80/81


  80 passing (5s)
  1 failing


npm ERR! Darwin 15.5.0
npm ERR! argv "/Users/glen/.nvm/versions/node/v6.2.2/bin/node" "/Users/glen/.nvm/versions/node/v6.2.2/bin/npm" "run" "test"
npm ERR! node v6.2.2
npm ERR! npm  v3.9.5
npm ERR! code ELIFECYCLE
npm ERR! [email protected] test: `standard && tap test/*.test.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] test script 'standard && tap test/*.test.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the autocannon package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     standard && tap test/*.test.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs autocannon
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls autocannon
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/glen/work/nearform/os/autocannon/npm-debug.log