When using the WPT web UI the addHeader method works great.

When using the node wrapper program I cannot get it to work.

This is the syntax I use in the WPT UI which works:
addHeader X-header: someheader
navigate https://some.url.com

In the node program it looks like this:

var script = wpt.scriptToString([
{addHeader:[‘X-Header:’, ‘someheader’]},
{navigate: "https://some.url.com"}
])

wpt.runTest(script, {"location": "us-west-2", "connectivity": "Cable"}, function(err, data) {
...
});

The headers do not show up in the requests when viewing the results on the WPT web UI

Is this disabled from the wrapper program, how can I get it to work?

Thanks,
Dan

Thanks for your work on this is been very helpful. I have some questions / possible bug reports.

Here you can see I've submitted two tests with the cli. One with the --noads option and one without. But both results clearly render an Apple ad. In comparison, I'm running Chrome with the AdBlock Plus plugin and the ad is being blocked as expected. So if the option is suppose to be using the AdBlock Plus block list, I think this should work.

1. Am I using the option correctly?
2. The README states Test (works for test command only) for all of the options in which the --noads option is listed. I assume this means it should also work for the node library .runtTest(<someurl>, { noads: true|false }, ...) equivalent as well ?
3. I don't see this option in the webpagetest.org UI but I assume that maybe you are injecting some variation of the adblock plus block list into the Block Requests Containing... option?

This library has some pretty excellent CI and appears to be passing all tests. If this is a bug, perhaps in addition to fixing it we could add some coverage for this?

Let me know if I can lend a hand.

I know the scripting can use to manipulate authentication, but the login DOM element is in an iframe, so how to get the element?. site example http://login.alibaba.com

Since I found this bug in the undocumented/inofficial proxy option it won't be of high priority. :)

I'm using your great api module in an environment that enforces proxy usage. So I'm glad I found the proxy option in the source code. It works great for the initial request your api does to the webpagetest.org service e.g. to run tests or fetch their status. But it fails for subsequent requests when I combine the runTest-command with the pollResults option. These request don't use the proxy and so they fail.

It's not a big problem since I could work around it by implementing polling with the getTestStatus command. Nevertheless it would be nice if this could be fixed and the proxy options goes official.

Thanks for your great work.

I am calling wpt.runTest with a spec. Is there any way to get the test id back? I'd like to do this so I can use it with other methods that require a testid like getHARData.

Additionally, it would be nice to get the response you get with the synchronous version, but still have the spec tests run. That way, I could dump out a message like "To view this test in the browser, go to ".

In order to let the caller hook into both of these events, it may be better to use something like an EventEmitter:

var testEmitter = wpt.runTest(url);

testEmitter.on('testStarted', function(testStarted) {
    console.log('View the test at: ' + testStarted.userUrl);
});

testEmitter.on('testComplete', function() {
    // proceed now that specs have passed
});

For now, to work around this, I'm going to re-implement the polling for results logic on my own.

I'm trying to figure out if there exists a parameter that will generate the filmstrip view of a test.

I'm not seeing the filmstrip in the test I generated via this script.

Do I need to do something special for webpagetest to generate a filmstrip for a test?

I'd appreciate any help!

Thanks

catchpoint/WebPageTest#318 added a new parameter 'tsview_configs' for tsviewdb to pass arbitrary configs to enhance the data in the timeseries

WebPageTest's export.php which provides the har dump, contains a parameter of pretty which determines whether the json is indented. Using the webpagetest har <id> command does not expose this parameter and seems to always request the prettified json which often has a much larger file size.

I would like a parameter, or for it to be the default, to get the non-prettified har file.

Is there any convenient way to get this information at the moment? It displays on the Content Breakdown tab of the results page, but none of the JSON outputs seem to have it.

Specifically, I am looking for the summed size in bytes of all the images, javascript, text, etc on a given page. All the .json results seem to return is the size of all images, specifically for calculating compression savings.

The agent machine the tests ran on just shows the history of jobs submitted manually. Where are the node.js tests?

Hi

I install npm and node via apt:
apt-get install npm node

After I install webpagetest:
npm install webpagetest -g

It installed but not work:

webpagetest --help

No helps givens, no commands executed

Hello,

First of all, thanks for this great module that make things much easier.

Does the waitResults work as expected when testing against the public instance of webpagetest?

Used as in the example,

wpt.runTest('http://twitter.com/marcelduran', {waitResults: 'http://wpt.bk.wope-framework.com:80'}, function(err, data) {
console.log(err || data);
});

My callback is never called.

Thanks,

i want test

webpagetest test http://www.baidu.com/s?usm=2&wd=58%E5%90%8C%E5%9F%8E&ie=utf-8&rsv_cq=%E6%99%BA%E8%81%9&rsv_dl=0_right_recommends_merge_21091&euri=1585257 -s xxx.xxx.xxx.xxx -k Key123 --location paris-wptdriver 

an error, help me

https://sites.google.com/a/webpagetest.org/docs/advanced-features/webpagetest-restful-apis

They are particularly useful for controlling variability by using the same test agent for a given page or for debugging a test agent (affinity tends to be a lot more useful than specifying a specific tester).

Facing below on executing

webpagetest test http://www.google.com

{
"error": {
"name": "WPTAPIError",
"code": 403,
"message": "Forbidden"
}

Any help is appreciated.

How to Set Cookie value in scriptToString ?

Hello,

Unable to set cookie value in scriptToString method , below is my syntax , Can you guys please validate and let me know the fix ?

script = wpt.scriptToString([
{navigate: http://www.xyz.com},
{setViewportSize: [1280,1024]},
{setCookie: [ http://www.xyz.com, 'SEGMENT=%7B%22EXPERIMENT%22%3A%5B1063%5D%7D']} ]);

Hi Marcel,

I am having an issue with launching a test against this url

http://beta.michelin.com.cn/CN/zh/tires/%E4%B8%9C%E9%A3%8E%E6%A0%87%E8%87%B4/3008/1__6T/%E8%87%AA%E5%8A%A8%20%20%E7%BD%97%E5%85%B0%E5%8A%A0%E6%B4%9B%E6%96%AF%E7%89%88/2013.html

return code and error are:

{
"statusCode": 400,
"statusText": "Invalid Script (make sure there is at least one navigate command and that the commands are tab-delimited). Please contact us if you need help with your test script."
}

launching a test against the same url works fine with the WEB UI.

a parsing issue with the node wrapper?

I also cannot make it test any url with parameters ( ?xxx= ).

Thank you
Alex

It would be nice to have a way to pass in the tsview_id when issuing a command to run a test. This parameter is used for private instances that wish to integrate with Google's tsviewdb.

The REST API docs have been recently updated with info on this parameter.

Hi, just wonder if you can publish http://marcelduran.com/webpagetest-api/console/ to github? Thank you.

I run the following command

webpagetest test http://plugin.restive.io --key {my_api_key}

And I get the following result:

{
  "error": {
    "bytesParsed": 0,
    "code": "HPE_INVALID_CONSTANT"
  }
}

It doesn't matter what URL I use, I keep getting this error. Is this from my end or an issue with the software?!

Hi,

Is there any chances to have some guidance in order to implement the nolimit GET querystring ?
From what I've quickly seen , there is a few issue related to the data volume exchanged , i.e. once data too big , it trunks the json output.

Regards,

Regis A. Despres

There is Import an existing Dev Tools Trace page available for importing an existing trace, but there is no way to import through the webpagetest-api client.

Can it be done in the future? Thank you.

Hi,
The connectivity value in the webpagetest test result (JSON) is not present on a test exectued on mobile agent. Is this something that is encountered by any one?

Mobile Agent :

"data": {

            "location": "Example_Nexus4:Nexus4 - Chrome",
            "from": "Nexus 4 - <b>Nexus4 - Chrome</b>",
            "completed": "Wed, 21 Oct 2015 00:01:05 +0000",
            "tester": "007886d5600d492f-172.29.236.53",
            "runs": 1,

Desktop Agent:

           "location": "WebPage2:Firefox",
            "from": "WebPage2 - <b>Firefox</b> - <b>3G</b>",
            "connectivity": "3G",
            "bwDown": 1600,
            "bwUp": 768,
            "latency": 300,
            "plr": 0,
            "completed": "Mon, 19 Oct 2015 23:04:13 +0000",

Notice the presence of connectivity object in Desktop version. Am I missing anything? My mobile tests run fine but I need the connectivity value to report the tests using sitespeed.io.

Command :

webpagetest test hrrp://www.google.com

{
"error": {
"name": "WPTAPIError",
"code": 403,
"message": "Forbidden"
}
}

Using the following code:

var WebPageTest = require('webpagetest');

var wpt = new WebPageTest('server', 'key');

var options = {
    pollResults: 5,
    timeout: 60
}

function runTest(url) {
    wpt.runTest(url, options, function(err, data) {
        runTest('some-other-url');
    });
}

runTest('some-url');

The options object has been modified after the first test run is complete, meaning for the 2nd run the following is passed:

{
 pollResults: 5000,
 timeout: 60000
}

This is because of:

options.timeout = (parseInt(options.timeout, 10) || 0) * 1000;
and 
options.pollResults = parseInt(options.pollResults * 1000, 10) || 5000;

This can be worked around by passing a new object each time but using the same object is fairly common. It would be better if the options object was not modified.

Use-case is here is gathering data from multiple urls with a single script.

I've try to get page speed score. But it seems to be dose not work even if "pageSpeed" option set true.

Could you tell me how to get?
What is the property name for pageSpeed on response object?

When running wpt.runTest with custom attributes, those attributes are seemingly ignored.

wpt.runTest('https://www.refme.com', {"bandwidthDown":"1600","bandwidthUp":"768","connectivity":"custom","latency":"300"}, function(err, response) {
  jsonUrl = response.data.jsonUrl;
  wpID = response.data.testId;
  console.log(jsonUrl, wpID);

  _pollStatus();
});

Hello!

I'm running a private instance of webpagetest behind HTTP auth. It works through the browser. However, if I enter the same stuff in the command line:

$ webpagetest test https://www.lostmy.name/ --server http://local-wpt/ -g user -w pass

It doesn't work:

{
  "error": {
    "name": "WPTAPIError",
    "code": 401,
    "message": "Unauthorized"
  }
}

Any ideas? Is this a problem with the module, or with how I'm using it?

Not released yet but available on the trunk:
catchpoint/WebPageTest@9519ca5

Allows for saving of the body of just the base page instead of all of the text resources.

xubuntu 14.04
node 0.10.35
npm 2.2.0
webpagetest-api 0.3.1
webpagetest 2.1.6
private instance

When I use runTest(script, options, callback) then the callback just gets a test status json response and never waits for the actual test result.

I haven't used this set up since Oct 14 (when it was last working) and upgraded everything this morning.

Hi there! I work at @wikimedia and we're using this and it's great! We are currently trying to improve things for our 2G users and this library doesn't support it.

Also how actively is this extension being worked on? Wikimedia would be interesting in helping maintaining it if that is useful. cc @soulgalore

For example while using grunt-sitespeedio, which uses webpagetest-api, the task does not receive a callback from the api if the chosen WebPageTest Test location is not reachable from the WebPageTest server. The timeout only works if the WebPageTest server is not reachable!

With a brand new API-key, I can't get any of the locations to work.

I'm located in Denmark, so using e. g. Copenhagen:Chrome would seem like an obvious choice

$ webpagetest test --key $API_KEY --location "Copenhagen:Chrome.DSL" --poll --runs 3 -p https://www.example.dk                                                       [15:38:11]
{
  "error": {
    "statusCode": 400,
    "statusText": "Invalid Location, please try submitting your test request again."
  }
}
$ webpagetest test --key $API_KEY --location "Copenhagen:Chrome.Cable" --poll --runs 3 -p https://www.example.dk                                                     [15:38:17]
{
  "error": {
    "statusCode": 400,
    "statusText": "Invalid Location, please try submitting your test request again."
  }
}
$ webpagetest test --key $API_KEY --location "Copenhagen" --poll --runs 3 -p https://www.example.dk                                                                  [15:38:32]
{
  "error": {
    "statusCode": 400,
    "statusText": "Invalid Location, please try submitting your test request again."
  }
}

I've tried with quite a lot of the location ids found in the webpagetest locations - No luck.

If I don't specify a location, I get the default Dulles, and the test runs just fine.

Help? - What am I doing wrong?

New to Node and came across webpagetest. Trying to run this in my Jenkins but looking for the script to save the csv file alone so then with a post build plugin I have, I can just get the pageload result in the column.

Looking to see how this can be done.

Is there a reporter that just gets the information similar to the summaryCSV url?

Hello..

Trying to test your webpagetest-api but still not found location:

webpagetest test http://www.obat24.com --key A.1231231 --location Singapore_IE8
{
"statusCode": 400,
"statusText": "Invalid Location, please try submitting your test request again."

any advices ?

I run your demo but it give me error.

Hi,

Thanks first for the great tool.

I might be overlooking the documentation, but I am not able to find a way to specify a location + browser combination to fire a test against our private WPT instance.

For example, my $ webpagetest locations results in something like this for all the browsers.

    {
      "id": "sfo-load-webpagetest-01_WPT_wptdriver:Safari",
      "Label": "SFO 1",
      "location": "sfo-load-webpagetest-01_WPT_wptdriver",
      "Browser": "Safari",
      "PendingTests": {
        "p1": 0,
        "p2": 0,
        "p3": 0,
        "p4": 0,
        "p5": 0,
        "p6": 0,
        "p7": 0,
        "p8": 0,
        "p9": 0,
        "Total": 0,
        "HighPriority": 0,
        "LowPriority": 0,
        "Testing": 0,
        "Idle": 1
      }

When I try to fire a test using the location option "-l". But this always picks up my default chrome browser. How do I specify the "id" tag which will differentiate my browsers for a test?

$webpagetest test https://mypage.com -l sfo-load-webpagetest-01_WPT_wptdriver

Update: Not a bug.

Sorry! This was my error. NON-ISSUE. Please close. Setting the env var works per the docs.
I hope I caught this in time to avoid wasting your time looking into it.
Thanks again for this tool, which looks like it will be very helpful! Have a great day!

Installed the webpagetest cli and executed test and it throws an error Missing API key

 webpagetest test http://twitter.com/marcelduran
{
  "statusCode": 400,
  "statusText": "An error occurred processing your request.  Please reload the testing page and try submitting your test request again. (missing API key)"
}%

I'm using node v0.12.6
Any webpagetest commands fails with below error

c:\temp\test>webpagetest locations
{
"error": {
"code": "ECONNREFUSED",
"errno": "ECONNREFUSED",
"syscall": "connect"
}
}

has anyone seen this issue before?

Environment:
node -v -> v4.2.2
npm -v -> 2.9.1

Here's the command I'm executing:

webpagetest.cmd waterfall 151106_M0_P5Z

Here's the error I'm getting:

net.js:617
    throw new TypeError('invalid data');
    ^

TypeError: invalid data
    at WriteStream.Socket.write (net.js:617:11)
    at output (C:\Users\Oliver\AppData\Roaming\npm\node_modules\webpagetest\bin\webpagetest:27:18)
    at Function.formatData (C:\Users\Oliver\AppData\Roaming\npm\node_modules\webpagetest\bin\webpagetest:68:5)
    at callbackYield (C:\Users\Oliver\AppData\Roaming\npm\node_modules\webpagetest\lib\webpagetest.js:138:14)
    at WebPageTest.apiCallback (C:\Users\Oliver\AppData\Roaming\npm\node_modules\webpagetest\lib\webpagetest.js:207:7)
    at IncomingMessage.onEnd (C:\Users\Oliver\AppData\Roaming\npm\node_modules\webpagetest\lib\webpagetest.js:123:11)
    at emitNone (events.js:72:20)
    at IncomingMessage.emit (events.js:166:7)
    at endReadableNT (_stream_readable.js:905:12)
    at doNTCallback2 (node.js:441:9)

Result:
{ [Error: connect ETIMEDOUT] code: 'ETIMEDOUT', errno: 'ETIMEDOUT', syscall: 'co
nnect' }

Running:
var wpt = new WebPageTest('www.webpagetest.org', '[my key here]');
wpt.runTest('www.twitter.com', function callback(err, data) {
console.log(err || data);
});

Not sure what's up, but getting nothing but timeout. Its a windows machine running Node.

When the api fails because of an error from WPT, it'd be nice to return more than just the response code and also include the error description.

E.g.:
instead of:
{ error: 400 }

return
{ error: 400, desc: Invalid Location, please try submitting your test request again. }

I'm executing:
webpagetest results test_id -r tap
webpagetest results test_id --reporter markdown

But it always returns a JSON.

facing below error

sitespeed.io -u http://www1.xxx.com -d 0 -b chrome -n 3
info: OS: 'darwin 14.3.0', Node.js: 'v4.2.2', sitespeed.io: '3.2.5', PhantomJS: '1.9.8', java: '1.7.0_75', browsertime: '0.9.3'
info: Will crawl from start point http://www1.macys.com/shop/product/american-living-cap-sleeve-ruffled-dress?ID=1358999 with crawl depth 0
error: Error from the crawl: The start url: http://www1.macys.com/shop/product/american-living-cap-sleeve-ruffled-dress?ID=1358999 couldn't be fetched, response code 401

info: Didn't get any URLs
error: Error: No URLs to analyze
at Object.module.exports.fineTuneUrls (/usr/local/lib/node_modules/sitespeed.io/lib/util/util.js:341:13)
at /usr/local/lib/node_modules/sitespeed.io/lib/analyzeOneSite.js:50:12
at fn (/usr/local/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:641:34)
at Immediate._onImmediate (/usr/local/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:557:34)
at processImmediate as _immediateCallback

I'm using the spec feature to automatically run a test and compare it against a range of values. It's pretty slick, but I'm getting the following error:

  1) WebPageTest median.firstView.requests:
     Error: not found
      at /Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/specs.js:126:51
      at Array.forEach (native)
      at traverse (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/specs.js:117:24)
      at /Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/specs.js:131:9
      at Array.forEach (native)
      at traverse (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/specs.js:117:24)
      at /Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/specs.js:131:9
      at Array.forEach (native)
      at traverse (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/specs.js:117:24)
      at WebPageTest.specsRunner (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/specs.js:153:5)
      at callbackYield (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/webpagetest.js:128:14)
      at Function.asyncParserCallback (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/webpagetest.js:134:3)
      at /Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/helper.js:59:5
      at Parser.<anonymous> (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/node_modules/xml2js/lib/xml2js.js:368:20)
      at Parser.EventEmitter.emit (events.js:95:17)
      at Object.saxParser.onclosetag (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/node_modules/xml2js/lib/xml2js.js:338:24)
      at emit (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/node_modules/xml2js/node_modules/sax/lib/sax.js:615:33)
      at emitNode (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/node_modules/xml2js/node_modules/sax/lib/sax.js:620:3)
      at closeTag (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/node_modules/xml2js/node_modules/sax/lib/sax.js:861:5)
      at Object.write (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/node_modules/xml2js/node_modules/sax/lib/sax.js:1293:29)
      at Parser.exports.Parser.Parser.parseString (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/node_modules/xml2js/lib/xml2js.js:386:29)
      at Parser.parseString (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/node_modules/xml2js/lib/xml2js.js:6:61)
      at Object.xmlToObj (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/helper.js:53:10)
      at WebPageTest.apiCallback (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/webpagetest.js:184:29)
      at unzip (/Users/nick.heiner/opower/x-web-frontend-perf-check/node_modules/webpagetest/lib/webpagetest.js:105:15)
      at Unzip.onEnd (zlib.js:166:5)
      at Unzip.EventEmitter.emit (events.js:117:20)
      at _stream_readable.js:920:16
      at process._tickDomainCallback (node.js:459:13)

My spec is:

{
    "median": {
        "firstView": {
            "requests": 200,
            "render": 4000,
            "loadTime": 30000
        }
    }
}

With some debugging, I found that it's because the webpagetest.org api isn't returning the path my spec wants to make an assertion about. (Frustratingly, this only happens intermittently, but that's another issue.) It would be nice if the error message were something more descriptive, like:

tests.push({text: path.join('.'), result: new Error('Spec path `' + path.join('.') +
    '` not found in webpagetest results response `' + JSON.stringify(data) + '`'});

Additionally, I would recommend using substack's excellent js-traverse to simplify the code in this area somewhat.