mattbostock / webdriver_exporter Goto Github PK

View Code? Open in Web Editor NEW

81.0 8.0 21.0 232 KB

Probes a web page using the WebDriver protocol and exposes metrics for Prometheus such as Navigation Timings

License: Apache License 2.0

Go 100.00%

prometheus-exporter prometheus webdriver-protocol chromedriver navigation-timing probe metrics webdriver-exporter

webdriver_exporter's Introduction

WebDriver Exporter

Probes a web page using the WebDriver protocol and exposes metrics for Prometheus such as Navigation Timings.

Alpha quality

This is alpha-quality code, without tests. Run it in Production at your own risk.

Example output

navigation_timing_secure_connection_start_seconds 1468151331.286000
navigation_timing_dom_complete_seconds 1468151332.641000
navigation_timing_dom_content_loaded_event_end_seconds 1468151332.167000
navigation_timing_load_event_end_seconds 1468151332.643000
navigation_timing_response_end_seconds 1468151331.645000
navigation_timing_redirect_start_seconds 0.000000
navigation_timing_request_start_seconds 1468151331.465000
navigation_timing_response_start_seconds 1468151331.643000
navigation_timing_dom_content_loaded_event_start_seconds 1468151332.158000
navigation_timing_dom_interactive_seconds 1468151332.158000
navigation_timing_dom_loading_seconds 1468151331.647000
navigation_timing_fetch_start_seconds 1468151331.221000
navigation_timing_connect_end_seconds 1468151331.465000
navigation_timing_connect_start_seconds 1468151331.277000
navigation_timing_domain_lookup_end_seconds 1468151331.277000
navigation_timing_unload_event_start_seconds 0.000000
navigation_timing_unload_event_end_seconds 0.000000
navigation_timing_domain_lookup_start_seconds 1468151331.271000
navigation_timing_load_event_start_seconds 1468151332.641000
navigation_timing_navigation_start_seconds 1468151330.960000
navigation_timing_redirect_end_seconds 0.000000
browser_log_warning_count 0
browser_log_severe_count 1
probe_duration_seconds 2.596026
probe_success 1

Building and running

Prerequisites

You'll need chromedriver:

# On Mac OS X using Homebrew
brew install chromedriver

To run WebDriver Exporter on a server with a headless Chrome browser, you'll need something like xvfb.

Building locally

go get ./...
go build
./webdriver_exporter <flags>

Visiting http://localhost:9156/probe?target=https://prometheus.io/ will return metrics for prometheus.io.

Prometheus Configuration

The WebDriver Exporter needs to be passed the target as a parameter, this can be done with relabelling.

Example configuration:

scrape_configs:
  - job_name: 'webdriver'
    metrics_path: /probe
    static_configs:
      - targets:
        - https://prometheus.io/   # Target to probe
    relabel_configs:
      - source_labels: [__address__]
        regex: (.*)(:80)?
        target_label: __param_target
        replacement: ${1}
      - source_labels: [__param_target]
        regex: (.*)
        target_label: instance
        replacement: ${1}
      - source_labels: []
        regex: .*
        target_label: __address__
        replacement: 127.0.0.1:9156  # WebDriver Exporter

Example recording rules:

# Official time breakdown: https://www.w3.org/TR/navigation-timing/
instance:navigation_timing_back_end_seconds = navigation_timing_response_start_seconds{job="webdriver"} - navigation_timing_start_seconds{job="webdriver"}
instance:navigation_timing_dom_content_loaded_seconds = navigation_timing_dom_content_loaded_event_start_seconds{job="webdriver"} - navigation_timing_start_seconds{job="webdriver"}
instance:navigation_timing_dom_interactive_seconds = navigation_timing_dom_interactive_seconds{job="webdriver"} - navigation_timing_start_seconds{job="webdriver"}
instance:navigation_timing_domain_lookup_seconds = navigation_timing_domain_lookup_end_seconds{job="webdriver"} - navigation_timing_domain_lookup_start_seconds{job="webdriver"}
instance:navigation_timing_front_end_seconds = navigation_timing_load_event_start_seconds{job="webdriver"} - navigation_timing_response_end_seconds{job="webdriver"}
instance:navigation_timing_page_download_seconds = navigation_timing_response_end_seconds{job="webdriver"} - navigation_timing_response_start_seconds{job="webdriver"}
instance:navigation_timing_page_load_seconds = navigation_timing_load_event_start_seconds{job="webdriver"} - navigation_timing_start_seconds{job="webdriver"}
instance:navigation_timing_redirection_seconds = navigation_timing_fetch_start_seconds{job="webdriver"} - navigation_timing_start_seconds{job="webdriver"}
instance:navigation_timing_server_connection_seconds = navigation_timing_connect_end_seconds{job="webdriver"} - navigation_timing_connect_start_seconds{job="webdriver"}
# Other
instance:navigation_timing_latency_seconds = navigation_timing_response_start_seconds{job="webdriver"} - navigation_timing_fetch_start_seconds{job="webdriver"}
instance:navigation_timing_transfer_seconds = navigation_timing_response_end_seconds{job="webdriver"} - navigation_timing_response_start_seconds{job="webdriver"}
instance:navigation_timing_dom_processing_interactive_seconds = navigation_timing_dom_interactive_seconds{job="webdriver"} - navigation_timing_dom_loading_seconds{job="webdriver"}
instance:navigation_timing_dom_processing_complete_seconds = navigation_timing_dom_complete_seconds{job="webdriver"} - navigation_timing_dom_interactive_seconds{job="webdriver"}
instance:navigation_timing_onload_seconds = navigation_timing_load_event_end_seconds{job="webdriver"} - navigation_timing_load_event_start_seconds{job="webdriver"}

Limitations

Chromedriver only (currently)

Only Chromedriver is supported currently. Adding support for other webdrivers (e.g. Selenium, phantomjs) should be relatively trivial. Pull requests are encouraged.

At the time of writing, PhantomJS did not support probing of sites with strict Content Security Policies; see ariya/phantomjs#13114.

Requires exact URL as target

To ensure that the timings returned are for the target requested, the page URL is strictly matched against the requested target.

For example, if you try to probe https://example.com/, the probe will fail if the page redirects to https://example.com/foo. Similarly, make sure that you use https as the URL scheme if your target enforces HTTPS.

This requirement could be loosened if there's enough demand for it.

Probe timeout hardcoded at 5 seconds

Chrome is set to timeout after 5 seconds if it hasn't yet loaded the page. Pull requests to make this value configurable are welcomed.

Probes are slow to initialise

We start a new Chromedriver session for each probe to ensure that the cache, cookies and local storage are clean. If we could retain the session and clear the cache, the time to complete a probe would be significantly reduced.

webdriver_exporter's People

Contributors

Stargazers

Watchers

Forkers

syepes tkusienicki chlam4 songbinliu pleaswaite marcelhillmann vvhh2002 nouchka xilu0 mjboxboat turbonomic jenningsloy318 sheffercool zwb-github suqcnn diakreado crystal30 thanamat

webdriver_exporter's Issues

when i run the binary file webdriver, show error

ERRO[0640] failed to retrieve URL: unexpected response: {"sessionId":"206432afec3ba164339100d1b83283fb","status":6,"value":{"message":"invalid session id\n (Driver info: chromedriver=2.45.615279 (12b89733300bd268cff3b78fc76cb8f3a7cc44e5),platform=Linux 4.4.0-117-generic x86_64)"}} source=probe.go:71

and my prometheus.yml is ：
"""

job_name: 'webdriver'
metrics_path: /probe
static_configs:
- targets:
  - https://prometheus.io/ # Target to probe
    relabel_configs:
- source_labels: [address]
  regex: (.*)(:80)?
  target_label: __param_target
  replacement: ${1}
- source_labels: [__param_target]
  regex: (.*)
  target_label: instance
  replacement: ${1}
- source_labels: []
  regex: .*
  target_label: address
  replacement: 127.0.0.1:9156 # WebDriver Exporter
  """

stability status

Can anyone comment on the stability of this software? Looks like the last commit was 10 months ago. Has anyone been running this exporter continuously for this long?

Hmm, at least now you appear to have the exporter working. I'd suggest opening a separate issue for this

Originally posted by @benridley in #7 (comment)

Is it related to chrome driver issues as it is not scraping the values
Hi,
We tried with chrome driver and facing issues like this when we run the curl
-bash-4.2$ curl http://localhost:9156/probe?target=https://google.com
probe_duration_seconds 0.031578
probe_success 0

-bash-4.2$ ./webdriver_exporter
INFO[0000] Starting webdriver_exporter (version=0.0.2, branch=, revision=) sour ce=main.go:50
INFO[0000] Build context (go=go1.6.3, user=, date=) source=main.go:51
INFO[0000] Starting webdriver source=main.go:69
INFO[0000] Listening on localhost:9156 source=main.go:76
ERRO[0272] failed to retrieve URL: unexpected response: {"sessionId":"8b2eebfced 6859a6d36d1c49c5a4ee99","status":6,"value":{"message":"no such session\n (Drive r info: chromedriver=2.40.565383 (76257d1ab79276b2d53ee976b2c3e3b9f335cde7),plat form=Linux 4.14.35-1911.el7uek.x86_64 x86_64)"}} source=probe.go:71

Error with chromedriver on scraping

Hi,

Thanks for your work on this exporter. I got this error when I tried to curl this url curl http://localhost:9156/probe\?target\=https://prometheus.io/ :

ERRO[0024] failed to retrieve URL: unexpected response: {"sessionId":"9a00b5ced6cfe589a2252bb67544e545","status":6,"value":{"message":"no such session\n (Driver info: chromedriver=2.22.397932 (282ed7cf89cf0053b6542e0d0f039d4123bbb6ad),platform=Linux 3.16.0-4-amd64 x86_64)"}} source=probe.go:45

I am using the latest chromedriver http://chromedriver.storage.googleapis.com/index.html?path=2.22/ and I have built your exporter with GO 1.6.2.

Do you know what happen? I have zero knowledge of chromedriver :D

Regards,

Laurent

What flags do we need to pass while building

We are getting the below issues while building
NFO[0000] Starting webdriver_exporter (version=0.0.2, branch=, revision=) source=main.go:50
INFO[0000] Build context (go=go1.6.3, user=, date=) source=main.go:51
INFO[0000] Starting webdriver source=main.go:69
FATA[0000] failed to start webdriver: failed to start service: failed to run command: exec: "chromedriver": executable file not found in $PATH source=main.go:72

Appreciate your help on this.

Useful recording rules

Hello Matt,
Just wanted to mention that this project is awesome!
I was starting to writing something similar but in Ruby, are you going to continue the dev on this?

Is there any plans on removing the redirect limitation (Requires exact URL as target)?

Here are some recording rules that I am currently using:

# Web Official https://www.w3.org/TR/navigation-timing/
instance:web:backEnd:seconds = navigation_timing_response_start_seconds{job="webdriver"} - navigation_timing_start_seconds{job="webdriver"}
instance:web:domContentLoaded:seconds = navigation_timing_dom_content_loaded_event_start_seconds{job="webdriver"} - navigation_timing_start_seconds{job="webdriver"}
instance:web:domInteractive:seconds = navigation_timing_dom_interactive_seconds{job="webdriver"} - navigation_timing_start_seconds{job="webdriver"}
instance:web:domainLookup:seconds = navigation_timing_domain_lookup_end_seconds{job="webdriver"} - navigation_timing_domain_lookup_start_seconds{job="webdriver"}
instance:web:frontEnd:seconds = navigation_timing_load_event_start_seconds{job="webdriver"} - navigation_timing_response_end_seconds{job="webdriver"}
instance:web:pageDownload:seconds = navigation_timing_response_end_seconds{job="webdriver"} - navigation_timing_response_start_seconds{job="webdriver"}
instance:web:pageLoad:seconds = navigation_timing_load_event_start_seconds{job="webdriver"} - navigation_timing_start_seconds{job="webdriver"}
instance:web:redirection:seconds = navigation_timing_fetch_start_seconds{job="webdriver"} - navigation_timing_start_seconds{job="webdriver"}
instance:web:serverConnection:seconds = navigation_timing_connect_end_seconds{job="webdriver"} - navigation_timing_connect_start_seconds{job="webdriver"}

# Web Other
instance:web:latency:seconds = navigation_timing_response_start_seconds{job="webdriver"} - navigation_timing_fetch_start_seconds{job="webdriver"}
instance:web:transfer:seconds = navigation_timing_response_end_seconds{job="webdriver"} - navigation_timing_response_start_seconds{job="webdriver"}
instance:web:domProcessingInteractive:seconds = navigation_timing_dom_interactive_seconds{job="webdriver"} - navigation_timing_dom_loading_seconds{job="webdriver"}
instance:web:domProcessingComplete:seconds = navigation_timing_dom_complete_seconds{job="webdriver"} - navigation_timing_dom_interactive_seconds{job="webdriver"}
instance:web:onLoad:seconds = navigation_timing_load_event_end_seconds{job="webdriver"} - navigation_timing_load_event_start_seconds{job="webdriver"}

By the way, I made this all work in a docker container if your interested I could share the setup