prometheus / blackbox_exporter Goto Github PK

View Code? Open in Web Editor NEW

4.3K 61.0 1.0K 6.44 MB

Blackbox prober exporter

Home Page: https://prometheus.io

License: Apache License 2.0

Go 99.47% Makefile 0.33% Dockerfile 0.19%

blackbox-exporter icmp prometheus-exporter prometheus

blackbox_exporter's Introduction

Prometheus

Visit prometheus.io for the full documentation, examples and guides.

Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed.

The features that distinguish Prometheus from other metrics and monitoring systems are:

A multi-dimensional data model (time series defined by metric name and set of key/value dimensions)
PromQL, a powerful and flexible query language to leverage this dimensionality
No dependency on distributed storage; single server nodes are autonomous
An HTTP pull model for time series collection
Pushing time series is supported via an intermediary gateway for batch jobs
Targets are discovered via service discovery or static configuration
Multiple modes of graphing and dashboarding support
Support for hierarchical and horizontal federation

Architecture overview

Install

There are various ways of installing Prometheus.

Precompiled binaries

Precompiled binaries for released versions are available in the download section on prometheus.io. Using the latest production release binary is the recommended way of installing Prometheus. See the Installing chapter in the documentation for all the details.

Docker images

Docker images are available on Quay.io or Docker Hub.

You can launch a Prometheus container for trying it out with

docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus

Prometheus will now be reachable at http://localhost:9090/.

Building from source

To build Prometheus from source code, You need:

Start by cloning the repository:

git clone https://github.com/prometheus/prometheus.git
cd prometheus

You can use the go tool to build and install the prometheus and promtool binaries into your GOPATH:

GO111MODULE=on go install github.com/prometheus/prometheus/cmd/...
prometheus --config.file=your_config.yml

However, when using go install to build Prometheus, Prometheus will expect to be able to read its web assets from local filesystem directories under web/ui/static and web/ui/templates. In order for these assets to be found, you will have to run Prometheus from the root of the cloned repository. Note also that these directories do not include the React UI unless it has been built explicitly using make assets or make build.

An example of the above configuration file can be found here.

You can also build using make build, which will compile in the web assets so that Prometheus can be run from anywhere:

make build
./prometheus --config.file=your_config.yml

The Makefile provides several targets:

build: build the prometheus and promtool binaries (includes building and compiling in web assets)
test: run the tests
test-short: run the short tests
format: format the source code
vet: check the source code for common errors
assets: build the React UI

Service discovery plugins

Prometheus is bundled with many service discovery plugins. When building Prometheus from source, you can edit the plugins.yml file to disable some service discoveries. The file is a yaml-formated list of go import path that will be built into the Prometheus binary.

After you have changed the file, you need to run make build again.

If you are using another method to compile Prometheus, make plugins will generate the plugins file accordingly.

If you add out-of-tree plugins, which we do not endorse at the moment, additional steps might be needed to adjust the go.mod and go.sum files. As always, be extra careful when loading third party code.

Building the Docker image

The make docker target is designed for use in our CI system. You can build a docker image locally with the following commands:

make promu
promu crossbuild -p linux/amd64
make npm_licenses
make common-docker-amd64

Using Prometheus as a Go Library

Remote Write

We are publishing our Remote Write protobuf independently at buf.build.

You can use that as a library:

go get buf.build/gen/go/prometheus/prometheus/protocolbuffers/go@latest

This is experimental.

Prometheus code base

In order to comply with go mod rules, Prometheus release number do not exactly match Go module releases. For the Prometheus v2.y.z releases, we are publishing equivalent v0.y.z tags.

Therefore, a user that would want to use Prometheus v2.35.0 as a library could do:

go get github.com/prometheus/[email protected]

This solution makes it clear that we might break our internal Go APIs between minor user-facing releases, as breaking changes are allowed in major version zero.

React UI Development

For more information on building, running, and developing on the React-based UI, see the React app's README.md.

More information

Godoc documentation is available via pkg.go.dev. Due to peculiarities of Go Modules, v2.x.y will be displayed as v0.x.y.
See the Community page for how to reach the Prometheus developers and users on various communication channels.

Contributing

Refer to CONTRIBUTING.md

License

Apache License 2.0, see LICENSE.

blackbox_exporter's People

Contributors

Stargazers

Watchers

Forkers

fabxc afefelov lyda jbarratt ynnt richih stapelberg base2solutions caskey-zz zhenweiwang1990 dnile tux21b cviecco knyar fauxfaux cambridge-university-press the108 gitinsky-bot siso mkokho kalynvalentyn carlpett solarnz yosefy ton31337 drewhemm tcolgate rdtr prashant0712 rchicoli pj42 jetstack-experimental mattbostock luke-orden hasso aaronmfraser bagel aparcar ildarmf yanghongkjxy daniel-yavorovich briceburg conr amitbmas90 colstuwjx grandi23 nuaays barthv discordianfish dwdraju tsechin aeriff guilhem tjsampson mbarzilovich kblin hawliczek jazzhanns joker234 dabear dominik-k sbadia dmyerscough sysadmind elmeramigleo thz fleaz logingood f0rmatting latency-at aknuds1 shobhit93 rainf930125 lokho gpicky ageis daicoden kalivarapureshma gouthamve vdemonchy edmundkwok captainrinaldi brunodonassolo pruchai timonwong jiangsuwwj1 josephreynolds ninech tanner-bruce spmarin judexzhu ericmanlol snebel29 stephanvd dmeulen erinbourgeous-taulia kmd09 lpaulmp megaqube hounddog

blackbox_exporter's Issues

Docker level=error msg="Error listening to socket: listen ip4:icmp 0.0.0.0: socket: operation not permitted" file=icmp.go line=32

This is my service file: http://s.natalian.org/2015-11-20/blackbox-exporter.service

Note I am running with --privileged which should provide all CAPS.

http://sg.dabase.com:9115/probe?target=dabase.com&module=icmp
results in errors like:

Nov 20 08:06:17 rojak docker[8786]: time="2015-11-20T08:06:17Z" level=error msg="Error listening to socket: listen ip4:icmp 0.0.0.0: socket: operation not permitted" file=icmp.go line=32

Some what confusingly I get metrics like:

probe_duration_seconds 0.000251
probe_success 0

When the domain exists or not [1]. Is 0 success ?!

[1] http://sg.dabase.com:9115/probe?target=doesnotexist.example.com&module=icmp

Incorrect Content-Length header

Hi, it seems I'm getting a Content-Length that's 8 bytes less than the actual response payload:

$ curl -s --ignore-content-length "http://127.0.0.1:9091/probe?target=google.com&module=http_2xx" | wc
      7      12     152

$ curl -vs "http://127.0.0.1:9091/probe?target=google.com&module=http_2xx"
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9091 (#0)
> GET /probe?target=google.com&module=http_2xx HTTP/1.1
> Host: 127.0.0.1:9091
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: {Wed, 23 Nov 2016 13:20:51 GMT}
< Content-Length: 144
< Content-Type: {text/plain; charset=utf-8}
< Connection: close

will fork and try to fix, but no guarantees as to when that will happen :)

detect revoked ssl certs

$ curl 'http://127.0.0.1:9115/probe?target=https://revoked.grc.com/&module=http_2xx'
probe_ssl_earliest_cert_expiry 1492961872.000000
probe_http_status_code 200
probe_http_content_length -1
probe_http_redirects 0
probe_http_ssl 1
probe_duration_seconds 0.246233
probe_success 1

whitelist / blacklist for HTTP and TCP probing targets?

There're some services that should not be exposed outside of localhost - if I deploy blackbox_exporter on these hosts, maybe it would get exploited somehow to access these internal services?

Be able to specify source IP

For at least TCP and ICMP checks, it'd be useful to be able to specify source IP address

adding port and uri as parameter inside module config

so we may add n checks with different ports and uri's in same prometheus job
for now it very difficult path with long relabeling configs if at all
and with service discovery looks even impossible

may be there is better idea?

thanks

IPv6 support

ICMP module is IPv4 only.
While TCP/HTTP modules seem to support IPv6, it's unclear how the choice is made between IPv4 and IPv6. For monitoring it's often good idea to force to is it either one.

request canceled (Client.Timeout exceeded while awaiting headers)

when using http probe module, got some warning message as below randomly. and the value of probe_duration_seconds is much higher but no failure.
time="2016-09-27T09:08:42Z" level=warning msg="Error for HTTP request to http://xxx/proplus/home: Get http://xxxx/proplus/home: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" source="http.go:97"
how to trouble shoot this? any clue is welcome.

Add parameter to specify http proxy

Hello,

I'm trying to add a blackbox probe to test the availability of an http proxy. The http module could be perfect for such task if it was possible to pass a proxy URL in parameter.

It would be helpful to be able to do:

http://blackbox:9115/probe?target=http://example.com&proxy=http://myproxy:3128&module=http_2xx

Or simply being able to define a proxy in the module configuration and do:

http://blackbox:9115/probe?target=http://example.com&module=http_2xx_proxy

I can workaround that limitation by launching an instance of the blackbox exporter for each proxy I need to test (using the HTTP_PROXY env var to set the proxy), but it would be cleaner to have a single blackbox exporter instance for all checks.

Targeting multiple DNS records

The DNS prober currently allows a DNS name to be configured statically in blackbox_exporter's YAML configuration file. The target, specified by Prometheus, is the nameserver to query. This diverges from @brian-brazil's suggestion to make the DNS name part of the query string (rather than the nameserver).

This favours probing the same DNS record from multiple DNS servers, and makes probing multiple DNS records from the same namesever rather cumbersome.

Realistically I think both use cases are going to be common, depending on whether you're running DNS servers or just want to make sure that your SRV records are working correctly, for example.

That said, I think probing multiple records against the same nameserver is possibly the most common case, and configuring a DNS modules per nameserver should be less cumbersome than one per DNS record.

Interested if @knyar has any suggestions on how to handle this.

feature: regexp matching in the HTTP body

Before I send a pull request, I wanted to check whether the following feature would be accepted:

As a defence in depth mechanism, the HTTP module in probers such as kanla supports checking whether the returned body matches a certain regular expression. That way, you can be alerted when the webserver still returns an HTTP 200, but all the application prints is “could not connect to database”. Obviously, applications should return proper HTTP status codes, but I still think it’s a nice feature to get some more confidence in your alerting, especially when you’re running applications that you did not write yourself.

What do you think?

ICMP probe fails if port is present in target

Now that Prometheus defaults target ports to 80 if unspecified this fails.

I guess the ICMP probe should drop the port if present in the target (which it will be now I guess).

option to ignore certificate

When starting the blackbox exporter with the insecure_skip_verify, it fails on the certificate.
According to issue #41 this options was to allow self-signed certificates I imagine so this is to be expected.

Would it be possible to add an option to ignore the certificate alltogether?

In the below usecase I probe for machine.example.com instead of www.example.com to bypass DNS geo load balancing. Adding the domain to the certificate will make the domain known to everyone who checks the certificate so not a solution.

The curl equivalent would be
curl -k machine.example.com
wget would do it as follows
wget --no-check-certificate machine.example.com

time="2016-11-04T12:41:19Z" level=warning msg="Error for HTTP request to https://machine.example.com: Get https://machine.example.com: x509: certificate is valid for www.example.com, not machine.example.com" source="http.go:97"

Honor dns TTL

Hi there, i'm facing sometime follow issue:
blackbox_exporter log:

WARN[0675] Error for HTTP request to https://somefancyhost.com/home: Get https://somefancyhost.com/home: dial tcp4: lookup somefancyhost.com on some-dns-server:53: no such host  source=http.go:164
WARN[0675] Error for HTTP request to https://somefancyhost.com/home: Get https://somefancyhost.com/home: dial tcp4: lookup somefancyhost.com on some-dns-server:53: no such host  source=http.go:164
WARN[0675] Error for HTTP request to https://somefancyhost.com/home: Get https://somefancyhost.com/home: dial tcp4: lookup somefancyhost.com on some-dns-server:53: no such host  source=http.go:164
WARN[0675] Error for HTTP request to https://somefancyhost.com/home: Get https://somefancyhost.com/home: dial tcp4: lookup somefancyhost.com on some-dns-server:53: no such host  source=http.go:164
WARN[0675] Error for HTTP request to https://somefancyhost.com/home: Get https://somefancyhost.com/home: dial tcp4: lookup somefancyhost.com on some-dns-server:53: no such host  source=http.go:164

dig output:

somefancyhost.com.  3600    IN      A       10.0.0.2

Any chance to get some cache for dns queries and honor DNS record TTL, to not bomb dns server with requests, while measuring http/tcp?

Static Configuration

I would like to have the blackbox_exporter support static configuration. This means that the configuration such as:

target
period
module + target combination

can be specified in the configuration file and the blackbox_exporter will add labels to the metrics and scrape the combinations periodically. This could be exposed over a /metrics endpoint alongside the /probe endpoint already exposed.

The reason for this feature is mainly to avoid complex and specific relabelling configuration in prometheus. Also, for our use case, the relabelling doesn't easily work when the blackbox_exporter is exposed as a service over consul.

New release to include DNS probe

There's been a few changes since version 0.1.0, notably the DNS probe:
0.1.0...master

Could a new release be cut?

Test HTTPS only resources

Hi,

We have some URLs which are only accessible using https. When using the given module http_2xx, the check fail since a request with http schema is sent (getsockopt: connection refused" source="http.go:159").

How can i test HTTPS only resources?

Thx,
Sven

Question regarding probing remote instances having basic auth

Hi,

Some of our instances (e.g. staging servers) are protected using BASIC AUTH. Is there a way using blackbox-exporter to probe those servers (e.g. using module http_2xx)

Thx in Advance,
Sven

out of memory on make

It throws an out of memory option when i run the make command
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x698ab0, 0x16)
/usr/local/go/src/runtime/panic.go:527 +0x90
runtime.sysMap(0xc824970000, 0x100000, 0x68b00, 0x7af258)
/usr/local/go/src/runtime/mem_linux.go:143 +0x9b
runtime.mHeap_SysAlloc(0x790840, 0x100000, 0x0)
/usr/local/go/src/runtime/malloc.go:423 +0x160
runtime.mHeap_Grow(0x790840, 0x8, 0x0)
/usr/local/go/src/runtime/mheap.go:628 +0x63
runtime.mHeap_AllocSpanLocked(0x790840, 0x1, 0x2b4533105ce0)
/usr/local/go/src/runtime/mheap.go:532 +0x5f1
runtime.mHeap_Alloc_m(0x790840, 0x1, 0x13, 0x2b4533105ce0)
/usr/local/go/src/runtime/mheap.go:425 +0x1ac
runtime.mHeap_Alloc.func1()
/usr/local/go/src/runtime/mheap.go:484 +0x41
runtime.systemstack(0xc820041e50)
/usr/local/go/src/runtime/asm_amd64.s:278 +0xab
runtime.mHeap_Alloc(0x790840, 0x1, 0x10000000013, 0x40dde4)
/usr/local/go/src/runtime/mheap.go:485 +0x63
runtime.mCentral_Grow(0x799258, 0x0)
/usr/local/go/src/runtime/mcentral.go:190 +0x93
runtime.mCentral_CacheSpan(0x799258, 0x2b4533105ce0)
/usr/local/go/src/runtime/mcentral.go:86 +0x4d4
runtime.mCache_Refill(0x2b4532e3f000, 0xc800000013, 0x2b4533105ce0)
/usr/local/go/src/runtime/mcache.go:118 +0xcf
runtime.mallocgc.func2()
/usr/local/go/src/runtime/malloc.go:611 +0x2b
runtime.systemstack(0xc820022000)
/usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
/usr/local/go/src/runtime/proc1.go:674

goroutine 1 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc82003ac00 sp=0xc82003abf8
runtime.mallocgc(0x140, 0x664640, 0x0, 0xc8200877c0)
/usr/local/go/src/runtime/malloc.go:612 +0x842 fp=0xc82003acd0 sp=0xc82003ac00
runtime.newobject(0x664640, 0xc822063290)
/usr/local/go/src/runtime/malloc.go:760 +0x42 fp=0xc82003acf8 sp=0xc82003acd0
cmd/link/internal/ld.readsym(0xc820072000, 0xc8248b38e0, 0xc820eb3270, 0xd, 0xc8248aa150, 0x6d)
/usr/local/go/src/cmd/link/internal/ld/objfile.go:208 +0x76a fp=0xc82003b140 sp=0xc82003acf8
cmd/link/internal/ld.ldobjfile(0xc820072000, 0xc8248b38e0, 0xc820eb3270, 0xd, 0x34e38, 0xc8248aa150, 0x6d)
/usr/local/go/src/cmd/link/internal/ld/objfile.go:147 +0xa56 fp=0xc82003b350 sp=0xc82003b140
cmd/link/internal/ld.ldobj(0xc8248b38e0, 0xc820eb3270, 0xd, 0x34e60, 0xc8248aa150, 0x6d, 0xc820fe4d90, 0x65, 0x1)
/usr/local/go/src/cmd/link/internal/ld/lib.go:1208 +0x14e1 fp=0xc82003b5b8 sp=0xc82003b350
cmd/link/internal/ld.objfile(0xc820fe4e70)
/usr/local/go/src/cmd/link/internal/ld/lib.go:730 +0x10d5 fp=0xc82003b8c8 sp=0xc82003b5b8
cmd/link/internal/ld.loadlib()
/usr/local/go/src/cmd/link/internal/ld/lib.go:482 +0x5ce fp=0xc82003b9f0 sp=0xc82003b8c8
cmd/link/internal/ld.Ldmain()
/usr/local/go/src/cmd/link/internal/ld/pobj.go:212 +0x1c4c fp=0xc82003be70 sp=0xc82003b9f0
cmd/link/internal/amd64.Main()
/usr/local/go/src/cmd/link/internal/amd64/obj.go:44 +0x19 fp=0xc82003be78 sp=0xc82003be70
main.main()
/usr/local/go/src/cmd/link/main.go:26 +0x189 fp=0xc82003bf50 sp=0xc82003be78
runtime.main()
/usr/local/go/src/runtime/proc.go:111 +0x2b0 fp=0xc82003bfa0 sp=0xc82003bf50
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82003bfa8 sp=0xc82003bfa0
make: *** [blackbox_exporter] Error 2

Allow running external program to get probe results

blackbox_exporter would just start a configured process for a module, and return its output as the result of the probe. This would allow it to be very extensible, without needing to bloat the codebase.

It's also possible that this should be written as a separate exporter and not as a feature of blackbox_exporter, but it will be very useful either way.

Use custom registry

Now that the go client supports them, we should switch to custom registries rather than hand-creating the output.

ARM support

When pulling the Docker image, how do I get an ARM version for my Raspberry PI?

standard_init_linux.go:175: exec user process caused "exec format error"

blackbox-exporter memory leak with Grafana https endpoints?

Grafana shows prometheus memory going up (blue line: 4GB limit is yet to be reached, but close).
Meanwhile, blackbox-exporter is reaching its imposed mem_limit: 1G and gets killed and restarted (red line) and that cycle repeats every few hours.
The node and collectd exporters are working fine, in their 15-20MB range, as expected...

I tried both :latest and :master images from the docker hub, so it doesn't seem to have been fixed in any of those versions.

Could it be accumulating the [connection / header / socket] timeouts seen in the logs (below) prior to stack dumping because it's running out of ram?

docker-compose.yml section:

  blackbox_exporter:
    image: prom/blackbox-exporter:master
    restart: always
    expose:
      - 9115
    volumes:
      - ./blackbox-config/:/config/
    networks:
      - backend
    command:
      - '-config.file=/config/blackbox.yml'
    cap_drop:
      - all
    cap_add:
    - NET_RAW
    security_opt:
      - no-new-privileges
    mem_limit: 1G
    cpu_shares: 1024

blackbox.yml:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []  # Defaults to 2xx
      method: GET
      headers:
        Strict-Transport-Security: max-age=31536000; includeSubdomains
        #Host: example.com
        #Accept-Language: en-US
      #no_follow_redirects: false
      #fail_if_ssl: false
      fail_if_not_ssl: true
      #fail_if_matches_regexp:
      #- "Could not connect to database"
      fail_if_not_matches_regexp:
      - "My Title</title>"

  tcp_connect:
    prober: tcp
    timeout: 5s

  ssh_banner:
    prober: tcp
    timeout: 5s
    tcp:
      query_response:
      - expect: "^SSH-2.0-"

  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      protocol: "icmp"
      preferred_ip_protocol: "ip4"

prometheus.yml relevant section example:

  - job_name: 'probe_https_instances'
    metrics_path: /probe

    params:
      module: [http_2xx]  # Look for a HTTP 200 response.

    static_configs:
      - targets:
        - site1.example.com
        - site2.example.com
        - site3.example.com
        # etc.

    relabel_configs:
      - source_labels: [__address__]
        regex: (.*)(:80)?
        target_label: __param_target
        replacement: https://${1}

      - source_labels: [__param_target]
        regex: (.*)
        target_label: instance
        replacement: ${1}

      - source_labels: []
        regex: .*
        target_label: __address__
        replacement: blackbox_exporter:9115


  - job_name: 'probe_ping_instances'
    metrics_path: /probe

    params:
      module: [icmp]  # Look for a HTTP 200 response.

    static_configs:
      - targets:          # Target to probe
        - site1.example.com
        - site2.example.com
        - site3.example.com
        - site4.example.com
       # etc. ~20 servers.

    relabel_configs:
      - source_labels: [__address__]
        regex: (.*)(:80)?
        target_label: __param_target
        replacement: ${1}

      - source_labels: [__param_target]
        regex: (.*)
        target_label: instance
        replacement: ${1}

      - source_labels: []
        regex: .*
        target_label: __address__
        replacement: blackbox_exporter:9115  # Blackbox exporter.

blackbox.log:

Dec 17 17:27:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:28Z" level=info msg="Timeout reading from socket for site6.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:28Z" level=warning msg="Error for HTTP request to https://site6.example.com: Get https://site6.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:28Z" level=info msg="Timeout reading from socket for site12.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:30 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:29Z" level=info msg="Timeout reading from socket for site1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:30 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:30Z" level=warning msg="Error for HTTP request to https://site6-staging.example.com: Get https://site6-staging.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:31 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:29Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:31 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:30Z" level=info msg="Timeout reading from socket for site11.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:31 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:30Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:32 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:31Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:32 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:31Z" level=warning msg="Error for HTTP request to https://site9.example.com: Get https://site9.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:33Z" level=info msg="Timeout reading from socket for site9.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:34Z" level=warning msg="Error for HTTP request to https://example.com: Get https://example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:33Z" level=warning msg="Error for HTTP request to https://site8.example.com: Get https://site8.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:34Z" level=warning msg="Error for HTTP request to https://site7.example.com: Get https://site7.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:37 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:35Z" level=info msg="Timeout reading from socket for site3.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:37 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:35Z" level=info msg="Timeout reading from socket for host1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:41 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:40Z" level=warning msg="Error for HTTP request to https://site11.example.com: Get https://site11.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:42 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:41Z" level=info msg="Timeout reading from socket for example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:42 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:41Z" level=info msg="Timeout reading from socket for site5.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:43 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:41Z" level=warning msg="Error for HTTP request to https://site4.example.com: Get https://site4.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:45 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:45Z" level=warning msg="Error for HTTP request to https://site1.example.com: Get https://site1.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:46 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:45Z" level=warning msg="Error for HTTP request to https://site2.example.com: Get https://site2.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:47 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:46Z" level=warning msg="Error for HTTP request to https://site3.example.com: Get https://site3.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:48 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:46Z" level=warning msg="Error for HTTP request to https://site5.example.com: Get https://site5.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:49 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:48Z" level=info msg="Timeout reading from socket for site8.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:49 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:48Z" level=warning msg="Error for HTTP request to https://site10.example.com: Get https://site10.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:50 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:48Z" level=info msg="Timeout reading from socket for host0.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:53 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:52Z" level=info msg="Timeout reading from socket for site10.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:54 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:52Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:54 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:54Z" level=warning msg="Error for HTTP request to https://site6.example.com: Get https://site6.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:55 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:53Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:55 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:53Z" level=info msg="Timeout reading from socket for site6.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:56 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:55Z" level=warning msg="Error for HTTP request to https://site6-staging.example.com: Get https://site6-staging.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:27:57 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:55Z" level=info msg="Timeout reading from socket for site11.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:57 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:56Z" level=info msg="Timeout reading from socket for site12.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:27:59 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:57Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:00 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:59Z" level=info msg="Timeout reading from socket for site6-staging.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:00 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:59Z" level=warning msg="Error for HTTP request to https://site9.example.com: Get https://site9.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:00 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:59Z" level=warning msg="Error for HTTP request to https://site7.example.com: Get https://site7.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:01 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:00Z" level=info msg="Timeout reading from socket for site9.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:02 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:01Z" level=warning msg="Error for HTTP request to https://site8.example.com: Get https://site8.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:03 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:01Z" level=info msg="Timeout reading from socket for host1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:05 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:04Z" level=info msg="Timeout reading from socket for site2.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:08 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:07Z" level=info msg="Timeout reading from socket for site3.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:08 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:07Z" level=warning msg="Error for HTTP request to https://example.com: Get https://example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:09 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:08Z" level=warning msg="Error for HTTP request to https://site4.example.com: Get https://site4.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:10 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:08Z" level=warning msg="Error for HTTP request to https://site11.example.com: Get https://site11.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:11 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:09Z" level=info msg="Timeout reading from socket for example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:12 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:11Z" level=info msg="Timeout reading from socket for site4.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:13 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:12Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:15 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:14Z" level=warning msg="Error for HTTP request to https://site1.example.com: Get https://site1.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:18 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:17Z" level=warning msg="Error for HTTP request to https://site2.example.com: Get https://site2.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:18 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:16Z" level=warning msg="Error for HTTP request to https://site3.example.com: Get https://site3.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:19 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:17Z" level=warning msg="Error for HTTP request to https://site10.example.com: Get https://site10.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:20 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:19Z" level=info msg="Timeout reading from socket for site8.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:21 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:19Z" level=warning msg="Error for HTTP request to https://site5.example.com: Get https://site5.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:21 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:20Z" level=info msg="Timeout reading from socket for host0.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:23 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:20Z" level=info msg="Timeout reading from socket for site10.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:24 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:23Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:25 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:23Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:26 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:24Z" level=warning msg="Error for HTTP request to https://site6.example.com: Get https://site6.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:27Z" level=info msg="Timeout reading from socket for site12.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:26Z" level=info msg="Timeout reading from socket for site1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:27Z" level=warning msg="Error for HTTP request to https://site6-staging.example.com: Get https://site6-staging.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:28Z" level=info msg="Timeout reading from socket for site6.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:30 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:28Z" level=info msg="Timeout reading from socket for site11.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:32 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:31Z" level=warning msg="Error for HTTP request to https://site9.example.com: Get https://site9.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:33 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:30Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:32Z" level=warning msg="Error for HTTP request to https://site7.example.com: Get https://site7.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:32Z" level=info msg="Timeout reading from socket for site6-staging.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:32Z" level=warning msg="Error for HTTP request to https://site8.example.com: Get https://site8.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:38 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:37Z" level=info msg="Timeout reading from socket for site9.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:38 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:35Z" level=warning msg="Error for HTTP request to https://example.com: Get https://example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:42 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:40Z" level=info msg="Timeout reading from socket for host1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:42 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:40Z" level=info msg="Timeout reading from socket for site3.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:42 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:41Z" level=info msg="Timeout reading from socket for site4.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:43 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:41Z" level=info msg="Timeout reading from socket for site2.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:45 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:43Z" level=warning msg="Error for HTTP request to https://site11.example.com: Get https://site11.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:47 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:44Z" level=info msg="Timeout reading from socket for example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:55 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:51Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:28:56 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:49Z" level=warning msg="Error for HTTP request to https://site4.example.com: Get https://site4.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:56 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:53Z" level=warning msg="Error for HTTP request to https://site3.example.com: Get https://site3.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:57 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:53Z" level=warning msg="Error for HTTP request to https://site1.example.com: Get https://site1.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:59 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:55Z" level=warning msg="Error for HTTP request to https://site10.example.com: Get https://site10.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:28:59 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:57Z" level=warning msg="Error for HTTP request to https://site2.example.com: Get https://site2.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:29:02 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:56Z" level=info msg="Timeout reading from socket for site5.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:29:06 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:02Z" level=warning msg="Error for HTTP request to https://site5.example.com: Get https://site5.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:29:10 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:07Z" level=info msg="Timeout reading from socket for site8.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:29:15 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:11Z" level=warning msg="Error for HTTP request to https://site6.example.com: Get https://site6.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:29:19 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:15Z" level=info msg="Timeout reading from socket for site10.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:29:21 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:17Z" level=info msg="Timeout reading from socket for site7.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:29:22 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:17Z" level=warning msg="Error for HTTP request to https://site6-staging.example.com: Get https://site6-staging.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:29:23 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:17Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:29:25 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:21Z" level=info msg="Timeout reading from socket for host0.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:29:25 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:21Z" level=info msg="Timeout reading from socket for site6.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:29:26 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:22Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:29:32 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:28Z" level=info msg="Timeout reading from socket for site1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:29:35 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:33Z" level=warning msg="Error for HTTP request to https://site8.example.com: Get https://site8.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:29:37 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:33Z" level=info msg="Timeout reading from socket for site9.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:29:41 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:36Z" level=info msg="Timeout reading from socket for site12.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:29:42 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:38Z" level=warning msg="Error for HTTP request to https://site9.example.com: Get https://site9.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:29:45 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:40Z" level=info msg="Timeout reading from socket for site6-staging.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:29:49 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:45Z" level=warning msg="Error for HTTP request to https://example.com: Get https://example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:29:49 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:44Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:29:49 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:44Z" level=info msg="Timeout reading from socket for site11.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:29:54 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:48Z" level=warning msg="Error for HTTP request to https://site7.example.com: Get https://site7.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:30:09 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:04Z" level=info msg="Timeout reading from socket for example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:30:11 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:08Z" level=info msg="Timeout reading from socket for site4.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:30:15 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:10Z" level=info msg="Timeout reading from socket for site3.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:30:17 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:12Z" level=info msg="Timeout reading from socket for site5.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:30:17 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:12Z" level=info msg="Timeout reading from socket for site2.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:30:20 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:14Z" level=warning msg="Error for HTTP request to https://site4.example.com: Get https://site4.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:30:22 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:15Z" level=warning msg="Error for HTTP request to https://site1.example.com: Get https://site1.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:30:26 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:22Z" level=warning msg="Error for HTTP request to https://site11.example.com: Get https://site11.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:30:32 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:26Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:30:52 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:48Z" level=warning msg="Error for HTTP request to https://site3.example.com: Get https://site3.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:30:55 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:51Z" level=info msg="Timeout reading from socket for site11.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:30:55 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:51Z" level=info msg="Timeout reading from socket for site1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:30:56 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:51Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146" 
Dec 17 17:30:59 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:51Z" level=error msg="Error resolving address host1.example.com: no suitable address found" source="icmp.go:81" 
Dec 17 17:31:00 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:55Z" level=error msg="Error resolving address site8.example.com: no suitable address found" source="icmp.go:81" 
Dec 17 17:31:02 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:58Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: 2016/12/17 22:31:03 http: panic serving 172.19.0.3:51572: runtime error: invalid memory address or nil pointer dereference
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: goroutine 1177822 [running]:
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*conn).serve.func1(0xc43ad9c480)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1491 +0x12a
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: panic(0x83a940, 0xc420010020)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/runtime/panic.go:458 +0x243
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: main.probeICMP(0xc4584b1ede, 0x17, 0xab3000, 0xc44f452750, 0xc4200ee568, 0x4, 0x12a05f200, 0x0, 0x0, 0x0, ...)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/icmp.go:84 +0x1fc
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: main.probeHandler(0xab3000, 0xc44f452750, 0xc45604aff0, 0xc42001e0b0)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/main.go:130 +0x218
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: main.main.func1(0xab3000, 0xc44f452750, 0xc45604aff0)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/main.go:170 +0x4a
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: net/http.HandlerFunc.ServeHTTP(0xc4200eeed0, 0xab3000, 0xc44f452750, 0xc45604aff0)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1726 +0x44
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*ServeMux).ServeHTTP(0xad3ae0, 0xab3000, 0xc44f452750, 0xc45604aff0)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2022 +0x7f
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: net/http.serverHandler.ServeHTTP(0xc420058800, 0xab3000, 0xc44f452750, 0xc45604aff0)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2202 +0x7d
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*conn).serve(0xc43ad9c480, 0xab3980, 0xc4584b1e80)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1579 +0x4b7
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: created by net/http.(*Server).Serve
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2293 +0x44d
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: 2016/12/17 22:31:05 http: panic serving 172.19.0.3:51614: runtime error: invalid memory address or nil pointer dereference
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: goroutine 1177995 [running]:
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*conn).serve.func1(0xc44afe0b80)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1491 +0x12a
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: panic(0x83a940, 0xc420010020)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/runtime/panic.go:458 +0x243
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: main.probeICMP(0xc4511c521e, 0x15, 0xab3000, 0xc44f453520, 0xc4200ee568, 0x4, 0x12a05f200, 0x0, 0x0, 0x0, ...)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/icmp.go:84 +0x1fc
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: main.probeHandler(0xab3000, 0xc44f453520, 0xc453964780, 0xc42001e0b0)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/main.go:130 +0x218
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: main.main.func1(0xab3000, 0xc44f453520, 0xc453964780)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/main.go:170 +0x4a
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: net/http.HandlerFunc.ServeHTTP(0xc4200eeed0, 0xab3000, 0xc44f453520, 0xc453964780)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1726 +0x44
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*ServeMux).ServeHTTP(0xad3ae0, 0xab3000, 0xc44f453520, 0xc453964780)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2022 +0x7f
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: net/http.serverHandler.ServeHTTP(0xc420058800, 0xab3000, 0xc44f453520, 0xc453964780)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2202 +0x7d
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*conn).serve(0xc44afe0b80, 0xab3980, 0xc4511c51c0)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1579 +0x4b7
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: created by net/http.(*Server).Serve
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2293 +0x44d
Dec 17 17:31:09 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:31:00Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:31:10 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:31:04Z" level=error msg="Error resolving address site10.example.com: no suitable address found" source="icmp.go:81" 
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: 2016/12/17 22:31:14 http: panic serving 172.19.0.3:51616: runtime error: invalid memory address or nil pointer dereference
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: goroutine 1177996 [running]:
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*conn).serve.func1(0xc44afe1b00)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1491 +0x12a
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: panic(0x83a940, 0xc420010020)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/runtime/panic.go:458 +0x243
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: main.probeICMP(0xc45009da0e, 0x1c, 0xab3000, 0xc44f4535f0, 0xc4200ee568, 0x4, 0x12a05f200, 0x0, 0x0, 0x0, ...)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/icmp.go:84 +0x1fc
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: main.probeHandler(0xab3000, 0xc44f4535f0, 0xc4425a00f0, 0xc42001e0b0)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/main.go:130 +0x218
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: main.main.func1(0xab3000, 0xc44f4535f0, 0xc4425a00f0)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/main.go:170 +0x4a
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: net/http.HandlerFunc.ServeHTTP(0xc4200eeed0, 0xab3000, 0xc44f4535f0, 0xc4425a00f0)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1726 +0x44
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*ServeMux).ServeHTTP(0xad3ae0, 0xab3000, 0xc44f4535f0, 0xc4425a00f0)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2022 +0x7f
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: net/http.serverHandler.ServeHTTP(0xc420058800, 0xab3000, 0xc44f4535f0, 0xc4425a00f0)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2202 +0x7d
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*conn).serve(0xc44afe1b00, 0xab3980, 0xc456907540)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1579 +0x4b7
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: created by net/http.(*Server).Serve
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2293 +0x44d
Dec 17 17:31:17 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:31:12Z" level=warning msg="Error for HTTP request to https://site8.example.com: Get https://site8.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164" 
Dec 17 17:31:25 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:31:25Z" level=info msg="Starting blackbox_exporter (version=0.3.0, branch=master, revision=1722550ff00dc7435e8b39adf292bfc51b69c26f)" source="main.go:151" 
Dec 17 17:31:25 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:31:25Z" level=info msg="Build context (go=go1.7.3, user=root@b7e4cd3247af, date=20161215-10:43:56)" source="main.go:152" 
Dec 17 17:31:25 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:31:25Z" level=info msg="Listening on :9115" source="main.go:183"

DNS prober: Target nameserver(s) in resolv.conf

I'd like to add support to the DNS prober to be able to query nameservers in resolv.conf.

The dns library used by the DNS prober supports parsing resolv.conf, which is useful for emulating whether an application in userspace that relies on resolv.conf can successfully query a specified DNS record.

Some considerations to work out:

Should the DNS prober return a success if just one of the nameservers from resolv.conf succeeds? I think yes.
How should resolv.conf be defined as a target? Using a magic string constant specified in the query string? See related issue: #51

Using the documented relabel_configs results in false positive

Hi,

relabel_configs:
  - source_labels: [__address__]
    regex: (.*)(:5050)?
    target_label: __param_target
    replacement: ${1}
  - source_labels: [__param_target]
    regex: (.*)
    target_label: instance
    replacement: ${1}
  - source_labels: []
    regex: .*
    target_label: __address__
    replacement: 127.0.0.1:9115  # Blackbox exporter.

For some reason, this relabel config causes the http_2xx to report that a targeted service is available, even the host is inaccessible for whatever reason, such as firewall rules. Here's the output from a curl that shows the host is not reachable, yet when using the above relabel config, Prometheus reports it is 'Up':

`[aop-dev@prometheus-i-e588886d ~]$ curl -v content-service.si.example.org:5050

About to connect() to content-service.si.example.org port 5050 (#0)
Trying 54.555.55.55...
Connection timed out
Trying 55.555.55.55...
After 86359ms connect time, move on!
Failed connect to content-service.si.example.org:5050; Operation now in progress
Closing connection 0
curl: (7) Failed connect to content-service.si.example.org:5050; Operation now in progress`

(TLDs and IPs changed for privacy)

Removing the block that targets __address__ solves this issue, but leaves the wrong value displayed in the interface.

Prometheus metrics path with probe not working

It might be that I'm doing it wrong, but I want to add the probe path as a metrics path in the Prometheus configuration, something like:

- job_name: "blackbox_exporter"
  scheme: "http"
  metrics_path: "/probe?target=google.com&module=http_2xx"
  static_configs:
  - targets:
    - localhost:9115
    labels:
      target_name: "google.com"

This doesn't work. In Prometheus I get the error text format parsing error in line 1: invalid metric name. When I look at the prometheus server localhost:9090/targets I see the endpoint as I put it in the configuration, however, when I click on it, I get the standard blackbox page:

Which is probably causing the error. When I look at the url in the browser then, I see that it's not http://localhost:9115/probe?target=google.com&module=http_2xx, but http://localhost:9115/probe%3target=google.com&module=http_2xx. The ? in the metric_path seems to get encoded (%3). When I replace the %3 by a ? it works as expected.

Since nobody else seems to have this problem, I might be doing something wrong. Any help would be greatly appreciated.

Support HUP signal configuration reloading

It would be nice to support configuration reload via SIGHUP.

dial tcp6: no suitable address found

I'm using:

Digest: sha256:0eaf1811c018de2d87d6d14ad848cd5fb2333b7d17eeb9da19a3101796e8f865
Status: Image is up to date for prom/blackbox-exporter:master

http://s.natalian.org/2016-11-05/blackboxexporter.txt

I guess I need to force tcp4 somehow, somewhere?

blackbox exporter failing to recognised the service is not responding

I was quite amazed to see that the service is termed still up and ok while in the stdout logs of the blackbox exporter is showing errors like this

time="2016-11-21T10:46:40Z" level=warning msg="Error for HTTP request to http://svc.proj.svc:19080: Get http://svc.proj.svc:19080: dial tcp4 172.30.162.209:19080: getsockopt: no route to host" source="http.go:159"

I have intentionally , scaled a service to zero pods in openshift origin. and i was expecting a instance down alert from prometheus.. How ever form target view the blackbox exporter never seems to recognize that service is down and is not responding

while i try to directly query the backend service using the blackbox exportetservice ip

curl http://172.30.235.19/probe?target=svc.proj.svc:19080
probe_ip_protocol 4
probe_http_status_code 0
probe_http_content_length 0
probe_http_redirects 0
probe_http_ssl 0
probe_duration_seconds 3.031048
probe_success 0

i am using module http_2xx so since the status code is 0 , should not it have raised (lavel) service as down in targets view in prometheus Web UI.

Also additionally this query works

curl http://172.30.235.19/probe?target=svc.project.svc:19080&module=http_2xx

probe_http_status_code 0
probe_http_content_length 0
probe_http_redirects 0
probe_http_ssl 0
probe_duration_seconds 3.040584
probe_success 0

while

 curl http://172.30.235.19/probe?module=http_2xx&target=svc.project.svc:19080

produces
Target parameter is missing

and bails out .

Am i missing something on the alert part. if a service dont have any endpoints i am hoping that probing to it fails or i am missing something?

I am using blackbox exporter container version0.3,0
as it start it shows

"Starting blackbox_exporter (version=0.3.0, branch=master, revision=43cfa9854e595993b64925c8a5b5f39086b03dae)"

how to use blackbox_exporter inside kubernetes

my rc yaml:

apiVersion: v1
kind: ReplicationController
metadata:
  name: blackbox-exporter
spec:
  replicas: 1
  selector:
    app: blackbox-exporter
  template:
    metadata:
      name: blackbox-exporter
      labels:
        app: blackbox-exporter
    spec:
      containers:
      - image: ****/prometheus/blackbox-exporter:1.0
        name: blackbox
        ports:
        - containerPort: 9115
          name: blackbox

my svc yaml:

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  labels:
    name: rabbitmq
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
    prometheus.io/probe: "true"
spec:
  ports:
    - port: 9090
      name: exporter
      targetPort: exporter
      protocol: TCP
  selector:
    app: rabbitmq

but prometheus targets found down of probe

ICMP reports up=1 although host is down

Hi,

i have a problem that the ICMP probe reports up=1 although the host is down.
This is the snippet of my blackbox.yml:

modules:
  ping:
    prober: icmp
    timeout: 5s
    icmp:
      protocol: "icmp"
      preferred_ip_protocol: "ip4"

This is the according snippet of my prometheus.yml:

  - job_name: 'ping'
    metrics_path: /probe
    params: 
      module: [ping]
    static_configs:
      - targets:
        - cram
    relabel_configs:
      # Set the target's address as the target-GET-parameter for the blackbox exporter
      - source_labels: [__address__]
        regex: (.*)
        target_label: __param_target
        replacement: ${1}
      # Set the target's address as the instance name
      - source_labels: [__address__]
        regex: (.*)
        target_label: instance
        replacement: ${1}
      # Set the scrape address to the blackbox exporter
      - source_labels: []
        regex: .*
        target_label: __address__
        replacement: 127.0.0.1:9115

The relabel_configs are almost directly copy&pasted from the github README.

So the host cram did exist in the network, but is currently down, but the exporter reports it as up.

This is the output of the ping command:

$ ping cram
PING cram.home.simonszu.de (192.168.1.15) 56(84) bytes of data.
From tirn.home.simonszu.de (192.168.1.9) icmp_seq=1 Destination Host Unreachable
From tirn.home.simonszu.de (192.168.1.9) icmp_seq=2 Destination Host Unreachable
From tirn.home.simonszu.de (192.168.1.9) icmp_seq=3 Destination Host Unreachable
^C
--- cram.home.simonszu.de ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4005ms

Please note that the host tirn is my dedicated monitoring server, running prometheus and the blackbox exporter.

So i think this is a bug, or at least not the intended behaviour. I am no ICMP expert, but i think it has something to do with the fact that the ping command returns at least something.

I am using latest prometheus and blackbox_exporter on armv7.

Docker container versioning not implemented

Hi,
I am using blackbox_exporter and I would like to use the official prom/blackbox-exporter container from docker hub.
The versioning system is not implemented and currently prom/blackbox-exporter:latest contains blackbox-exporter 0.1.0. Is it possible to push containers with a sane versioning system?

Please support custom CAs for http prober

The tcp prober supports specifying a custom certificate authority store, but the http one does not.

cannot set proper header for 30. requests

+----------+     +-----------+
|          +---->|           |
|prometheus|     |bb-exporter|
|          |     |           |
+----------+     +----+------+
                      |
                      |
                      |
                      v
         +-----------------+
         |                 |
         |       nginx     |
         |                 |
         +-----------------+

I have multiple targets configured in prometheus.yml for blackbox_exporter, and when BB got request to go and scrape, BB sometimes does not use configured attributes:

modules:
  http_2xx:
    prober: http
    timeout: 30s
    http:
      valid_status_codes: [200,301,302]
      method: GET
      headers:
        User-Agent: bb-prober
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: false
      protocol: "tcp4"
      preffered_ip_protocol: "ip4"

how does this look like from web server log:

10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home HTTP/1.1" *301* 178 "-" "bb-prober" "-" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home HTTP/1.1" *301* 178 "-" "bb-prober" "-" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home HTTP/1.1" *301* 178 "-" "bb-prober" "-" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home HTTP/1.1" *301* 178 "-" "bb-prober" "-" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home HTTP/1.1" *301* 178 "-" "bb-prober" "-" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home HTTP/1.1" *301* 178 "-" "bb-prober" "-" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home/ HTTP/1.1" *200* 509 "https://somefancyhost.com/home" "Go-http-client/1.1" "2.71" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home/ HTTP/1.1" *200* 509 "https://somefancyhost.com/home" "Go-http-client/1.1" "2.71" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home/ HTTP/1.1" *200* 509 "https://somefancyhost.com/home" "Go-http-client/1.1" "2.71" - 0.000 -

Assuming my observations, when response code is not 200, header is not set.

icmp prober measurement includes DNS lookup time

See https://groups.google.com/forum/#!topic/prometheus-developers/XrTeEkb1WAU for context.

feature: IRC probing

Would a pull request be accepted which verifies that an IRC server is running on a specific address?

Thanks.

http parse content

Hi,

I would like to monitor 100+ web apps. But I need identify, that I monitor right site, so in all web app I have <!-- Site: {{fqdn}} --!>. I trying to create check with blackbox_exporter but I can't find way how to do that. Does blackbox_exporter support this kind of check? I hope that fail_if_not_matches_regexp can help me, but I don't know how to set it to parse diferent hostname for all my sites.

Thanks for help

support for payloads for http prober

it would be nice if the http prober supported a payload in the request, this would be similarly true for the tcp part of the prober.

Support HTTP request body

Could there please be an example of a POST with a JSON payload & a check on the response like looking for a "status": "success" in the JSON response.

more verbose messages for tcp prober when errors are encountered

feature/wishlist request from #69: when using tcp prober and port is missing (or other kind of errors e.g. address malformed) no message is logged at any level. IMO blackbox_exporter should at least signal when such errors are encountered.

Instance shown as "up" although status code 403 (module http_2xx)

Hi,

I'm using the latest Docker image of blackbox for probing targets. I'm having a target probe (module http_2xx, config as in https://github.com/prometheus/blackbox_exporter/blob/master/blackbox.yml) where the server returns status code 403 (due to IP resctriction), but the target instance is still shown as "up".I would expect status "down" since i'm using the defaul config for module http_2xx which checks for status code 200-300.

When looking at the value of probe_http_status_code for this instance, it shows 403.

Any idea why the instance is still shown as "up"?

Cheers Sven

Rationalise example config and included config

The config in the README is meant to show all the config options, however it also contains several useful modules that should be in blackbox.yml. Minimise it and move over whatever makes sense to blackbox.yml.

Negative Probe Duration

Hi there,

I have been scraping and monitoring a number of target urls for a while now and on 19 April 2016, a couple of them reflected negative probe duration (seconds). This happened for a brief period of 10 seconds around 5:47 PM. May i know if this is normal?

Thanks in advance

cert vs wildcard

I'm getting errors in log;
certificate is valid for *.foo.com, foo.com, not abc.foo.com

Shouldn't blackbox accept a wildcard cert without the extra noise?

run time error on make

I am using go1.6.2 and centos6.8
This is the error being reflected on screen when i run make.

panic: runtime error: slice bounds out of range

goroutine 1 [running]:
panic(0x93d400, 0xc8200100a0)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
main.vendoredImportPath(0xc8201dc800, 0xc820250331, 0x20, 0x0, 0x0)
    /usr/local/go/src/cmd/go/pkg.go:466 +0x713
main.loadImport(0xc820250331, 0x20, 0xc820053f20, 0x58, 0xc8201dc800, 0xc82049b788, 0xc820250630, 0x1, 0x1, 0x1, ...)
    /usr/local/go/src/cmd/go/pkg.go:336 +0xe52
main.(*Package).load(0xc8201dc800, 0xc82049b788, 0xc8202dc000, 0x0, 0x0, 0x4)
    /usr/local/go/src/cmd/go/pkg.go:953 +0x4397
main.loadImport(0xc820176121, 0x2e, 0xc820144690, 0x22, 0xc820027800, 0xc82049b788, 0xc820176330, 0x1, 0x1, 0x1, ...)
    /usr/local/go/src/cmd/go/pkg.go:377 +0x84e
main.(*Package).load(0xc820027800, 0xc82049b788, 0xc820420e00, 0x0, 0x0, 0xc)
    /usr/local/go/src/cmd/go/pkg.go:953 +0x4397
main.loadImport(0xc82013a3a3, 0x3, 0xc82000a034, 0x22, 0x0, 0xc82049b788, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/cmd/go/pkg.go:377 +0x84e
main.loadPackage(0xc82013a3a3, 0x3, 0xc82049b788, 0x0)
    /usr/local/go/src/cmd/go/pkg.go:1639 +0x13d8
main.packagesAndErrors(0xc82008c000, 0x14, 0x14, 0x0, 0x0, 0x0)
    /usr/local/go/src/cmd/go/pkg.go:1684 +0x531
main.packages(0xc82000a260, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/cmd/go/pkg.go:1652 +0x86
main.runList(0xc8a6a0, 0xc82000a260, 0x1, 0x1)
    /usr/local/go/src/cmd/go/list.go:186 +0x1f6
main.main()
    /usr/local/go/src/cmd/go/main.go:181 +0x783
>> formatting code
panic: runtime error: slice bounds out of range

goroutine 1 [running]:
panic(0x93d400, 0xc82000e0d0)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
main.vendoredImportPath(0xc820199000, 0xc8201ef4a1, 0x20, 0x0, 0x0)
    /usr/local/go/src/cmd/go/pkg.go:466 +0x713
main.loadImport(0xc8201ef4a1, 0x20, 0xc8203a99e0, 0x58, 0xc820199000, 0xc820179838, 0xc8201ef5f0, 0x1, 0x1, 0x1, ...)
    /usr/local/go/src/cmd/go/pkg.go:336 +0xe52
main.(*Package).load(0xc820199000, 0xc820179838, 0xc82016e380, 0x0, 0x0, 0x4)
    /usr/local/go/src/cmd/go/pkg.go:953 +0x4397
main.loadImport(0xc82014b651, 0x2e, 0xc82010bbf0, 0x22, 0xc820135000, 0xc820179838, 0xc82014b860, 0x1, 0x1, 0x1, ...)
    /usr/local/go/src/cmd/go/pkg.go:377 +0x84e
main.(*Package).load(0xc820135000, 0xc820179838, 0xc82012b500, 0x0, 0x0, 0xc)
    /usr/local/go/src/cmd/go/pkg.go:953 +0x4397
main.loadImport(0x9e0b58, 0x1, 0xc820014004, 0x22, 0x0, 0xc820179838, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/cmd/go/pkg.go:377 +0x84e
main.loadPackage(0x9e0b58, 0x1, 0xc820179838, 0x0)
    /usr/local/go/src/cmd/go/pkg.go:1639 +0x13d8
main.packagesAndErrors(0xc82013a1f0, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/cmd/go/pkg.go:1684 +0x531
main.packages(0xc82000a270, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/cmd/go/pkg.go:1652 +0x86
main.runFmt(0xc8a060, 0xc82000a270, 0x0, 0x0)
    /usr/local/go/src/cmd/go/fmt.go:38 +0x8d
main.main()
    /usr/local/go/src/cmd/go/main.go:181 +0x783
make: *** [format] Error 2

Metrics are randomly disappearing

blackbox_exporter 0.3.0 / prometheus 1.4.1

I've faced the issue when blackbox_exporter starts returning probe_success 0 for random hosts after several hours/days of correct functioning.

Prometheus' configuration:

global:
  scrape_interval: 60s
  scrape_timeout: 20s
  external_labels:
    monitor: 'tvdashboard'

scrape_configs:
  - job_name: 'blackbox_devmetrics'
    metrics_path: /probe
    params:
      module: [http_devmetrics]
    static_configs:
      - targets:
        - devmetrics.hostname.com
    relabel_configs: &relabel
      - source_labels: [__address__]
        regex: (.*)(:80)?
        target_label: __param_target
        replacement: ${1}
      - source_labels: [__param_target]
        regex: (.*)
        target_label: instance
        replacement: ${1}
      - source_labels: []
        regex: .*
        target_label: __address__
        replacement: 127.0.0.1:9115  # blackbox_exporter
  - job_name: 'blackbox_rally'
    metrics_path: /probe
    params:
      module: [http_rally]
    static_configs:
      - targets:
        - rally1.rallydev.com/slm/login.op
    relabel_configs: *relabel
  - job_name: 'blackbox_jira'
    metrics_path: /probe
    params:
      module: [http_jira]
    static_configs:
      - targets:
        - jira.hostname.com/login.jsp?os_destination=%2Fsecure%2FDashboard.jspa
    relabel_configs: *relabel
  - job_name: 'blackbox_confluence'
    metrics_path: /probe
    params:
      module: [http_confluence]
    static_configs:
      - targets:
        - confluence.hostname.com/login.action
    relabel_configs: *relabel

blackbox_exporter's configuration

modules:
  http_devmetrics:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: []
      method: GET
      headers: &headers
        Accept-Language: en-US
        User-Agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      fail_if_not_matches_regexp:
        - "username"
      tls_config: &tls_config
        insecure_skip_verify: true
      protocol: "tcp"
      preferred_ip_protocol: "ip4"
  http_rally:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: []
      method: GET
      headers: *headers
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      fail_if_not_matches_regexp:
        - "Sign in"
      tls_config: *tls_config
      protocol: "tcp"
      preferred_ip_protocol: "ip4"
  http_jira:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: []
      method: GET
      headers: *headers
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      fail_if_not_matches_regexp:
        - "login-form-username"
      tls_config: *tls_config
      protocol: "tcp"
      preferred_ip_protocol: "ip4"
  http_confluence:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: []
      method: GET
      headers: *headers
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      fail_if_not_matches_regexp:
        - "Username"
      tls_config: *tls_config
      protocol: "tcp"
      preferred_ip_protocol: "ip4"

I've tried using Prometheus' web UI and CLI to debug. I've noticed that all the targets are active ("up") in Prometheus, but at the same time when I want to get a value of probe_http_status_code metric, I see it only for several (randomly selected) targets, but not for all (this is my expected behavior).

When I'm using CLI, I got the same result for the hosts that aren't displaying in Prometheus UI:

$ curl "http://127.0.0.1:9115/probe?module=http_devmetrics&target=devmetrics.hostname.com"
probe_duration_seconds 0.001795
probe_success 0

So seems like the problem is in blackbox_exporter but not in Prometheus itself.

On the other hand, if using just curl, I see that my service is up and running:

$ curl -L -k "http://devmetrics.hostname.com" | grep -o username
username

I'm doing curl on the same machine where blackbox_exporter is running to eliminate any network issues.

This problem occurs periodically and in most cases is solved by blackbox_exporter restarting.

dns module?

hi,
is a dns module something you'd be interested in having in blackbox exporter? if so I'm happy to take a stab at it using e.g. https://github.com/miekg/dns

thanks!

Ping: Timeout reading from socket

Hi,

i have defined a ping-module which is basically just an icmp prober. I am probing around 17 hosts with it. Now i have a problem: The probe_success for one of this host is 0, although the host is pingable when executing ping in my shell.
Systemd journal shows the following error:

Nov 10 14:58:21 tirn blackbox_exporter[29216]: time="2016-11-10T14:58:21Z" level=info msg="Timeout reading from socket for grandstream: read ip4 0.0.0.0: i/o timeout" source="icmp.go:93"

The blackbox_exporter binary is run as root and has CAP_NET_RAW permissions. All other ping probes succeed, and since pinging this host succeeds as well via the shell, i think this could be blackbox_exporter's fault - couldn't it?

Add diagnostic information to webserver

It'd be handy if the details of the last ~10 probes to each target/module were available via the web UI so you could look at things like HTTP responses, rather than just having the metrics on the Prometheus side

ipv4 not working

it seems setting to ip4 is not working, even if I set in http module: preferred_ip_protocol: "ip4"
output is:
probe_ip_protocol 6
probe_http_status_code 0
probe_http_content_length 0
probe_http_redirects 0
probe_http_ssl 0
probe_duration_seconds 0.001879
probe_success 0

and error shows:

Error for HTTP request to http://10.1.0.57:9115: Get http://10.1.0.57:9115: dial tcp6 10.1.0.57:9115: connect: network is unreachable" source="http.go:159"

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.