prometheus / blackbox_exporter Goto Github PK
View Code? Open in Web Editor NEWBlackbox prober exporter
Home Page: https://prometheus.io
License: Apache License 2.0
Blackbox prober exporter
Home Page: https://prometheus.io
License: Apache License 2.0
Now that the go client supports them, we should switch to custom registries rather than hand-creating the output.
For at least TCP and ICMP checks, it'd be useful to be able to specify source IP address
hi,
is a dns module something you'd be interested in having in blackbox exporter? if so I'm happy to take a stab at it using e.g. https://github.com/miekg/dns
thanks!
Hi,
i have a problem that the ICMP probe reports up=1 although the host is down.
This is the snippet of my blackbox.yml:
modules:
ping:
prober: icmp
timeout: 5s
icmp:
protocol: "icmp"
preferred_ip_protocol: "ip4"
This is the according snippet of my prometheus.yml:
- job_name: 'ping'
metrics_path: /probe
params:
module: [ping]
static_configs:
- targets:
- cram
relabel_configs:
# Set the target's address as the target-GET-parameter for the blackbox exporter
- source_labels: [__address__]
regex: (.*)
target_label: __param_target
replacement: ${1}
# Set the target's address as the instance name
- source_labels: [__address__]
regex: (.*)
target_label: instance
replacement: ${1}
# Set the scrape address to the blackbox exporter
- source_labels: []
regex: .*
target_label: __address__
replacement: 127.0.0.1:9115
The relabel_configs
are almost directly copy&pasted from the github README.
So the host cram
did exist in the network, but is currently down, but the exporter reports it as up.
This is the output of the ping command:
$ ping cram
PING cram.home.simonszu.de (192.168.1.15) 56(84) bytes of data.
From tirn.home.simonszu.de (192.168.1.9) icmp_seq=1 Destination Host Unreachable
From tirn.home.simonszu.de (192.168.1.9) icmp_seq=2 Destination Host Unreachable
From tirn.home.simonszu.de (192.168.1.9) icmp_seq=3 Destination Host Unreachable
^C
--- cram.home.simonszu.de ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4005ms
Please note that the host tirn
is my dedicated monitoring server, running prometheus and the blackbox exporter.
So i think this is a bug, or at least not the intended behaviour. I am no ICMP expert, but i think it has something to do with the fact that the ping
command returns at least something.
I am using latest prometheus and blackbox_exporter on armv7.
I'd like to add support to the DNS prober to be able to query nameservers in resolv.conf.
The dns library used by the DNS prober supports parsing resolv.conf, which is useful for emulating whether an application in userspace that relies on resolv.conf can successfully query a specified DNS record.
Some considerations to work out:
it would be nice if the http prober supported a payload in the request, this would be similarly true for the tcp part of the prober.
so we may add n checks with different ports and uri's in same prometheus job
for now it very difficult path with long relabeling configs if at all
and with service discovery looks even impossible
may be there is better idea?
thanks
Now that Prometheus defaults target ports to 80 if unspecified this fails.
I guess the ICMP probe should drop the port if present in the target (which it will be now I guess).
I am using go1.6.2 and centos6.8
This is the error being reflected on screen when i run make.
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
panic(0x93d400, 0xc8200100a0)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
main.vendoredImportPath(0xc8201dc800, 0xc820250331, 0x20, 0x0, 0x0)
/usr/local/go/src/cmd/go/pkg.go:466 +0x713
main.loadImport(0xc820250331, 0x20, 0xc820053f20, 0x58, 0xc8201dc800, 0xc82049b788, 0xc820250630, 0x1, 0x1, 0x1, ...)
/usr/local/go/src/cmd/go/pkg.go:336 +0xe52
main.(*Package).load(0xc8201dc800, 0xc82049b788, 0xc8202dc000, 0x0, 0x0, 0x4)
/usr/local/go/src/cmd/go/pkg.go:953 +0x4397
main.loadImport(0xc820176121, 0x2e, 0xc820144690, 0x22, 0xc820027800, 0xc82049b788, 0xc820176330, 0x1, 0x1, 0x1, ...)
/usr/local/go/src/cmd/go/pkg.go:377 +0x84e
main.(*Package).load(0xc820027800, 0xc82049b788, 0xc820420e00, 0x0, 0x0, 0xc)
/usr/local/go/src/cmd/go/pkg.go:953 +0x4397
main.loadImport(0xc82013a3a3, 0x3, 0xc82000a034, 0x22, 0x0, 0xc82049b788, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/cmd/go/pkg.go:377 +0x84e
main.loadPackage(0xc82013a3a3, 0x3, 0xc82049b788, 0x0)
/usr/local/go/src/cmd/go/pkg.go:1639 +0x13d8
main.packagesAndErrors(0xc82008c000, 0x14, 0x14, 0x0, 0x0, 0x0)
/usr/local/go/src/cmd/go/pkg.go:1684 +0x531
main.packages(0xc82000a260, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/cmd/go/pkg.go:1652 +0x86
main.runList(0xc8a6a0, 0xc82000a260, 0x1, 0x1)
/usr/local/go/src/cmd/go/list.go:186 +0x1f6
main.main()
/usr/local/go/src/cmd/go/main.go:181 +0x783
>> formatting code
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
panic(0x93d400, 0xc82000e0d0)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
main.vendoredImportPath(0xc820199000, 0xc8201ef4a1, 0x20, 0x0, 0x0)
/usr/local/go/src/cmd/go/pkg.go:466 +0x713
main.loadImport(0xc8201ef4a1, 0x20, 0xc8203a99e0, 0x58, 0xc820199000, 0xc820179838, 0xc8201ef5f0, 0x1, 0x1, 0x1, ...)
/usr/local/go/src/cmd/go/pkg.go:336 +0xe52
main.(*Package).load(0xc820199000, 0xc820179838, 0xc82016e380, 0x0, 0x0, 0x4)
/usr/local/go/src/cmd/go/pkg.go:953 +0x4397
main.loadImport(0xc82014b651, 0x2e, 0xc82010bbf0, 0x22, 0xc820135000, 0xc820179838, 0xc82014b860, 0x1, 0x1, 0x1, ...)
/usr/local/go/src/cmd/go/pkg.go:377 +0x84e
main.(*Package).load(0xc820135000, 0xc820179838, 0xc82012b500, 0x0, 0x0, 0xc)
/usr/local/go/src/cmd/go/pkg.go:953 +0x4397
main.loadImport(0x9e0b58, 0x1, 0xc820014004, 0x22, 0x0, 0xc820179838, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/cmd/go/pkg.go:377 +0x84e
main.loadPackage(0x9e0b58, 0x1, 0xc820179838, 0x0)
/usr/local/go/src/cmd/go/pkg.go:1639 +0x13d8
main.packagesAndErrors(0xc82013a1f0, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/cmd/go/pkg.go:1684 +0x531
main.packages(0xc82000a270, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/cmd/go/pkg.go:1652 +0x86
main.runFmt(0xc8a060, 0xc82000a270, 0x0, 0x0)
/usr/local/go/src/cmd/go/fmt.go:38 +0x8d
main.main()
/usr/local/go/src/cmd/go/main.go:181 +0x783
make: *** [format] Error 2
when using http probe module, got some warning message as below randomly. and the value of probe_duration_seconds is much higher but no failure.
time="2016-09-27T09:08:42Z" level=warning msg="Error for HTTP request to http://xxx/proplus/home: Get http://xxxx/proplus/home: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" source="http.go:97"
how to trouble shoot this? any clue is welcome.
Hi,
Some of our instances (e.g. staging servers) are protected using BASIC AUTH. Is there a way using blackbox-exporter to probe those servers (e.g. using module http_2xx
)
Thx in Advance,
Sven
Hi there, i'm facing sometime follow issue:
blackbox_exporter log:
WARN[0675] Error for HTTP request to https://somefancyhost.com/home: Get https://somefancyhost.com/home: dial tcp4: lookup somefancyhost.com on some-dns-server:53: no such host source=http.go:164
WARN[0675] Error for HTTP request to https://somefancyhost.com/home: Get https://somefancyhost.com/home: dial tcp4: lookup somefancyhost.com on some-dns-server:53: no such host source=http.go:164
WARN[0675] Error for HTTP request to https://somefancyhost.com/home: Get https://somefancyhost.com/home: dial tcp4: lookup somefancyhost.com on some-dns-server:53: no such host source=http.go:164
WARN[0675] Error for HTTP request to https://somefancyhost.com/home: Get https://somefancyhost.com/home: dial tcp4: lookup somefancyhost.com on some-dns-server:53: no such host source=http.go:164
WARN[0675] Error for HTTP request to https://somefancyhost.com/home: Get https://somefancyhost.com/home: dial tcp4: lookup somefancyhost.com on some-dns-server:53: no such host source=http.go:164
dig output:
somefancyhost.com. 3600 IN A 10.0.0.2
Any chance to get some cache for dns queries and honor DNS record TTL, to not bomb dns server with requests, while measuring http/tcp?
The tcp
prober supports specifying a custom certificate authority store, but the http
one does not.
Hi,
I'm using the latest Docker image of blackbox for probing targets. I'm having a target probe (module http_2xx
, config as in https://github.com/prometheus/blackbox_exporter/blob/master/blackbox.yml) where the server returns status code 403
(due to IP resctriction), but the target instance is still shown as "up".I would expect status "down" since i'm using the defaul config for module http_2xx
which checks for status code 200-300.
When looking at the value of probe_http_status_code
for this instance, it shows 403
.
Any idea why the instance is still shown as "up"?
Cheers Sven
Before I send a pull request, I wanted to check whether the following feature would be accepted:
As a defence in depth mechanism, the HTTP module in probers such as kanla supports checking whether the returned body matches a certain regular expression. That way, you can be alerted when the webserver still returns an HTTP 200, but all the application prints is “could not connect to database”. Obviously, applications should return proper HTTP status codes, but I still think it’s a nice feature to get some more confidence in your alerting, especially when you’re running applications that you did not write yourself.
What do you think?
Hi there,
I have been scraping and monitoring a number of target urls for a while now and on 19 April 2016, a couple of them reflected negative probe duration (seconds). This happened for a brief period of 10 seconds around 5:47 PM. May i know if this is normal?
Thanks in advance
Would a pull request be accepted which verifies that an IRC server is running on a specific address?
Thanks.
I would like to have the blackbox_exporter support static configuration. This means that the configuration such as:
can be specified in the configuration file and the blackbox_exporter will add labels to the metrics and scrape the combinations periodically. This could be exposed over a /metrics endpoint alongside the /probe endpoint already exposed.
The reason for this feature is mainly to avoid complex and specific relabelling configuration in prometheus. Also, for our use case, the relabelling doesn't easily work when the blackbox_exporter is exposed as a service over consul.
When pulling the Docker image, how do I get an ARM version for my Raspberry PI?
standard_init_linux.go:175: exec user process caused "exec format error"
It'd be handy if the details of the last ~10 probes to each target/module were available via the web UI so you could look at things like HTTP responses, rather than just having the metrics on the Prometheus side
it seems setting to ip4 is not working, even if I set in http module: preferred_ip_protocol: "ip4"
output is:
probe_ip_protocol 6
probe_http_status_code 0
probe_http_content_length 0
probe_http_redirects 0
probe_http_ssl 0
probe_duration_seconds 0.001879
probe_success 0
and error shows:
Error for HTTP request to http://10.1.0.57:9115: Get http://10.1.0.57:9115: dial tcp6 10.1.0.57:9115: connect: network is unreachable" source="http.go:159"
This is my service file: http://s.natalian.org/2015-11-20/blackbox-exporter.service
Note I am running with --privileged
which should provide all CAPS.
http://sg.dabase.com:9115/probe?target=dabase.com&module=icmp
results in errors like:
Nov 20 08:06:17 rojak docker[8786]: time="2015-11-20T08:06:17Z" level=error msg="Error listening to socket: listen ip4:icmp 0.0.0.0: socket: operation not permitted" file=icmp.go line=32
Some what confusingly I get metrics like:
probe_duration_seconds 0.000251
probe_success 0
When the domain exists or not [1]. Is 0 success ?!
[1] http://sg.dabase.com:9115/probe?target=doesnotexist.example.com&module=icmp
I'm using:
Digest: sha256:0eaf1811c018de2d87d6d14ad848cd5fb2333b7d17eeb9da19a3101796e8f865
Status: Image is up to date for prom/blackbox-exporter:master
http://s.natalian.org/2016-11-05/blackboxexporter.txt
I guess I need to force tcp4 somehow, somewhere?
I was quite amazed to see that the service is termed still up and ok while in the stdout logs of the blackbox exporter is showing errors like this
time="2016-11-21T10:46:40Z" level=warning msg="Error for HTTP request to http://svc.proj.svc:19080: Get http://svc.proj.svc:19080: dial tcp4 172.30.162.209:19080: getsockopt: no route to host" source="http.go:159"
I have intentionally , scaled a service to zero pods in openshift origin. and i was expecting a instance down alert from prometheus.. How ever form target view the blackbox exporter never seems to recognize that service is down and is not responding
while i try to directly query the backend service using the blackbox exportetservice ip
curl http://172.30.235.19/probe?target=svc.proj.svc:19080
probe_ip_protocol 4
probe_http_status_code 0
probe_http_content_length 0
probe_http_redirects 0
probe_http_ssl 0
probe_duration_seconds 3.031048
probe_success 0
i am using module http_2xx so since the status code is 0 , should not it have raised (lavel) service as down in targets view in prometheus Web UI.
Also additionally this query works
curl http://172.30.235.19/probe?target=svc.project.svc:19080&module=http_2xx
probe_http_status_code 0
probe_http_content_length 0
probe_http_redirects 0
probe_http_ssl 0
probe_duration_seconds 3.040584
probe_success 0
while
curl http://172.30.235.19/probe?module=http_2xx&target=svc.project.svc:19080
produces
Target parameter is missing
and bails out .
Am i missing something on the alert part. if a service dont have any endpoints i am hoping that probing to it fails or i am missing something?
I am using blackbox exporter container version0.3,0
as it start it shows
"Starting blackbox_exporter (version=0.3.0, branch=master, revision=43cfa9854e595993b64925c8a5b5f39086b03dae)"
blackbox_exporter 0.3.0 / prometheus 1.4.1
I've faced the issue when blackbox_exporter starts returning probe_success 0 for random hosts after several hours/days of correct functioning.
Prometheus' configuration:
global:
scrape_interval: 60s
scrape_timeout: 20s
external_labels:
monitor: 'tvdashboard'
scrape_configs:
- job_name: 'blackbox_devmetrics'
metrics_path: /probe
params:
module: [http_devmetrics]
static_configs:
- targets:
- devmetrics.hostname.com
relabel_configs: &relabel
- source_labels: [__address__]
regex: (.*)(:80)?
target_label: __param_target
replacement: ${1}
- source_labels: [__param_target]
regex: (.*)
target_label: instance
replacement: ${1}
- source_labels: []
regex: .*
target_label: __address__
replacement: 127.0.0.1:9115 # blackbox_exporter
- job_name: 'blackbox_rally'
metrics_path: /probe
params:
module: [http_rally]
static_configs:
- targets:
- rally1.rallydev.com/slm/login.op
relabel_configs: *relabel
- job_name: 'blackbox_jira'
metrics_path: /probe
params:
module: [http_jira]
static_configs:
- targets:
- jira.hostname.com/login.jsp?os_destination=%2Fsecure%2FDashboard.jspa
relabel_configs: *relabel
- job_name: 'blackbox_confluence'
metrics_path: /probe
params:
module: [http_confluence]
static_configs:
- targets:
- confluence.hostname.com/login.action
relabel_configs: *relabel
blackbox_exporter's configuration
modules:
http_devmetrics:
prober: http
timeout: 10s
http:
valid_status_codes: []
method: GET
headers: &headers
Accept-Language: en-US
User-Agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
no_follow_redirects: false
fail_if_ssl: false
fail_if_not_ssl: false
fail_if_not_matches_regexp:
- "username"
tls_config: &tls_config
insecure_skip_verify: true
protocol: "tcp"
preferred_ip_protocol: "ip4"
http_rally:
prober: http
timeout: 10s
http:
valid_status_codes: []
method: GET
headers: *headers
no_follow_redirects: false
fail_if_ssl: false
fail_if_not_ssl: false
fail_if_not_matches_regexp:
- "Sign in"
tls_config: *tls_config
protocol: "tcp"
preferred_ip_protocol: "ip4"
http_jira:
prober: http
timeout: 10s
http:
valid_status_codes: []
method: GET
headers: *headers
no_follow_redirects: false
fail_if_ssl: false
fail_if_not_ssl: false
fail_if_not_matches_regexp:
- "login-form-username"
tls_config: *tls_config
protocol: "tcp"
preferred_ip_protocol: "ip4"
http_confluence:
prober: http
timeout: 10s
http:
valid_status_codes: []
method: GET
headers: *headers
no_follow_redirects: false
fail_if_ssl: false
fail_if_not_ssl: false
fail_if_not_matches_regexp:
- "Username"
tls_config: *tls_config
protocol: "tcp"
preferred_ip_protocol: "ip4"
I've tried using Prometheus' web UI and CLI to debug. I've noticed that all the targets are active ("up") in Prometheus, but at the same time when I want to get a value of probe_http_status_code metric, I see it only for several (randomly selected) targets, but not for all (this is my expected behavior).
When I'm using CLI, I got the same result for the hosts that aren't displaying in Prometheus UI:
$ curl "http://127.0.0.1:9115/probe?module=http_devmetrics&target=devmetrics.hostname.com"
probe_duration_seconds 0.001795
probe_success 0
So seems like the problem is in blackbox_exporter but not in Prometheus itself.
On the other hand, if using just curl, I see that my service is up and running:
$ curl -L -k "http://devmetrics.hostname.com" | grep -o username
username
I'm doing curl on the same machine where blackbox_exporter is running to eliminate any network issues.
This problem occurs periodically and in most cases is solved by blackbox_exporter restarting.
The DNS prober currently allows a DNS name to be configured statically in blackbox_exporter's YAML configuration file. The target, specified by Prometheus, is the nameserver to query. This diverges from @brian-brazil's suggestion to make the DNS name part of the query string (rather than the nameserver).
This favours probing the same DNS record from multiple DNS servers, and makes probing multiple DNS records from the same namesever rather cumbersome.
Realistically I think both use cases are going to be common, depending on whether you're running DNS servers or just want to make sure that your SRV records are working correctly, for example.
That said, I think probing multiple records against the same nameserver is possibly the most common case, and configuring a DNS modules per nameserver should be less cumbersome than one per DNS record.
Interested if @knyar has any suggestions on how to handle this.
When starting the blackbox exporter with the insecure_skip_verify, it fails on the certificate.
According to issue #41 this options was to allow self-signed certificates I imagine so this is to be expected.
Would it be possible to add an option to ignore the certificate alltogether?
In the below usecase I probe for machine.example.com instead of www.example.com to bypass DNS geo load balancing. Adding the domain to the certificate will make the domain known to everyone who checks the certificate so not a solution.
The curl equivalent would be
curl -k machine.example.com
wget would do it as follows
wget --no-check-certificate machine.example.com
time="2016-11-04T12:41:19Z" level=warning msg="Error for HTTP request to https://machine.example.com: Get https://machine.example.com: x509: certificate is valid for www.example.com, not machine.example.com" source="http.go:97"
Hi,
I would like to monitor 100+ web apps. But I need identify, that I monitor right site, so in all web app I have <!-- Site: {{fqdn}} --!>. I trying to create check with blackbox_exporter but I can't find way how to do that. Does blackbox_exporter support this kind of check? I hope that fail_if_not_matches_regexp can help me, but I don't know how to set it to parse diferent hostname for all my sites.
Thanks for help
+----------+ +-----------+
| +---->| |
|prometheus| |bb-exporter|
| | | |
+----------+ +----+------+
|
|
|
v
+-----------------+
| |
| nginx |
| |
+-----------------+
I have multiple targets configured in prometheus.yml for blackbox_exporter, and when BB got request to go and scrape, BB sometimes does not use configured attributes:
modules:
http_2xx:
prober: http
timeout: 30s
http:
valid_status_codes: [200,301,302]
method: GET
headers:
User-Agent: bb-prober
fail_if_not_ssl: true
tls_config:
insecure_skip_verify: false
protocol: "tcp4"
preffered_ip_protocol: "ip4"
how does this look like from web server log:
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home HTTP/1.1" *301* 178 "-" "bb-prober" "-" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home HTTP/1.1" *301* 178 "-" "bb-prober" "-" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home HTTP/1.1" *301* 178 "-" "bb-prober" "-" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home HTTP/1.1" *301* 178 "-" "bb-prober" "-" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home HTTP/1.1" *301* 178 "-" "bb-prober" "-" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home HTTP/1.1" *301* 178 "-" "bb-prober" "-" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home/ HTTP/1.1" *200* 509 "https://somefancyhost.com/home" "Go-http-client/1.1" "2.71" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home/ HTTP/1.1" *200* 509 "https://somefancyhost.com/home" "Go-http-client/1.1" "2.71" - 0.000 -
10.0.0.1 somefancyhost.com - [23/Dec/2016:12:27:57] "GET /home/ HTTP/1.1" *200* 509 "https://somefancyhost.com/home" "Go-http-client/1.1" "2.71" - 0.000 -
Assuming my observations, when response code is not 200, header is not set.
Hi, it seems I'm getting a Content-Length that's 8 bytes less than the actual response payload:
$ curl -s --ignore-content-length "http://127.0.0.1:9091/probe?target=google.com&module=http_2xx" | wc
7 12 152
$ curl -vs "http://127.0.0.1:9091/probe?target=google.com&module=http_2xx"
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9091 (#0)
> GET /probe?target=google.com&module=http_2xx HTTP/1.1
> Host: 127.0.0.1:9091
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: {Wed, 23 Nov 2016 13:20:51 GMT}
< Content-Length: 144
< Content-Type: {text/plain; charset=utf-8}
< Connection: close
will fork and try to fix, but no guarantees as to when that will happen :)
It throws an out of memory option when i run the make command
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0x698ab0, 0x16)
/usr/local/go/src/runtime/panic.go:527 +0x90
runtime.sysMap(0xc824970000, 0x100000, 0x68b00, 0x7af258)
/usr/local/go/src/runtime/mem_linux.go:143 +0x9b
runtime.mHeap_SysAlloc(0x790840, 0x100000, 0x0)
/usr/local/go/src/runtime/malloc.go:423 +0x160
runtime.mHeap_Grow(0x790840, 0x8, 0x0)
/usr/local/go/src/runtime/mheap.go:628 +0x63
runtime.mHeap_AllocSpanLocked(0x790840, 0x1, 0x2b4533105ce0)
/usr/local/go/src/runtime/mheap.go:532 +0x5f1
runtime.mHeap_Alloc_m(0x790840, 0x1, 0x13, 0x2b4533105ce0)
/usr/local/go/src/runtime/mheap.go:425 +0x1ac
runtime.mHeap_Alloc.func1()
/usr/local/go/src/runtime/mheap.go:484 +0x41
runtime.systemstack(0xc820041e50)
/usr/local/go/src/runtime/asm_amd64.s:278 +0xab
runtime.mHeap_Alloc(0x790840, 0x1, 0x10000000013, 0x40dde4)
/usr/local/go/src/runtime/mheap.go:485 +0x63
runtime.mCentral_Grow(0x799258, 0x0)
/usr/local/go/src/runtime/mcentral.go:190 +0x93
runtime.mCentral_CacheSpan(0x799258, 0x2b4533105ce0)
/usr/local/go/src/runtime/mcentral.go:86 +0x4d4
runtime.mCache_Refill(0x2b4532e3f000, 0xc800000013, 0x2b4533105ce0)
/usr/local/go/src/runtime/mcache.go:118 +0xcf
runtime.mallocgc.func2()
/usr/local/go/src/runtime/malloc.go:611 +0x2b
runtime.systemstack(0xc820022000)
/usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
/usr/local/go/src/runtime/proc1.go:674
goroutine 1 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc82003ac00 sp=0xc82003abf8
runtime.mallocgc(0x140, 0x664640, 0x0, 0xc8200877c0)
/usr/local/go/src/runtime/malloc.go:612 +0x842 fp=0xc82003acd0 sp=0xc82003ac00
runtime.newobject(0x664640, 0xc822063290)
/usr/local/go/src/runtime/malloc.go:760 +0x42 fp=0xc82003acf8 sp=0xc82003acd0
cmd/link/internal/ld.readsym(0xc820072000, 0xc8248b38e0, 0xc820eb3270, 0xd, 0xc8248aa150, 0x6d)
/usr/local/go/src/cmd/link/internal/ld/objfile.go:208 +0x76a fp=0xc82003b140 sp=0xc82003acf8
cmd/link/internal/ld.ldobjfile(0xc820072000, 0xc8248b38e0, 0xc820eb3270, 0xd, 0x34e38, 0xc8248aa150, 0x6d)
/usr/local/go/src/cmd/link/internal/ld/objfile.go:147 +0xa56 fp=0xc82003b350 sp=0xc82003b140
cmd/link/internal/ld.ldobj(0xc8248b38e0, 0xc820eb3270, 0xd, 0x34e60, 0xc8248aa150, 0x6d, 0xc820fe4d90, 0x65, 0x1)
/usr/local/go/src/cmd/link/internal/ld/lib.go:1208 +0x14e1 fp=0xc82003b5b8 sp=0xc82003b350
cmd/link/internal/ld.objfile(0xc820fe4e70)
/usr/local/go/src/cmd/link/internal/ld/lib.go:730 +0x10d5 fp=0xc82003b8c8 sp=0xc82003b5b8
cmd/link/internal/ld.loadlib()
/usr/local/go/src/cmd/link/internal/ld/lib.go:482 +0x5ce fp=0xc82003b9f0 sp=0xc82003b8c8
cmd/link/internal/ld.Ldmain()
/usr/local/go/src/cmd/link/internal/ld/pobj.go:212 +0x1c4c fp=0xc82003be70 sp=0xc82003b9f0
cmd/link/internal/amd64.Main()
/usr/local/go/src/cmd/link/internal/amd64/obj.go:44 +0x19 fp=0xc82003be78 sp=0xc82003be70
main.main()
/usr/local/go/src/cmd/link/main.go:26 +0x189 fp=0xc82003bf50 sp=0xc82003be78
runtime.main()
/usr/local/go/src/runtime/proc.go:111 +0x2b0 fp=0xc82003bfa0 sp=0xc82003bf50
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82003bfa8 sp=0xc82003bfa0
make: *** [blackbox_exporter] Error 2
It might be that I'm doing it wrong, but I want to add the probe path as a metrics path in the Prometheus configuration, something like:
- job_name: "blackbox_exporter"
scheme: "http"
metrics_path: "/probe?target=google.com&module=http_2xx"
static_configs:
- targets:
- localhost:9115
labels:
target_name: "google.com"
This doesn't work. In Prometheus I get the error text format parsing error in line 1: invalid metric name
. When I look at the prometheus server localhost:9090/targets
I see the endpoint as I put it in the configuration, however, when I click on it, I get the standard blackbox page:
Which is probably causing the error. When I look at the url in the browser then, I see that it's not http://localhost:9115/probe?target=google.com&module=http_2xx
, but http://localhost:9115/probe%3target=google.com&module=http_2xx
. The ? in the metric_path seems to get encoded (%3). When I replace the %3 by a ? it works as expected.
Since nobody else seems to have this problem, I might be doing something wrong. Any help would be greatly appreciated.
Hi,
i have defined a ping-module which is basically just an icmp prober. I am probing around 17 hosts with it. Now i have a problem: The probe_success for one of this host is 0, although the host is pingable when executing ping in my shell.
Systemd journal shows the following error:
Nov 10 14:58:21 tirn blackbox_exporter[29216]: time="2016-11-10T14:58:21Z" level=info msg="Timeout reading from socket for grandstream: read ip4 0.0.0.0: i/o timeout" source="icmp.go:93"
The blackbox_exporter binary is run as root and has CAP_NET_RAW
permissions. All other ping probes succeed, and since pinging this host succeeds as well via the shell, i think this could be blackbox_exporter's fault - couldn't it?
There's been a few changes since version 0.1.0, notably the DNS probe:
0.1.0...master
Could a new release be cut?
blackbox_exporter would just start a configured process for a module, and return its output as the result of the probe. This would allow it to be very extensible, without needing to bloat the codebase.
It's also possible that this should be written as a separate exporter and not as a feature of blackbox_exporter, but it will be very useful either way.
Hello,
I'm trying to add a blackbox probe to test the availability of an http proxy. The http module could be perfect for such task if it was possible to pass a proxy URL in parameter.
It would be helpful to be able to do:
http://blackbox:9115/probe?target=http://example.com&proxy=http://myproxy:3128&module=http_2xx
Or simply being able to define a proxy in the module configuration and do:
http://blackbox:9115/probe?target=http://example.com&module=http_2xx_proxy
I can workaround that limitation by launching an instance of the blackbox exporter for each proxy I need to test (using the HTTP_PROXY env var to set the proxy), but it would be cleaner to have a single blackbox exporter instance for all checks.
my rc yaml:
apiVersion: v1
kind: ReplicationController
metadata:
name: blackbox-exporter
spec:
replicas: 1
selector:
app: blackbox-exporter
template:
metadata:
name: blackbox-exporter
labels:
app: blackbox-exporter
spec:
containers:
- image: ****/prometheus/blackbox-exporter:1.0
name: blackbox
ports:
- containerPort: 9115
name: blackbox
my svc yaml:
apiVersion: v1
kind: Service
metadata:
name: rabbitmq
labels:
name: rabbitmq
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/probe: "true"
spec:
ports:
- port: 9090
name: exporter
targetPort: exporter
protocol: TCP
selector:
app: rabbitmq
feature/wishlist request from #69: when using tcp prober and port is missing (or other kind of errors e.g. address malformed) no message is logged at any level. IMO blackbox_exporter should at least signal when such errors are encountered.
There're some services that should not be exposed outside of localhost
- if I deploy blackbox_exporter
on these hosts, maybe it would get exploited somehow to access these internal services?
Could there please be an example of a POST with a JSON payload & a check on the response like looking for a "status": "success"
in the JSON response.
Hi,
I am using blackbox_exporter and I would like to use the official prom/blackbox-exporter container from docker hub.
The versioning system is not implemented and currently prom/blackbox-exporter:latest contains blackbox-exporter 0.1.0. Is it possible to push containers with a sane versioning system?
It would be nice to support configuration reload via SIGHUP.
Hi,
relabel_configs:
- source_labels: [__address__]
regex: (.*)(:5050)?
target_label: __param_target
replacement: ${1}
- source_labels: [__param_target]
regex: (.*)
target_label: instance
replacement: ${1}
- source_labels: []
regex: .*
target_label: __address__
replacement: 127.0.0.1:9115 # Blackbox exporter.
For some reason, this relabel config causes the http_2xx to report that a targeted service is available, even the host is inaccessible for whatever reason, such as firewall rules. Here's the output from a curl that shows the host is not reachable, yet when using the above relabel config, Prometheus reports it is 'Up':
`[aop-dev@prometheus-i-e588886d ~]$ curl -v content-service.si.example.org:5050
(TLDs and IPs changed for privacy)
Removing the block that targets __address__
solves this issue, but leaves the wrong value displayed in the interface.
I'm getting errors in log;
certificate is valid for *.foo.com, foo.com, not abc.foo.com
Shouldn't blackbox accept a wildcard cert without the extra noise?
The config in the README is meant to show all the config options, however it also contains several useful modules that should be in blackbox.yml. Minimise it and move over whatever makes sense to blackbox.yml.
$ curl 'http://127.0.0.1:9115/probe?target=https://revoked.grc.com/&module=http_2xx'
probe_ssl_earliest_cert_expiry 1492961872.000000
probe_http_status_code 200
probe_http_content_length -1
probe_http_redirects 0
probe_http_ssl 1
probe_duration_seconds 0.246233
probe_success 1
Hi,
We have some URLs which are only accessible using https. When using the given module http_2xx
, the check fail since a request with http schema is sent (getsockopt: connection refused" source="http.go:159"
).
How can i test HTTPS only resources?
Thx,
Sven
Grafana shows prometheus memory going up (blue line: 4GB limit is yet to be reached, but close).
Meanwhile, blackbox-exporter is reaching its imposed mem_limit: 1G and gets killed and restarted (red line) and that cycle repeats every few hours.
The node and collectd exporters are working fine, in their 15-20MB range, as expected...
I tried both :latest
and :master
images from the docker hub, so it doesn't seem to have been fixed in any of those versions.
Could it be accumulating the [connection / header / socket] timeouts seen in the logs (below) prior to stack dumping because it's running out of ram?
docker-compose.yml section:
blackbox_exporter:
image: prom/blackbox-exporter:master
restart: always
expose:
- 9115
volumes:
- ./blackbox-config/:/config/
networks:
- backend
command:
- '-config.file=/config/blackbox.yml'
cap_drop:
- all
cap_add:
- NET_RAW
security_opt:
- no-new-privileges
mem_limit: 1G
cpu_shares: 1024
blackbox.yml:
modules:
http_2xx:
prober: http
timeout: 5s
http:
valid_status_codes: [] # Defaults to 2xx
method: GET
headers:
Strict-Transport-Security: max-age=31536000; includeSubdomains
#Host: example.com
#Accept-Language: en-US
#no_follow_redirects: false
#fail_if_ssl: false
fail_if_not_ssl: true
#fail_if_matches_regexp:
#- "Could not connect to database"
fail_if_not_matches_regexp:
- "My Title</title>"
tcp_connect:
prober: tcp
timeout: 5s
ssh_banner:
prober: tcp
timeout: 5s
tcp:
query_response:
- expect: "^SSH-2.0-"
icmp:
prober: icmp
timeout: 5s
icmp:
protocol: "icmp"
preferred_ip_protocol: "ip4"
prometheus.yml relevant section example:
- job_name: 'probe_https_instances'
metrics_path: /probe
params:
module: [http_2xx] # Look for a HTTP 200 response.
static_configs:
- targets:
- site1.example.com
- site2.example.com
- site3.example.com
# etc.
relabel_configs:
- source_labels: [__address__]
regex: (.*)(:80)?
target_label: __param_target
replacement: https://${1}
- source_labels: [__param_target]
regex: (.*)
target_label: instance
replacement: ${1}
- source_labels: []
regex: .*
target_label: __address__
replacement: blackbox_exporter:9115
- job_name: 'probe_ping_instances'
metrics_path: /probe
params:
module: [icmp] # Look for a HTTP 200 response.
static_configs:
- targets: # Target to probe
- site1.example.com
- site2.example.com
- site3.example.com
- site4.example.com
# etc. ~20 servers.
relabel_configs:
- source_labels: [__address__]
regex: (.*)(:80)?
target_label: __param_target
replacement: ${1}
- source_labels: [__param_target]
regex: (.*)
target_label: instance
replacement: ${1}
- source_labels: []
regex: .*
target_label: __address__
replacement: blackbox_exporter:9115 # Blackbox exporter.
blackbox.log:
Dec 17 17:27:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:28Z" level=info msg="Timeout reading from socket for site6.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:28Z" level=warning msg="Error for HTTP request to https://site6.example.com: Get https://site6.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:28Z" level=info msg="Timeout reading from socket for site12.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:30 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:29Z" level=info msg="Timeout reading from socket for site1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:30 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:30Z" level=warning msg="Error for HTTP request to https://site6-staging.example.com: Get https://site6-staging.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:31 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:29Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:31 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:30Z" level=info msg="Timeout reading from socket for site11.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:31 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:30Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:32 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:31Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:32 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:31Z" level=warning msg="Error for HTTP request to https://site9.example.com: Get https://site9.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:33Z" level=info msg="Timeout reading from socket for site9.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:34Z" level=warning msg="Error for HTTP request to https://example.com: Get https://example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:33Z" level=warning msg="Error for HTTP request to https://site8.example.com: Get https://site8.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:34Z" level=warning msg="Error for HTTP request to https://site7.example.com: Get https://site7.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:37 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:35Z" level=info msg="Timeout reading from socket for site3.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:37 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:35Z" level=info msg="Timeout reading from socket for host1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:41 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:40Z" level=warning msg="Error for HTTP request to https://site11.example.com: Get https://site11.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:42 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:41Z" level=info msg="Timeout reading from socket for example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:42 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:41Z" level=info msg="Timeout reading from socket for site5.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:43 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:41Z" level=warning msg="Error for HTTP request to https://site4.example.com: Get https://site4.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:45 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:45Z" level=warning msg="Error for HTTP request to https://site1.example.com: Get https://site1.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:46 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:45Z" level=warning msg="Error for HTTP request to https://site2.example.com: Get https://site2.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:47 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:46Z" level=warning msg="Error for HTTP request to https://site3.example.com: Get https://site3.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:48 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:46Z" level=warning msg="Error for HTTP request to https://site5.example.com: Get https://site5.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:49 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:48Z" level=info msg="Timeout reading from socket for site8.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:49 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:48Z" level=warning msg="Error for HTTP request to https://site10.example.com: Get https://site10.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:50 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:48Z" level=info msg="Timeout reading from socket for host0.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:53 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:52Z" level=info msg="Timeout reading from socket for site10.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:54 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:52Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:54 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:54Z" level=warning msg="Error for HTTP request to https://site6.example.com: Get https://site6.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:55 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:53Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:55 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:53Z" level=info msg="Timeout reading from socket for site6.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:56 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:55Z" level=warning msg="Error for HTTP request to https://site6-staging.example.com: Get https://site6-staging.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:27:57 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:55Z" level=info msg="Timeout reading from socket for site11.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:57 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:56Z" level=info msg="Timeout reading from socket for site12.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:27:59 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:57Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:00 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:59Z" level=info msg="Timeout reading from socket for site6-staging.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:00 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:59Z" level=warning msg="Error for HTTP request to https://site9.example.com: Get https://site9.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:00 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:27:59Z" level=warning msg="Error for HTTP request to https://site7.example.com: Get https://site7.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:01 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:00Z" level=info msg="Timeout reading from socket for site9.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:02 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:01Z" level=warning msg="Error for HTTP request to https://site8.example.com: Get https://site8.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:03 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:01Z" level=info msg="Timeout reading from socket for host1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:05 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:04Z" level=info msg="Timeout reading from socket for site2.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:08 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:07Z" level=info msg="Timeout reading from socket for site3.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:08 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:07Z" level=warning msg="Error for HTTP request to https://example.com: Get https://example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:09 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:08Z" level=warning msg="Error for HTTP request to https://site4.example.com: Get https://site4.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:10 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:08Z" level=warning msg="Error for HTTP request to https://site11.example.com: Get https://site11.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:11 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:09Z" level=info msg="Timeout reading from socket for example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:12 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:11Z" level=info msg="Timeout reading from socket for site4.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:13 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:12Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:15 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:14Z" level=warning msg="Error for HTTP request to https://site1.example.com: Get https://site1.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:18 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:17Z" level=warning msg="Error for HTTP request to https://site2.example.com: Get https://site2.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:18 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:16Z" level=warning msg="Error for HTTP request to https://site3.example.com: Get https://site3.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:19 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:17Z" level=warning msg="Error for HTTP request to https://site10.example.com: Get https://site10.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:20 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:19Z" level=info msg="Timeout reading from socket for site8.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:21 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:19Z" level=warning msg="Error for HTTP request to https://site5.example.com: Get https://site5.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:21 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:20Z" level=info msg="Timeout reading from socket for host0.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:23 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:20Z" level=info msg="Timeout reading from socket for site10.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:24 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:23Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:25 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:23Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:26 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:24Z" level=warning msg="Error for HTTP request to https://site6.example.com: Get https://site6.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:27Z" level=info msg="Timeout reading from socket for site12.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:26Z" level=info msg="Timeout reading from socket for site1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:27Z" level=warning msg="Error for HTTP request to https://site6-staging.example.com: Get https://site6-staging.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:29 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:28Z" level=info msg="Timeout reading from socket for site6.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:30 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:28Z" level=info msg="Timeout reading from socket for site11.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:32 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:31Z" level=warning msg="Error for HTTP request to https://site9.example.com: Get https://site9.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:33 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:30Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:32Z" level=warning msg="Error for HTTP request to https://site7.example.com: Get https://site7.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:32Z" level=info msg="Timeout reading from socket for site6-staging.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:34 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:32Z" level=warning msg="Error for HTTP request to https://site8.example.com: Get https://site8.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:38 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:37Z" level=info msg="Timeout reading from socket for site9.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:38 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:35Z" level=warning msg="Error for HTTP request to https://example.com: Get https://example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:42 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:40Z" level=info msg="Timeout reading from socket for host1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:42 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:40Z" level=info msg="Timeout reading from socket for site3.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:42 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:41Z" level=info msg="Timeout reading from socket for site4.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:43 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:41Z" level=info msg="Timeout reading from socket for site2.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:45 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:43Z" level=warning msg="Error for HTTP request to https://site11.example.com: Get https://site11.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:47 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:44Z" level=info msg="Timeout reading from socket for example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:55 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:51Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:28:56 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:49Z" level=warning msg="Error for HTTP request to https://site4.example.com: Get https://site4.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:56 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:53Z" level=warning msg="Error for HTTP request to https://site3.example.com: Get https://site3.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:57 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:53Z" level=warning msg="Error for HTTP request to https://site1.example.com: Get https://site1.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:59 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:55Z" level=warning msg="Error for HTTP request to https://site10.example.com: Get https://site10.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:28:59 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:57Z" level=warning msg="Error for HTTP request to https://site2.example.com: Get https://site2.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:29:02 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:28:56Z" level=info msg="Timeout reading from socket for site5.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:29:06 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:02Z" level=warning msg="Error for HTTP request to https://site5.example.com: Get https://site5.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:29:10 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:07Z" level=info msg="Timeout reading from socket for site8.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:29:15 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:11Z" level=warning msg="Error for HTTP request to https://site6.example.com: Get https://site6.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:29:19 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:15Z" level=info msg="Timeout reading from socket for site10.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:29:21 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:17Z" level=info msg="Timeout reading from socket for site7.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:29:22 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:17Z" level=warning msg="Error for HTTP request to https://site6-staging.example.com: Get https://site6-staging.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:29:23 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:17Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:29:25 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:21Z" level=info msg="Timeout reading from socket for host0.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:29:25 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:21Z" level=info msg="Timeout reading from socket for site6.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:29:26 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:22Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:29:32 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:28Z" level=info msg="Timeout reading from socket for site1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:29:35 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:33Z" level=warning msg="Error for HTTP request to https://site8.example.com: Get https://site8.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:29:37 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:33Z" level=info msg="Timeout reading from socket for site9.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:29:41 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:36Z" level=info msg="Timeout reading from socket for site12.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:29:42 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:38Z" level=warning msg="Error for HTTP request to https://site9.example.com: Get https://site9.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:29:45 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:40Z" level=info msg="Timeout reading from socket for site6-staging.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:29:49 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:45Z" level=warning msg="Error for HTTP request to https://example.com: Get https://example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:29:49 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:44Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:29:49 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:44Z" level=info msg="Timeout reading from socket for site11.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:29:54 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:29:48Z" level=warning msg="Error for HTTP request to https://site7.example.com: Get https://site7.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:30:09 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:04Z" level=info msg="Timeout reading from socket for example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:30:11 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:08Z" level=info msg="Timeout reading from socket for site4.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:30:15 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:10Z" level=info msg="Timeout reading from socket for site3.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:30:17 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:12Z" level=info msg="Timeout reading from socket for site5.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:30:17 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:12Z" level=info msg="Timeout reading from socket for site2.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:30:20 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:14Z" level=warning msg="Error for HTTP request to https://site4.example.com: Get https://site4.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:30:22 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:15Z" level=warning msg="Error for HTTP request to https://site1.example.com: Get https://site1.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:30:26 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:22Z" level=warning msg="Error for HTTP request to https://site11.example.com: Get https://site11.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:30:32 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:26Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:30:52 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:48Z" level=warning msg="Error for HTTP request to https://site3.example.com: Get https://site3.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:30:55 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:51Z" level=info msg="Timeout reading from socket for site11.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:30:55 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:51Z" level=info msg="Timeout reading from socket for site1.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:30:56 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:51Z" level=info msg="Timeout reading from socket for prom.example.com: read ip4 0.0.0.0: i/o timeout" source="icmp.go:146"
Dec 17 17:30:59 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:51Z" level=error msg="Error resolving address host1.example.com: no suitable address found" source="icmp.go:81"
Dec 17 17:31:00 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:55Z" level=error msg="Error resolving address site8.example.com: no suitable address found" source="icmp.go:81"
Dec 17 17:31:02 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:30:58Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: 2016/12/17 22:31:03 http: panic serving 172.19.0.3:51572: runtime error: invalid memory address or nil pointer dereference
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: goroutine 1177822 [running]:
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*conn).serve.func1(0xc43ad9c480)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1491 +0x12a
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: panic(0x83a940, 0xc420010020)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/runtime/panic.go:458 +0x243
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: main.probeICMP(0xc4584b1ede, 0x17, 0xab3000, 0xc44f452750, 0xc4200ee568, 0x4, 0x12a05f200, 0x0, 0x0, 0x0, ...)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/icmp.go:84 +0x1fc
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: main.probeHandler(0xab3000, 0xc44f452750, 0xc45604aff0, 0xc42001e0b0)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/main.go:130 +0x218
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: main.main.func1(0xab3000, 0xc44f452750, 0xc45604aff0)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/main.go:170 +0x4a
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: net/http.HandlerFunc.ServeHTTP(0xc4200eeed0, 0xab3000, 0xc44f452750, 0xc45604aff0)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1726 +0x44
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*ServeMux).ServeHTTP(0xad3ae0, 0xab3000, 0xc44f452750, 0xc45604aff0)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2022 +0x7f
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: net/http.serverHandler.ServeHTTP(0xc420058800, 0xab3000, 0xc44f452750, 0xc45604aff0)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2202 +0x7d
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*conn).serve(0xc43ad9c480, 0xab3980, 0xc4584b1e80)
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1579 +0x4b7
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: created by net/http.(*Server).Serve
Dec 17 17:31:04 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2293 +0x44d
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: 2016/12/17 22:31:05 http: panic serving 172.19.0.3:51614: runtime error: invalid memory address or nil pointer dereference
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: goroutine 1177995 [running]:
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*conn).serve.func1(0xc44afe0b80)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1491 +0x12a
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: panic(0x83a940, 0xc420010020)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/runtime/panic.go:458 +0x243
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: main.probeICMP(0xc4511c521e, 0x15, 0xab3000, 0xc44f453520, 0xc4200ee568, 0x4, 0x12a05f200, 0x0, 0x0, 0x0, ...)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/icmp.go:84 +0x1fc
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: main.probeHandler(0xab3000, 0xc44f453520, 0xc453964780, 0xc42001e0b0)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/main.go:130 +0x218
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: main.main.func1(0xab3000, 0xc44f453520, 0xc453964780)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/main.go:170 +0x4a
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: net/http.HandlerFunc.ServeHTTP(0xc4200eeed0, 0xab3000, 0xc44f453520, 0xc453964780)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1726 +0x44
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*ServeMux).ServeHTTP(0xad3ae0, 0xab3000, 0xc44f453520, 0xc453964780)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2022 +0x7f
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: net/http.serverHandler.ServeHTTP(0xc420058800, 0xab3000, 0xc44f453520, 0xc453964780)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2202 +0x7d
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*conn).serve(0xc44afe0b80, 0xab3980, 0xc4511c51c0)
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1579 +0x4b7
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: created by net/http.(*Server).Serve
Dec 17 17:31:06 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2293 +0x44d
Dec 17 17:31:09 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:31:00Z" level=warning msg="Error for HTTP request to https://prom.example.com: Get https://prom.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:31:10 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:31:04Z" level=error msg="Error resolving address site10.example.com: no suitable address found" source="icmp.go:81"
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: 2016/12/17 22:31:14 http: panic serving 172.19.0.3:51616: runtime error: invalid memory address or nil pointer dereference
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: goroutine 1177996 [running]:
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*conn).serve.func1(0xc44afe1b00)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1491 +0x12a
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: panic(0x83a940, 0xc420010020)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/runtime/panic.go:458 +0x243
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: main.probeICMP(0xc45009da0e, 0x1c, 0xab3000, 0xc44f4535f0, 0xc4200ee568, 0x4, 0x12a05f200, 0x0, 0x0, 0x0, ...)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/icmp.go:84 +0x1fc
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: main.probeHandler(0xab3000, 0xc44f4535f0, 0xc4425a00f0, 0xc42001e0b0)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/main.go:130 +0x218
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: main.main.func1(0xab3000, 0xc44f4535f0, 0xc4425a00f0)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/go/src/github.com/prometheus/blackbox_exporter/main.go:170 +0x4a
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: net/http.HandlerFunc.ServeHTTP(0xc4200eeed0, 0xab3000, 0xc44f4535f0, 0xc4425a00f0)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1726 +0x44
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*ServeMux).ServeHTTP(0xad3ae0, 0xab3000, 0xc44f4535f0, 0xc4425a00f0)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2022 +0x7f
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: net/http.serverHandler.ServeHTTP(0xc420058800, 0xab3000, 0xc44f4535f0, 0xc4425a00f0)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2202 +0x7d
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: net/http.(*conn).serve(0xc44afe1b00, 0xab3980, 0xc456907540)
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:1579 +0x4b7
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: created by net/http.(*Server).Serve
Dec 17 17:31:14 localhost docker/example_blackbox_exporter_1[1210]: #011/usr/local/go/src/net/http/server.go:2293 +0x44d
Dec 17 17:31:17 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:31:12Z" level=warning msg="Error for HTTP request to https://site8.example.com: Get https://site8.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="http.go:164"
Dec 17 17:31:25 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:31:25Z" level=info msg="Starting blackbox_exporter (version=0.3.0, branch=master, revision=1722550ff00dc7435e8b39adf292bfc51b69c26f)" source="main.go:151"
Dec 17 17:31:25 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:31:25Z" level=info msg="Build context (go=go1.7.3, user=root@b7e4cd3247af, date=20161215-10:43:56)" source="main.go:152"
Dec 17 17:31:25 localhost docker/example_blackbox_exporter_1[1210]: time="2016-12-17T22:31:25Z" level=info msg="Listening on :9115" source="main.go:183"
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.