monitoringartist / dockbix-agent-xxl Goto Github PK

:whale: Dockerized Zabbix agent with Docker metrics and host metrics support for CoreOS, RHEL, CentOS, Ubuntu, Debian, Fedora, Boot2docker, Photon OS, Amazon Linux, ...

Home Page: https://hub.docker.com/r/monitoringartist/dockbix-agent-xxl-limited/

License: Other

zabbix-agent docker-monitoring docker monitoring devops zabbix-docker dockbix

dockbix-agent-xxl's Introduction

Dockbix Agent XXL - Dockerized Zabbix Agent with Docker, host, systemd, sockets monitoring

dockbix-agent-xxl-limited is a 2nd generation of successful (500k+ pulls) zabbix-agent-xxl-limited Docker image - it's a standard Zabbix agent with Docker/host/systemd/socket/stress metrics support, which can be used with any Zabbix server. Tested on CoreOS, RHEL, CentOS, Ubuntu, Debian, Fedora, Boot2docker, Photon OS.

If you like or use this project, please provide feedback to author - Star it ★ or submit feedback form.

Overview of Monitoring Artist (dockerized) monitoring ecosystem:

Dockbix XXL - Zabbix server/proxy/UI/snmpd/java gateway with additional extensions
Dockbix agent XXL - Zabbix agent with Docker (Kubernetes/Mesos/Chronos/Marathon) monitoring module
Zabbix templates - tiny Docker image for simple template deployment of selected Zabbix monitoring templates
Zabbix extension - all templates - storage image for Dockbix XXL with 200+ community templates
Kubernetized Zabbix - containerized Zabbix cluster based on Kubernetes
Grafana XXL - dockerized Grafana with all community plugins
Grafana dashboards - Grafana dashboard collection for AWS and Zabbix
Monitoring Analytics - graphic analytic tool for Zabbix data from data scientists
Docker killer - Docker image for Docker stress and Docker orchestration testing

Start Dockbix agent container and monitor all Docker containers on your host. Module Zabbix Docker monitoring is used - all docker module container metrics are available except docker.xnet. Import/assign Dockbix Agent XXL templates + then just start Dockbix agent container:

docker run \
  --name=dockbix-agent-xxl \
  --net=host \
  --privileged \
  -v /:/rootfs \
  -v /var/run:/var/run \
  --restart unless-stopped \
  -e "ZA_Server=<ZABBIX SERVER IP/DNS NAME/IP_RANGE>" \
  -e "ZA_ServerActive=<ZABBIX SERVER IP/DNS NAME>" \
  -d monitoringartist/dockbix-agent-xxl-limited:latest

docker run --rm -t monitoringartist/dockbix-agent-xxl-limited version

Please donate to the author, so he can continue to publish other awesome projects for free:

Available tags

Docker image tag	Docker image description
latest 3.4-3	Recommended version Zabbix 3.4 agent + Dockbix v3
3.2-2	Zabbix 3.2 agent + Dockbix v2
3.2-1	Zabbix 3.2 agent + Dockbix v1

Public image tags on Docker Hub | Private image tags on Docker Hub

Public limited vs private paid image

Public limited Docker image monitoringartist/dockbix-agent-xxl-limited:latest has almost the same functionality as private paid Docker image monitoringartist/dockbix-agent-xxl:latest. However, limited version:

doesn't support custom UserParameter(s)
doesn't support command execution in the container; for example, you can't use system.run[] item
doesn't support any shell access in the container; for example, you can't use docker exec -ti dockbix-agent-xxl bash
collects Google Analytics statistics (Docker version, kernel version, execution driver, Dockbix agent container start-up errors), which are used for improvements; statistics may also be used for public presentations

General limitations:

Dockbix agent container doesn't support docker stop/start container commands
Dockbix agent container doesn't support docker.xnet container metrics
host metric support is not complete; CPU and net host metrics are available, filesystem metrics with /rootfs prefix, process (proc.*) are not available due to cgroup limitation

Environment configuration variables

You can use almost any agent config parameter, just add prefix ZA_. If you don't specify a custom setting variable, then default Zabbix agent setting will be used. For example, you want to use StartAgents=10, just add environment variable -e "ZA_StartAgents=10".

Example:

docker run \
  --name=dockbix-agent-xxl \
  --net=host \
  --privileged \
  -v /:/rootfs \
  -v /var/run:/var/run \
  --restart unless-stopped \
  -e "ZA_Server=<ZABBIX SERVER IP/DNS NAME/IP RANGE>" \
  -e "ZA_ServerActive=<ZABBIX SERVER IP/DNS NAME>" \
  -e "ZA_StartAgents=10" \
  -e "ZA_Timeout=30" \
  -d monitoringartist/dockbix-agent-xxl-limited:latest

You can't override some parameters: AllowRoot, LoadModulePath, LoadModule, LogType, because Docker monitoring module is used. Also, Zabbix agent configuration from the config file is not supported.

Zabbix templates

Example of Zabbix templates, which can be used with Dockbix agent:

Templates for Dockbix Agent XXL - discovers and monitors (some) host and Docker metrics
Templates App systemd services - discovers and monitors systemd services metrics
Template App TCP Sockets - monitors TCP socket metrics
Template App UDP Sockets - monitors UDP socket metrics
Template App UNIX Sockets - monitors UNIX socket metrics
Template App Zabbix Server Stress 5k passive A - template for Zabbix stress testing

Migration

There is no particular action required to migrate from older Docker image monitoringartist/zabbix-agent-xxl-limited. Just change Docker image name. Environment variables are still the same.

Public test instance of Dockbix Agent XXL

A public test instance of Dockbix agent XXL is available on the address play.monitoringart.com:10050*. It's available for anyone as a public part of the monitoringartist playground. Just create a new host in your Zabbix server with this configuration, and you will see what Dockbix can provide:

Note: There is no guarantee of availability of public test instance of Dockbix Agent container.

How it works

No standard rpm/deb package installation or Zabbix module compilation. Just start dockbix-agent-xxl container and Docker container metrics will be collected from the Docker daemon API or cgroups.

How to monitor dockerized apps

Ideas about monitoring of dockerized apps:

health endpoint: publish app health as an HTTP endpoint, which can be used for monitoring (item keys: web.page.get, web.page.regexp)
monitor them as a service: use standard Zabbix TCP/HTTP check (item keys: net.tcp.port, net.tcp.service, net.tcp.service.perf)
monitor them as a process: use standard Zabbix process check (item key: proc.num)

Still no idea how to monitor your dockerized app? Hire us!

Included projects

zabbix-module-systemd - loadbable agent module for systemd monitoring
zabbix-module-sockets - loadbable agent module for sockets monitoring
zabbix_agent_bench - utility to test performance of Zabbix agent
zabbix-server-stress-test - loadbable agent module for stress testing

Zabbix agent stress testing:

$ docker run \
  --name=dockbix-agent-xxl \
  --net=host \
  --privileged \
  -v /:/rootfs \
  -v /var/run:/var/run \
  -e "ZA_Server=<ZABBIX SERVER IP/DNS NAME/IP RANGE>" \
  -e "ZA_ServerActive=<ZABBIX SERVER IP/DNS NAME>" \
  -d monitoringartist/dockbix-agent-xxl-limited:latest

# used HW - 8xCPU Intel(R) Xeon(R) CPU E31270 @ 3.40GHz:
$ docker exec -ti dockbix-agent-xxl zabbix_agent_bench -host 127.0.0.1 -timelimit 30 -key stress.ping --threads 50
Testing 1 keys with 50 threads (press Ctrl-C to cancel)...
stress.ping :	1225787	0	0

=== Totals ===

Total values processed:		1225787
Total unsupported values:	0
Total transport errors:		0
Total key list iterations:	1225787

Finished! Processed 1225787 values across 50 threads in 30.002141605s (40856.650040 NVPS)

Integrations/Docker orchestrations

Kubernetes - please visit https://github.com/monitoringartist/kubernetes-zabbix for inspiration
Puppet for dockbix-agent-xxl-limited
Ansible for dockbix-agent-xxl-limited
docker-compose for dockbix-agent-xxl-limited
systemd service unit file - see section below

Please feel free to create a pull request for other Docker orchestration tools: AWS ECS, Docker Swarm, Mesos/Marathon, Cloud Foundry, CoreOS Fleet, Azure ACS, Nomad, Zenoss Control Center, ...

3rd party projects:

https://github.com/ThomasVdBerge/zabbix-docker-terraform

Dockbix Agent XXL service managed by systemd

Example of systemd service unit file for Dockbix agent XXL - don't forget to edit environment variables. Then you can manage Dockbix agent XXL as a standard OS service:

$ wget -P /usr/lib/systemd/system/ https://raw.githubusercontent.com/monitoringartist/dockbix-agent-xxl/master/systemd/docker-dockbix-agent-xxl.service
# edit env variables: vi /usr/lib/systemd/system/docker-dockbix-agent-xxl.service
$ systemctl enable docker-dockbix-agent-xxl.service
Created symlink from /etc/systemd/system/multi-user.target.wants/docker-dockbix-agent-xxl.service to /usr/lib/systemd/system/docker-dockbix-agent-xxl.service.
$ systemctl start docker-dockbix-agent-xxl.service
$ systemctl status docker-dockbix-agent-xxl.service
● docker-dockbix-agent-xxl.service - Dockbix Agent XXL www.monitoringartist.com
   Loaded: loaded (/usr/lib/systemd/system/docker-dockbix-agent-xxl.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2016-08-20 00:17:04 BST; 5s ago
  Process: 72712 ExecStartPre=/usr/bin/docker pull monitoringartist/dockbix-agent-xxl-limited:latest (code=exited, status=0/SUCCESS)
 Main PID: 72719 (docker)
   Memory: 8.6M
   CGroup: /system.slice/docker-dockbix-agent-xxl.service
           └─72719 /usr/bin/docker run --rm -e ZA_Server=<ZABBIX SERVER IP/DNS NAME> --name docker-dockbix-agent-xxl.service -h dockerhost --net=host --privileged -v /:/rootfs...

Aug 20 00:17:05 dockerhost docker[72719]: 11:20160819:231705.575 IPv6 support:          YES
Aug 20 00:17:05 dockerhost docker[72719]: 11:20160819:231705.575 TLS support:           YES
Aug 20 00:17:05 dockerhost docker[72719]: 11:20160819:231705.575 **************************
Aug 20 00:17:05 dockerhost docker[72719]: 11:20160819:231705.575 using configuration file: /etc/zabbix/zabbix_agentd.conf
Aug 20 00:17:05 dockerhost docker[72719]: 11:20160819:231705.581 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so
Aug 20 00:17:05 dockerhost docker[72719]: 11:20160819:231705.583 agent #0 started [main process]
Aug 20 00:17:05 dockerhost docker[72719]: 12:20160819:231705.583 agent #1 started [collector]
Aug 20 00:17:05 dockerhost docker[72719]: 14:20160819:231705.583 agent #3 started [listener #2]
Aug 20 00:17:05 dockerhost docker[72719]: 13:20160819:231705.584 agent #2 started [listener #1]
Aug 20 00:17:05 dockerhost docker[72719]: 15:20160819:231705.586 agent #4 started [listener #3]
$ docker ps | grep dockbix-agent-xxl
b2ecbf3a7df0        monitoringartist/dockbix-agent-xxl-limited:latest   "/dockbix-agent-xxl"       25 seconds ago      Up 24 seconds       0.0.0.0:10050->10050/tcp   docker-zabbix-agent-xxl.service

Support

First try to troubleshoot problems yourself. Increase debug level -e 'ZA_DebugLevel=5' and check the container logs docker logs dockbix-agent-xxl. Try to obtain raw values from the agent, for example:

$ docker exec -ti dockbix-agent-xxl zabbix_get -s 127.0.0.1 -k docker.mem[/dockbix-agent-xxl,rss]
2977792

Other options:

Try to ask Zabbix community http://www.zabbix.org/wiki/Getting_help
If you need support directly from the author, then don't hesitate to contact him and ask for paid support

All reported issues, which are not real issues, but requests for support will be closed with reference to this README section.

Author

Devops Monitoring Expert, who loves monitoring systems and cutting/bleeding edge technologies: Docker, Kubernetes, ECS, AWS, Google GCP, Terraform, Lambda, Zabbix, Grafana, Elasticsearch, Kibana, Prometheus, Sysdig, ...

Summary:

2000+ GitHub stars
10 000+ Grafana dashboard downloads
1 000 000+ Docker image pulls

Professional devops / monitoring / consulting services:

dockbix-agent-xxl's People

Contributors

Stargazers

Watchers

dockbix-agent-xxl's Issues

Agent not running after host restart

After Docker host restart all containers return to running state. But dockbix-agent-xxl is in restarting state.
Error log:
Dockbix Agent XXL v0.0.1b limited version
Copyright (C) 2014-2017 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only.
Error #12 - problem during agent execution

Cannot open metric file: '/sys/fs/cgroup/memory/.../memory.stat'

My system is RHEL 7.3, docker version 1.12.5, build 047e51b/1.12.5, docker-compose version 1.10.0, build 4bd6f1a.
Selinux & user namespace's are enabled.

docker-compose.yml:
version: '2.1'
services:
    zocker:
        container_name: zocker
        image: monitoringartist/zabbix-agent-xxl-limited
        ports:
            - "10060:10050"
        hostname: $HOSTNAME 
        volumes: 
            - /:/rootfs:ro,z
            - /var/run:/var/run:ro,z
        environment:
            - ZA_Server=10.0.0.1
        userns_mode: host
        privileged: true

In logs:

13:20170220:133750.978 Cannot open metric file: '/sys/fs/cgroup/memory/d85758f142afcc4887f3bc2dcbae682e9b60cc63e487d530e225835e953d5747/memory.stat'
11:20170220:133751.981 Cannot open metric file: '/sys/fs/cgroup/memory/174aaa981703620fec6c8695289b6623390d1c04cd28a369bc1f1479106f2131/memory.stat'

Actually this 2 files are here:

/sys/fs/cgroup/memory/system.slice/docker-d85758f142afcc4887f3bc2dcbae682e9b60cc63e487d530e225835e953d5747.scope/memory.stat
/sys/fs/cgroup/memory/system.slice/docker-174aaa981703620fec6c8695289b6623390d1c04cd28a369bc1f1479106f2131.scope/memory.stat

Can you suggest some solutions?

Privileged flag not documented

In the readme all examples use the
--privileged
flag but its not documented.

Docker Swarm does not support the privileged flag, does this mean we cannot use this image?

moby/moby#24862

Cannot open Docker container metric file: '/rootfs/sys/fs/cgroup/memory/docker/.../memory.stat'

cat /etc/centos-release

CentOS Linux release 7.2.1511 (Core)

uname -a

Linux docker 3.10.0-327.4.5.el7.x86_64 #1 SMP Mon Jan 25 22:07:14 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

docker -v

Docker version 1.10.1, build 9e83765

docker version

Client:
Version: 1.10.1
API version: 1.22
Go version: go1.5.3
Git commit: 9e83765
Built: Thu Feb 11 19:18:46 2016
OS/Arch: linux/amd64

Server:
Version: 1.10.1
API version: 1.22
Go version: go1.5.3
Git commit: 9e83765
Built: Thu Feb 11 19:18:46 2016
OS/Arch: linux/amd64

TLS support

Dear developer, is there any chance to see TLS support ?

Can't see any data about containers

I deployed dockbix-agent-xxl in my K8s node and connect to Zabbix 4.0 server, the containers can be discovered, but no data is updated, even for the container created by docker itself.

increase startpoller

after edit /usr/local/etc/zabbix_server.conf all returned.

Restart not working

I have the dockbix-agent-xxl (limited) working on my environments.

When the server (or docker) restarts. the container should also restart automatically.

However, it stays stuck in the following state:

Restarting (1) 32 seconds ago

Docker logs reveals the following:

Dockbix Agent XXL v0.0.5b limited version Docker image: monitoringartist/dockbix-agent-xxl-limited Copyright (C) 2014-2017 Jan Garaj - www.monitoringartist.com Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only. Error #12 - problem during agent execution

Any idea what might cause this?

Ghost Container

Hello guys!

I use docker stack and my agents xxl are works fine.

My only problem is LLD in zabbix, because some containers that i already removed, still are monitoring by zabbix like a ghost container and sending triggers for me...

Do you know how to adjust this?

thanks so much...

How could i monitor container?

Hello,
I went through the read.me but to be honest, it was still very difficult to understand.
So basically, at this moment i have some container like this

I run the dockbix-agent-xxl container with Zabbix server IP (installed separately in different instance EC2), also i imported the Dockbix Agent XXL templates.

Now how could i add each container as host in Zabbix server console?

use proot with a recent kernel (4.8.4 and above) I got a segv

As a known issue, you get a segv with a kernel higher than 4.8.4
If you would update your README to include the following information it would eliminate your known issue

adding -e "ROOT_NO_SECCOMP=1" to disable SEGV it will work (of course you disable SEGV though).

Issue is brought up here - proot-me/proot#106

Cannot open memory.stat file and cannot get statistic

Hi,
I'm running dockerized version of dockbix-agent-xxl.
Zabbix can discovery the items, but only items working is the one do not use memory.stat.

Docker info:

`Containers: 14
Running: 14
Paused: 0
Stopped: 0
Images: 63
Server Version: 17.12.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.0-8-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.91GiB
Name: Docker2
ID: UAYJ:J33F:HHF3:2YQH:V5Y4:S2DH:Y4UX:7LJJ:DB7W:GWYL:G7SW:QORB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
`

OS:
`PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

I enabled Debug Level of the agent at 5.
Attached the log file, running for some minutes.
docker.zip

Thanks

How to get custom content, such as how many times to get docker restart

I want to get the restart times of the container. I have checked the docker API, but it seems that there is no parameter to get the restart times. Is there any other way to get it?

license is not clear if it's allowed to be copied in a form of docker copy

License document states that one cannot copy binary files dockbix-agent-xxl and zabbix_agentd included in Docker image monitoringartist/dockbix-agent-xxl-limited.
However, it's not clear if it's allowed to copy the docker image itself.

More clearly, is it allowed to execute following commands to monitoringartist/dockbix-agent-xxl-limited ?

docker pull
docker run

I am going to use the docker image in our environment and would like to confirm it does not violate the license.
My apologies for asking trivial questions, but my legal department requested me to make it clear.

Thank you in advance.

Kobayashi

no network and filesystem metrics

Containers: 20
 Running: 14
 Paused: 0
 Stopped: 6
Images: 98
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: s9i2in04wm6g1s333qr69ljxh
 Is Manager: true
 ClusterID: vl170qc80rt8yvbi80vo46e3o
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 10
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 192.168.32.XXX
 Manager Addresses:
  192.168.32.XXX:2377
  192.168.32.XXX:2377
  192.168.32.XXX:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: XXXXXXXX
runc version: XXXXXXXXXXXXX
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 8.615GiB
Name: nodem1.XXXX
ID: VIH6:MIWJ:JOL2:6FI5:GCP2:H6TH:IH5C:MW4G:443V:OUOL:KSH2:QSJU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: XXXXX
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

using template https://github.com/monitoringartist/dockbix-agent-xxl/blob/master/Zabbix-Templates-for-Dockbix-Agent-XXL.xml in my a zabbix server. All the metrics works fine, but the zabbix server do not get any the filesystem and network metrics. If i'm using an APP template + OS linux, the metric problem does not exists.

how to use key docker.xnet?

I deployed zabbix-agent-xxl container on CentOS7.1.
And I set item like this:
docker.xnet[/{#HCONTAINERID},all,TX-OK]

$docker logs zabbix-agent-xxl
got this error message: "Cannot create /var/run/netns".

too many levels of symbolic links

Environment -

Kubernetes / Docker cluster
Latest Dockbix image running with K8 yaml from github

Output -

Get http://unix.sock/info: dial unix /rootfs/var/run/docker.sock: connect: too many levels of symbolic links
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x661e52]

goroutine 1 [running]:
main.main()
/dockbix-agent-xxl.go:381 +0x11b2

Cannot connect to standard docker's socket /var/run/docker.sock

Start container (also try with --privileged option, same result)

docker run --name=zabbix-agent-xxl -h `hostname` -p 10050:10050 -v /:/rootfs  -e "ZA_Server=zabbix.srv"  -it monitoringartist/zabbix-agent-xxl-limited:latest

Output:

 Zabbix Agent XXL v0.8b public limited version
 Copyright (C) 2014-2016 Jan Garaj - www.monitoringartist.com
 Freeware licence - Usage of this binary is restricted to oficial monitoringartist Docker images only.
 Starting Zabbix Agent [swarm2]. Zabbix 3.0.0 XXL (2016-02-22) (revision {ZABBIX_REVISION}).
 Press Ctrl+C to exit.

     16:20160223:084954.710 Starting Zabbix Agent [swarm2]. Zabbix 3.0.0 XXL (2016-02-22) (revision {ZABBIX_REVISION}).
     16:20160223:084954.711 **** Enabled features ****
     16:20160223:084954.711 IPv6 support:          YES
     16:20160223:084954.711 TLS support:           YES
     16:20160223:084954.711 **************************
     16:20160223:084954.711 using configuration file: /etc/zabbix/zabbix_agentd.conf
     16:20160223:084954.711 Cannot connect to standard docker's socket /var/run/docker.sock
*** Error in `': munmap_chunk(): invalid pointer: 0x00007fb1773eae30 ***

Docker daemon running with parameters

daemon -H fd:// --log-driver=gelf --log-opt gelf-address=udp://192.168.15.93:12201 --log-opt tag=node1 --label enviroment=staging --label node=node2 --cluster-store=consul://192.168.15.66:8500 --cluster-advertise=eth0:2375 -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock

Cannot open metric file: '/sys/fs/cgroup/memory/system.slice

I am setting up Zabbix server in our environment. Connections from the Zabbix server to non-docker Hosts/servers is working flawlessly. We are deploying the Dockbix agent via Kubernetes, and are unable to get stats from containers, and it is flooding our central logging Server.

The most prominent errors are

Cannot find the [Id] item in the Received JSON object
Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//*Insert Docker instance name here */memory.stat'	
Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice//*Insert Docker instance name here */cpuacct.stat

During startup of the Dockbix instance, it seems to correctly initialize and find it's configuration.
It creates 6 Agents -
#0 = main process
#1 = Collector
#2-4 = listener
#5 = Active checks

I am finding it difficult to troubleshoot without access to the container via Docker exec, but I have exported the logs from the central logging server and doctored them to remove personally identifying information.

"Dockbix Agent XXL v0.0.1b limited version""," ,2017-06-19T08:59:25.542-0500,http:docker,_json
"{""line"":""Copyright (C) 2014-2017 Jan Garaj - www.monitoringartist.com""," ,2017-06-19T08:59:25.554-0500,http:docker,_json
"{""line"":""Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only.""," ,2017-06-19T08:59:25.559-0500,http:docker,_json
"{""line"":""Starting Zabbix Agent [Hostname]. Zabbix 3.2.4 Dockbix Agent XXL (2017-03-25) (revision ).""," ,2017-06-19T08:59:25.563-0500,http:docker,_json
"{""line"":""Press Ctrl+C to exit.""," ,2017-06-19T08:59:25.567-0500,http:docker,_json
"{""line"":""""," ,2017-06-19T08:59:25.572-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.556 Starting Zabbix Agent [Hostname]. Zabbix 3.2.4 Dockbix Agent XXL (2017-03-25) (revision ).""," ,2017-06-19T08:59:25.576-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.557 **** Enabled features ****""," ,2017-06-19T08:59:25.583-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.557 IPv6 support:          YES""," ,2017-06-19T08:59:25.588-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.557 TLS support:            NO""," ,2017-06-19T08:59:25.594-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.557 **************************""," ,2017-06-19T08:59:25.598-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.557 using configuration file: /etc/zabbix/zabbix_agentd.conf""," ,2017-06-19T08:59:25.604-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.562 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so""," ,2017-06-19T08:59:25.604-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.563 agent #0 started [main process]""," ,2017-06-19T08:59:25.607-0500,http:docker,_json
"{""line"":"" 10316:20170619:135925.564 agent #1 started [collector]""," ,2017-06-19T08:59:25.609-0500,http:docker,_json
"{""line"":"" 10317:20170619:135925.564 agent #2 started [listener #1]""," ,2017-06-19T08:59:25.619-0500,http:docker,_json
"{""line"":"" 10318:20170619:135925.565 agent #3 started [listener #2]""," ,2017-06-19T08:59:25.627-0500,http:docker,_json
"{""line"":"" 10319:20170619:135925.565 agent #4 started [listener #3]""," ,2017-06-19T08:59:25.631-0500,http:docker,_json
"{""line"":"" 10320:20170619:135925.565 agent #5 started [active checks #1]""," ,2017-06-19T08:59:25.634-0500,http:docker,_json
"{""line"":"" 10320:20170619:135925.568 active check configuration update from [shp-corp-zab.corporate:10051] started to fail (cannot resolve [shp-corp-zab.corporate])""," ,2017-06-19T08:59:25.636-0500,http:docker,_json
"{""line"":"" 10317:20170619:140034.310 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:34.311-0500,http:docker,_json
"{""line"":"" 10317:20170619:140034.311 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:34.343-0500,http:docker,_json
"{""line"":"" 10318:20170619:140034.635 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:34.636-0500,http:docker,_json
"{""line"":"" 10318:20170619:140034.636 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance-904c-005056bd0e22_25a1f918/memory.stat'""," ,2017-06-19T09:00:34.638-0500,http:docker,_json
"{""line"":"" 10318:20170619:140034.653 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:34.653-0500,http:docker,_json
"{""line"":"" 10318:20170619:140034.653 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice//$DockerInstance/cpuacct.stat'""," ,2017-06-19T09:00:34.656-0500,http:docker,_json
"{""line"":"" 10319:20170619:140035.713 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:35.713-0500,http:docker,_json
"{""line"":"" 10319:20170619:140035.713 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice//DockerInstance/cpuacct.stat'""," ,2017-06-19T09:00:35.715-0500,http:docker,_json
"{""line"":"" 10318:20170619:140035.727 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:35.727-0500,http:docker,_json
"{""line"":"" 10318:20170619:140035.727 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:35.728-0500,http:docker,_json
"{""line"":"" 10317:20170619:140036.731 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:36.731-0500,http:docker,_json
"{""line"":"" 10317:20170619:140036.731 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:36.733-0500,http:docker,_json
"{""line"":"" 10318:20170619:140036.781 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:36.781-0500,http:docker,_json
"{""line"":"" 10318:20170619:140036.781 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice//DockerInstance/cpuacct.stat'""," ,2017-06-19T09:00:36.783-0500,http:docker,_json
"{""line"":"" 10319:20170619:140037.840 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:37.841-0500,http:docker,_json
"{""line"":"" 10319:20170619:140037.840 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:37.842-0500,http:docker,_json
"{""line"":"" 10318:20170619:140037.841 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:37.851-0500,http:docker,_json
"{""line"":"" 10318:20170619:140037.841 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice//DockerInstance/cpuacct.stat'""," ,2017-06-19T09:00:37.855-0500,http:docker,_json
"{""line"":"" 10319:20170619:140038.880 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:38.880-0500,http:docker,_json
"{""line"":"" 10319:20170619:140038.880 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:38.882-0500,http:docker,_json
"{""line"":"" 10317:20170619:140038.880 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:38.883-0500,http:docker,_json
"{""line"":"" 10317:20170619:140038.881 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice//DockerInstance/cpuacct.stat'""," ,2017-06-19T09:00:38.884-0500,http:docker,_json
"{""line"":"" 10318:20170619:140039.891 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:39.891-0500,http:docker,_json
"{""line"":"" 10318:20170619:140039.891 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-/dockbix.scope/memory.stat'""," ,2017-06-19T09:00:39.895-0500,http:docker,_json
"{""line"":"" 10317:20170619:140039.903 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:39.903-0500,http:docker,_json
"{""line"":"" 10317:20170619:140039.903 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:39.907-0500,http:docker,_json
"{""line"":"" 10319:20170619:140040.912 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:40.913-0500,http:docker,_json
"{""line"":"" 10319:20170619:140040.913 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:40.916-0500,http:docker,_json
"{""line"":"" 10317:20170619:140040.914 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:40.918-0500,http:docker,_json
"{""line"":"" 10317:20170619:140040.914 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:40.918-0500,http:docker,_json
"{""line"":"" 10317:20170619:140040.952 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:40.953-0500,http:docker,_json
"{""line"":"" 10318:20170619:140040.954 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:40.956-0500,http:docker,_json
"{""line"":"" 10318:20170619:140040.954 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:40.960-0500,http:docker,_json
"{""line"":"" 10317:20170619:140041.928 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:41.928-0500,http:docker,_json
"{""line"":"" 10318:20170619:140041.928 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:41.932-0500,http:docker,_json

Cannot see Docker containers on Zabbix UI

Hi, first of all thanks for doing this. It's a great deal to have dockerized monitoring of the docker containers!
I was trying to run this, and launched first the zabbix-db
container and the zabbix server (localhost:10050). Then I launched this container, pointing to ZA_Server=http://localhost:10051 . There are no errors, and everything seems to be running as expected, but when I access the web interface in localhost:80, my containers are nowhere to be seen! Is there something that I need to configure in the UI, maybe? I apologize if this is a configuration problem, but I am not very experienced with Zabbix either. Bellow you can find some additional info about my system:


docker logs zabbix-agent-xxl

   11:20160428:075906.155 Starting Zabbix Agent [invismac]. Zabbix 3.0.1 XXL (2016-03-19) (revision {ZABBIX_REVISION}).
    11:20160428:075906.156 **** Enabled features ****
    11:20160428:075906.156 IPv6 support:          YES
    11:20160428:075906.156 TLS support:           YES
    11:20160428:075906.156 **************************
    11:20160428:075906.156 using configuration file: /etc/zabbix/zabbix_agentd.conf
    11:20160428:075906.157 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so
    11:20160428:075906.158 agent #0 started [main process]
    12:20160428:075906.158 agent #1 started [collector]
    13:20160428:075906.158 agent #2 started [listener #1]
    14:20160428:075906.158 agent #3 started [listener #2]
    15:20160428:075906.158 agent #4 started [listener #3]

docker logs zabbix

Nginx status page: allowed address set to 127.0.0.1.
PHP-FPM status page: allowed address set to 127.0.0.1.
[LOG 09:52:46] Preparing server configuration
[LOG 09:52:47] Config updated.
[LOG 09:52:47] Enabling logging and pid management
[LOG 09:52:47] Done
[LOG 09:52:47] Waiting for database server
[LOG 09:52:47] Database server is available
[LOG 09:52:47] Checking if database exists or SQL import is required
[WARNING 09:52:47] Zabbix database doesn't exist. Installing and importing default settings

[LOG 09:52:47] Database and user created, importing default SQL

[LOG 09:53:07] Import finished, starting
[LOG 09:53:07] Starting Zabbix version 3.0.1
2016
2016-04-28 09:53:07,928 CRIT Supervisor running as root (no user in config file)
2016-04-28 09:53:07,928 WARN Included extra file "/etc/supervisor.d/zabbix-agent.conf" during parsing
2016-04-28 09:53:07,928 WARN Included extra file "/etc/supervisor.d/zabbix-server.conf" during parsing
2016-04-28 09:53:07,928 WARN Included extra file "/etc/supervisor.d/php-fpm.conf" during parsing
2016-04-28 09:53:07,929 WARN Included extra file "/etc/supervisor.d/nginx.conf" during parsing
2016-04-28 09:53:07,949 INFO RPC interface 'supervisor' initialized
2016-04-28 09:53:07,949 INFO supervisord started with pid 969
2016-04-28 09:53:08,951 INFO spawned: 'nginx' with pid 972
2016-04-28 09:53:08,953 INFO spawned: 'nginx-reload' with pid 973
2016-04-28 09:53:08,955 INFO spawned: 'php-fpm' with pid 974
2016-04-28 09:53:08,957 INFO spawned: 'zabbix-agent' with pid 975
2016-04-28 09:53:08,959 INFO spawned: 'zabbix-server' with pid 976
2016-04-28 09:53:08,980 DEBG 'nginx' stderr output:
2016/04/28 09:53:08 [notice] 972#0: using the "epoll" event method
2016/04/28 09:53:08 [notice] 972#0: nginx/1.8.1
2016/04/28 09:53:08 [notice] 972#0: built by gcc 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) 
2016/04/28 09:53:08 [notice] 972#0: OS: Linux 4.2.0-35-generic
2016/04/28 09:53:08 [notice] 972#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576

2016-04-28 09:53:08,980 DEBG 'nginx' stderr output:
2016/04/28 09:53:08 [notice] 972#0: start worker processes
2016/04/28 09:53:08 [notice] 972#0: start worker process 983

2016-04-28 09:53:08,980 DEBG 'nginx' stderr output:
2016/04/28 09:53:08 [notice] 972#0: start worker process 984

2016-04-28 09:53:08,980 DEBG 'nginx' stderr output:
2016/04/28 09:53:08 [notice] 972#0: start worker process 985
2016/04/28 09:53:08 [notice] 972#0: start worker process 986

2016-04-28 09:53:08,989 DEBG 'zabbix-agent' stdout output:
Starting Zabbix Agent [Zabbix server]. Zabbix 3.0.1 (2016-04-26) (revision {ZABBIX_REVISION}).
Press Ctrl+C to exit.

   988:20160428:095308.989 Starting Zabbix Agent [Zabbix server]. Zabbix 3.0.1 (2016-04-26) (revision {ZABBIX_REVISION}).
   988:20160428:095308.989 **** Enabled features ****
   988:20160428:095308.989 IPv6 support:          YES
   988:20160428:095308.989 TLS support:           YES
   988:20160428:095308.989 **************************
   988:20160428:095308.989 using configuration file: /usr/local/etc/zabbix_agentd.conf

2016-04-28 09:53:08,990 DEBG 'zabbix-agent' stdout output:
   988:20160428:095308.989 agent #0 started [main process]

2016-04-28 09:53:08,990 DEBG 'zabbix-agent' stdout output:
   989:20160428:095308.990 agent #1 started [collector]

2016-04-28 09:53:08,990 DEBG 'zabbix-agent' stdout output:
   990:20160428:095308.990 agent #2 started [listener #1]

2016-04-28 09:53:08,990 DEBG 'zabbix-agent' stdout output:
   991:20160428:095308.990 agent #3 started [listener #2]

2016-04-28 09:53:08,994 DEBG 'zabbix-agent' stdout output:
   992:20160428:095308.993 agent #4 started [listener #3]

2016-04-28 09:53:08,994 DEBG 'zabbix-agent' stdout output:
   993:20160428:095308.994 agent #5 started [active checks #1]

2016-04-28 09:53:09,017 DEBG 'zabbix-agent' stdout output:
   993:20160428:095309.016 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 09:53:09,066 DEBG fd 14 closed, stopped monitoring <POutputDispatcher at 34415464 for <Subprocess at 33820040 with name php-fpm in state STARTING> (stdout)>
2016-04-28 09:53:09,067 DEBG fd 18 closed, stopped monitoring <POutputDispatcher at 34415176 for <Subprocess at 33820040 with name php-fpm in state STARTING> (stderr)>
2016-04-28 09:53:10,068 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-04-28 09:53:10,069 INFO success: nginx-reload entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-04-28 09:53:10,069 INFO success: php-fpm entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-04-28 09:53:10,069 INFO success: zabbix-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-04-28 09:53:10,069 INFO success: zabbix-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-04-28 09:55:09,034 DEBG 'zabbix-agent' stdout output:
   993:20160428:095509.033 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 09:57:09,049 DEBG 'zabbix-agent' stdout output:
   993:20160428:095709.049 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 09:59:09,064 DEBG 'zabbix-agent' stdout output:
   993:20160428:095909.063 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:01:09,079 DEBG 'zabbix-agent' stdout output:
   993:20160428:100109.079 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:03:09,096 DEBG 'zabbix-agent' stdout output:
   993:20160428:100309.096 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:05:09,110 DEBG 'zabbix-agent' stdout output:
   993:20160428:100509.110 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:07:09,124 DEBG 'zabbix-agent' stdout output:
   993:20160428:100709.124 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:09:09,139 DEBG 'zabbix-agent' stdout output:
   993:20160428:100909.139 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:11:09,154 DEBG 'zabbix-agent' stdout output:
   993:20160428:101109.154 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:13:09,169 DEBG 'zabbix-agent' stdout output:
   993:20160428:101309.169 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:15:09,181 DEBG 'zabbix-agent' stdout output:
   993:20160428:101509.181 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:17:09,195 DEBG 'zabbix-agent' stdout output:
   993:20160428:101709.195 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:19:09,208 DEBG 'zabbix-agent' stdout output:
   993:20160428:101909.208 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:21:09,224 DEBG 'zabbix-agent' stdout output:
   993:20160428:102109.224 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:23:09,242 DEBG 'zabbix-agent' stdout output:
   993:20160428:102309.242 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:25:09,256 DEBG 'zabbix-agent' stdout output:
   993:20160428:102509.256 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:27:09,270 DEBG 'zabbix-agent' stdout output:
   993:20160428:102709.269 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:29:09,283 DEBG 'zabbix-agent' stdout output:
   993:20160428:102909.282 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:31:09,297 DEBG 'zabbix-agent' stdout output:
   993:20160428:103109.296 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:33:09,313 DEBG 'zabbix-agent' stdout output:
   993:20160428:103309.313 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:35:09,328 DEBG 'zabbix-agent' stdout output:
   993:20160428:103509.327 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:37:09,338 DEBG 'zabbix-agent' stdout output:
   993:20160428:103709.338 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:39:09,349 DEBG 'zabbix-agent' stdout output:
   993:20160428:103909.349 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:41:09,362 DEBG 'zabbix-agent' stdout output:
   993:20160428:104109.362 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:43:09,380 DEBG 'zabbix-agent' stdout output:
   993:20160428:104309.379 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:45:09,396 DEBG 'zabbix-agent' stdout output:
   993:20160428:104509.395 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored

2016-04-28 10:47:09,410 DEBG 'zabbix-agent' stdout output:
   993:20160428:104709.410 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not monitored


docker ps

5c211d4ac54e        monitoringartist/zabbix-agent-xxl-limited:latest   "/zocker-xxl"            51 minutes ago      Up 51 minutes       0.0.0.0:10050->10050/tcp                                           zabbix-agent-xxl
a2ae8b1e50c4        zabbix/zabbix-3.0:latest                           "/config/bootstrap.sh"   58 minutes ago      Up 58 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:10051->10051/tcp, 162/udp, 10052/tcp   zabbix
ac3d36ae5af7        zabbix/zabbix-db-mariadb                           "/run.sh"                About an hour ago   Up About an hour    3306/tcp                                                           zabbix-db
0df55c1c7b89        livecompose_gn                                     "catalina.sh run"        45 hours ago        Up 34 minutes       8080/tcp, 0.0.0.0:8441->8443/tcp                                   livecompose_gn_1
fcfad4b77a4b        livecompose_gs                                     "catalina.sh run"        45 hours ago        Up 34 minutes       8080/tcp, 0.0.0.0:8442->8443/tcp                                   livecompose_gs_1
3284fabfaac3        livecompose_db                                     "/usr/lib/postgresql/"   45 hours ago        Up 34 minutes                                                                          livecompose_db_1
2d9cde554594        livecompose_data                                   "/bin/sh -c './etc/in"   45 hours ago        Up 34 minutes       0.0.0.0:9102->9102/tcp

Cannot open metric file: '/sys/fs/cgroup/memory/docker//4cef1a63087d/memory.stat

The stat file path is full container id, like this

panic: runtime error: invalid memory address or nil pointer dereference

Hello, after I deployed the agent on my host, the following error was shown.
Would someone tell me how to solve it? Thank you.
8:20180924:013104.498 Starting Zabbix Agent [website]. Zabbix 3.0.4 XXL (2016-08-21) (revision 61185).
8:20180924:013104.499 **** Enabled features ****
8:20180924:013104.499 IPv6 support: YES
8:20180924:013104.499 TLS support: YES
8:20180924:013104.499 **************************
8:20180924:013104.499 using configuration file: /etc/zabbix/zabbix_agentd.conf
8:20180924:013104.500 Cannot connect to standard docker's socket /var/run/docker.sock
8:20180924:013104.500 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so
8:20180924:013104.503 agent #0 started [main process]
13:20180924:013104.503 agent #4 started [listener #3]
12:20180924:013104.504 agent #3 started [listener #2]
11:20180924:013104.504 agent #2 started [listener #1]
10:20180924:013104.509 agent #1 started [collector]
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x40596a]

goroutine 1 [running]:
panic(0x7eb920, 0xc82000a070)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
main.main()
/root/git/zabbix-agent-xxl-source-code-private/zabbix-agent-3.0-limited/zocker-xxl.go:191 +0x496a
Zocker XXL v0.9.4b public limited version
Copyright (C) 2014-2016 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only.
Error #10

how to deploy zabbix-agent on real host but not zabbix-agent-xxl?

I have deployed zabbix-agent(3.0.0) on my real host(not by container).

$cat /etc/zabbix/zabbix_agentd.conf
PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=0
Server=10.43.1.198
ServerActive=10.43.1.198
Hostname=vhs10-43-1-193.ubank365.local
Include=/etc/zabbix/zabbix_agentd.d/
AllowRoot=1
LoadModulePath=/etc/zabbix/zabbix_agentd.d/modules
LoadModule=zabbix_module_docker.so
LoadModule=zabbix_module_stress.so

*.so I have download and chmod 755

zabbix server said on the dashboard:
Received empty response from Zabbix Agent at [10.43.1.193]. Assuming that agent dropped connection because of access permissions.

$docker logs zabbix
Zabbix agent item "docker.discovery" on host "vhs10-43-1-193.ubank365.local" failed: another network error, wait for 15 seconds

use zabbix-agent-xll-limited worked well. But I can not use any command. like:
docker exec zabbix-agent-xxl ps -ef (not found ps in $PATH)

Stops working after container restart

docker run \
  --name=zabbix-agent-xxl \
  -p 10050:10050 \
  -v /:/rootfs \
  -v /var/run:/var/run \
  -e "ZA_Server=server" \
  -e "ZA_ServerActive=server" \
  -e "ZA_StartAgents=5" \
  -e "ZA_Timeout=30" \
  -e "ZA_ListenIP=`ifconfig tun0 | grep "inet addr" | awk -F: '{print $2}' | awk '{print $1}'`" \
  -e "ZA_Hostname=$(hostname)"  \
  --net=host \
  -d --restart=always --name zabbix-agent monitoringartist/zabbix-agent-xxl-limited:latest

It starts and works but after i stop and start the container is breaking:

Zocker XXL v0.9.2b public limited version
Copyright (C) 2014-2016 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to oficial monitoringartist Docker images only.
Error #10
Zocker XXL v0.9.2b public limited version
Copyright (C) 2014-2016 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to oficial monitoringartist Docker images only.
Error #10
Zocker XXL v0.9.2b public limited version
Copyright (C) 2014-2016 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to oficial monitoringartist Docker images only.
Error #10

Doesn't support Ubuntu 16.04.1 LTS?

I started one zabbix-agent-xxl container in Ubuntu 16.04.1 LTS.

and got this result from zabbix-3.0-xxl:

docker.up/systemd.up check is not available at the moment - no cpu_cgroup directory

and got these logs from zabbix-agent-xxl container:

Starting Zabbix Agent [c5cb80f7158b]. Zabbix 3.0.3 XXL (2016-06-26) (revision {ZABBIX_REVISION}).
Press Ctrl+C to exit.

 8:20160823:150908.591 Starting Zabbix Agent [c5cb80f7158b]. Zabbix 3.0.3 XXL (2016-06-26) (revision {ZABBIX_REVISION}).
 8:20160823:150908.591 **** Enabled features ****
 8:20160823:150908.591 IPv6 support:          YES
 8:20160823:150908.591 TLS support:           YES
 8:20160823:150908.591 **************************
 8:20160823:150908.591 using configuration file: /etc/zabbix/zabbix_agentd.conf
 8:20160823:150908.592 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so
 8:20160823:150908.593 agent #0 started [main process]
12:20160823:150908.593 agent #4 started [listener #3]
11:20160823:150908.593 agent #3 started [listener #2]
10:20160823:150908.596 agent #2 started [listener #1]
 9:20160823:150908.598 agent #1 started [collector]
11:20160823:150959.933 Cannot open metric file: 'memory/c5cb80f7158bb80d4a254af264ded1412e8e74a57c88b61904797ca11a8fb003/memory.stat'
10:20160823:150959.936 Cannot open metric file: 'memory/c0f3ffe8120a93712c2e3e6ac7e8fd6d3beeb63f88be8344f391623a717adb28/memory.stat'

my docker version:
Docker version 1.12.1, build 23cf638

add custom low level discovery rule in python

Hi I want to thank you for handling this awesome project.

I want to ask a little detail about how should I do that. Where place my .py file and which enviroment variable need to set to use as a discovery key in zabbix server?

Cannot find the [Id] item in the received JSON object

Hi,

My logs for dockbix-agent-xxl (not limited) container flooding me with following content:

16:20170814:110500.780 Cannot find the [Id] item in the received JSON object
16:20170814:110501.854 Cannot find the [Id] item in the received JSON object
14:20170814:110501.877 Cannot find the [Id] item in the received JSON object
16:20170814:110502.920 Cannot find the [Id] item in the received JSON object
15:20170814:110502.980 Cannot find the [Id] item in the received JSON object
15:20170814:110503.581 Cannot find the [Id] item in the received JSON object
14:20170814:110503.638 Cannot find the [Id] item in the received JSON object
15:20170814:110505.030 Cannot find the [Id] item in the received JSON object
14:20170814:110505.038 Cannot find the [Id] item in the received JSON object
16:20170814:110505.051 Cannot find the [Id] item in the received JSON object
14:20170814:110505.091 Cannot find the [Id] item in the received JSON object
16:20170814:110505.130 Cannot find the [Id] item in the received JSON object
14:20170814:110506.135 Cannot find the [Id] item in the received JSON object
16:20170814:110506.149 Cannot find the [Id] item in the received JSON object
15:20170814:110506.181 Cannot find the [Id] item in the received JSON object
16:20170814:110507.206 Cannot find the [Id] item in the received JSON object
16:20170814:110507.285 Cannot find the [Id] item in the received JSON object
14:20170814:110509.510 Cannot find the [Id] item in the received JSON object
16:20170814:110509.510 Cannot find the [Id] item in the received JSON object
14:20170814:110510.569 Cannot find the [Id] item in the received JSON object
14:20170814:110511.653 Cannot find the [Id] item in the received JSON object
15:20170814:110511.674 Cannot find the [Id] item in the received JSON object
14:20170814:110512.715 Cannot find the [Id] item in the received JSON object
16:20170814:110512.772 Cannot find the [Id] item in the received JSON object
16:20170814:110513.831 Cannot find the [Id] item in the received JSON object
14:20170814:110513.839 Cannot find the [Id] item in the received JSON object
16:20170814:110514.877 Cannot find the [Id] item in the received JSON object
14:20170814:110514.882 Cannot find the [Id] item in the received JSON object
14:20170814:110514.889 Cannot find the [Id] item in the received JSON object
15:20170814:110515.926 Cannot find the [Id] item in the received JSON object
16:20170814:110515.968 Cannot find the [Id] item in the received JSON object
15:20170814:110516.976 Cannot find the [Id] item in the received JSON object
14:20170814:110516.981 Cannot find the [Id] item in the received JSON object
16:20170814:110517.014 Cannot find the [Id] item in the received JSON object
16:20170814:110517.038 Cannot find the [Id] item in the received JSON object
16:20170814:110517.086 Cannot find the [Id] item in the received JSON object
15:20170814:110519.220 Cannot find the [Id] item in the received JSON object
16:20170814:110519.221 Cannot find the [Id] item in the received JSON object
14:20170814:110520.249 Cannot find the [Id] item in the received JSON object

Got like hundreds of this every minute and the memory of the container is peaking.
I've set memory limit of 500Mb for the container but it crash after a while.

Sometimes it give me a message like:
14:20170814:110813.103 Cannot open metric file: '/sys/fs/cgroup/memory/docker//2b528c5d9b5cdd362a761d2b78e8be50845e6187c417f37ca3e4a44c85e57177/memory.stat'

"-v /var/run:/var/run" required in docker-compose.yml?

Hi. Thank you for sharing your great project. I noticed something. Your readme says it needs "-v /var/run:/var/run " but your docker-compose.yml doesn't have it.
Which is correct?

Zabbix-agent-xxl error #10

@jangaraj
I am trying to setup zabbix for our docker containers.
I have setup Zabbix server and DB by following - https://www.zabbix.org/wiki/Dockerized_Zabbix
Zabbix server is up and running fine.

I am having trouble in setting up zabbix xxl agent.
followed this link but no luck

Docker command to create agent container
tried both dockbix agent and zabbix agent xxl
dzdo docker run
--name=dockbix-agent-xxl
--net=host
--privileged
-v /:/rootfs
-v /var/run:/var/run
-e "ZA_Server=*****"
-d monitoringartist/dockbix-agent-xxl-limited:latestst

For Dockbix i get - Error #12 - Problem during agent execuion

zabbix agent -
dzdo docker run -i
--name=zabbix-agent-xxl
-h $(hostname)
-p 10050:10050
-v /:/rootfs
-e "ZA_Server=<>"
-d monitoringartist/zabbix-agent-xxl-limited:latest

Error log

Zocker XXL v0.9.4b public limited version
Copyright (C) 2014-2016 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only.
Starting Zabbix Agent [dcndkr10u]. Zabbix 3.0.4 XXL (2016-08-21) (revision 61185).
Press Ctrl+C to exit.

 9:20170427:181424.744 Starting Zabbix Agent [dcndkr10u]. Zabbix 3.0.4 XXL (2016-08-21) (revision 61185).
 9:20170427:181424.744 **** Enabled features ****
 9:20170427:181424.744 IPv6 support:          YES
 9:20170427:181424.744 TLS support:           YES
 9:20170427:181424.744 **************************
 9:20170427:181424.744 using configuration file: /etc/zabbix/zabbix_agentd.conf
 9:20170427:181424.745 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so
 9:20170427:181424.747 agent #0 started [main process]
10:20170427:181424.747 agent #1 started [collector]
11:20170427:181424.748 agent #2 started [listener #1]
12:20170427:181424.750 agent #3 started [listener #2]
13:20170427:181424.750 agent #4 started [listener #3]

remove /lib/x86_64-linux-gnu//security: directory not empty
Zocker XXL v0.9.4b public limited version
Copyright (C) 2014-2016 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only.
Error #10

Docker info

Containers: 35
Running: 34
Paused: 0
Stopped: 1
Images: 27
Server Version: 1.13.1
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: active
NodeID: *********
Is Manager: true
ClusterID: *********
Managers: 1
Nodes: 1
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
External CAs:
cfssl: https://*1/api/v1/cfssl/sign
Node Address:
Manager Addresses:

Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.10.2.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 15.51 GiB
Name: **********
ID: YD3E:UU4G:6HJ6:NXC7:CN3S:QNSE:F46P:AONP:ZP46:FVX4:VBDF:6XNF
Docker Root Dir: /"***************
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:

Live Restore Enabled: false

Not receiving mounted file systems discovery

I'm not able to monitor my disk usage on my docker hosts. Getting:

item "pr-nodedocker2in:vfs.fs.discovery" became not supported: Unsupported item key.

Is there another solution?

private paid image limitations

I found three "General limitaions" in README in github. However, in your product page, it's only one "General limitations" about TLS. What does the "General limitations" mean? I wonder which description is for private paid image.

With private paid image, can I use docker stop/start container command or not ?

docker container crash

I try to use your docker image to monitor my docker containers.

My system - ubuntu 14.04
Docker version - 1.10.2

Run command:

docker run --name=zabbix-agent-xxl -h my-hostname -p 31050:10050 -v /:/rootfs -e "ZA_Server=zabbix.marathon.mesos" -e "ZA_ServerActive=zabbix.marathon.mesos" -e "ZA_StartAgents=10" -e "ZA_Timeout=30" --rm monitoringartist/zabbix-agent-xxl-limited:latest

Start error:

Zocker XXL v0.9.1b public limited version
Copyright (C) 2014-2016 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to oficial monitoringartist Docker images only.
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x8 pc=0x4e77d7]

goroutine 1 [running]:
github.com/fsouza/go-dockerclient.(*Env).Map(0x0, 0x1b)
        /usr/lib/golang/src/github.com/fsouza/go-dockerclient/env.go:159 +0x27
github.com/fsouza/go-dockerclient.(*Env).Get(0x0, 0x7e07e0, 0x4, 0x0, 0x0)
        /usr/lib/golang/src/github.com/fsouza/go-dockerclient/env.go:20 +0x2f
main.main()
        /root/zabbix-agent-docker-commercial/zocker-xxl.go:50 +0xc27

goroutine 10 [select]:
net/http.(*persistConn).readLoop(0xc8200a60b0)
        /usr/lib/golang/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
        /usr/lib/golang/src/net/http/transport.go:685 +0xc78

goroutine 11 [select]:
net/http.(*persistConn).writeLoop(0xc8200a60b0)
        /usr/lib/golang/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/lib/golang/src/net/http/transport.go:686 +0xc9d

Error #3 on host with userns enabled

Hello. I'm trying to launch zabbix agent on CentOS 7.2 host with enabled user namespaces for docker. And agent container falls with Error #3 message.

docker -v

Docker version 1.11.2, build b9f10c9

docker daemon flags

docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --userns-remap=dcu --storage-driver devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt dm.use_deferred_removal=true --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic

docker-compose.yml

version: "2"

services:
  agent:
    image: monitoringartist/zabbix-agent-xxl-limited
    container_name: monitoring_agent
    hostname: hope.ua
    ports:
      - "10050:10050"
    volumes:
      - /:/rootfs
      - /var/run:/var/run
    environment:
      ZA_Server: <zabbix.server>
    restart: always

Logs of docker container:

Zocker XXL v0.9.4b public limited version
Copyright (C) 2014-2016 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only.
Error #3

Compatibility with Marathon

Any way I can get this to work with Mesos/Marathon?

--container="mesos-ed031b0b-a489-4455-9a03-d11b83a98929-S4.a5eddd84-f247-4f96-85c6-2ac85f8e31c7" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/opt/mesosphere/packages/mesos--0335ca0d3700ea88ad8b808f3b1b84d747ed07f0/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/var/lib/mesos/slave/slaves/ed031b0b-a489-4455-9a03-d11b83a98929-S4/frameworks/ed031b0b-a489-4455-9a03-d11b83a98929-0000/executors/zabbix-agent.d6972ff9-64b1-11e6-965f-0242e15b1d74/runs/a5eddd84-f247-4f96-85c6-2ac85f8e31c7" --stop_timeout="0ns"
--container="mesos-ed031b0b-a489-4455-9a03-d11b83a98929-S4.a5eddd84-f247-4f96-85c6-2ac85f8e31c7" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/opt/mesosphere/packages/mesos--0335ca0d3700ea88ad8b808f3b1b84d747ed07f0/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/var/lib/mesos/slave/slaves/ed031b0b-a489-4455-9a03-d11b83a98929-S4/frameworks/ed031b0b-a489-4455-9a03-d11b83a98929-0000/executors/zabbix-agent.d6972ff9-64b1-11e6-965f-0242e15b1d74/runs/a5eddd84-f247-4f96-85c6-2ac85f8e31c7" --stop_timeout="0ns"
Registered docker executor on 10.31.1.12
Starting task zabbix-agent.d6972ff9-64b1-11e6-965f-0242e15b1d74
Zocker XXL v0.9.4b public limited version
Copyright (C) 2014-2016 Jan Garaj - www.monitoringartist.com 
Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only.
Starting Zabbix Agent [dcosagent003]. Zabbix 3.0.3 XXL (2016-06-26) (revision {ZABBIX_REVISION}).
Press Ctrl+C to exit.

    11:20160817:193606.321 Starting Zabbix Agent [dcosagent003]. Zabbix 3.0.3 XXL (2016-06-26) (revision {ZABBIX_REVISION}).
    11:20160817:193606.321 **** Enabled features ****
    11:20160817:193606.321 IPv6 support:          YES
    11:20160817:193606.321 TLS support:           YES
    11:20160817:193606.321 **************************
    11:20160817:193606.321 using configuration file: /etc/zabbix/zabbix_agentd.conf
    11:20160817:193606.321 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so
    11:20160817:193606.322 agent #0 started [main process]
    12:20160817:193606.322 agent #1 started [collector]
    13:20160817:193606.322 agent #2 started [listener #1]
    14:20160817:193606.322 agent #3 started [listener #2]
    15:20160817:193606.322 agent #4 started [listener #3]
remove /lib/x86_64-linux-gnu//security: directory not empty
Received killTask
Shutting down

CentOS Linux release 7.2.1511 (Core)

Cannot open Docker container metric file: '/rootfs/sys/fs/cgroup/memory/docker/.../memory.stat'

docker info

Containers: 7
Running: 4
Paused: 0
Stopped: 3
Images: 10
Server Version: 1.10.3
Storage Driver: aufs
Root Dir: /data/docker/aufs
Backing Filesystem: extfs
Dirs: 129
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 23.6 GiB
Name: bla-bla
ID: DWGA:Q5HB:VP4Q:HHFD:BPL6:VBR4:LE2Y:KSEE:J3NZ:JDYK:LGIL:VPGF
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

ls -1 /sys/fs/cgroup/

blkio
cpu
cpuacct
cpu,cpuacct
cpuset
devices
freezer
net_cls
net_cls,net_prio
net_prio
perf_event
systemd

cat docker-compose.yml

zabbix-agent-xxl:
image: monitoringartist/zabbix-agent-xxl-limited:latest
hostname: bla-bla
ports:
- "10050:10050"
volumes:
- /:/rootfs
- /var/run:/var/run
environment:
- ZA_Server=bla-bla
- ZA_ServerActive=bla-bla
- ZA_Timeout=30

cpu and status get correctly

Dockbix agent XXL - enable TLS configuration

Hi,

I want to enabled TLSConnect=psk and TLSAccept=psk, so I have tried setting environment variable ZA_TLSConnect=psk, ZA_TLSAccept=psk, ZA_TLSPSKIdentity and ZA_TLSPSKFile but this has no effect.
The agent log contains a messaging saying "failed to accept an incoming connection: from ...: TLS connections are not allowed"

I used the above environment variables as "https://hub.docker.com/r/monitoringartist/dockbix-agent-xxl-limited/" states "You can use almost any agent config parameter, just add prefix ZA_ " with a link to agent configuration page.

Global regular expression "Exclude Docker related file systems" does not exist.

Discovery error: Global regular expression "Exclude Docker related file systems" does not exist.
Template require regular expression.
This part is missing in docs.

Long and not friendly container names

I'm testing dockbix to monitor google cloud container engine (kubernetes)

Dockbix discover items like
k8s_nginx.eb9a4879_nginx-701339712-8rcxq_default_e1bca122-3650-11e7-83f6-42010a840180_b89a0cd9

and with some items it cannot create them
Cannot create graph: value "State of k8s_kube-proxy.1dc32eb9_kube-proxy-gke-zabbix-test-default-pool-d2963b18-l63q_kube-system_cb6d4d0528f97c31a8e3c883bfdc0994_cda8690c container" is too long.

Can be modified to capture the friendly name (as kubectl get pods)

Thanks

Error response from daemon: Unable to remove filesystem for ab68b000adfca007 ...

Hello.

docker-compose.yml

zabbix-agent-xxl:
    image: monitoringartist/zabbix-agent-xxl-limited:latest
    hostname: agent1
    ports:
        - "10050:10050"
    volumes:
        - /:/rootfs
        - /var/run:/var/run
    environment:
        - ZA_Server=192.168.2.176
        - ZA_ServerActive=192.168.2.176

docker-compose up -d

output:

[root@localhost zabbix-agent]# docker-compose up -d
Creating zabbixagent_zabbix-agent-xxl_1
Error response from daemon: Unable to remove filesystem for ab68b000adfca007cfa18d2f4c12c31c60818ace86c27c3a61afd53d6d05e073: remove /var/lib/docker/containers/ab68b000adfca007cfa18d2f4c12c31c60818ace86c27c3a61afd53d6d05e073/shm: device or resource busy
[root@localhost zabbix-agent]#

OS:

LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.2.1511 (Core) 
Release:        7.2.1511
Codename:       Core

[root@localhost zabbix-agent]# docker logs b5792366ef21
Zocker XXL v0.9.3b public limited version
Copyright (C) 2014-2016 Jan Garaj - www.monitoringartist.com 
Freeware licence - Usage of this binary is restricted to oficial monitoringartist Docker images only.
Starting Zabbix Agent [agent1]. Zabbix 3.0.3 XXL (2016-06-16) (revision {ZABBIX_REVISION}).
Press Ctrl+C to exit.

     8:20160619:181500.031 Starting Zabbix Agent [agent1]. Zabbix 3.0.3 XXL (2016-06-16) (revision {ZABBIX_REVISION}).
     8:20160619:181500.032 **** Enabled features ****
     8:20160619:181500.032 IPv6 support:          YES
     8:20160619:181500.032 TLS support:           YES
     8:20160619:181500.032 **************************
     8:20160619:181500.032 using configuration file: /etc/zabbix/zabbix_agentd.conf
     8:20160619:181500.040 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so
     8:20160619:181500.041 agent #0 started [main process]
    10:20160619:181500.042 agent #1 started [collector]
    11:20160619:181500.043 agent #2 started [listener #1]
    12:20160619:181500.043 agent #3 started [listener #2]
    13:20160619:181500.044 agent #4 started [listener #3]
    14:20160619:181500.044 agent #5 started [active checks #1]
    14:20160619:181503.058 active check configuration update from [192.168.2.176:10051] started to fail (ZBX_TCP_READ() timed out)

[root@localhost zabbix-agent]# docker -v
Docker version 1.11.2, build b9f10c9

[root@localhost zabbix-agent]# telnet 192.168.2.176 10051
Trying 192.168.2.176...
Connected to 192.168.2.176.
Escape character is '^]'.
Connection closed by foreign host.

Server can't get any info from agent

Hi!
We installed Dockbix-Agent-XXL (latest version) on our Kubernetes cluster today, according to the logs it's just running fine, and our zabbix server (which is not hosted on the cluster) can detect the agent.

However, when trying to get any information from the agent, we just receive this : "invalid item key format" (we even tried to get the agent.version value, which also returns invalid item key format).

Is it impossible to deploy Dockbix on Kubernetes yet? I also want to mention our server is using Zabbix 4, but according to Zabbix's doc, older agents are supposed to work..

Thank you for your reply!

zabbix-agent start failed

I can not start the zabbix-agent-xxl.here is the logs
Zocker XXL v0.9.4b public limited version
Copyright (C) 2014-2016 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only.
Starting Zabbix Agent [docker02]. Zabbix 3.0.4 XXL (2016-08-21) (revision 61185).
Press Ctrl+C to exit.

10:20160907:080011.397 Starting Zabbix Agent [docker02]. Zabbix 3.0.4 XXL (2016-08-21) (revision 61185).
10:20160907:080011.397 **** Enabled features ****
10:20160907:080011.397 IPv6 support:          YES
10:20160907:080011.397 TLS support:           YES
10:20160907:080011.397 **************************
10:20160907:080011.397 using configuration file: /etc/zabbix/zabbix_agentd.conf
10:20160907:080011.398 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so
10:20160907:080011.398 agent #0 started [main process]
11:20160907:080011.398 agent #1 started [collector]
12:20160907:080011.399 agent #2 started [listener #1]
13:20160907:080011.400 agent #3 started [listener #2]
14:20160907:080011.401 agent #4 started [listener #3]

remove /lib/x86_64-linux-gnu//security: directory not empty

how can i reslove this problem,ths!!!

Cannot get process number in container with dockbix-agent

hi everyone~
I install dockbix-xxl in centos 7.4 then use it to monitor a container which is running apache.
but when i use "zabbix_get -s 172.16.0.195 -k 'proc.num[apache]'" ,the zabbix-server gets 0.where am i wrong?

[root@lcshop-zabbix ~]# zabbix_get -s 172.16.0.195 -k 'net.tcp.listen[9099]'
1
[root@lcshop-zabbix ~]# zabbix_get -s 172.16.0.195 -k 'proc.num[apache]'
0

letsencrypt ca missing in docker container

i had the following error when trying to import a configuration template:

docker run --rm \

-e XXL_apiurl=https://zabbix.lynx.website
-e XXL_apiuser=xxxxx
-e XXL_apipass=xxxxxx
monitoringartist/zabbix-templates

Post https://zabbix.lynx.website/api_jsonrpc.php: x509: failed to load system roots and no roots provided

according to the following blogpost its probably due to ssl certificate issues.

http://blog.cloud66.com/x509-error-when-using-https-inside-a-docker-container/

i want to add that we use lets encrypt certificates on the server. maybe thats a special root certificate?

cheers!

No Docker Metrics Available in zabbix

below are info from logs

Docker image: monitoringartist/dockbix-agent-xxl-limited
Copyright (C) 2014-2017 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only.
Starting Zabbix Agent [jira02.sg.aws.grabpay.com]. Zabbix 3.4.2 Dockbix Agent XXL (2017-09-26) (revision 72885).
Press Ctrl+C to exit.

     7:20171214:074845.476 Starting Zabbix Agent [jira02.sg.aws.grabpay.com]. Zabbix 3.4.2 Dockbix Agent XXL (2017-09-26) (revision 72885).
     7:20171214:074845.478 **** Enabled features ****
     7:20171214:074845.478 IPv6 support:          YES
     7:20171214:074845.478 TLS support:           YES
     7:20171214:074845.478 **************************
     7:20171214:074845.478 using configuration file: /etc/zabbix/zabbix_agentd.conf
     7:20171214:074845.478 In zbx_load_modules()
     7:20171214:074845.478 loading module "/var/lib/zabbix/modules/zabbix_module_docker.so"
     7:20171214:074845.478 In zbx_module_api_version()
     7:20171214:074845.478 In zbx_module_init()
     7:20171214:074845.478 zabbix_module_docker v0.6.8, compilation time: Sep 26 2017 22:48:15
     7:20171214:074845.478 In zbx_docker_dir_detect()
     7:20171214:074845.479 Cannot detect used docker driver
     7:20171214:074845.479 In zbx_docker_api_detect()
     7:20171214:074845.479 In zbx_module_docker_socket_query()
     7:20171214:074845.479 Docker's socket query: GET /_ping HTTP/1.0
     7:20171214:074845.479 Docker's socket response: OK
     7:20171214:074845.479 Docker's socket works - extended docker metrics are available
     7:20171214:074845.479 In zbx_module_item_list()
     7:20171214:074845.479 In zbx_module_item_timeout()
     7:20171214:074845.479 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_docker.so": /var/lib/zabbix/modules/zabbix_module_docker.so: undefined symbol: zbx_module_history_write_cbs
     7:20171214:074845.479 loading module "/var/lib/zabbix/modules/zabbix_module_stress.so"
     7:20171214:074845.479 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_stress.so": /var/lib/zabbix/modules/zabbix_module_stress.so: undefined symbol: zbx_module_history_write_cbs
     7:20171214:074845.479 loading module "/var/lib/zabbix/modules/zabbix_module_systemd.so"
     7:20171214:074845.481 [systemd] starting v3.4.2, compiled: Sep 26 2017 22:47:47
     7:20171214:074845.481 [systemd] in cgroup_dir_detect()
     7:20171214:074845.481 [systemd] detected cgroup mount directory: /rootfs/var/lib/docker/overlay2/b3d89b8f3be768a040338eae39ca78d0ae2453df289ad80bac9665b4d761ea61/merged/sys/fs/cgroup/
     7:20171214:074845.481 [systemd] cpu_cgroup is cpuacct
     7:20171214:074845.481 [systemd] set timeout to 3000ms
     7:20171214:074845.481 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_systemd.so": /var/lib/zabbix/modules/zabbix_module_systemd.so: undefined symbol: zbx_module_history_write_cbs
     7:20171214:074845.481 loading module "/var/lib/zabbix/modules/zabbix_module_sockets.so"
     7:20171214:074845.481 [sockets] starting v3.4.2
     7:20171214:074845.481 [sockets] set timeout to 3000ms
     7:20171214:074845.481 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_sockets.so": /var/lib/zabbix/modules/zabbix_module_sockets.so: undefined symbol: zbx_module_history_write_cbs
     7:20171214:074845.481 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so, zabbix_module_systemd.so, zabbix_module_sockets.so
     7:20171214:074845.481 End of zbx_load_modules():SUCCEED
     7:20171214:074845.481 In init_collector_data()
     7:20171214:074845.481 In zbx_dshm_create() size:0
     7:20171214:074845.481 End of zbx_dshm_create():SUCCEED shmid:-1
     7:20171214:074845.481 End of init_collector_data()
     7:20171214:074845.481 agent #0 started [main process]
    12:20171214:074845.485 agent #3 started [listener #2]
    11:20171214:074845.485 agent #2 started [listener #1]
    12:20171214:074845.485 In zbx_tls_init_child()
    15:20171214:074845.485 agent #5 started [active checks #1]
    11:20171214:074845.485 In zbx_tls_init_child()
    14:20171214:074845.486 agent #4 started [listener #3]
    15:20171214:074845.486 In zbx_tls_init_child()
    14:20171214:074845.486 In zbx_tls_init_child()
    11:20171214:074845.491 OpenSSL library (version OpenSSL 1.0.2g  1 Mar 2016) initialized
    10:20171214:074845.503 agent #1 started [collector]
    12:20171214:074845.503 OpenSSL library (version OpenSSL 1.0.2g  1 Mar 2016) initialized
    10:20171214:074845.503 In init_cpu_collector()
    11:20171214:074845.504 End of zbx_tls_init_child()
    15:20171214:074845.504 OpenSSL library (version OpenSSL 1.0.2g  1 Mar 2016) initialized
    12:20171214:074845.504 End of zbx_tls_init_child()
    10:20171214:074845.504 End of init_cpu_collector():SUCCEED
    11:20171214:074845.504 __zbx_zbx_setproctitle() title:'listener #1 [waiting for connection]'
    14:20171214:074845.504 OpenSSL library (version OpenSSL 1.0.2g  1 Mar 2016) initialized
    15:2017121

the above logs are from debug level 5

also please find the logs From normal logging

14:20171214:074414.510 Cannot open metric file: '/rootfs/var/lib/docker/overlay2/c889f774dc895802e1ae039183bbd1d65228868e480e1a23d4a53be89e71ede9/merged/sys/fs/cgroup/memory/1a732c7feb38d73087ab7b2d2f02ce3bcb148e8f2429bd1a6c0c45b1448a8357/memory.stat'`

docker info output

docker info
Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 17.03.2-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.43-17.38.amzn1.x86_64
Operating System: Amazon Linux AMI 2017.03
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 993.5 MiB
Name: jira02.sg.aws.grabpay.com
ID: L75C:365N:J3CD:RGXY:NOTU:WBAA:SOHM:O5ZT:B7PJ:NDLP:W7UX:YUGO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

OS we are using is AWS linux

further debugging output

[root@jira02 ~]# docker exec -ti dockbix-agent-xxl zabbix_get -s 127.0.0.1 -k docker.mem[/dockbix-agent-xxl,rss]
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "no such file or directory"`

help would be appreciated

Question: How to set AllowRoot

Hi
I'm useing the zabbix-agent-xxl monitor docker container.
I monitor the interface net metrics with the key: docker.xnet[/{#HCONTAINERID},all,TX-OK]. But can't get any values.
I want modify the config file zabbix_agent.cnf in the zabbix-agent-xxl container, But appear follow error:

docker ps -a

fbf1e9182507 monitoringartist/zabbix-agent-xxl-limited:latest "/zocker-xxl" 4 seconds ago Up 3 seconds 0.0.0.0:10050->10050/tcp zabbix-agent-xxl

docker exec -it fbf1e9182507 /bin/bash

rpc error: code = 2 desc = oci runtime error: exec failed: exec: "/bin/bash": stat /bin/bash: no such file or directory

How can I do set the AllowRoot=1

Can DockBix Agent work on AWS Fargate

With AWS Fargate there is no more concept of docker host... so if we need to monitor the containers ...is it possible to deploy dockbix agent as a task in the application container say for instance a simple Web Application to monitor the container metrics ? nothing much is available on AWS site or zabbix site on this .... need help on this ? or only way is to pump the metrics from aws cloudwatch to zabbix via the integration available on the zabbix site.

Has this been tested with Amazon ECS?

I cannot for the life of me get it to work...wondering if it's even possible?

Does this replace the zabbix agent running on the host?

Hi,

Great project, thanks for the effort. I assume the Zabbix agent running in the container cannot report metrics about the host system and therefor we need to keep the agent on the host system as well as the agent in the zabbix-agent-xxl container. If so how does Zabbix server deal with two agents from the same host? and is the containerized agent compatible with the new Zabbix agent/server encryption?

net.tcp.service is cannot resolve dns names

I try to monitor web pages with this agent and I can't make it work with DNS names. With IP addresses it works like a charm.

13:20170302:140616.538 Requested [net.tcp.service[http,www.google.com,80]] 13:20170302:140616.538 TCP expect network error: cannot resolve [www.google.com]

The container is on a custom network, other containers can resolve dns names on that network.

Any idea what cause this issue?

Open file handles prevent containers from being removed

I'm using dockbix-agent-xxl-limited on one of our CI systems for evaluation. While working nicely, I encountered an issue when containers are removed (either by hand or by CI):

Removing my_app_staging_db ... error

ERROR: for my_app_staging_db  Unable to remove filesystem for 4170b684a2bd193d6e5f73717d20fd4266afcf04ba0c5737d6649bc1e3365107: remove /var/lib/docker/containers/4170b684a2bd193d6e5f73717d20fd4266afcf04ba0c5737d6649bc1e3365107/shm: device or resource busy

Digging deeper revealed dockbix-agent-xxl as the blocking process:

$ grep -l 4170b684a2 /proc/*/mountinfo
/proc/14571/mountinfo
$ ps -f 14571
UID        PID  PPID  C STIME TTY      STAT   TIME CMD
root     14571 14556  0 Mai02 ?        Ssl    0:00 /dockbix-agent-xxl

After stopping the agent-container, removing other containers works again.

Is there something I can do about this?