Comments (11)
One way I could think of is to inject a label with the Docker Swarm Node ID on each metric in node-exporter. This way, even if the node-exporter changes IP you have the same ID, since this is generated at Swarm init. I'm trying to port dockprom to swarm this week and I'll post on the blog if I manage to run it properly.
from dockprom.
@Hermain if you use the dns for node-exporter than you'll not be able to link the source to the hosts. every time you'll restart node-export it will get another IP, how will you match the source to the real host name?
from dockprom.
Hello @xeor
When running the Prometheus stack on Docker Swarm you need to decide if you want to use the same Prometheus database for all Swarm nodes.
If that's the case, then you need to do the following:
- change the monitor network to overlay
- run Prometheus and Grafana services as a single replica and pin them to a single node (since Prometheus and Grafana need to persist data on the host you can't allow Swarm to redistribute them to a random node)
- run Alertmanager service as a single replica (since this service is stateless, it doesn't matter if Swarm moves it around, but you need to embed the Alertmanager config in the container)
- run NodeExporter and cAdvisor services as global services (Swarm will set one replica of each service on each node)
The challenge with this setup is that Swarm will load balance the Prometheus scraping calls to NodeExporter and cAdvisor. So you can't use the host name of these 2 services in the Prometheus config.
Docker Swarm doesn't have a Prometheus discovery integration like Kubernetes has. I know that Docker plans to do this, but right now it's not available.
A workaround would be to use the Prometheus dns_sd_config
for service discovery, or you could deploy a dedicated NodeExporter and cAdvisor service for each Swarm node and use the node constrain to prevent Swarm from moving your services between nodes.
I plan to extend dockprom to Docker Swarm in the future.
from dockprom.
Thanks for a very good answer, it's all logical now :)
Looking forward to a future update..! Keep it up!
from dockprom.
I had to change a few of the queries on the container grafana dashboard to do sums, but now it's working pretty decently. I'm still trying to figure out how to get container (or service) names to show up instead of the generated id's, but wonder if that might be a cAdvisor/newest swarm issue. Thanks for your tips.
from dockprom.
Swarm Mode +1
from dockprom.
"A workaround would be to use the Prometheus dns_sd_config for service discovery"
Could you please explain what speaks against this workaround? It seems to solve the problem of the swarm loadbalancing the requests.
from dockprom.
@stefanprodan I only wanted to understand why you call it a workaround and not a solution. Now I understand. The problem is, you can't link the node-exporter instance to a host. Thank you for the explanation.
from dockprom.
Sounds promising!
from dockprom.
IMHO, we could split Swarm Version to another project. At this point, I assume this project is for a single host monitoring
from dockprom.
pinging in to see where this might go... also interested.
from dockprom.
Related Issues (20)
- Adding my own dashboards HOT 1
- Removed
- Alerting for any docker container HOT 3
- How to clean all data without remove all dashboard from grafana
- Expose cAdvisor web interface in caddy HOT 1
- Caddy generated password error
- Grafana v9.1.1 (9e60a1bff1) show no preconfigured dashboards by default HOT 2
- invalid interpolation format for services.caddy.environment.[]: "$2a$14$1l.IozJ...". You may need to escape any $ with another $. HOT 8
- container disk usage so high HOT 3
- How to have Docker Container dashboard show the containers used for monitoring? HOT 1
- Can't login to prometheus HOT 1
- Adding a log scraper
- Duplicate container names HOT 1
- Where is the dockprom_network configuration? HOT 1
- Help please HOT 2
- Only 7 days of logging. HOT 2
- Server temperatures and fan speeds. HOT 1
- docker-compose up -d 报错:ERROR HOT 1
- understaing the meaning of cpu usage graph in grafana HOT 2
- Storage Load N/A HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dockprom.