Comments (13)
Hmmm, I guess support for multiple instances will need to be added to fix #37, so I think it's reasonable to just allow passing multiple instances, update those concurrently and tag alerts with alertmanager instance name + add a filter for it.
For HA we should probably de-duplicate alerts, so maybe alerts should be tagged with list of instances reporting same alert.
Should be the next big feature to work on, I'll start working on it once I have some time. Thanks!
from unsee.
Now that everyone mentioned it I really want to have that too ;)
I also need to do some refactoring around the code that fetches data from Alertmanager API, so that's a good opportunity to do that.
from unsee.
I looked into it a bit and even got some code written, things that need to be changed:
- configuration syntax - need to support multiple URIs, should allow naming AM instances, if possible configuring timeouts, having a config file starts to be a good idea
- silence and alert ids/fingerprints need to contain AM instance
- silence API endpoint URI (used by silence form) needs to be instance aware, probably best if each alert was tagged with
instane: [am1, am2]
, name -> uri mapping provided in the json reponse and form would send silences to all instance where alert is spotted (or even allow use to select instances where it should be silenced, probably the best) - internal metrics will need to have another dimension with the instance name
- upstream errors are now per instance, not global
- error handling needs to be refactored - it's no longer binary, one instance might be down but other up, so instead of fullscreen error we only need a top bar warning if it's only partial issue (fullscreen only if all am instances are down)
- alert source button needs to be per instance, this probably means that there should be a single button/label that will trigger a modal with list of instances and details (like link to source per instance).
All of that on top of the obvious changes, so a bit of work and will probably take me a week or two.
from unsee.
Got some code in multi-upstream branch, it's working (as in should collect alerts from multiple Alertmanagers and deduplicate them), but it's not done, some features are missing, need to write new tests and clean up that branch. But it's usable for testing, ALERTMANAGER_URI now accepts multiple values separated by spaces, format is "name:uri name2:uri2", e.g. ALERTMANAGER_URI="prod:http://localhost staging:http://foo.bar"
It should be mergeable in a week or so, there's UI work needed which will require a bit of time, I don't want to overload it with yet another label, so will add some other UI elements to indicate which Alertmanager instance it was collected from (color bar on the side or something equally non-intrusive, but that must still allow filtering actions).
from unsee.
Similar use case here: multiple alertmanagers on different environments and we would like to have a single dashboard where we could see, at a glance, all the alerts that have triggered in all the environments.
It would be amazing to be able to use unsee for this job.
from unsee.
Wanted to second OP and vincepii — we also need data to be sourced from multiple alertmanagers.
from unsee.
Thanks for working on this! I'd like to have it too
But it's usable for testing, ALERTMANAGER_URI now accepts multiple values separated by spaces, format is "name:uri name2:uri2", e.g. ALERTMANAGER_URI="prod:http://localhost staging:http://foo.bar"
Random nitpick: I'd use comma-separated list of values for alertmanager_uri, it would match what prometheus expects and it is easier to quote e.g. in the shell :D
from unsee.
All multi-value env options are space separated, so we would have to change everything for consistency. I'd rather keep it as is right now.
from unsee.
@prymitive ah, if space separated is already the rule then yes 👍
from unsee.
What's on multi-upstream is almost feature complete, a filter support is missing and some UI tweaks are needed, but it should be usable. Long weekend might allow me to finish it, or maybe will make me lazy, no promises.
from unsee.
I think I'm done and all pieces are in place, PR awaits a review now.
from unsee.
PR merged, this will be in the next release, which will be tagged after some testing (and likely some fixes), a week or two will likely be needed.
You can use it today via docker using cloudflare/unsee:latest
image.
from unsee.
unsee 0.7.0
was jut tagged and it includes those changes, see https://github.com/cloudflare/unsee/releases/tag/v0.7.0
from unsee.
Related Issues (20)
- Silence on proxied alertmanagers give a 404 HOT 4
- Feature: HTML safe annotations
- Feature: Create silences HOT 3
- Feature: Take username from http header HOT 3
- using nginx HOT 2
- Upgrade moment.js
- Silence comments only appear after a while HOT 4
- How to add Alert manager name for each instance HOT 2
- [0.9.1] Silence on Proxied Alertmanagers returns 404 HOT 16
- WEB_PREFIX ENV not working HOT 2
- gin request metrics contain arbitrary URLs, resulting in high cardinality
- Credentials in URLs are not masked HOT 3
- Panic in semantic version parsing HOT 9
- Silences POST fails with 307 Temporary Redirect due to double-slash in URL HOT 2
- Support alertmanager 0.15.x HOT 5
- Add tls certs to the "curl" example command for silencing
- TLS works for getting alerts, but fails for sending silences? HOT 1
- Basic auth credentials in stdout HOT 1
- Hide top bar and optionally replace with customizable headline HOT 2
- Incompatibility with alertmanager 0.16.x HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from unsee.