Comments (16)
prymitive
commented on August 26, 2024
1
Host header got fixed, now it's time to fix basic auth as well - #256
from unsee.
prymitive
commented on August 26, 2024
POST https://unsee.mydomain.com/proxy/alertmanager/staging/api/v1/silences 404
That's not the correct path, see https://github.com/cloudflare/unsee/blob/master/proxy.go#L17, it's proxy/alertmanager/${name}, so the URI should be https://unsee.mydomain.com/proxy/alertmanager/test/api/v1/silences, not sure why you're seeing errors with staging instead of test. Likely you have more than one alertmanager server configured in unsee, can you post the config part for staging if that's correct?
time="2018-04-12T15:50:02Z" level=debug msg="[test] Proxy request for /api/v1/silences" that seems to suggest that requests for test alertmanager are proxied.
Do you have a single instance of unsee or do you have multiple each for a different alertmanager server?
from unsee.
svenmueller
commented on August 26, 2024
Hi,
Sorry, that was only a copy/paste error in this ticket. I replaced "staging" with test in my example but forgot the part you've mentioned. i adjusted it now in my question above.
- yes, we have multiple Alertmanager defined in the configuration, since all Alertmanager are behind Nginx with Basic Auth and therefore username/password needs to be part of the url
- we are only running a single unsee instance
Thx Sven
The complete configuration looks like this (credentials/domains replaced by random strings)
alertmanager:
interval: 60s
servers:
- name: test
uri: https://username:[email protected]
timeout: 30s
proxy: true
- name: production-1
uri: https://username:[email protected]
timeout: 30s
proxy: true
- name: production-2
uri: https://username:[email protected]
timeout: 30s
proxy: true
- name: misc
uri: https://username:[email protected]
timeout: 30s
proxy: true
- name: hosting-2
uri: https://username:[email protected]
timeout: 30s
proxy: true
- name: hosting-1
uri: https://username:[email protected]
timeout: 30s
proxy: true
annotations:
default:
hidden: false
hidden: []
visible: []
debug: true
filters:
default: []
labels:
color:
static:
- job
unique:
- cluster
- instance
- "@receiver"
keep: []
strip: []
listen:
address: "0.0.0.0"
port: 8080
prefix: /
log:
config: true
level: debug
jira:
- regex: OPS-[0-9]+
uri: https://jira.example.com
receivers:
keep: []
strip: []
from unsee.
prymitive
commented on August 26, 2024
I'm having trouble reproducing this with my setup. You have log:debug so you should see:
[GIN-debug] POST /proxy/alertmanager/test/api/v1/silences --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (5 handlers)
[GIN-debug] DELETE /proxy/alertmanager/test/api/v1/silence/*id --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (5 handlers)
in logs on startup, can you verify if that's there?
Also what version of alertmanager are you using with test instance? Can you try to:
curl -XPOST https://username:[email protected]/api/v1/silences
?
time="2018-04-12T15:50:02Z" level=debug msg="[test] Proxy request for /api/v1/silences" seems to suggest that request is send upstream, it's just that alertmanager1.mydomain.com/api/v1/silences returns 404 for some reason
from unsee.
svenmueller
commented on August 26, 2024
- we are running Alertmanager version
0.9.1 - method
POSTreturns an error
{"status":"error","errorType":"bad_data","error":"EOF"}
- method
GETworks
{"status":"success","data":[...]}
- logged events during start
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
time="2018-04-13T19:33:17Z" level=info msg="reqCnt registered."
time="2018-04-13T19:33:17Z" level=info msg="reqDur registered."
time="2018-04-13T19:33:17Z" level=info msg="reqSz registered."
time="2018-04-13T19:33:17Z" level=info msg="resSz registered."
[GIN-debug] GET /metrics --> github.com/cloudflare/unsee/vendor/github.com/mcuadros/go-gin-prometheus.prometheusHandler.func1 (2 handlers)
[GIN-debug] GET /debug/pprof/ --> github.com/cloudflare/unsee/vendor/github.com/DeanThompson/ginpprof.IndexHandler.func1 (2 handlers)
[GIN-debug] GET /debug/pprof/heap --> github.com/cloudflare/unsee/vendor/github.com/DeanThompson/ginpprof.HeapHandler.func1 (2 handlers)
[GIN-debug] GET /debug/pprof/goroutine --> github.com/cloudflare/unsee/vendor/github.com/DeanThompson/ginpprof.GoroutineHandler.func1 (2 handlers)
[GIN-debug] GET /debug/pprof/block --> github.com/cloudflare/unsee/vendor/github.com/DeanThompson/ginpprof.BlockHandler.func1 (2 handlers)
[GIN-debug] GET /debug/pprof/threadcreate --> github.com/cloudflare/unsee/vendor/github.com/DeanThompson/ginpprof.ThreadCreateHandler.func1 (2 handlers)
[GIN-debug] GET /debug/pprof/cmdline --> github.com/cloudflare/unsee/vendor/github.com/DeanThompson/ginpprof.CmdlineHandler.func1 (2 handlers)
[GIN-debug] GET /debug/pprof/profile --> github.com/cloudflare/unsee/vendor/github.com/DeanThompson/ginpprof.ProfileHandler.func1 (2 handlers)
[GIN-debug] GET /debug/pprof/symbol --> github.com/cloudflare/unsee/vendor/github.com/DeanThompson/ginpprof.SymbolHandler.func1 (2 handlers)
[GIN-debug] POST /debug/pprof/symbol --> github.com/cloudflare/unsee/vendor/github.com/DeanThompson/ginpprof.SymbolHandler.func1 (2 handlers)
[GIN-debug] GET /debug/pprof/trace --> github.com/cloudflare/unsee/vendor/github.com/DeanThompson/ginpprof.TraceHandler.func1 (2 handlers)
[GIN-debug] GET /debug/pprof/mutex --> github.com/cloudflare/unsee/vendor/github.com/DeanThompson/ginpprof.MutexHandler.func1 (2 handlers)
[GIN-debug] GET /favicon.ico --> main.favicon (4 handlers)
[GIN-debug] GET / --> main.index (4 handlers)
[GIN-debug] GET /help --> main.help (4 handlers)
[GIN-debug] GET /alerts.json --> main.alerts (4 handlers)
[GIN-debug] GET /autocomplete.json --> main.autocomplete (4 handlers)
[GIN-debug] POST /proxy/alertmanager/production-1/api/v1/silences --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
[GIN-debug] DELETE /proxy/alertmanager/production-1/api/v1/silence/*id --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
[GIN-debug] POST /proxy/alertmanager/production-2/api/v1/silences --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
[GIN-debug] DELETE /proxy/alertmanager/production-2/api/v1/silence/*id --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
[GIN-debug] POST /proxy/alertmanager/misc/api/v1/silences --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
[GIN-debug] DELETE /proxy/alertmanager/misc/api/v1/silence/*id --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
[GIN-debug] POST /proxy/alertmanager/hosting-1/api/v1/silences --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
[GIN-debug] DELETE /proxy/alertmanager/hosting-1/api/v1/silence/*id --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
[GIN-debug] POST /proxy/alertmanager/hosting-2/api/v1/silences --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
[GIN-debug] DELETE /proxy/alertmanager/hosting-2/api/v1/silence/*id --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
[GIN-debug] POST /proxy/alertmanager/test/api/v1/silences --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
[GIN-debug] DELETE /proxy/alertmanager/test/api/v1/silence/*id --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
time="2018-04-13T19:33:17Z" level=info msg="Listening on 0.0.0.0:8080"
[GIN-debug] Listening and serving HTTP on 0.0.0.0:8080
from unsee.
svenmueller
commented on August 26, 2024
Hi,
I just tried updating Alertmanager to version 0.14.0, but still we see the same result.
-- Sven
from unsee.
svenmueller
commented on August 26, 2024
In front of each alertmanager service, we have a nginx as reverse proxy running (for Basic Auth). All components are deployed in a k8s cluster.
Here is the used nginx config:
geo $google_lb {
default 0;
10.0.0.0/8 1;
}
map "$google_lb:$http_x_forwarded_for" $lb {
default 0;
"1:" 1;
}
server {
listen 80;
server_name alertmanager1.mydomain.com;
if ( $http_x_forwarded_proto = "http" ) {
return 301 https://$host$request_uri;
}
location /health {
if ( $lb ) {
rewrite ^/(.*) /\#/status break ;
proxy_pass http://alertmanager1;
}
}
location / {
auth_basic "Restricted Content";
auth_basic_user_file /etc/nginx/auth/htpasswd;
proxy_pass http://alertmanager1;
}
}
from unsee.
svenmueller
commented on August 26, 2024
Using the Alertmanager API directly works:
curl https://username:[email protected]/api/v1/silences \
-X POST --data @- << EOF
{
"matchers": [
...
],
"startsAt": "2018-04-14T10:35:41.436Z",
"endsAt": "2018-04-14T11:35:41.722Z",
"createdBy": "...",
"comment": "test"
}
EOF
{"status":"success","data":{"silenceId":"a09f4706-6129-425c-b773-f071e126bf42"}}
Does the proxy maybe have issues when the username/password is part of the proxied url?
-- Sven
from unsee.
svenmueller
commented on August 26, 2024
when i try the exact same request from unsee via curl (copied from firefox console):
curl -v 'https://unsee.mydomain.com/proxy/alertmanager/test/api/v1/silences' -H 'Accept: application/json, text/javascript, */*; q=0.01' --compressed -H 'Accept-Language: de,en-US;q=0.7,en;q=0.3' -H 'Connection: keep-alive' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Cookie: optimizelySegments=...; optimizelyEndUserId=...; optimizelyBuckets=...; _ga=...; amplitude_idmydomain.com=...; intercom-lou-fnb5rw5o=...; tk_tc=...; GCP_IAAP_XSRF_NONCE=...; GCP_IAAP_AUTH_TOKEN=...; _gid=...' -H 'Host: unsee.mydomain.com' -H 'Referer: https://unsee.mydomain.com/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0' -H 'X-Requested-With: XMLHttpRequest' --data ''
* Trying xxx.xxx.xxx.xxx...
* TCP_NODELAY set
* Connected to unsee.mydomain.com (xxx.xxx.xxx.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.mydomain.com
* start date: Oct 19 00:00:00 2017 GMT
* expire date: Dec 18 23:59:59 2019 GMT
* subjectAltName: host "unsee.mydomain.com" matched cert's "*.mydomain.com"
* issuer: C=US; O=GeoTrust Inc.; CN=RapidSSL SHA256 CA - G2
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7feb06006000)
> POST /proxy/alertmanager/staging/api/v1/silences HTTP/2
> Host: unsee.mydomain.com
> Accept-Encoding: deflate, gzip
> Accept: application/json, text/javascript, */*; q=0.01
> Accept-Language: de,en-US;q=0.7,en;q=0.3
> Connection: keep-alive
> Content-Type: application/x-www-form-urlencoded; charset=UTF-8
> Cookie: optimizelySegments=...; optimizelyEndUserId=...; optimizelyBuckets=%7B%7D; _ga=...; amplitude_idmydomain.com=eyJkZXZpY2VJZCI6Ijc5Y2Q2NTM0LTkyMDQtNGQzMy05MDRiLTRhZDRkYjljMThhNlIiLCJ1c2VySWQiOiJmNWMwMTIwNy0wNjQ4LTQ5NDgtOWI1Ny1kNmYyMmViZWZlZGQiLCJvcHRPdXQiOmZhbHNlLCJzZXNzaW9uSWQiOjE0NzkxMzU2Mjc2NzEsImxhc3RFdmVudFRpbWUiOjE0NzkxMzg2MTkxMTUsImV2ZW50SWQiOjI5NSwiaWRlbnRpZnlJZCI6ODMsInNlcXVlbmNlTnVtYmVyIjozNzh9; intercom-lou-fnb5rw5o=1; tk_tc=1523532035454; GCP_IAAP_XSRF_NONCE=...; GCP_IAAP_AUTH_TOKEN=...; _gid=...
> Referer: https://unsee.mydomain.com/
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0
> X-Requested-With: XMLHttpRequest
> Content-Length: 0
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404
< alt-svc: clear
< alt-svc: clear
< content-encoding: gzip
< content-type: text/plain; charset=utf-8
< date: Sat, 14 Apr 2018 10:50:15 GMT
< vary: Accept-Encoding
< content-length: 45
< via: 1.1 google, 1.1 google
* HTTP error before end of send, stop sending
<
* Connection #0 to host unsee.mydomain.com left intact
default backend - 404
The unsee service is running behind Google IAP (https://cloud.google.com/iap/docs/?hl=en), looks like the XMR request from the client fails for some reasons i don't understand yet.
from unsee.
svenmueller
commented on August 26, 2024
Ok, i turned off Google IAP in front off unsee service but still the same issue ("default backend - 404"). Must be something related to the request done from the client (browser) and google ingress/loadbalancer.
from unsee.
svenmueller
commented on August 26, 2024
@prymitive Ok, i think i found an issue. Looks like the unsee proxy thingy somehow sets a wrong hostname when proxying the requests.
Example:
curl -v 'https://unsee.mydomain.com/proxy/alertmanager/test/api/v1/silences' -H 'Accept: application/json, text/javascript, */*; q=0.01' --compressed -H 'Accept-Language: de,en-US;q=0.7,en;q=0.3' -H 'Connection: keep-alive' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Cookie: optimizelySegments=...; optimizelyEndUserId=...; optimizelyBuckets=...; _ga=...; amplitude_idmydomain.com=...; intercom-lou-fnb5rw5o=...; tk_tc=...; GCP_IAAP_XSRF_NONCE=...; GCP_IAAP_AUTH_TOKEN=...; _gid=...' -H 'Host: unsee.mydomain.com' -H 'Referer: https://unsee.mydomain.com/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0' -H 'X-Requested-With: XMLHttpRequest' --data ''
When i look into the logs of the target Alertmanager loadbalancer, i can see that a log like this is created:
xxx.xxx.xxx.xxx - "POST https://unsee.mydomain.com/api/v1/silences" 404 81 "https://unsee.mydomain.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0"
The domain is wrong! It should be something like this:
xxx.xxx.xxx.xxx - "POST https://alertmanager1.mydomain.com/api/v1/silences" 200 81 "https://unsee.mydomain.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0"
Any idea how this happens?
from unsee.
prymitive
commented on August 26, 2024
Nice find!
https://github.com/cloudflare/unsee/blob/master/proxy.go#L33
I need to look into this part of code and see what's happening there, not sure if I'll have time for this today but I'll get back to you soon. It seems like a simple bug with request headers.
from unsee.
svenmueller
commented on August 26, 2024
I guess removing the host header from the proxied request could solve the issue: https://github.com/cloudflare/unsee/blob/master/proxy.go#L37
from unsee.
prymitive
commented on August 26, 2024
#256 is merged, can you try the cloudflare/unsee:latest docker image and verify if everything works for you now?
from unsee.
svenmueller
commented on August 26, 2024
@prymitive i can confirm that it is working for us now 👍
from unsee.
prymitive
commented on August 26, 2024
Tagged 0.9.2 with those fixes - https://github.com/cloudflare/unsee/releases/tag/v0.9.2
Thanks!
from unsee.
Related Issues (20)
- Silence on proxied alertmanagers give a 404 HOT 4
- Feature: HTML safe annotations
- Feature: Create silences HOT 3
- Feature: Take username from http header HOT 3
- using nginx HOT 2
- Upgrade moment.js
- Silence comments only appear after a while HOT 4
- How to add Alert manager name for each instance HOT 2
- WEB_PREFIX ENV not working HOT 2
- gin request metrics contain arbitrary URLs, resulting in high cardinality
- Credentials in URLs are not masked HOT 3
- Panic in semantic version parsing HOT 9
- Silences POST fails with 307 Temporary Redirect due to double-slash in URL HOT 2
- Support alertmanager 0.15.x HOT 5
- Add tls certs to the "curl" example command for silencing
- TLS works for getting alerts, but fails for sending silences? HOT 1
- Basic auth credentials in stdout HOT 1
- Hide top bar and optionally replace with customizable headline HOT 2
- Incompatibility with alertmanager 0.16.x HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from unsee.