rewe-digital / cortex-gateway Goto Github PK

It seems that one or two of our distributor pods are getting an uneven amount of requests.

Hi,
i have a question about the jwt. How i set in prometheus this token? Could you help me about it?

Tkz

I tried to build the project, got the following error

github.com/weaveworks/common/logging
../../weaveworks/common/logging/logging.go:25:16: cannot use hook (type *promrus.PrometheusHook) as type "github.com/sirupsen/logrus".Hook in argument to "github.com/sirupsen/logrus".AddHook:
   *promrus.PrometheusHook does not implement "github.com/sirupsen/logrus".Hook (wrong type for Fire method)
   	have Fire(*"github.com/weaveworks/promrus/vendor/github.com/sirupsen/logrus".Entry) error
   	want Fire(*"github.com/sirupsen/logrus".Entry) error

Hey team, great project here!

I have an interesting use case that I think this gateway could solve with new functionality on the audience part of the token.

Some context:

I run a Cortex cluster as a HA and long-term metric solution. But my multi-tenancy set-up is different to most; I instead run a fixed cluster of Prometheus & Cortex components, and my 'tenants' are actually specified in metric labels. (As an example, my http_request_duration_seconds metric would be divided into series by a client label, being sourced from multiple applications. (e.g. application a serves metrics for client alpha and client bravo. And application b serves metrics for client charlie)

Proposal

My proposal to help with this albeit obscure usecase is to utilise the Audience claim as a label key-value map to enforce a set of rules around read and write paths.

Read Path
I foresee the most popular use for this feature being in the read path. Being able to narrow the query result set by enforcing customisable labels. An example token may look like:

{
  "tenant_id": "organization",
  "aud": {
    "client": "alpha"
  }
}

The gateway is then responsible for parsing the incoming query, parsing the labels, and enforcing the instance="customer-alpha" label on all incoming queries. A project that exists already in a similar fashion to this is prom-label-proxy however labels are configured on startup of the service and cannot be dynamically processed.

Write Path
The write path will likely not benefit from this feature, as the existing concept of relabelling from the Remote Write API exists. There might be a use case for setting an audience label map on writing, but I cannot identify it currently

Note: I am mostly looking for feedback and input to validate whether this would be a valid contribution, and gauge whether this use case exists in the wild, before forking and looking at working on a feature here.

I have provided the bearer_token in scrap_configs but still getting no bearer token provided. Here is my prometheus config.
On port 8071 cortex-gateway is running.

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
  bearer_token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.mUhhZNrgCmfEdk2wXmyNjFjOipw-0ks5X7FLfr4QuPY

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

  static_configs:
    - targets: ['localhost:9090']
remote_write:
  - url: http://localhost:8071/api/prom/push

Getting the following error
12:28:16 http: proxy error: unsupported protocol scheme ""
when I configured it with prometheus remote_write
Here is the prometheus cofig

remote_write:
    - url: http://localhost:8071/api/prom/push
      bearer_token: eyJhbGciOiJ

I used this token in Prometheus config.
using tcpdump checked HTTP headers from prometheus to gateway:
User-Agent: Prometheus/2.10.0
Content-Length: 8805
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJPbmxpbmUgSldUIEJ1aWxkZXIiLCJpYXQiOjE1NzE4NDg3ODYsImV4cCI6MTYwMzM4NDc5OSwiYXVkIjoid3d3LmV4YW1wbGUuY29tIiwic3ViIjoianJvY2tldEBleGFtcGxlLmNvbSIsIm5hbWUiOiJteV9wcm9tb18wMSJ9.09gkgEpCu9ryGCRc2DolJDNh5gVY7HG8otAIt0XmdFU
but still getting "Invalid bearer token" in response. Signing key is provided to gateway. What is wrong with y JWT? What gateway is expecting in JWT payload? Is there any debug log flag which will provide more information from gateway?

Hi,

Thank you for writing this simple and powerful piece of software.
My problem
Currently, the cortex-gw replaces the X-Scope-OrgID without checking if it is already set in the header. Here is the code that is doing it link
I am sending the correct dynamically configured X-Scope-OrgID and do not want it to be replaced after the jwt token verification.

Can you please suggest on how I can prevent the override without adding any other proxy route ?

Is it possible to check for a preset X-Scope-OrgID?