"The specified bucket does not exist" but all buckets are present in minio about mimir HOT 9 OPEN

This may be because there is a minio section in the values file. If you're setting up your minio seperately then at the top-level you could counter-intuitively set

minio:
  enabled: false

One way you could verify if this is the interaction you're experiencing is by looking at the config that is actually being generated with:

kubectl describe configmap/mimir-config

For instance, if you see mimir-tsdb in the config and not your mimir-blocks, then that's the issue.

from mimir.

Thanks for your reply, @andyasp!

Yes, I tried to use embedded minio, and it works fine. It seems like some policies are not set properly. Is there a guide to set up the junction of mimir and minio?

from mimir.

It seems like some policies are not set properly.

I'm not sure what this means, but it sounds like maybe you figured it out?

Is there a guide to set up the junction of mimir and minio?

Not that I'm aware of. There's the object storage configuration documentation, but it's not specifically about minio. From Mimir's perspective it is only aware of the fact that it is talking to an s3 backend. There's nothing special about minio in this sense. As long as it knows the buckets and how to talk to it/how to auth to it that's basically it.

The helm chart has that embedded minio included by default so it can work out of the box, but it's suggested to override it for production use cases. You can see an artifact of that in some of the sample value files, like here.

from mimir.

@andyasp . I just found a root cause: that was a confusing error message.

my settings are:

mimir:
  structuredConfig:
    common:
      storage:
        s3:
          insecure: true
          http:
            insecure_skip_verify: true
          endpoint: minio.minio-tenant-mimir.svc.cluster.local
          ...

key point here is insecure: true. This endpoint receives only HTTPS. For some reason, I got the error message

blocks storage: unable to successfully send a request to object storage: The specified bucket does not exist.

instead of

blocks storage: unable to successfully send a request to object storage: HTTP request made to HTTPS server

I removed insecure: true, and everything started to work fine.

Is it a bug?

from mimir.

Interesting. I don't think that's a bug with the sanity check. It's rather simple and creates the bucket client and GETs a single object (code reference). Here the error it's getting from minio has the code NoSuchBucket and it's reflecting that back.

If there was a simple reproduction procedure (like with curl or minio-go directly) to verify that swapping HTTP/HTTPS is all that's going on then filing an issue with minio may help improve that.

from mimir.

blocks storage: unable to successfully send a request to object storage: The specified bucket does not exist.

I think this is a "non-ideal" behaviour in mimio-go (the s3 API client, we use through the objstore package): they treat all responses with 404 as "bucket not exists" (ref the code in mimio-go).

from mimir.

they treat all responses with 404 as "bucket not exists"

That isn't true and would be very incorrect behavior. Requesting an object that doesn't exist results in 404 and a NoSuchKey code. The code you linked shows that in the else case. That's what objstore looks for too: objstore ref

from mimir.

they treat all responses with 404 as "bucket not exists"

That isn't true and would be very incorrect behavior. Requesting an object that doesn't exist results in 404 and a NoSuchKey code. The code you linked shows that in the else case

What I wanted to say is that a request to an object, needs to locate the object's bucket, first, by calling to S3's GetBucketLocation API (ref to this chunk in minio-go).

I suspect, the error reported in the issue came from such request, as it returned 404 due to wrong protocol.

from mimir.

Ah, gotcha.

from mimir.