Remove chunks storage support about mimir HOT 16 CLOSED

I have started removing chunks of storage in the ingester code path.

All things that should be addressed as part of this work, but not in a particular PR I have marked with TODO: (remove-chunks)

I think those are the next next tasks I could foresee:

• #510
• #524
• #525
• #526
• #527
• #528 (not strictly related to removing chunks storage support)
• #529
• Update helm chart to avoid setting any chunks-storage fields

from mimir.

@simonswine should the list above include anything about the purger, which I believe is only relevant for chunks?

from mimir.

Beware that purger exposes API for deleting tenants when using blocks storage.

from mimir.

So does that mean the purger can't be removed @pstibrany ?

I was going off of this docs page:
https://cortexmetrics.io/docs/guides/deleting-series/
which says deletion is only supported for chunks and that deletion involves the use of the "purger" service.

from mimir.

Ah I just re-read your comment...
so are you saying that the purger can't be used in blocks storage for targeted deletion of time series, but it can be used to delete an entire tenant?
When you say "delete an entire tenant", I assume you mean all blocks that belong to that tenant?

from mimir.

so are you saying that the purger can't be used in blocks storage for targeted deletion of time series, but it can be used to delete an entire tenant?

Correct.

When you say "delete an entire tenant", I assume you mean all blocks that belong to that tenant?

Correct.

from mimir.

@pstibrany so re: the purger, everything can be removed except https://cortexmetrics.io/docs/api/#tenant-delete-request and https://cortexmetrics.io/docs/api/#tenant-delete-status?

from mimir.

do you intend to keep these endpoints around now that there is the ability to configure per-tenant retention? I see they are still marked as "experimental"

from mimir.

everything can be removed except https://cortexmetrics.io/docs/api/#tenant-delete-request and https://cortexmetrics.io/docs/api/#tenant-delete-status?

I think so.

do you intend to keep these endpoints around now that there is the ability to configure per-tenant retention?

Yes, we do.

from mimir.

Thanks @pracucci . I created #555 and added it to the list.

from mimir.

@simonswine would any of the existing issues under this epic cover removing limits.max_chunks_per_query ? I see this marked as a flag to remove in our spreadsheet with a pointer to this issue. Just want to make sure it doesn't get lost - I can always make another issue if needed.

It seems like if we did this, we would also want to set querier.max-fetched-chunks-per-query': '1500000, but only for the ruler (not for any other components). Seems like we would leave the default to 0 (i.e. unset), but selectively set this on the ruler. Then again, I'm not even sure how easy that is to do since you'd have to do it with the CLI flag. At minimum, seems like you couldn't do it if you were running as target=all.

from mimir.

PR to remove chunks purger: #743

from mimir.

PR to remove table manager logic: #744

from mimir.

PR to remove tombstones and cache gen support (used by series deletion supported only by chunks storage): #748

from mimir.

PR to remove chunks storage support from querier: #753

from mimir.

From now on I will keep track of remaining work in this comment, that I will edit over time.

• Remove chunks storage logic (PR)
  • -store.cardinality-limit
  • Remove from flusher
• Move purger package (PR)
• Second storage engine support (PR)
  • t.Cfg.Querier.SecondStoreEngine
• Remove storage engine config (PR)
  • storage.StorageEngineChunks
• Remove unused config from flusher (PR)
• Remove unused metrics from pkg/ingester/metrics.go (PR)
• Remove unused chunk encodings (PR)
• Revisit all config to find more stuff to delete
• Update all CLI flags mentioning "blocks storage" (PR)
• Remove unused event logger (PR)
• Update -ingester.stream-chunks-when-using-blocks CLI flag description (PR)
• Revisit all packages to find more stuff to delete
• Look for code to remove in: (PR)
  • ingester/locker.go
  •  wal.proto
  • labelPairs
  • extract.go
  • hash_fp.go
  • SampleStreamIterator
  • matchers.go
  • priority_queue.go
  • sync.go
• Remove "TransferChunks" (PR)
• Remove prom1 package (PR)
• Cleanup ingester_v2.go
  1. Rename user_state.go and move userTSDB to dedicated file (PR)
  2. Merge ingester.go and ingester_v2.go (PR)
  3. #830
  4. #836
• Remove FIFO + Redis + Tiered cache support from query-frontend (PR)
  • Discussing about dropping fifo and redis support
• #794
• Remove final references to "chunks storage" across the whole repo (PR)
• pkg/chunk/encoding (PR)
• #811
• #812
  • https://github.com/grafana/deployment_tools/pull/23370
• Remove storage chunks clients #527
• Ensure all limits left are in use

from mimir.