ppy / osu-elastic-indexer Goto Github PK

The form of some of the queries currently being used won't work with ES7+ (especially the index_meta ones).
This metadata should probably be moved into the high_score indices themselves; the metadata queries can be replaced by listing and filtering for matching indices.

Running pppy/osu-elastic-indexer:2023.823.0 in production, we are observing CPU and RAM usage growing on every instance, until it hits a StackExchange.Redis.RedisTimeoutException and crashes. This happens about every 30 minutes.

The new indexer does not check this flag.

We can just check the current alias on ES instead. Should simplify things.

Not just the current schema used, but any versions being reindexed as well.

The use case for this is during a reindex, if a score with an id lower than the current indexing progress gets deleted, it will get missed since other processors pushing to the queue will still be pushing to the old queue.
This should let processors know there is more than one queue to push to.

Maybe also allow pushing just the id? So it's easier to push scores for reindexing. Mainly for user restriction/unrestriction and deleting scores on beatmapset state change (and testing).

Also the current json being pushed doesn't match the score data structure anymore so either the indexer need to deal with incomplete json structure or everything else pushing scores for indexing will need to sync their json as needed.

• Defaults should be in code for simplicity (one less level of redirection).
• Double underscores should not be required for envvar variables (currently is)
• SetEnvironmentVariable should not be used in the code to forward envvars to dependencies.

Needed for beatmap pack completion state check as it only counts plays on non-converted difficulties. Splitting search per ruleset is possible but that means one query per ruleset will be run for the check.

beatmapset_id would also be nice although not strictly needed.

For instance, queue will look like it is processing items successfully when it's actually doing nothing.

osu-queue-score-statistics will push to the queue externally. It will query osu-queue:score-index:{AppSettings.Prefix}schema to get the version.

This version is set after an all command completes. There may be a few scores missed between the time during which the all command is iterating scores and when the schema version is updated.

Easy solution is to do one final index operation after switching the schema version, to catch any new scores which may have been missed.