Club tags are available through TMIO leaderboards. We should include this data anywhere we're including a username / user_id.

When scraping TMX and TMIO, we are currently grabbing data exclusively relating to the current TOTD. It would be nice to grab information about the previous day's TOTD (to refresh our leaderboards and world records).

This should only require alterations to the collect_tmio_enhancements DAG. It is currently hardcoded to pull the top 10 leaderboard entries for the current TOTD. I would like to:

• Make the lookup configurable with a top N entries
• Use the top 2 map_uid values (rather than top 1)

To accomplish both:

• Modify the get_totd_map_uid task to use LIMIT {f-string'ed variable} rather than LIMIT 1. Make the variable default to 2 so the DAG doesn't error out if it isn't set. This will also enable (manual) full refreshes.
• Rewrite scrape_leaderboards_today to be iterative over multiple days, still pushing to one XCom object
• Refactor push_leaderboards_to_postgres to UNNEST() the JSON it pulls from XCom

Haven't been able to test consumption layer objects because TMX is down tonight. Need to do this and merge resource-names back into main.

This raises an interesting conversation about fault-tolerance. I've opted for the TMX and OpenPlanet APIs to avoid dealing with Nadeo's API. Not having data 100% of the time is the price you pay for a much faster development cycle and a more stable API. We rely on both TMX and TMIO, and our AI model will require both to accurately make predictions. We don't want to re-train our model with bad or partial data.

In other words, if TMX or OpenPlanet is down, we don't want to do anything, so fault tolerance is not a problem on the data ingestion front.

We're going to want a variety of tables in the Consume layer. This is what I've come up with off the top of my head:

• TOTD
  track name, exchange_id, authors, totd date, upload date, days from upload to totd, medal times
• TOTD_Tags
  exchange_id, tag name (repeated for multiple tags on one map)
• TOTD_WorldRecords
  exchange_id, wr_time, wr_player, date driven
• TOTD_Authors
  exchange_id, author user_id, all tags on all of author's maps

• Create DAG for ingesting TMIO data into Collection layer
• Create DAG for cleaning TMIO data and promoting to Conform
• Create DAG for ingesting TMX data into Collection layer
• Create DAG for cleaning TMX data and promoting to Conform
• Create master orchestration DAG to trigger all aforementioned DAGs sequentially
• Pay down technical debt and use volumes to persist data properly
• Perform one-time load of TMX tags data into Conform layer XREF table
• Create DAGs for loading additional TMX data such as multiple authors, leaderboards, etc
• Create DAGs for loading TMIO leaderboards data to supplement TMX data
• #2
• Revisit technical debt for persistent volumes; create volumes independently from docker-compose stack
• Perform historical load of past TMIO and TMX data
• Train AI models using views on data from Consume, select one, and bake it into the Airflow environment
• Create DAG for running AI model with data from Consume and publishing predictions to Application layer
• Create DAG for demoting old predictions from the Application layer down to an archive in Consume
• Build FastAPI route /predictions/{date} to expose predictions from the Application layer
• Build FastAPI route /totd/{date} to expose actual / historical data from the Consume layer