This project contains all the code needed to export slack messages (events) to BigQuery. It's made up of two core pipelines:
- Batch import of historical slack export as JSON files
- ingest bucket
- load function reacting to bucket events
- Streaming import based on the Slack Events API
- Ingest function
- ingest pubsub topic
- load function
All infrastructure is defined as TF code in infra. Unit tests are in the test_...
files of each component.
Taken inspration from this video