To build a system which fetches Youtube video data for a particular subject periodically in the background, and an API to retrieve it. Keeping in mind the performance, scalability and fault tolerance too.
🌼 Celery
- Background job to fetch youtube videos periodically.
🥕 Redis
- Broker for celery
🌿 MongoDB
- The data we are storing is not very relational. We are also retrieving the whole document all at once, hence non-relational document storage seems right for this particular use case.
🚀 Flask
- Flask API to query the mongoDB for the youtube data
🐳 Docker / Docker compose
- Containerised each service into separate containers, allowing us to scale individual service independently.
✅ Background job to load youtube video data into mongoDB
✅ Implemented proper indexing
- Created indexes on videoId and publishTime.
- Hence good performance on the ranged queries using published_before, published_after dates.
- Text indexes on video title and description, hence faster filtering based on query through express dashboard.
✅ Background job to load youtube video data into mongoDB
- Celery beat to schedule periodic tasks.
✅ Scalable, optimised and fault tolerant
- Dockerized the whole system.
- Celery worker containers can be horizontally scaled to handle load.
- Also the Flask API service is horizontally scalable.
- In case if database goes down and comes back after sometime, the tasks will resume adding videos from the last execution date, so that no data is lost.
✅ Support for multiple API keys
- Use the API to add the keys, or you can manually add it through mongo express dashboard.
✅ Dashboard
- Mongo express container provides a dashboard to view all the collections.
- User can sort, query the videos fetched.
- Visit http://localhost:28081 to view the express dashboard.
Inside the repository, run these commands to launch the services
docker-compose build
docker-compose up
After all the containers are up,
use the postman collection to send the request to the /add_api_key api to add the youtube api key
POST http://localhost:8000/add_api_key
Content-Type: application/json
{
"key": "API_KEY_HERE"
}
Now the celery workers should start fetching the youtube videos.
Use /get to retrieve the data in descending order of the published date
By default page of size 5 is returned, but you can specify the page size upto 50 items.
For ranged queries, use published_after and published_before params with dates in ISO format.
GET http://localhost:8000/get
# With addtional params
# page, limit, published_after, published_before
GET http://localhost:8000/get?page=5&limit=10&published_after=2023-01-03T04:24:53Z&published_before=2023-01-03T05:19:52Z