reddit-etl

Description

A data pipeline for extracting and visualizing basic activity in r/ukraine Reddit community.

(details here)

Download docker-compose.yaml bash curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.3.2/docker-compose.yaml'
Add the required libraries to docker-compose.yaml
```
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:- praw pandas pygsheet}
```
This is a development/test feature only. This should never be used in production( build a custom image as described in the docs), but it would be enough for our project.
In the project/airflow directory run
```
mkdir -p ./dags ./logs ./plugins
```
Run the next command to create .env file next to docker-compose.yaml:
```
echo -e "AIRFLOW_UID=$(id -u)" > .env
```
Initialize the database to run database migrations and create the first user account
```
docker-compose up airflow-init
```
The created account has the login airflow and the password airflow
Now start the Airflow
```
docker-compose up
```
The Airflow web server is available at http://localhost:8080

First, you need an active Reddit account. If you don't have one, create
Go to reddit.com/prefs/apps. If the previous link doesn't work for you, try old.reddit.com/prefs/apps/
Select create another app. Make sure you select the Script option
Fill in the description and optional fields. Click create app
Next, you will see your client id and secret. These values will be needed in the next step
Create pipeline.conf file in the dags/extraction directory with the credentials from the previous step
```
[reddit]
client_id = your_script_client_id
client_secret = your_script_client_secret
```