Data pipelines based on dbt-core
project structure
Data Stack:
- DBT
- Snowflake (as target Cloud DW)
- Fivetran
- Quickly installation (preferably in a virtual environment):
pip install -r requirements.txt
dbt --version
- Configure profile.yml with Snowflake connection details
- Run dbt commands:
dbt deps
dbt seed
# run and test all models
dbt build
- Goal: Data startup analysys
- Sources (previously ingested via Fivetran):
yc_startups.json
: startups that have received funding from Y Combinator up to the year 2024unicorns.csv
: Startups Valued at $1 Billion or More
- Overall Architecture: