Giter Club home page Giter Club logo

rapid-onboarding-exemplar's Introduction

dbt Training - Rapid Onboarding Exemplar

This is a dbt project for dbt Lab's Rapid Onboarding training.

Our analytics stack:

  • Loader: Snowflake's TPCH sample data
  • Warehouse: Snowflake
  • Transformation: dbt

Permissions

Access to the Snowflake warehouse is managed on a per-user basis by the training team. If you need access, open a request in the #training-rapid-onboarding Slack channel.

Using This Project

Developing in the Cloud IDE

The easiest way to contribute to this project is by developing in dbt Cloud. If you need access, contact the training team in the #training-rapid-onboarding Slack channel.

Once you have access, navigate to the develop tab in the menu and fill out any required information to get connected.

In the command line bar at the bottom of the interface, run the following commands one at a time:

  • dbt deps - installs any packages defined in the packages.yml file.
  • dbt seed - builds any .csv files as tables in the warehouse. These are located in the data folder of the project.
  • dbt run - builds the models found in the project into your dev schema in the warehouse.
Local Development

  1. Install Requirements

    Install dbt.
    Optionally, you can set up venv to allow for environment switching.

  2. Setup

    Open your terminal and navigate to your profiles.yml. This is in the .dbt hidden folder on your computer, located in your home directory.

    On macOS, you can open the file from your terminal similar to this (which is using the Atom text editor to open the file):

    $ atom ~/.dbt/profiles.yml

    Insert the following into your profiles.yml file and change out the bracketed lines with your own information. Here is further documentation for setting up your profile.

    my_project:                                          
     target: dev                                         
     outputs:                 
       dev:                                              
         type: [warehouse name]                                 
         threads: 8                                      
         account: [abc12345.us-west-1]                   
         user: [your_username]                           
         password: [your_password]                       
         role: transformer                               
         database: analytics                             
         warehouse: transforming                         
         schema: dbt_[your_name]                         
    Configuration Key Definition
    my_project This is defining a profile - this specific name should be the profile that is referenced in our dbt_project.yml
    target: dev This is the default environment that will be used during our runs.
    outputs: This is a prompt to start defining targets and their configurations. You likely won't need more than dev, but this and any other targets you define can be used to accomplish certain functionalities throughout dbt.
    dev: This is defining a target named dev.
    type: [warehouse_name] This is the type of target connection we are using, based on our warehouse.
    threads: 8 This is the amount of concurrent models that can run against our warehouse, for this user, at one time when conducting a dbt run
    account: [abc12345.us-west-1] Change this out to the warehouse's account.
    user: [your_username] Change this to use your own username that you use to log in to the warehouse
    password: [your_password] Change this to use your own password for the warehouse
    role: transformer This is the role that has the correct permissions for working in this project.
    database: analytics This is the database name where our models will build
    schema: dbt_[your_name] Change this to a custom name. Follow the convention dbt_[first initial][last_name]. This is the schema that models will build into / test from when conducting runs locally.
  3. Running dbt

    Run the following commands one at a time from your command line:

    • dbt debug - tests your connection. If this fails, check your profiles.yml.
    • dbt deps - installs any packages defined in the packages.yml file.
    • dbt seed - builds any .csv files as tables in the warehouse. These are located in the data folder of the project.
    • dbt run - builds the models found in the project into your dev schema in the warehouse.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.