This project demonstrates the usage of dagster-pipes
from an R runtime environment. It supports handling of context
extracts
, as well as logging events back to Dagster. It showcases a simple pipeline that loads in NYC vehicle collision data and generates a plot of annual collision per borough.
Extras can be specified as a parameter to the pipes_subprocess_client
run
method.
return pipes_subprocess_client.run(
command=cmd,
context=context,
extras={
"vehicle_collisions_url": "https://data.cityofnewyork.us/api/views/h9gi-nx95/rows.csv",
"vehicle_collisions_cache_path": "data/nyc-collisions.csv",
"plot_output_path": "nyc-vehicle-collisions-by-borough.png",
},
).get_materialize_result()
They can then be accessed from the context in R like so:
context <- load_context(params$context_params)
A logger can be used to send events back to the Dagster instance.
logger <- PipesLogger$new(params)
logger$open()
logger$info(sprintf("saving plot to: %s", plot_output_path))
![image](https://private-user-images.githubusercontent.com/5807118/336609824-52667fa0-2c02-4200-b422-8ff37b98f9d4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE0NDYxOTIsIm5iZiI6MTcyMTQ0NTg5MiwicGF0aCI6Ii81ODA3MTE4LzMzNjYwOTgyNC01MjY2N2ZhMC0yYzAyLTQyMDAtYjQyMi04ZmYzN2I5OGY5ZDQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcyMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MjBUMDMyNDUyWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZDVmYzk2YmE3YmU0MjM5MGIzNTkwNzQ3OGMzZjg2NTMwNzI5MzAzMzI5MDQwOWNmM2FiNWQ0ZDg0ODFhOTYwZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.vHmDsxyiSC3WUZwkjHo1dEPa9YfHSiJC3yJiQ-3g_SQ)
pip install -e ".[dev]"
Then, start the Dagster UI web server:
dagster dev
Open http://localhost:3000 with your browser to see the project.
This is the resulting plot that is generated from the pipeline.