Comments (4)
Hi @JenspederM, thanks for flagging this issue. Can I ask what your use case is for printing the result of find_pipelines()
?
This method has been added to enable auto discovery of pipelines and does some stuff in the back to make sure your project and its modules are discoverable (https://docs.kedro.org/en/stable/nodes_and_pipelines/pipeline_registry.html). It's meant to run as part of a "regular" Kedro flow where it's preceded by certain project setup methods. You can fix your script by calling bootstrap_project()
before find_pipelines()
(https://docs.kedro.org/en/stable/kedro_project_setup/session.html#bootstrap-project-and-configure-project). However, I would only recommend doing that for exploration and not if you're planning to run that code in production.
Let me know if this makes sense!
from kedro.
Hi @merelcht,
Thank you for your reply.
I am using find_pipelines()
to generate databricks assets bundle resources. I am working on a template for asset bundles that uses Kedro for defining pipelines and dependencies and databricks workflows for scheduling. You can find the project here
Thanks for the suggesting bootstrap_project()
. For now, I have been using configure_project(<package-name>)
as used in databricks_run.py
in the databricks-iris
starter.
You can see my exact usage right here
from kedro.
I have been thinking of making a cookiecutter for Kedro as well. Do you think there would be any interest in this?
I made the template based on my own experience of running large scale Databricks projects in production with many contributors of varying levels of experience.
from kedro.
I'd say, regardless of use case, raising an UnboundLocalError
from internal code should not happen, but a more informative error instead.
I have been thinking of making a cookiecutter for Kedro as well. Do you think there would be any interest in this?
Of course! When you get to do it, we can promote it on https://github.com/kedro-org/awesome-kedro
Also consider exploring https://github.com/copier-org/copier/, a modern alternative to cookiecutter
from kedro.
Related Issues (20)
- ci: Nightly build failure on `develop` HOT 7
- Deprecate micropackaging HOT 10
- DataCatalog.shallow_copy() destroys any `CustomDataCatalog` type object HOT 1
- ci: Nightly build failure on `main` HOT 3
- Backfill some old documentation versions with static assets HOT 15
- Assess Kedro performance for complex pipelines
- Release 0.19.6
- Broken link in docs guidelines/standards
- ci: Nightly build failure on `main`
- Use `click` choice options to simplify validation of project creation workflow
- Implement version fallback for starters & framework
- Spike: Make `cookiecutter` optional / not a core dependency of `kedro` HOT 1
- Default to `raise_errors=True` in `find_pipelines` HOT 1
- Allow returning modified dataset in `after_dataset_loaded` and `before_dataset_saved` hook HOT 5
- ci: Nightly build failure on `main` HOT 2
- Investigate performance of config loading for big projects HOT 4
- Logs for packaged Kedro projects are confusing
- Move PySpark to end of Tools prompt HOT 1
- Monthly issue metrics report
- ci: Nightly build failure on `develop` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kedro.