salompas / credo Goto Github PK

Create a configuration file that has default specifications for connecting to certain databases and allows users to extend it to connect to others. For example, a YAML file with:

Redshift:
  - connection:
    - driver:
    - host:
    - port:
    - database:
    - timeout:
  - catalog:
    - name
    - load
    - save 

Project has to be simple enough that it could be built in a single day workshop.

Have a file that has global parameters that can also control the behavior of credo. For example, a parameter that changes the location where files are saved by default. Instead of using a folder in the current directory named data, the parameters would allow you to specify a path that will be prepended to the data folder.

Have a way to check new parameters files given a template parameters. For example, you only create one template where you explain what each parameter does, then have a function that creates a new parameters file from the template and checks existing files against the template.

Use roxygen comments to obtain a summary of what each node does, use that to explain what a pipeline does. Potentially, use GenAI to generate a compelling explanation. The idea is for a new user to quickly have an overview of each pipeline and automate documentation.

Have a way to set up the default parameters file to be used when working, this way you don't have to keep typing the parameters file name all the time.

Check if all parameters in the data catalog are as expected. At a minimum, each entry should have a type and filepath. If an entry doesn't have it, be very clear on the messaging that it needs to be updated.

credo::make runs the required nodes and pipelines to generate the requested data

Have a way of specifying a default parameters file for a pipeline or a hook, so that it doesn't have to be specified at run time. At run time you can always overwrite it.

Improved logging with indications for node, function, inputs, outputs, runtime.
For example:
(NODE)
(FUNCTION)
(READ)
(WRITE)
Have an option that also shows time stamps and time to run each node.

Option (or parameter) that will print out time to load and save data, and execute nodes, pipelines. Also generate a clean log for debugging and showing what are the largest time sinks.

Projects that use remote tables will have outputs with a specific schema and name for its tables. If another person wants to run the same project, they will overwrite the tables, which could be used in production. Have a parameter that allows a person to turn all remote table saves into temporary saves instead.

When debugging or building pipelines, it is often necessary to run many nodes, and then continue from there. You can store the output of the nodes that were ran in a context, but then you need to keep calling context$variable to use the values. Have a way of setting up the .GlobalEnv to be the context in such case. Either by using .GlobalEnv as the context itself, or having a utility function to offload the variables to that environment.

Automatically create connections if not available when connections$<connection_type> is referenced as an input.