DS Dict about sagerx HOT 8 CLOSED

Right now all dependencies are imported through sagerx.py and user_defined_macros...which should allow us to move things outside of dynamic dag pretty easily.

Another consideration here using JSON files to store this data. JSON would allow us to define functions in it to pass to dataset_dicts if needed as one offs. There are pros and cons (I think mostly cons) but just an alternative

https://stackoverflow.com/questions/51936785/store-python-function-in-json/51938459

from sagerx.

Could also consider moving dynamic DAG into its own folder with settings files for each DS also stored in same folder. This way moving them out of dynamic DAG will be easy

from sagerx.

Sorry totally missed these comments earlier. Im still somewhat torn about the best way to handle this. The JSON alt makes sense but I do think the cons outway the pros if we are using eval and functions as strings. One of the comments kinda points out that this is dangerous. Im not sure how risky it actually is for our use case from a security perspective, but everywhere else I have read you want to be extremely cautious about this.

from sagerx.

Alternative consider making independent DAGs for each data source. Then using either functional or object paradigm make a generate tasks code block that can be placed as a single line of code to generate all tasks for each DAG. This will module and abstract the DAG functions, organize everything.

Looking at the datasets we currently have there is benefit on a few (CMS, RxNorm) to adjust our process and grab historical files. So I feel like in the short future we will be pulling alot of these out of the dynamic DAG anyways, so my thought is how do we do that and still keep the code maintainable and able to share a common set of processes.

This is pretty complex and might be above my software engineering skills at the moment.

from sagerx.

As example the Download_dataset should be a class in its own module. With subclasses and methods that can be adjusted from a dataset info class.

Then a SQL class should be formed in its own module

Then a DS class should call both of these as part of composition

from sagerx.

@Bridg109 / @yevgenybulochnik - does this issue represent the huge topic of re-structuring the dynamic DAG into like functional programming or a class system? What would be a better name for this issue than "DS Dict" to capture that? Feel free to rename it.

from sagerx.

See notes about RxNorm and NADAC on this PR when we tackle this issue: #158

from sagerx.

Fixed by #215

from sagerx.