Similarity with snakemake? Domestic use? about ziggy HOT 2 CLOSED

I'm not familiar with snakemake or Apache airflow. After reading their introductions, they seem very similar to Jenkins to me: workflows for building things. That's where the similarity ends.

Ziggy is designed as a factory for processing data that’s coming in from a current mission. It handles large volumes of data on thousands of processors and a pipeline may take hours or days to run. Ziggy preserves records, data products, and provides data accountability across a long span of time. In contrast, the Apache airflow documentation said you could run your workflow hourly or daily. Although they can do extremely large analyses, they seem to be aimed at a much more informal setting, such as a laboratory or research group.

Ziggy also predates these tools by a decade, and is certified to process data for NASA missions. I don't see these tools on our groups accredited tool list.

That said, it would be worthwhile to describe similar tools in our README and more thoroughly answer your question. Thank you.

I didn't quite follow the second half of your question. We've released the Kepler pipeline to GitHub (github.com/nasa/kepler-pipeline) for study only as this version can't be compiled or run due to licensing issues. Note that this code also includes the bulk of the K2 work as well.

We are also applying Ziggy to the TESS pipeline as we speak. While the Ziggy version that does this will be pushed to GitHub, the TESS pipeline will not. Once we decide to make the TESS source public, in our experience it will take at least a year to authorize it for deployment to GitHub. So don't hold your breath for that one!

from ziggy.

Thank you for your detailed response. I overlooked the certification requirements that are inherent to NASA and the importance of ensuring data prosperity (something not tackled by the tools I mentioned).

Regarding my second question, I appreciate the clarification that released pipelines cannot be directly compiled. Since I am only interested in specific tasks (such as TPS) I will focus on separate implementations that can be run locally.

from ziggy.