allenai / pipeline Goto Github PK

Library for building reproducible data pipelines to support experimentation

Scala 93.02% HTML 6.24% Shell 0.74%

pipeline's Introduction

#Allen-AI Pipeline Framework

The Allen-AI Pipeline (AIP) framework is a library that facilitates collaborative on data-driven projects. It allows users to define workflows that share data resources transparently while maintaining complete freedom over the environment in which those workflows execute. AIP falls somewhere between unix make and KNIME. Unlike make, it can operate on cloud storage resources and execute in a distributed environment. Unlike KNIME, it does not lock you into any particular repository for storing your data or execution environment in which the workflows must run.

AIP can be used in two ways:

As PipeScript, a binary that interprets a simple scripting language to execute native commands locally and store the results remotely
Via the Scala API to define workflows that produce strongly-typed objects and execute within any JVM-based environment.

In summary, AIP provides the following benefits:

Intermediate data is cached and is sharable by different users on different systems.
A record of past runs is maintained, with navigable links to all inputs/output of the pipeline.
A pipeline can be visualized before running.
Output resource naming is managed to eliminate naming collisions.
Input/output data is always compatible with the code reading the data.

Send questions to [email protected]

pipeline's People

Contributors

Stargazers

Watchers

Forkers

rodneykinney dirkgr markschaake andrewlmurray aria42 oyvindtafjord codeaudit sbhaktha robmcdan canve atulny awesome-archive m8e

pipeline's Issues

Spark support (docs link is broken)

The docs are really nice. I wonder what kind of support for Spark might be helpful, however in ―

For distributed execution, support for running pipelines using Apache Spark can be found in the spark sub-project

the hyperlink seems to be broken. Was there something specific for Spark which is now removed?
Link to the quoted docs page - https://github.com/allenai/pipeline/blob/master/docs/ParallelExecution.md

Log beginning and end of each pipeline step

debug or info?

Log inside get.

Add timing information to pipeline diagrams

@kwhumphreys requested timing information to be added to the pipeline diagrams. This seems like an excellent addition--although it might not be easy to implement for the Spark components.

@rodneykinney are you OK if I try and implement this on Monday?

Make a PersistedProducer interface

In src/main/scala/org/allenai/pipeline/Producer.scala:

> @@ -125,7 +156,7 @@ trait CachingDisabled extends CachingEnabled {
>    override def cachingEnabled: Boolean = false
>  }
>  
> -class PersistedProducer[T, -A <: Artifact](
> +class PersistedProducer[T, -A <: Artifact] private[pipeline] (

The Producer.persist method returns a PersistedProducer not a simple Producer This really should be changed to return an interface instead of a concrete class. The only additional thing that PersistedProducer exposes is the Artifact that the data was saved to.

Move StreamClosingIterator to common

Show pipeline output dynamically

Presently the pipeline produces a static HTML file after a pipeline run completes.

We could decouple the HTML generation from the underlying pipeline graph data. This way we could use an external tool to view the pipeline graph data and that tool could update in real time. For example, it could highlight stages of the pipeline once they complete (if they have output data).

One wrinkle is that timing data would be hard to add to the pipeline data unless the pipeline updated it's pipeline graph data as it ran.

Document why "The most magical of the factory methods" is useful

https://github.com/allenai/pipeline/blob/io-refactor/src/main/scala/org/allenai/pipeline/PipelineStep.scala#L77

Dynamically highlighted links

It would be awesome to be able to highlight the links between nodes on a click or mouse-over of a link or node. E.g. http://ai2-s2.s3.amazonaws.com/pipeline/output/publish/summary/BuildIndexFromPdfs-2015-04-29-00-53-22.html.

At the same time, it would be nice to return to the smooth-spline link rendering instead of the sharp corners that came with the dagre upgrade.

sbt 0.13.9 support

Off a fresh clone: this is likely very minor, but tweaking build.properties to use sbt 0.13.9 rather than 0.13.8 seems to break over scalarifrom. One word of disclaimer is I've not locally set bintray credentials ― but I'll venture these are not related. Presumably this stems from the allenai sbt plugin toolchain and not specific to pipeline nor very exciting either, so, just a heads up.

[error] Scalariform parser error in file /repos/pipeline/src/main/scala/org/allenai/pipeline/ReplicateResource.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,594,<)
[error] Scalariform parser error in file /repos/pipeline/src/main/scala/org/allenai/pipeline/Pipeline.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,3320,<)
[error] Scalariform parser error in file /repos/pipeline/src/main/scala/org/allenai/pipeline/ReadFromArtifact.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,63,<)
[error] Scalariform parser error in file /repos/pipeline/src/main/scala/org/allenai/pipeline/PipelineStep.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,3736,<)
[error] Scalariform parser error in file /repos/pipeline/src/main/scala/org/allenai/pipeline/ArtifactFactory.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,449,<)
[error] Scalariform parser error in file /repos/pipeline/src/main/scala/org/allenai/pipeline/IoHelpers.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,304,<)
[error] Scalariform parser error in file /repos/pipeline/src/main/scala/org/allenai/pipeline/ColumnFormats.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,425,<)
[error] Scalariform parser error in file /repos/pipeline/src/main/scala/org/allenai/pipeline/Producer.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,5411,<)
[error] Scalariform parser error in file /repos/pipeline/src/main/scala/org/allenai/pipeline/SaveToArtifact.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,61,<)
[error] Scalariform parser error in file /repos/pipeline/src/main/scala/org/allenai/pipeline/ArtifactIo.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,223,<)
[error] Scalariform parser error in file /repos/pipeline/src/main/scala/org/allenai/pipeline/s3/CreateCoreArtifacts.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,454,<)
[error] /repos/pipeline/src/main/scala/org/allenai/pipeline/ReplicateResource.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,594,<)
[error] /repos/pipeline/src/main/scala/org/allenai/pipeline/Pipeline.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,3320,<)
[error] /repos/pipeline/src/main/scala/org/allenai/pipeline/ReadFromArtifact.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,63,<)
[error] /repos/pipeline/src/main/scala/org/allenai/pipeline/PipelineStep.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,3736,<)
[error] /repos/pipeline/src/main/scala/org/allenai/pipeline/ArtifactFactory.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,449,<)
[error] /repos/pipeline/src/main/scala/org/allenai/pipeline/IoHelpers.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,304,<)
[error] /repos/pipeline/src/main/scala/org/allenai/pipeline/ColumnFormats.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,425,<)
[error] /repos/pipeline/src/main/scala/org/allenai/pipeline/Producer.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,5411,<)
[error] /repos/pipeline/src/main/scala/org/allenai/pipeline/SaveToArtifact.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,61,<)
[error] /repos/pipeline/src/main/scala/org/allenai/pipeline/ArtifactIo.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,223,<)
[error] /repos/pipeline/src/main/scala/org/allenai/pipeline/s3/CreateCoreArtifacts.scala: Expected token RBRACKET but got Token(XML_START_OPEN,<,454,<)

allenai / pipeline Goto Github PK

pipeline's Introduction

pipeline's People

Contributors

Stargazers

Watchers

Forkers

pipeline's Issues

Spark support (docs link is broken)

Log beginning and end of each pipeline step

Add timing information to pipeline diagrams

Make a PersistedProducer interface

Move StreamClosingIterator to common

Show pipeline output dynamically

Document why "The most magical of the factory methods" is useful

Dynamically highlighted links

sbt 0.13.9 support

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent