Giter Club home page Giter Club logo

spring-cloud-dataflow's Introduction

Spring Cloud Data Flow Build Status Stories Ready Stores In Progress

The Spring Cloud Data Flow project provides orchestration for data microservices, including long lived stream applications and short lived task applications.

Components

The Core domain module includes the concept of a stream that is a composition of spring-cloud-stream modules in a linear pipeline from a source to a sink, optionally including processor module(s) in between. The domain also includes the concept of a task, which may be any process that does not run indefinitely, including Spring Batch jobs.

The Module Registry maintains the set of available modules, and their mappings to Maven coordinates.

The Module Deployer SPI provides the abstraction layer for deploying the modules of a given stream across a variety of runtime environments. Current SPI implementations support Local deployment, YARN, Cloud Foundry, Mesos and Kubernetes.

The Data Flow Server is a Spring Boot application that provides a common REST API, UI, and an implementation of the module deployer SPI interface for the runtime environment. For each runtime environment there is a different version of the Data Flow server. The github locations for these Data Flow servers are:

The Shell connects to the Admin’s REST API and supports a DSL that simplifies the process of defining a stream and managing its lifecycle.

Instructions to run the Admin Server for each runtime environments can be found in their respective github repositories.

Contributing

We love contributions. Follow this link for more information on how to contribute.

Building

Clone the repo and type

$ ./mvnw clean install

If you are running behind a proxy, then you need to specify the proxy settings via environment properties like this: (choose the ones that are appropriate for your settings)

export MAVEN_LOCAL_REPOSITORY=mylocalMavenRepo
export MAVEN_REMOTE_REPOSITORIES=repo1,repo2
export MAVEN_OFFLINE=true
export MAVEN_PROXY_PROTOCOL=https
export MAVEN_PROXY_HOST=host1
export MAVEN_PROXY_PORT=8090
export MAVEN_PROXY_NON_PROXY_HOSTS='host2|host3'
export MAVEN_PROXY_AUTH_USERNAME=user1
export MAVEN_PROXY_AUTH_PASSWORD=passwd

For more information on building, see this link.

spring-cloud-dataflow's People

Contributors

markfisher avatar ilayaperumalg avatar ericbottard avatar markpollack avatar pperalta avatar trisberg avatar jvalkeal avatar mbogoevici avatar mminella avatar artembilan avatar garyrussell avatar ghillert avatar twoseat avatar joshlong avatar sabbyanandan avatar alexvengrovsk avatar chochreiner avatar jeanbza avatar mross-pivotal avatar mwea avatar

Watchers

James Cloos avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.