Giter Club home page Giter Club logo

S4 - Stream Computing Platform

For more information, see s4.io

Requirements

  • Linux
  • Java 1.6

Build Instructions

  • Build using Gradle

    • USAGE: gradlew TASK1, TASK2, ...

    • Main Tasks:

      • clean: deletes all the build dirs
      • jar: creates all the jars in PROJECT/build/libs
      • binImage: creates an image in the ./build subdir that includes jars, scripts, and other resources required to run the S4 server and sample applications.
      • binTgz: same in a gzipped tar file in ./build/distributions.
      • allImage: creates an image in the ./build subdir that includes what is in binImage plus sources, javadoc, and other documents.
      • allTgz: same in a gzipped tar file in ./build/distributions.
      • install: installs jars and POMs in local Maven repo (eg. ~/.m2)

The S4 Image Structure

build/s4-image/
            bin/
            s4-ext/
            s4-example-apps/
            s4-apps/
                    app-name1/
                              lib/
                                  (app-name1-*.jar)
                              app-name1-conf.xml
            s4-core/      
                    lib/
                    conf/
                    lock/
                    logs

            s4-driver/lib/
                           driver jar
                           dependencies jars
                           java examples jar
                      bin/
                           python, perl, and shell scripts

Running the Twitter Topic Count Example

#  Download the project files from the repository.
git clone https://github.com/s4/s4.git

# Create image
./gradlew allImage

# set the S4_IMAGE environmental variable
cd build/s4-image/
export S4_IMAGE=`pwd`

# get the sample application
git clone git://github.com/s4/twittertopiccount.git

# build the sample application
./gradlew install

# deploy the sample application into the S4 image (relies in the S4_IMAGE environmental variable)
./gradlew deploy

# set the TWIT_LISTENER environmental variable
cd build/install/twitter_feed_listener
export TWIT_LISTENER=`pwd`

# Start server with twittertopiccount app
$S4_IMAGE/scripts/start-s4.sh -r client-adapter &

# start the client adapter
$S4_IMAGE/scripts/run-client-adapter.sh -s client-adapter -g s4 -d $S4_IMAGE/s4-core/conf/default/client-stub-conf.xml &

# run a client to send events into the S4 cluster. Replace *your-twitter-user* and *your-twitter-password* with your Twitter userid and password.
$TWIT_LISTENER/bin/twitter_feed_listener *your-twitter-user* *your-twitter-password* &

# Check output
cat /tmp/top_n_hashtags

Developing with Eclipse

The command gradle eclipse will create an eclipse project that you can import from the Eclipse IDE.

There is now a Gradle plugin for the Eclipse IDE. To install Gradle without installing the full Spring development environment follow the instructions under the heading "Installing from update site". There is also a discussion in the Gradle mailing list.

The S4 Project's Projects

core icon core

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data.

sandbox icon sandbox

Sandbox repository for experimenting with git

zeno icon zeno

Dynamic load balancing using Zookeeper

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.