Giter Club home page Giter Club logo

kafka-spark-streaming-hive-example's Introduction

kafka-spark-streaming-hive-example

Execution

./bin/spark-submit \
--packages org.apache.spark:spark-streaming-kafka_2.10:1.6.3 \
--jars /usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar, \
/usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar, \
/usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar \
kafkaConsumer.py master:2181 test

Input Log Format

{"data": {"CH_22": 93, "CH_39": 5, "CH_48": 81, "CH_67": 33, "CH_84": 15, "CH_98": 87,
"CH_68": 72, "CH_70": 97, "CH_17": 62}, "time": 1491808854}
{"data": {"CH_8": 49, "CH_48": 66, "CH_90": 88, "CH_53": 82, "CH_50": 94, "CH_74": 66,
"CH_75": 31, "CH_69": 76, "CH_71": 45, "CH_14": 29}, "time": 1491808855}

Ambari Dashboard Hive View 

SELECT key, SUM(value) FROM word_count WHERE time BETWEEN '2017-04-10 16:19:00' AND
 '2017-04-10 16:20:00' GROUP BY key;

Result

kafka-spark-streaming-hive-example's People

Contributors

jinhan avatar

Stargazers

avatar

Watchers

avatar avatar

Forkers

mainak24 newfan tuhinsharma121 igorgorbenko yzopen ahkhaniki

kafka-spark-streaming-hive-example's Issues

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.