Giter Club home page Giter Club logo

event-hub-kafka-poc's Introduction

event-hub-kafka-poc

POC of Producer/Consumer implementation for Azure EventHub Kafka interface.

Build instructions

This project is build using SBT:

sbt assembly

The output jar will be located in dist/event-hub-kafka-poc.jar

Usage

java -jar dist/event-hub-kafka-poc.jar --help

EventHub consumer/producer using Kafka interface
  -e, --eventhub  <arg>    EventHub name
  -i, --input  <arg>       Input data file
  -m, --mode  <arg>        Consume or produce messages to the desired EventHub
                           Choices: consumer, producer
  -n, --namespace  <arg>   EventHub namespace name
  -o, --output  <arg>      Output data file
  -s, --sas  <arg>         SAS connection string between ""
  -h, --help               Show help message

- Producer example: --mode=producer --namespace=ns-test --eventhub=eh-test --sas="Endpoint=sb://ns-test.servicebus.windows.net/;SharedAccessKeyName=sas-keys;SharedAccessKey=xxx;EntityPath=eh-test" --input=/path/to/data/1gb.json
- Consumer example: --mode=consumer --namespace=ns-test --eventhub=eh-test --sas="Endpoint=sb://ns-test.servicebus.windows.net/;SharedAccessKeyName=sas-keys;SharedAccessKey=xxx;EntityPath=eh-test" --output=/path/to/data/out.json

Getting started: EventHub Throughput POC

  1. Create a sample of data using the utils/json-generator util:
  • Resolve util dependencies: npm i utils/json-generator
  • Build data sample running: node utils/json-generator/main.js -r 409600
  • Display util options: node utils/json-generator/main.js help
Usage: main.js [options] [command]

Commands:
  help     Display help

Options:
  -h, --help         Output usage information
  -o, --out [value]  Output file (defaults to "event-hub-kafka-poc/data/data.json")
  -r, --records <n>  Amount of records (defaults to 409600; 409600 == 1GB)
  1. Assembly application jar:
sbt assembly  

The output jar will be located in dist/event-hub-kafka-poc.jar

  1. Create an EventHub with manage shared access policies at the Azure developers console:
  • Standard EventHub:
    • 1 TU
    • Enable Auto-Inflate
    • Auto-Inflate Maximum Throughput Units: 20
    • Partition count: 32
  • Dedicated EventHub:
    • 1 CU
    • Partition count: 600
  1. Run application:
export NAMESPACE="namespace-name"
export EVENTHUB="eventhub-name"
export SASK_KEY="eventhub-shared-connection-string"

java -jar /opt/event-hub-kafka-poc/event-hub-kafka-poc.jar \
  --mode=producer \
  --namespace=${NAMESPACE} \
  --eventhub=${EVENTHUB} \
  --sas="${SASK_KEY}" \
  --input=event-hub-kafka-poc/data/data.json
  1. Check the EventHub namespace throughput metrics

event-hub-kafka-poc's People

Contributors

edrevo avatar pradomota avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.