kafka_load_generator can be used to generate stub data of cpu log of multiple nodes in a cluster on a per second basis and send the data to a kafka broker. Learn more about kafka here.

The data generated does not represent actual cpu usage, but generates data which has some continuous sequences of 9 and 10 seconds of cpu usage of a single node above 80%. The data generated will contain cpu usage from 1 to 100.

• Some data will be <= 80, which is assumed to be normal data.
• Some data will be > 80, which is considered anomalous.

For testing purposes, all the sequences will not be of 10 seconds. Few will be of 9 seconds.

• If the first second modulo 20 is 0, then that sequence will have 10 seconds of anomalous data.
• Otherwise(i.e. first second modulo 20 is 10), the sequence will have 9 seconds.

Anomalous data will be generated in sequences of 9 secs or 10 secs, which might be consecutive as well. The gap between two sequences is random.

json-data-generator is used here to generate the json and publish them to a kafka broker. kafka_load_generator can generate stub cpu logs for multiple nodes of the same cluster. If scaling up the number of nodes creates performance issue on one machine, multiple instances of this generator can be started on various machines after configuring the clusterConfig.json's firstNode and lastNode.

If you want to generate data for 2000 nodes belonging to a same cluster, but the machine you are using can generate only upto 1000 points per second before creating performance issues, two machines can be used to achieve the same. The clusterConfig files will look like the following. Notice the firstNode and lastNode:

{
  "topic": "cpu-usage",

  "broker": {
    "server": "127.0.0.1",
    "port": 9092
  },

  "cluster": {
    "clusterName": "firstCluster",
    "firstNode": 1,
    "lastNode": 1000
  }
}

{
  "topic": "cpu-usage",

  "broker": {
    "server": "127.0.0.1",
    "port": 9092
  },

  "cluster": {
    "clusterName": "firstCluster",
    "firstNode": 1001,
    "lastNode": 2000
  }
}

clusterConfig is the main configuration file which contains:

• topic - Name of the Kafka topic to publish to.
• Kafka broker's details:

• Details of the cluster

1. Edit clusterConfig.json according to the details mentioned above.
2. Run python3 configGenerator.py in terminal. If there are no errors, it will generate two types of json files inside json-data-generator/conf.

• cpuUsageConfig_clusterName.json
• n instances of cpuUsageWorkflow_clusterName_nodeNumber.json where n is (lastNode-firstNode+1) and nodeNumber ranges from firstNode to lastNode.

3. Run java -jar json-data-generator-1.2.2-SNAPSHOT/json-data-generator-1.2.2-SNAPSHOT.jar cpuUsageConfig_*clusterName*.json

Note: Kafka broker should be running before executing the jar.

1. Python3
2. Java8

• Script to take broker and cluster details as arguments, and start producer.
• Containerize kafka_load_generator.