nathan-k- / mqttbeat Goto Github PK

View Code? Open in Web Editor NEW

19.0 3.0 16.0 47.31 MB

Elasticsearch beat to save messages received with the MQTT protocol.

License: Other

Makefile 12.53% Go 79.27% Python 8.21%

mqtt elasticsearch elasticbeats beats broker

mqttbeat's Introduction

Mqttbeat

Welcome to Mqttbeat.

This beat will allow you to put MQTT messages in an elasticsearch instance.

Ensure that this folder is at the following location: ${GOPATH}/github.com/nathan-k-/mqttbeat

Support

Hello, this project is no longer under work. Feel free to fork it, adapt it to your need, and improve it ! Cheers,

Getting Started with Mqttbeat

Requirements

Golang 1.7 (1.8 recommended)

Init Project

To get running with Mqttbeat and also install the dependencies, run the following command, with glide installed:

glide install
pip install virtualenv (for the testsuite)
make setup

It will create a clean git history for each major step. Note that you can always rewrite the history if you wish before pushing your changes.

To push Mqttbeat in the git repository, run the following commands:

git remote set-url origin https://github.com/nathan-k-/mqttbeat
git push origin master

For further development, check out the beat developer guide.

Build

To build the binary for Mqttbeat run the command below. This will generate a binary in the same directory with the name mqttbeat.

make

Run

To run Mqttbeat with debugging output enabled, run:

./mqttbeat -c mqttbeat.yml -e -d "*"

Test

To test Mqttbeat, run the following command:

make testsuite

alternatively:

make unit-tests
make system-tests
make integration-tests
make coverage-report

The test coverage is reported in the folder ./build/coverage/

Update

Each beat has a template for the mapping in elasticsearch and a documentation for the fields which is automatically generated based on etc/fields.yml. To generate etc/mqttbeat.template.json and etc/mqttbeat.asciidoc

make update

Cleanup

To clean Mqttbeat source code, run the following commands:

make fmt
make simplify

To clean up the build directory and generated artifacts, run:

make clean

Clone

To clone Mqttbeat from the git repository, run the following commands:

mkdir -p ${GOPATH}/github.com/nathan-k-/
cd ${GOPATH}/github.com/nathan-k-/
git clone https://github.com/nathan-k-/mqttbeat

For further development, check out the beat developer guide.

Packaging

The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires docker and vendoring as described above. To build packages of your beat, run the following command:

make package

This will fetch and create all images required for the build process. The hole process to finish can take several minutes.

mqttbeat's People

Contributors

Stargazers

Watchers

Forkers

rainbowcolours pchevali masgari brightids plinde kanatsux oddsockmachine kreastr freek felix-lessoer tmechen bthom2 vincenzo-pinto jetli123

mqttbeat's Issues

Cannot build with latest beats

I get the following:

Updating generated files for mqttbeat
make[2]: Entering directory '/root/go/src/github.com/nathan-k-/mqttbeat/vendor/github.com/elastic/beats/libbeat'
make[2]: Leaving directory '/root/go/src/github.com/nathan-k-/mqttbeat/vendor/github.com/elastic/beats/libbeat'
# github.com/nathan-k-/mqttbeat/vendor/github.com/elastic/beats/libbeat/logp
vendor/github.com/elastic/beats/libbeat/logp/log.go:46:5: _log redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/core.go:22:2
vendor/github.com/elastic/beats/libbeat/logp/log.go:126:49: Debug redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:31:49
vendor/github.com/elastic/beats/libbeat/logp/log.go:130:49: MakeDebug redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:11:49
vendor/github.com/elastic/beats/libbeat/logp/log.go:136:31: IsDebug redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:25:31
vendor/github.com/elastic/beats/libbeat/logp/log.go:140:35: HasSelector redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:18:35
vendor/github.com/elastic/beats/libbeat/logp/log.go:151:31: Info redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:40:31
vendor/github.com/elastic/beats/libbeat/logp/log.go:155:31: Warn redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:49:31
vendor/github.com/elastic/beats/libbeat/logp/log.go:159:30: Err redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:58:30
vendor/github.com/elastic/beats/libbeat/logp/log.go:163:35: Critical redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:68:35
vendor/github.com/elastic/beats/libbeat/logp/log.go:186:30: WTF redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:79:30
vendor/github.com/elastic/beats/libbeat/logp/log.go:186:30: too many errors
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:290: recipe for target 'update' failed
make[1]: *** [update] Error 2
make[1]: Leaving directory '/root/go/src/github.com/nathan-k-/mqttbeat'
Makefile:18: recipe for target 'setup' failed
make: *** [setup] Error 2

make errors

I am trying to make, but get the following errors:

rclarke@es-rclarke:mqttbeat$ make
go build -i -ldflags "-X github.com/elastic/beats/libbeat/version.buildTime=2018-02-19T15:15:29Z -X github.com/elastic/beats/libbeat/version.commit=64e3bf19a45686a5cc6aecae58d9d2e140e7c915"
# github.com/nathan-k-/mqttbeat/vendor/github.com/elastic/beats/libbeat/logp
vendor/github.com/elastic/beats/libbeat/logp/log.go:46:5: _log redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/core.go:22:2
vendor/github.com/elastic/beats/libbeat/logp/log.go:125:49: Debug redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:31:49
vendor/github.com/elastic/beats/libbeat/logp/log.go:129:49: MakeDebug redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:11:49
vendor/github.com/elastic/beats/libbeat/logp/log.go:135:31: IsDebug redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:25:31
vendor/github.com/elastic/beats/libbeat/logp/log.go:139:35: HasSelector redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:18:35
vendor/github.com/elastic/beats/libbeat/logp/log.go:150:31: Info redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:40:31
vendor/github.com/elastic/beats/libbeat/logp/log.go:154:31: Warn redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:49:31
vendor/github.com/elastic/beats/libbeat/logp/log.go:158:30: Err redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:58:30
vendor/github.com/elastic/beats/libbeat/logp/log.go:162:35: Critical redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:68:35
vendor/github.com/elastic/beats/libbeat/logp/log.go:168:30: WTF redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:79:30
vendor/github.com/elastic/beats/libbeat/logp/log.go:168:30: too many errors
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:96: recipe for target 'mqttbeat' failed
make: *** [mqttbeat] Error 2

I have pulled the latest from github.com/elastic/beats before doing make setup / make

And go:

rclarke@es-rclarke:mqttbeat$ go version
go version go1.9.4 linux/amd64

Unable to build with go 1.8, libbeat 5.5.0

What version of beats did you build against? Perhaps something changed...

make[2]: Entering directory '/root/go/src/github.com/nathan-k-/mqttbeat/vendor/github.com/elastic/beats/libbeat'
cat _meta/fields.common.yml > _meta/fields.generated.yml
cat processors/*/_meta/fields.yml >> _meta/fields.generated.yml
make[2]: Leaving directory '/root/go/src/github.com/nathan-k-/mqttbeat/vendor/github.com/elastic/beats/libbeat'
# Update docs
. /root/go/src/github.com/nathan-k-/mqttbeat/build/python-env/bin/activate && python ./vendor/github.com/elastic/beats/libbeat/scripts/generate_fields_docs.py /root/go/src/github.com/nathan-k-/mqttbeat mqttbeat ./vendor/github.com/elastic/beats
# Generate index templates
. /root/go/src/github.com/nathan-k-/mqttbeat/build/python-env/bin/activate && python ./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py --es2x /root/go/src/github.com/nathan-k-/mqttbeat mqttbeat ./vendor/github.com/elastic/beats
Traceback (most recent call last):
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 381, in <module>
    fields_to_es_template(args, fields, output, args.beatname + "-*", version_data['version'])
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 110, in fields_to_es_template
    defaults, "")
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 180, in fill_section_properties
    prop, dynamic = fill_field_properties(args, field, defaults, path)
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 313, in fill_field_properties
    prop, dynamic = fill_section_properties(args, field, defaults, path)
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 180, in fill_section_properties
    prop, dynamic = fill_field_properties(args, field, defaults, path)
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 313, in fill_field_properties
    prop, dynamic = fill_section_properties(args, field, defaults, path)
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 180, in fill_section_properties
    prop, dynamic = fill_field_properties(args, field, defaults, path)
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 338, in fill_field_properties
    raise ValueError("Unknown type found: " + field.get("type"))
ValueError: Unknown type found: object

Error loading Elasticsearch template: could not load template: couldn't load template: couldn't load json

I am trying it using the latest 7.5.0 Elasticsearch, and I get the error like:
ERR Failed to connect: Connection marked as failed because the onConnect callback failed: Error loading Elasticsearch template: could not load template: couldn't load template: couldn't load json. Error: 400 Bad Request. Response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters: [default : {_meta={version=1.0.0}, dynamic_templates=[{fields={path_match=fields., mapping={type=keyword}, match_mapping_type=string}}, {docker.container.labels={path_match=docker.container.labels., mapping={type=keyword}, match_mapping_type=string}}, {fields={path_match=fields., mapping={type=keyword}, match_mapping_type=string}}, {docker.container.labels={path_match=docker.container.labels., mapping={type=keyword}, match_mapping_type=string}}, {fields={path_match=fields., mapping={type=keyword}, match_mapping_type=string}}, {docker.container.labels={path_match=docker.container.labels., mapping={type=keyword}, match_mapping_type=string}}, {strings_as_keyword={mapping={ignore_above=1024, type=keyword}, match_mapping_type=string}}], date_detection=false, properties={kubernetes={properties={container={properties={image={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}}}, pod={properties={name={ignore_above=1024, ty ...

Add client-id to config parameters

When switching to verneMQ (and using a stricter authentication), any MQTT connection requires to use a client-id known to the MQTT server.

There should be an option like "broker_clientid" to specify the client-id. If empty, it should generate a random ID (32 chars).

_timestamp field in mqtt message causes logstash ingestion failure

I have some mqtt messages coming in that already have a _timestamp field, and logstash complains about these.

2018/09/27 17:38:42.251161 mqttbeat.go:87: DBG MQTT MESSAGE RECEIVED {"_timestamp": 1538069922, "actual": 21.6, "target": 0.0}
2018/09/27 17:38:42.251298 mqttbeat.go:147: DBG Payload decoded - json
2018/09/27 17:38:42.251421 client.go:203: DBG Publish: {
  "@timestamp": "2018-09-27T17:38:42.251Z",
  "_timestamp": 1538069922,
  "actual": 21.6,
  "beat": {
    "index": "mqttbeat",
    "type": "message"

I don't know what the answer is since I can't change my incoming messages, but I at least wanted to log it.

Sanity check for topics_subscribe

Running with this config:

  topics_subscribe:
    - /events

produces this error:

rclarke@es-rclarke:mqttbeat$ mqttbeat -e -v -c ~/elastic/temp/mqttbeat.yml 
2018/02/20 11:01:55.252501 beat.go:470: INFO Home path: [/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat] Config path: [/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat] Data path: [/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/data] Logs path: [/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/logs]
2018/02/20 11:01:55.252532 beat.go:495: INFO Beat metadata path: /home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/data/meta.json
2018/02/20 11:01:55.252582 beat.go:477: INFO Beat UUID: 067ea9df-546b-4d58-8009-37205673334c
2018/02/20 11:01:55.252595 beat.go:239: INFO Setup Beat: mqttbeat; Version: 1.0.0
2018/02/20 11:01:55.252607 processor.go:49: DBG Processors: 
2018/02/20 11:01:55.252623 beat.go:250: DBG Initializing output plugins
2018/02/20 11:01:55.252690 metrics.go:23: INFO Metrics logging every 30s
2018/02/20 11:01:55.252759 client.go:123: INFO Elasticsearch url: https://*****:****
2018/02/20 11:01:55.252965 logger.go:18: DBG start pipeline event consumer
2018/02/20 11:01:55.252982 pipeline.go:96: INFO Publisher name: es-rclarke
2018/02/20 11:01:55.252991 publish.go:100: INFO Publisher name: es-rclarke
2018/02/20 11:01:55.253024 mqttbeat.go:34: INFO BROKER url: tcp://hassio:8883
2018/02/20 11:01:55.253033 mqttbeat.go:39: INFO BROKER username: hassio
2018/02/20 11:01:55.257520 mqttbeat.go:53: INFO MQTT Client connected: true
panic: runtime error: index out of range

goroutine 29 [running]:
github.com/nathan-k-/mqttbeat/beater.ParseTopics(0xc42007f270, 0x1, 0x1, 0x0)
	/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/beater/mqttbeat.go:161 +0x20b
github.com/nathan-k-/mqttbeat/beater.(*Mqttbeat).subscribeOnConnect(0xc42011a200, 0x1e254e0, 0xc4200ab180)
	/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/beater/mqttbeat.go:58 +0x45
github.com/nathan-k-/mqttbeat/beater.(*Mqttbeat).(github.com/nathan-k-/mqttbeat/beater.subscribeOnConnect)-fm(0x1e254e0, 0xc4200ab180)
	/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/beater/mqttbeat.go:36 +0x3e
created by github.com/nathan-k-/mqttbeat/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*client).Connect.func1
	/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/vendor/github.com/eclipse/paho.mqtt.golang/client.go:242 +0xa32

The correct configuration should define the QoS:

  topics_subscribe:
    - /events?0

This is a very confusing error and it can be difficult to identify the root cause - would it be possible to put in some sanity check for the configuration at this point and indicate that the topics_subscribe is invalid at this point?

index out of range