Giter Club home page Giter Club logo

mqttbeat's Issues

Cannot build with latest beats

I get the following:

Updating generated files for mqttbeat
make[2]: Entering directory '/root/go/src/github.com/nathan-k-/mqttbeat/vendor/github.com/elastic/beats/libbeat'
make[2]: Leaving directory '/root/go/src/github.com/nathan-k-/mqttbeat/vendor/github.com/elastic/beats/libbeat'
# github.com/nathan-k-/mqttbeat/vendor/github.com/elastic/beats/libbeat/logp
vendor/github.com/elastic/beats/libbeat/logp/log.go:46:5: _log redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/core.go:22:2
vendor/github.com/elastic/beats/libbeat/logp/log.go:126:49: Debug redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:31:49
vendor/github.com/elastic/beats/libbeat/logp/log.go:130:49: MakeDebug redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:11:49
vendor/github.com/elastic/beats/libbeat/logp/log.go:136:31: IsDebug redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:25:31
vendor/github.com/elastic/beats/libbeat/logp/log.go:140:35: HasSelector redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:18:35
vendor/github.com/elastic/beats/libbeat/logp/log.go:151:31: Info redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:40:31
vendor/github.com/elastic/beats/libbeat/logp/log.go:155:31: Warn redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:49:31
vendor/github.com/elastic/beats/libbeat/logp/log.go:159:30: Err redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:58:30
vendor/github.com/elastic/beats/libbeat/logp/log.go:163:35: Critical redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:68:35
vendor/github.com/elastic/beats/libbeat/logp/log.go:186:30: WTF redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:79:30
vendor/github.com/elastic/beats/libbeat/logp/log.go:186:30: too many errors
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:290: recipe for target 'update' failed
make[1]: *** [update] Error 2
make[1]: Leaving directory '/root/go/src/github.com/nathan-k-/mqttbeat'
Makefile:18: recipe for target 'setup' failed
make: *** [setup] Error 2

Add client-id to config parameters

When switching to verneMQ (and using a stricter authentication), any MQTT connection requires to use a client-id known to the MQTT server.

There should be an option like "broker_clientid" to specify the client-id. If empty, it should generate a random ID (32 chars).

_timestamp field in mqtt message causes logstash ingestion failure

I have some mqtt messages coming in that already have a _timestamp field, and logstash complains about these.

2018/09/27 17:38:42.251161 mqttbeat.go:87: DBG MQTT MESSAGE RECEIVED {"_timestamp": 1538069922, "actual": 21.6, "target": 0.0}
2018/09/27 17:38:42.251298 mqttbeat.go:147: DBG Payload decoded - json
2018/09/27 17:38:42.251421 client.go:203: DBG Publish: {
  "@timestamp": "2018-09-27T17:38:42.251Z",
  "_timestamp": 1538069922,
  "actual": 21.6,
  "beat": {
    "index": "mqttbeat",
    "type": "message"

I don't know what the answer is since I can't change my incoming messages, but I at least wanted to log it.

Unable to build with go 1.8, libbeat 5.5.0

What version of beats did you build against? Perhaps something changed...

make[2]: Entering directory '/root/go/src/github.com/nathan-k-/mqttbeat/vendor/github.com/elastic/beats/libbeat'
cat _meta/fields.common.yml > _meta/fields.generated.yml
cat processors/*/_meta/fields.yml >> _meta/fields.generated.yml
make[2]: Leaving directory '/root/go/src/github.com/nathan-k-/mqttbeat/vendor/github.com/elastic/beats/libbeat'
# Update docs
. /root/go/src/github.com/nathan-k-/mqttbeat/build/python-env/bin/activate && python ./vendor/github.com/elastic/beats/libbeat/scripts/generate_fields_docs.py /root/go/src/github.com/nathan-k-/mqttbeat mqttbeat ./vendor/github.com/elastic/beats
# Generate index templates
. /root/go/src/github.com/nathan-k-/mqttbeat/build/python-env/bin/activate && python ./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py --es2x /root/go/src/github.com/nathan-k-/mqttbeat mqttbeat ./vendor/github.com/elastic/beats
Traceback (most recent call last):
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 381, in <module>
    fields_to_es_template(args, fields, output, args.beatname + "-*", version_data['version'])
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 110, in fields_to_es_template
    defaults, "")
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 180, in fill_section_properties
    prop, dynamic = fill_field_properties(args, field, defaults, path)
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 313, in fill_field_properties
    prop, dynamic = fill_section_properties(args, field, defaults, path)
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 180, in fill_section_properties
    prop, dynamic = fill_field_properties(args, field, defaults, path)
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 313, in fill_field_properties
    prop, dynamic = fill_section_properties(args, field, defaults, path)
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 180, in fill_section_properties
    prop, dynamic = fill_field_properties(args, field, defaults, path)
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_template.py", line 338, in fill_field_properties
    raise ValueError("Unknown type found: " + field.get("type"))
ValueError: Unknown type found: object

make errors

I am trying to make, but get the following errors:

rclarke@es-rclarke:mqttbeat$ make
go build -i -ldflags "-X github.com/elastic/beats/libbeat/version.buildTime=2018-02-19T15:15:29Z -X github.com/elastic/beats/libbeat/version.commit=64e3bf19a45686a5cc6aecae58d9d2e140e7c915"
# github.com/nathan-k-/mqttbeat/vendor/github.com/elastic/beats/libbeat/logp
vendor/github.com/elastic/beats/libbeat/logp/log.go:46:5: _log redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/core.go:22:2
vendor/github.com/elastic/beats/libbeat/logp/log.go:125:49: Debug redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:31:49
vendor/github.com/elastic/beats/libbeat/logp/log.go:129:49: MakeDebug redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:11:49
vendor/github.com/elastic/beats/libbeat/logp/log.go:135:31: IsDebug redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:25:31
vendor/github.com/elastic/beats/libbeat/logp/log.go:139:35: HasSelector redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:18:35
vendor/github.com/elastic/beats/libbeat/logp/log.go:150:31: Info redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:40:31
vendor/github.com/elastic/beats/libbeat/logp/log.go:154:31: Warn redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:49:31
vendor/github.com/elastic/beats/libbeat/logp/log.go:158:30: Err redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:58:30
vendor/github.com/elastic/beats/libbeat/logp/log.go:162:35: Critical redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:68:35
vendor/github.com/elastic/beats/libbeat/logp/log.go:168:30: WTF redeclared in this block
	previous declaration at vendor/github.com/elastic/beats/libbeat/logp/global.go:79:30
vendor/github.com/elastic/beats/libbeat/logp/log.go:168:30: too many errors
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:96: recipe for target 'mqttbeat' failed
make: *** [mqttbeat] Error 2

I have pulled the latest from github.com/elastic/beats before doing make setup / make

And go:

rclarke@es-rclarke:mqttbeat$ go version
go version go1.9.4 linux/amd64

Sanity check for topics_subscribe

Running with this config:

  topics_subscribe:
    - /events

produces this error:

rclarke@es-rclarke:mqttbeat$ mqttbeat -e -v -c ~/elastic/temp/mqttbeat.yml 
2018/02/20 11:01:55.252501 beat.go:470: INFO Home path: [/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat] Config path: [/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat] Data path: [/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/data] Logs path: [/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/logs]
2018/02/20 11:01:55.252532 beat.go:495: INFO Beat metadata path: /home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/data/meta.json
2018/02/20 11:01:55.252582 beat.go:477: INFO Beat UUID: 067ea9df-546b-4d58-8009-37205673334c
2018/02/20 11:01:55.252595 beat.go:239: INFO Setup Beat: mqttbeat; Version: 1.0.0
2018/02/20 11:01:55.252607 processor.go:49: DBG Processors: 
2018/02/20 11:01:55.252623 beat.go:250: DBG Initializing output plugins
2018/02/20 11:01:55.252690 metrics.go:23: INFO Metrics logging every 30s
2018/02/20 11:01:55.252759 client.go:123: INFO Elasticsearch url: https://*****:****
2018/02/20 11:01:55.252965 logger.go:18: DBG start pipeline event consumer
2018/02/20 11:01:55.252982 pipeline.go:96: INFO Publisher name: es-rclarke
2018/02/20 11:01:55.252991 publish.go:100: INFO Publisher name: es-rclarke
2018/02/20 11:01:55.253024 mqttbeat.go:34: INFO BROKER url: tcp://hassio:8883
2018/02/20 11:01:55.253033 mqttbeat.go:39: INFO BROKER username: hassio
2018/02/20 11:01:55.257520 mqttbeat.go:53: INFO MQTT Client connected: true
panic: runtime error: index out of range

goroutine 29 [running]:
github.com/nathan-k-/mqttbeat/beater.ParseTopics(0xc42007f270, 0x1, 0x1, 0x0)
	/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/beater/mqttbeat.go:161 +0x20b
github.com/nathan-k-/mqttbeat/beater.(*Mqttbeat).subscribeOnConnect(0xc42011a200, 0x1e254e0, 0xc4200ab180)
	/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/beater/mqttbeat.go:58 +0x45
github.com/nathan-k-/mqttbeat/beater.(*Mqttbeat).(github.com/nathan-k-/mqttbeat/beater.subscribeOnConnect)-fm(0x1e254e0, 0xc4200ab180)
	/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/beater/mqttbeat.go:36 +0x3e
created by github.com/nathan-k-/mqttbeat/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*client).Connect.func1
	/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/vendor/github.com/eclipse/paho.mqtt.golang/client.go:242 +0xa32

The correct configuration should define the QoS:

  topics_subscribe:
    - /events?0

This is a very confusing error and it can be difficult to identify the root cause - would it be possible to put in some sanity check for the configuration at this point and indicate that the topics_subscribe is invalid at this point?

index out of range

Running with this config:

mqttbeat:
  broker_url: "tcp://hassio:8883"
  broker_username: "hassio"
  broker_password: "***"
  decode_payload: true
  topics_subscribe:
    - /events
output.elasticsearch:
  hosts: ["https://my-elasticsearch:9200"]
  protocol: "https"
  username: "elastic"
  password: "****"
  index: "hassio-%{+yyyy.MM.dd}"
logging.level: debug

I get this error:

rclarke@es-rclarke:mqttbeat$ mqttbeat -e -v -c ~/elastic/temp/mqttbeat.yml 
2018/02/20 11:01:55.252501 beat.go:470: INFO Home path: [/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat] Config path: [/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat] Data path: [/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/data] Logs path: [/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/logs]
2018/02/20 11:01:55.252532 beat.go:495: INFO Beat metadata path: /home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/data/meta.json
2018/02/20 11:01:55.252582 beat.go:477: INFO Beat UUID: 067ea9df-546b-4d58-8009-37205673334c
2018/02/20 11:01:55.252595 beat.go:239: INFO Setup Beat: mqttbeat; Version: 1.0.0
2018/02/20 11:01:55.252607 processor.go:49: DBG Processors: 
2018/02/20 11:01:55.252623 beat.go:250: DBG Initializing output plugins
2018/02/20 11:01:55.252690 metrics.go:23: INFO Metrics logging every 30s
2018/02/20 11:01:55.252759 client.go:123: INFO Elasticsearch url: https://*****:****
2018/02/20 11:01:55.252965 logger.go:18: DBG start pipeline event consumer
2018/02/20 11:01:55.252982 pipeline.go:96: INFO Publisher name: es-rclarke
2018/02/20 11:01:55.252991 publish.go:100: INFO Publisher name: es-rclarke
2018/02/20 11:01:55.253024 mqttbeat.go:34: INFO BROKER url: tcp://hassio:8883
2018/02/20 11:01:55.253033 mqttbeat.go:39: INFO BROKER username: hassio
2018/02/20 11:01:55.257520 mqttbeat.go:53: INFO MQTT Client connected: true
panic: runtime error: index out of range

goroutine 29 [running]:
github.com/nathan-k-/mqttbeat/beater.ParseTopics(0xc42007f270, 0x1, 0x1, 0x0)
	/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/beater/mqttbeat.go:161 +0x20b
github.com/nathan-k-/mqttbeat/beater.(*Mqttbeat).subscribeOnConnect(0xc42011a200, 0x1e254e0, 0xc4200ab180)
	/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/beater/mqttbeat.go:58 +0x45
github.com/nathan-k-/mqttbeat/beater.(*Mqttbeat).(github.com/nathan-k-/mqttbeat/beater.subscribeOnConnect)-fm(0x1e254e0, 0xc4200ab180)
	/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/beater/mqttbeat.go:36 +0x3e
created by github.com/nathan-k-/mqttbeat/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*client).Connect.func1
	/home/rclarke/elastic/repos/go/src/github.com/nathan-k-/mqttbeat/vendor/github.com/eclipse/paho.mqtt.golang/client.go:242 +0xa32

Is this a config error, or code error?

Error loading Elasticsearch template: could not load template: couldn't load template: couldn't load json

I am trying it using the latest 7.5.0 Elasticsearch, and I get the error like:
ERR Failed to connect: Connection marked as failed because the onConnect callback failed: Error loading Elasticsearch template: could not load template: couldn't load template: couldn't load json. Error: 400 Bad Request. Response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters: [default : {_meta={version=1.0.0}, dynamic_templates=[{fields={path_match=fields., mapping={type=keyword}, match_mapping_type=string}}, {docker.container.labels={path_match=docker.container.labels., mapping={type=keyword}, match_mapping_type=string}}, {fields={path_match=fields., mapping={type=keyword}, match_mapping_type=string}}, {docker.container.labels={path_match=docker.container.labels., mapping={type=keyword}, match_mapping_type=string}}, {fields={path_match=fields., mapping={type=keyword}, match_mapping_type=string}}, {docker.container.labels={path_match=docker.container.labels., mapping={type=keyword}, match_mapping_type=string}}, {strings_as_keyword={mapping={ignore_above=1024, type=keyword}, match_mapping_type=string}}], date_detection=false, properties={kubernetes={properties={container={properties={image={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}}}, pod={properties={name={ignore_above=1024, ty ...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.