e-travel / cloudwatchlogsbeat Goto Github PK

View Code? Open in Web Editor NEW

64.0 8.0 25.0 27.58 MB

A beat for AWS CloudWatch Logs

License: MIT License

Go 98.60% Dockerfile 1.40%

golang elasticsearch logstash aws elastic cloudwatch logs

cloudwatchlogsbeat's People

Stargazers

Watchers

cloudwatchlogsbeat's Issues

Command line arguments

Where are the command line arguments documented for cloudwatchlogsbeat?

I would like to have multiple instances of it running using a different cloudwatchlogsbeat.yml file. Can you set the config file path as a argument?
I see ./cloudwatchlogsbeat -e -d '*' as the example but I can't find documentation on what that means.

My use case is I want to have a different clodwatchlogsbeat.yml for stage, dev and set a different name or tag and different log groups to monitor.

Thanks!

default templates are missing, error on startup

relevant config:

  template.enabled: true
  template.overwrite: true
  template.versions.2x.enabled: false
  template.path: "/cwlogsbeat.template.json"

2018/01/24 22:04:12.423175 beat.go:285: INFO Home path: [/go/src/github.com/e-travel/cloudwatchlogsbeat] Config path: [/go/src/github.com/e-travel/cloudwatchlogsbeat] Data path: [/go/src/github.com/e-travel/cloudwatchlogsbeat/data] Logs path: [/go/src/github.com/e-travel/cloudwatchlogsbeat/logs]
2018/01/24 22:04:12.423324 beat.go:186: INFO Setup Beat: cloudwatchlogsbeat; Version: 5.4.2
2018/01/24 22:04:12.423919 metrics.go:23: INFO Metrics logging every 30s
2018/01/24 22:04:12.424603 output.go:258: INFO Loading template enabled. Reading template file: /cwlogsbeat.template.json
2018/01/24 22:04:12.424879 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /go/src/github.com/e-travel/cloudwatchlogsbeat/cloudwatchlogsbeat.template-es6x.json
2018/01/24 22:04:12.425642 outputs.go:102: ERR failed to initialize elasticsearch plugin as output: Error loading template /go/src/github.com/e-travel/cloudwatchlogsbeat/cloudwatchlogsbeat.template-es6x.json: open /go/src/github.com/e-travel/cloudwatchlogsbeat/cloudwatchlogsbeat.template-es6x.json: no such file or directory
2018/01/24 22:04:12.426091 beat.go:339: CRIT Exiting: error initializing publisher: Error loading template /go/src/github.com/e-travel/cloudwatchlogsbeat/cloudwatchlogsbeat.template-es6x.json: open /go/src/github.com/e-travel/cloudwatchlogsbeat/cloudwatchlogsbeat.template-es6x.json: no such file or directory
Exiting: error initializing publisher: Error loading template /go/src/github.com/e-travel/cloudwatchlogsbeat/cloudwatchlogsbeat.template-es6x.json: open /go/src/github.com/e-travel/cloudwatchlogsbeat/cloudwatchlogsbeat.template-es6x.json: no such file or directory

I get that unless I set template.enabled: false. You can see that I'm specifying a template, I don't know why it's looking for the es6x template instead.

No Elasticsearch output

Hi,

I have Elasticsearch output enabled in cloudwatchlogsbeat.yml, but I'm not getting anything into my cluster. I have checked that the cluster is reachable from the worker node.

output.elasticsearch:

enabled: true
hosts: ["http://elasticsearch.elasticsearch.svc.cluster.local:9200"]
worker: 1
index: "cloudwatchlogsbeat-%{+yyyy.MM.dd}"

Any ideas?

Thanks

ResourceNotFoundException not sure what to do next

Ok, next hurdle

I have a Log Group called "/gocd" it appars the logs find it but then I get this error

2018/11/12 16:33:37.128945 group.go:64: ERR /gocd ResourceNotFoundException: The specified log group does not exist.
	status code: 400, request id: b46ade63-e698-11e8-a555-4b6b8e681f34

I have full access to both CWLogs and S3

  prospectors:
    - id: application-name
      groupnames:
        - /gocd

I have tried many different options /gocd /gocd/ /gocd/*

I know the logs are coming in but can't seem to get past this step and not sure what I should be looking at

[ec2-user@ip-10-136-26-235 ~]$ ./cloudwatchlogsbeat -c cloudwatchlogsbeat.yml -e
2018/11/12 16:33:22.069731 beat.go:285: INFO Home path: [/home/ec2-user] Config path: [/home/ec2-user] Data path: [/home/ec2-user/data] Logs path: [/home/ec2-user/logs]
2018/11/12 16:33:22.069763 beat.go:186: INFO Setup Beat: cloudwatchlogsbeat; Version: 5.4.2
2018/11/12 16:33:22.069790 outputs.go:108: INFO Activated console as output plugin.
2018/11/12 16:33:22.069813 metrics.go:23: INFO Metrics logging every 30s
2018/11/12 16:33:22.069915 outputs.go:108: INFO Activated kafka as output plugin.
2018/11/12 16:33:22.069994 publish.go:295: INFO Publisher name: cloudwatchlogsbeat
2018/11/12 16:33:22.070112 async.go:63: INFO Flush Interval set to: 1s
2018/11/12 16:33:22.070119 async.go:64: INFO Max Bulk Size set to: 2048
2018/11/12 16:33:22.070165 async.go:63: INFO Flush Interval set to: 1s
2018/11/12 16:33:22.070171 async.go:64: INFO Max Bulk Size set to: 2048
2018/11/12 16:33:22.070330 cloudwatchlogsbeat.go:39: INFO settings: s3_bucket_name=ed-cloudwatchbeat|s3_key_prefix=|aws_region=us-east-1|group_refresh_frequency=10s|stream_refresh_frequency=5s|report_frequency=5m0s|stream_event_horizon=3h0m0s|stream_event_refresh_frequency=10s|hot_stream_event_horizon=5m0s|hot_stream_event_refresh_frequency=1s
2018/11/12 16:33:22.070339 cloudwatchlogsbeat.go:43: INFO Hot streams activated
2018/11/12 16:33:22.070426 cloudwatchlogsbeat.go:55: INFO Working with s3 registry in bucket ed-cloudwatchbeat
2018/11/12 16:33:22.070549 beat.go:221: INFO cloudwatchlogsbeat start running.
2018/11/12 16:33:22.070559 cloudwatchlogsbeat.go:82: INFO cloudwatchlogsbeat is running! Hit CTRL-C to stop it.
2018/11/12 16:33:32.070700 group.go:92: INFO [group] /gocd started
2018/11/12 16:33:37.128945 group.go:64: ERR /gocd ResourceNotFoundException: The specified log group does not exist.
	status code: 400, request id: b46ade63-e698-11e8-a555-4b6b8e681f34
2018/11/12 16:33:42.138365 group.go:64: ERR /gocd ResourceNotFoundException: The specified log group does not exist.
	status code: 400, request id: b7675554-e698-11e8-bfe1-9d72a1f63eff
^C2018/11/12 16:33:43.087593 metrics.go:51: INFO Total non-zero values: 
2018/11/12 16:33:43.087612 metrics.go:52: INFO Uptime: 21.020151703s
2018/11/12 16:33:43.087617 beat.go:225: INFO cloudwatchlogsbeat stopped.

Xpack metrics

I think the xpack metrics are missing from this beat. I am using the configuration in all of my beats, and I can watch their performance on the Kibana dashboard. But it does not appear this beat has it as an available feature. I don't see any error messages nor does it show up in kibana. (or am i just blind?)

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch:
  hosts: [ "http://<hostname>:<port>"]

elasticsearch 7

Any chance to get an update so it works with elastic 7. Might just need a new template.

Question: Amazon Kinesis

I searched for a way to get my vpc flow log data into elasticsearch and just stumbled across cloudwatch-logs-subscription-consumer.

You write in the description that cloudwatchlogsbeat's "operation is subject to AWS limitations and throttling policies".

Out of pure curiosity: Would this throttling policies also apply when using Amazon Kinesis - as cloudwatch-logs-subscription-consumer does?

crashing: fatal error: stack overflow

cloudwatchlogsbeat_1  | runtime: goroutine stack exceeds 1000000000-byte limit
cloudwatchlogsbeat_1  | fatal error: stack overflow
cloudwatchlogsbeat_1  |
cloudwatchlogsbeat_1  | runtime stack:
cloudwatchlogsbeat_1  | runtime.throw(0xbac188, 0xe)
cloudwatchlogsbeat_1  | 	/usr/local/go/src/runtime/panic.go:608 +0x72
cloudwatchlogsbeat_1  | runtime.newstack()
cloudwatchlogsbeat_1  | 	/usr/local/go/src/runtime/stack.go:1008 +0x729
cloudwatchlogsbeat_1  | runtime.morestack()
cloudwatchlogsbeat_1  | 	/usr/local/go/src/runtime/asm_amd64.s:429 +0x8f
cloudwatchlogsbeat_1  |
cloudwatchlogsbeat_1  | goroutine 1 [running]:
cloudwatchlogsbeat_1  | strings.genSplit(0xc0000211e2, 0xd, 0xba2ed4, 0x1, 0x0, 0xffffffffffffffff, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/usr/local/go/src/strings/strings.go:240 +0x2ec fp=0xc021a0a358 sp=0xc021a0a350 pc=0x4e18dc
cloudwatchlogsbeat_1  | strings.Split(0xc0000211e2, 0xd, 0xba2ed4, 0x1, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/usr/local/go/src/strings/strings.go:303 +0x5b fp=0xc021a0a3b0 sp=0xc021a0a358 pc=0x4e19eb
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.parsePath(0xc0000211e2, 0xd, 0xba2ed4, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/path.go:52 +0x8f fp=0xc021a0a498 sp=0xc021a0a3b0 pc=0x8848af
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*expansionDefault).eval(0xc0001afd40, 0xc000204ab0, 0xc00011a880, 0xc0002066c0, 0xc000040e00, 0xc021a0a5d0, 0x885311)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:206 +0x119 fp=0xc021a0a568 sp=0xc021a0a498 pc=0x8973e9
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.spliceDynValue.getValue(0xc75300, 0xc0001afd40, 0xc0000a2ff0, 0xc00011a880, 0x8a097b, 0xc0000211ea, 0x5, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:503 +0x6f fp=0xc021a0a5b8 sp=0xc021a0a568 pc=0x890b7f
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*spliceDynValue).getValue(0xc0001b1ba0, 0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0a6a8, 0x8851c5, 0xc0006f1cd0)
cloudwatchlogsbeat_1  | 	<autogenerated>:1 +0x5d fp=0xc021a0a608 sp=0xc021a0a5b8 pc=0x8a0c0d
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue.func1(0xadb880, 0xc000204db0, 0xc000026e10, 0x25)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:472 +0x41 fp=0xc021a0a650 sp=0xc021a0a608 pc=0x899f51
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.valueCache.cachedValue(0xc000204db0, 0xc000026e10, 0x25, 0xc021a0a6f0, 0x0, 0x0, 0xc0002047e0, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/opts.go:132 +0xc6 fp=0xc021a0a6b0 sp=0xc021a0a650 pc=0x884476
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue(0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0a798, 0x8969a3, 0xc021a0a870)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:471 +0x7b fp=0xc021a0a718 sp=0xc021a0a6b0 pc=0x8908bb
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).withValue(0xc0000a2ff0, 0xc021a0a7c8, 0xc00011a880, 0xc021a0a778)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:465 +0x35 fp=0xc021a0a758 sp=0xc021a0a718 pc=0x8907d5
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).toString(0xc0000a2ff0, 0xc00011a880, 0x0, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:429 +0x80 fp=0xc021a0a7a8 sp=0xc021a0a758 pc=0x890500
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*reference).eval(0xc021a0a870, 0xc000204ab0, 0xc00011a880, 0x1, 0xc041b324a0, 0x2, 0x2)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:160 +0x79 fp=0xc021a0a800 sp=0xc021a0a7a8 pc=0x896be9
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*expansionDefault).eval(0xc0001afd40, 0xc000204ab0, 0xc00011a880, 0xc0002066c0, 0xc000040e00, 0xc021a0a938, 0x885311)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:207 +0x1c9 fp=0xc021a0a8d0 sp=0xc021a0a800 pc=0x897499
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.spliceDynValue.getValue(0xc75300, 0xc0001afd40, 0xc0000a2ff0, 0xc00011a880, 0x8a097b, 0xc0000211ea, 0x5, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:503 +0x6f fp=0xc021a0a920 sp=0xc021a0a8d0 pc=0x890b7f
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*spliceDynValue).getValue(0xc0001b1ba0, 0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0aa10, 0x8851c5, 0xc0006f1cb0)
cloudwatchlogsbeat_1  | 	<autogenerated>:1 +0x5d fp=0xc021a0a970 sp=0xc021a0a920 pc=0x8a0c0d
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue.func1(0xadb880, 0xc000204db0, 0xc000026e10, 0x25)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:472 +0x41 fp=0xc021a0a9b8 sp=0xc021a0a970 pc=0x899f51
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.valueCache.cachedValue(0xc000204db0, 0xc000026e10, 0x25, 0xc021a0aa58, 0x0, 0x0, 0xc0002047e0, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/opts.go:132 +0xc6 fp=0xc021a0aa18 sp=0xc021a0a9b8 pc=0x884476
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue(0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0ab00, 0x8969a3, 0xc021a0abd8)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:471 +0x7b fp=0xc021a0aa80 sp=0xc021a0aa18 pc=0x8908bb
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).withValue(0xc0000a2ff0, 0xc021a0ab30, 0xc00011a880, 0xc021a0aae0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:465 +0x35 fp=0xc021a0aac0 sp=0xc021a0aa80 pc=0x8907d5
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).toString(0xc0000a2ff0, 0xc00011a880, 0x0, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:429 +0x80 fp=0xc021a0ab10 sp=0xc021a0aac0 pc=0x890500
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*reference).eval(0xc021a0abd8, 0xc000204ab0, 0xc00011a880, 0x1, 0xc041b323a0, 0x2, 0x2)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:160 +0x79 fp=0xc021a0ab68 sp=0xc021a0ab10 pc=0x896be9
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*expansionDefault).eval(0xc0001afd40, 0xc000204ab0, 0xc00011a880, 0xc0002066c0, 0xc000040e00, 0xc021a0aca0, 0x885311)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:207 +0x1c9 fp=0xc021a0ac38 sp=0xc021a0ab68 pc=0x897499
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.spliceDynValue.getValue(0xc75300, 0xc0001afd40, 0xc0000a2ff0, 0xc00011a880, 0x8a097b, 0xc0000211ea, 0x5, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:503 +0x6f fp=0xc021a0ac88 sp=0xc021a0ac38 pc=0x890b7f
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*spliceDynValue).getValue(0xc0001b1ba0, 0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0ad78, 0x8851c5, 0xc0006f1c90)
cloudwatchlogsbeat_1  | 	<autogenerated>:1 +0x5d fp=0xc021a0acd8 sp=0xc021a0ac88 pc=0x8a0c0d
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue.func1(0xadb880, 0xc000204db0, 0xc000026e10, 0x25)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:472 +0x41 fp=0xc021a0ad20 sp=0xc021a0acd8 pc=0x899f51
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.valueCache.cachedValue(0xc000204db0, 0xc000026e10, 0x25, 0xc021a0adc0, 0x0, 0x0, 0xc0002047e0, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/opts.go:132 +0xc6 fp=0xc021a0ad80 sp=0xc021a0ad20 pc=0x884476
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue(0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0ae68, 0x8969a3, 0xc021a0af40)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:471 +0x7b fp=0xc021a0ade8 sp=0xc021a0ad80 pc=0x8908bb
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).withValue(0xc0000a2ff0, 0xc021a0ae98, 0xc00011a880, 0xc021a0ae48)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:465 +0x35 fp=0xc021a0ae28 sp=0xc021a0ade8 pc=0x8907d5
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).toString(0xc0000a2ff0, 0xc00011a880, 0x0, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:429 +0x80 fp=0xc021a0ae78 sp=0xc021a0ae28 pc=0x890500
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*reference).eval(0xc021a0af40, 0xc000204ab0, 0xc00011a880, 0x1, 0xc041b322a0, 0x2, 0x2)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:160 +0x79 fp=0xc021a0aed0 sp=0xc021a0ae78 pc=0x896be9
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*expansionDefault).eval(0xc0001afd40, 0xc000204ab0, 0xc00011a880, 0xc0002066c0, 0xc000040e00, 0xc021a0b008, 0x885311)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:207 +0x1c9 fp=0xc021a0afa0 sp=0xc021a0aed0 pc=0x897499
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.spliceDynValue.getValue(0xc75300, 0xc0001afd40, 0xc0000a2ff0, 0xc00011a880, 0x8a097b, 0xc0000211ea, 0x5, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:503 +0x6f fp=0xc021a0aff0 sp=0xc021a0afa0 pc=0x890b7f
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*spliceDynValue).getValue(0xc0001b1ba0, 0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0b0e0, 0x8851c5, 0xc0006f1c70)
cloudwatchlogsbeat_1  | 	<autogenerated>:1 +0x5d fp=0xc021a0b040 sp=0xc021a0aff0 pc=0x8a0c0d
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue.func1(0xadb880, 0xc000204db0, 0xc000026e10, 0x25)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:472 +0x41 fp=0xc021a0b088 sp=0xc021a0b040 pc=0x899f51
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.valueCache.cachedValue(0xc000204db0, 0xc000026e10, 0x25, 0xc021a0b128, 0x0, 0x0, 0xc0002047e0, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/opts.go:132 +0xc6 fp=0xc021a0b0e8 sp=0xc021a0b088 pc=0x884476
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue(0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0b1d0, 0x8969a3, 0xc021a0b2a8)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:471 +0x7b fp=0xc021a0b150 sp=0xc021a0b0e8 pc=0x8908bb
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).withValue(0xc0000a2ff0, 0xc021a0b200, 0xc00011a880, 0xc021a0b1b0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:465 +0x35 fp=0xc021a0b190 sp=0xc021a0b150 pc=0x8907d5
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).toString(0xc0000a2ff0, 0xc00011a880, 0x0, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:429 +0x80 fp=0xc021a0b1e0 sp=0xc021a0b190 pc=0x890500
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*reference).eval(0xc021a0b2a8, 0xc000204ab0, 0xc00011a880, 0x1, 0xc041b321a0, 0x2, 0x2)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:160 +0x79 fp=0xc021a0b238 sp=0xc021a0b1e0 pc=0x896be9
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*expansionDefault).eval(0xc0001afd40, 0xc000204ab0, 0xc00011a880, 0xc0002066c0, 0xc000040e00, 0xc021a0b370, 0x885311)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:207 +0x1c9 fp=0xc021a0b308 sp=0xc021a0b238 pc=0x897499
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.spliceDynValue.getValue(0xc75300, 0xc0001afd40, 0xc0000a2ff0, 0xc00011a880, 0x8a097b, 0xc0000211ea, 0x5, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:503 +0x6f fp=0xc021a0b358 sp=0xc021a0b308 pc=0x890b7f
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*spliceDynValue).getValue(0xc0001b1ba0, 0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0b448, 0x8851c5, 0xc0006f1c50)
cloudwatchlogsbeat_1  | 	<autogenerated>:1 +0x5d fp=0xc021a0b3a8 sp=0xc021a0b358 pc=0x8a0c0d
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue.func1(0xadb880, 0xc000204db0, 0xc000026e10, 0x25)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:472 +0x41 fp=0xc021a0b3f0 sp=0xc021a0b3a8 pc=0x899f51
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.valueCache.cachedValue(0xc000204db0, 0xc000026e10, 0x25, 0xc021a0b490, 0x0, 0x0, 0xc0002047e0, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/opts.go:132 +0xc6 fp=0xc021a0b450 sp=0xc021a0b3f0 pc=0x884476
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue(0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0b538, 0x8969a3, 0xc021a0b610)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:471 +0x7b fp=0xc021a0b4b8 sp=0xc021a0b450 pc=0x8908bb
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).withValue(0xc0000a2ff0, 0xc021a0b568, 0xc00011a880, 0xc021a0b518)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:465 +0x35 fp=0xc021a0b4f8 sp=0xc021a0b4b8 pc=0x8907d5
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).toString(0xc0000a2ff0, 0xc00011a880, 0x0, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:429 +0x80 fp=0xc021a0b548 sp=0xc021a0b4f8 pc=0x890500
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*reference).eval(0xc021a0b610, 0xc000204ab0, 0xc00011a880, 0x1, 0xc041b320a0, 0x2, 0x2)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:160 +0x79 fp=0xc021a0b5a0 sp=0xc021a0b548 pc=0x896be9
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*expansionDefault).eval(0xc0001afd40, 0xc000204ab0, 0xc00011a880, 0xc0002066c0, 0xc000040e00, 0xc021a0b6d8, 0x885311)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:207 +0x1c9 fp=0xc021a0b670 sp=0xc021a0b5a0 pc=0x897499
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.spliceDynValue.getValue(0xc75300, 0xc0001afd40, 0xc0000a2ff0, 0xc00011a880, 0x8a097b, 0xc0000211ea, 0x5, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:503 +0x6f fp=0xc021a0b6c0 sp=0xc021a0b670 pc=0x890b7f
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*spliceDynValue).getValue(0xc0001b1ba0, 0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0b7b0, 0x8851c5, 0xc0006f1c30)
cloudwatchlogsbeat_1  | 	<autogenerated>:1 +0x5d fp=0xc021a0b710 sp=0xc021a0b6c0 pc=0x8a0c0d
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue.func1(0xadb880, 0xc000204db0, 0xc000026e10, 0x25)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:472 +0x41 fp=0xc021a0b758 sp=0xc021a0b710 pc=0x899f51
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.valueCache.cachedValue(0xc000204db0, 0xc000026e10, 0x25, 0xc021a0b7f8, 0x0, 0x0, 0xc0002047e0, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/opts.go:132 +0xc6 fp=0xc021a0b7b8 sp=0xc021a0b758 pc=0x884476
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue(0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0b8a0, 0x8969a3, 0xc021a0b978)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:471 +0x7b fp=0xc021a0b820 sp=0xc021a0b7b8 pc=0x8908bb
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).withValue(0xc0000a2ff0, 0xc021a0b8d0, 0xc00011a880, 0xc021a0b880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:465 +0x35 fp=0xc021a0b860 sp=0xc021a0b820 pc=0x8907d5
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).toString(0xc0000a2ff0, 0xc00011a880, 0x0, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:429 +0x80 fp=0xc021a0b8b0 sp=0xc021a0b860 pc=0x890500
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*reference).eval(0xc021a0b978, 0xc000204ab0, 0xc00011a880, 0x1, 0xc041b39fa0, 0x2, 0x2)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:160 +0x79 fp=0xc021a0b908 sp=0xc021a0b8b0 pc=0x896be9
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*expansionDefault).eval(0xc0001afd40, 0xc000204ab0, 0xc00011a880, 0xc0002066c0, 0xc000040e00, 0xc021a0ba40, 0x885311)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:207 +0x1c9 fp=0xc021a0b9d8 sp=0xc021a0b908 pc=0x897499
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.spliceDynValue.getValue(0xc75300, 0xc0001afd40, 0xc0000a2ff0, 0xc00011a880, 0x8a097b, 0xc0000211ea, 0x5, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:503 +0x6f fp=0xc021a0ba28 sp=0xc021a0b9d8 pc=0x890b7f
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*spliceDynValue).getValue(0xc0001b1ba0, 0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0bb18, 0x8851c5, 0xc0006f1c10)
cloudwatchlogsbeat_1  | 	<autogenerated>:1 +0x5d fp=0xc021a0ba78 sp=0xc021a0ba28 pc=0x8a0c0d
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue.func1(0xadb880, 0xc000204db0, 0xc000026e10, 0x25)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:472 +0x41 fp=0xc021a0bac0 sp=0xc021a0ba78 pc=0x899f51
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.valueCache.cachedValue(0xc000204db0, 0xc000026e10, 0x25, 0xc021a0bb60, 0x0, 0x0, 0xc0002047e0, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/opts.go:132 +0xc6 fp=0xc021a0bb20 sp=0xc021a0bac0 pc=0x884476
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue(0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0bc08, 0x8969a3, 0xc021a0bce0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:471 +0x7b fp=0xc021a0bb88 sp=0xc021a0bb20 pc=0x8908bb
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).withValue(0xc0000a2ff0, 0xc021a0bc38, 0xc00011a880, 0xc021a0bbe8)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:465 +0x35 fp=0xc021a0bbc8 sp=0xc021a0bb88 pc=0x8907d5
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).toString(0xc0000a2ff0, 0xc00011a880, 0x0, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:429 +0x80 fp=0xc021a0bc18 sp=0xc021a0bbc8 pc=0x890500
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*reference).eval(0xc021a0bce0, 0xc000204ab0, 0xc00011a880, 0x1, 0xc041b39ea0, 0x2, 0x2)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:160 +0x79 fp=0xc021a0bc70 sp=0xc021a0bc18 pc=0x896be9
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*expansionDefault).eval(0xc0001afd40, 0xc000204ab0, 0xc00011a880, 0xc0002066c0, 0x7fb9e4816270, 0xc021a0bda8, 0x885311)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:207 +0x1c9 fp=0xc021a0bd40 sp=0xc021a0bc70 pc=0x897499
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.spliceDynValue.getValue(0xc75300, 0xc0001afd40, 0xc0000a2ff0, 0xc00011a880, 0x8a097b, 0xc0000211ea, 0x5, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:503 +0x6f fp=0xc021a0bd90 sp=0xc021a0bd40 pc=0x890b7f
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*spliceDynValue).getValue(0xc0001b1ba0, 0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0be80, 0x8851c5, 0xc0006f1bf0)
cloudwatchlogsbeat_1  | 	<autogenerated>:1 +0x5d fp=0xc021a0bde0 sp=0xc021a0bd90 pc=0x8a0c0d
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue.func1(0xadb880, 0xc000204db0, 0xc000026e10, 0x25)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:472 +0x41 fp=0xc021a0be28 sp=0xc021a0bde0 pc=0x899f51
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.valueCache.cachedValue(0xc000204db0, 0xc000026e10, 0x25, 0xc021a0bec8, 0x0, 0x0, 0xc0002047e0, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/opts.go:132 +0xc6 fp=0xc021a0be88 sp=0xc021a0be28 pc=0x884476
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue(0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0bf70, 0x8969a3, 0xc021a0c048)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:471 +0x7b fp=0xc021a0bef0 sp=0xc021a0be88 pc=0x8908bb
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).withValue(0xc0000a2ff0, 0xc021a0bfa0, 0xc00011a880, 0xc021a0bf50)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:465 +0x35 fp=0xc021a0bf30 sp=0xc021a0bef0 pc=0x8907d5
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).toString(0xc0000a2ff0, 0xc00011a880, 0x0, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:429 +0x80 fp=0xc021a0bf80 sp=0xc021a0bf30 pc=0x890500
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*reference).eval(0xc021a0c048, 0xc000204ab0, 0xc00011a880, 0x1, 0xc041b39da0, 0x2, 0x2)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:160 +0x79 fp=0xc021a0bfd8 sp=0xc021a0bf80 pc=0x896be9
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*expansionDefault).eval(0xc0001afd40, 0xc000204ab0, 0xc00011a880, 0xc0002066c0, 0xc000040e00, 0xc021a0c110, 0x885311)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:207 +0x1c9 fp=0xc021a0c0a8 sp=0xc021a0bfd8 pc=0x897499
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.spliceDynValue.getValue(0xc75300, 0xc0001afd40, 0xc0000a2ff0, 0xc00011a880, 0x8a097b, 0xc0000211ea, 0x5, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:503 +0x6f fp=0xc021a0c0f8 sp=0xc021a0c0a8 pc=0x890b7f
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*spliceDynValue).getValue(0xc0001b1ba0, 0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0c1e8, 0x8851c5, 0xc0006f1bd0)
cloudwatchlogsbeat_1  | 	<autogenerated>:1 +0x5d fp=0xc021a0c148 sp=0xc021a0c0f8 pc=0x8a0c0d
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue.func1(0xadb880, 0xc000204db0, 0xc000026e10, 0x25)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:472 +0x41 fp=0xc021a0c190 sp=0xc021a0c148 pc=0x899f51
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.valueCache.cachedValue(0xc000204db0, 0xc000026e10, 0x25, 0xc021a0c230, 0x0, 0x0, 0xc0002047e0, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/opts.go:132 +0xc6 fp=0xc021a0c1f0 sp=0xc021a0c190 pc=0x884476
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue(0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0c2d8, 0x8969a3, 0xc021a0c3b0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:471 +0x7b fp=0xc021a0c258 sp=0xc021a0c1f0 pc=0x8908bb
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).withValue(0xc0000a2ff0, 0xc021a0c308, 0xc00011a880, 0xc021a0c2b8)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:465 +0x35 fp=0xc021a0c298 sp=0xc021a0c258 pc=0x8907d5
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).toString(0xc0000a2ff0, 0xc00011a880, 0x0, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:429 +0x80 fp=0xc021a0c2e8 sp=0xc021a0c298 pc=0x890500
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*reference).eval(0xc021a0c3b0, 0xc000204ab0, 0xc00011a880, 0x1, 0xc041b39ca0, 0x2, 0x2)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:160 +0x79 fp=0xc021a0c340 sp=0xc021a0c2e8 pc=0x896be9
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*expansionDefault).eval(0xc0001afd40, 0xc000204ab0, 0xc00011a880, 0xc0002066c0, 0xc000040e00, 0xc021a0c478, 0x885311)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:207 +0x1c9 fp=0xc021a0c410 sp=0xc021a0c340 pc=0x897499
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.spliceDynValue.getValue(0xc75300, 0xc0001afd40, 0xc0000a2ff0, 0xc00011a880, 0x8a097b, 0xc0000211ea, 0x5, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:503 +0x6f fp=0xc021a0c460 sp=0xc021a0c410 pc=0x890b7f
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*spliceDynValue).getValue(0xc0001b1ba0, 0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0c550, 0x8851c5, 0xc0006f1bb0)
cloudwatchlogsbeat_1  | 	<autogenerated>:1 +0x5d fp=0xc021a0c4b0 sp=0xc021a0c460 pc=0x8a0c0d
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue.func1(0xadb880, 0xc000204db0, 0xc000026e10, 0x25)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:472 +0x41 fp=0xc021a0c4f8 sp=0xc021a0c4b0 pc=0x899f51
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.valueCache.cachedValue(0xc000204db0, 0xc000026e10, 0x25, 0xc021a0c598, 0x0, 0x0, 0xc0002047e0, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/opts.go:132 +0xc6 fp=0xc021a0c558 sp=0xc021a0c4f8 pc=0x884476
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue(0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0c640, 0x8969a3, 0xc021a0c718)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:471 +0x7b fp=0xc021a0c5c0 sp=0xc021a0c558 pc=0x8908bb
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).withValue(0xc0000a2ff0, 0xc021a0c670, 0xc00011a880, 0xc021a0c620)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:465 +0x35 fp=0xc021a0c600 sp=0xc021a0c5c0 pc=0x8907d5
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).toString(0xc0000a2ff0, 0xc00011a880, 0x0, 0x0, 0x0, 0x0)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:429 +0x80 fp=0xc021a0c650 sp=0xc021a0c600 pc=0x890500
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*reference).eval(0xc021a0c718, 0xc000204ab0, 0xc00011a880, 0x1, 0xc041b39ba0, 0x2, 0x2)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:160 +0x79 fp=0xc021a0c6a8 sp=0xc021a0c650 pc=0x896be9
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*expansionDefault).eval(0xc0001afd40, 0xc000204ab0, 0xc00011a880, 0xc0002066c0, 0xc000040e00, 0xc021a0c7e0, 0x885311)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/variables.go:207 +0x1c9 fp=0xc021a0c778 sp=0xc021a0c6a8 pc=0x897499
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.spliceDynValue.getValue(0xc75300, 0xc0001afd40, 0xc0000a2ff0, 0xc00011a880, 0x8a097b, 0xc0000211ea, 0x5, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:503 +0x6f fp=0xc021a0c7c8 sp=0xc021a0c778 pc=0x890b7f
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*spliceDynValue).getValue(0xc0001b1ba0, 0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0c8b8, 0x8851c5, 0xc0006f1b90)
cloudwatchlogsbeat_1  | 	<autogenerated>:1 +0x5d fp=0xc021a0c818 sp=0xc021a0c7c8 pc=0x8a0c0d
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue.func1(0xadb880, 0xc000204db0, 0xc000026e10, 0x25)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:472 +0x41 fp=0xc021a0c860 sp=0xc021a0c818 pc=0x899f51
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.valueCache.cachedValue(0xc000204db0, 0xc000026e10, 0x25, 0xc021a0c900, 0x0, 0x0, 0xc0002047e0, 0xc00011a880)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/opts.go:132 +0xc6 fp=0xc021a0c8c0 sp=0xc021a0c860 pc=0x884476
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).getValue(0xc0000a2ff0, 0xc00011a880, 0x0, 0xc021a0c9a8, 0x8969a3, 0xc021a0ca80)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:471 +0x7b fp=0xc021a0c928 sp=0xc021a0c8c0 pc=0x8908bb
cloudwatchlogsbeat_1  | github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*cfgDynamic).withValue(0xc0000a2ff0, 0xc021a0c9d8, 0xc00011a880, 0xc021a0c988)
cloudwatchlogsbeat_1  | 	/go/src/github.com/e-travel/cloudwatchlogsbeat/vendor/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/types.go:465 +0x35 fp=0xc021a0c968 sp=0xc021a0c928 pc=0x8907d5
cloudwatchlogsbeat_1  | ...additional frames elided...
cloudwatchlogsbeat_1  |
cloudwatchlogsbeat_1  | goroutine 17 [syscall]:
cloudwatchlogsbeat_1  | os/signal.signal_recv(0x0)
cloudwatchlogsbeat_1  | 	/usr/local/go/src/runtime/sigqueue.go:139 +0x9c
cloudwatchlogsbeat_1  | os/signal.loop()
cloudwatchlogsbeat_1  | 	/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
cloudwatchlogsbeat_1  | created by os/signal.init.0
cloudwatchlogsbeat_1  | 	/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

config:

name: cloudwatchlogsbeat

cloudwatchlogsbeat:

  s3_bucket_name: logs
  s3_key_prefix: cloudwatchlogsbeat/

  group_refresh_frequency: 60s
  stream_refresh_frequency: 10s
  report_frequency: 5m
  aws_region: eu-west-1

  # === HOT STREAMS ===
  # hot streams are streams whose last event is earlier than this value
  # a value of zero deactivates hot streams
  hot_stream_event_horizon: 5m
  # defines the refresh frequency of log events for hot streams
  # AWS API call: GetLogEvents
  hot_stream_event_refresh_frequency: 1s

  # === STANDARD STREAMS ===
  # log events earlier than this value are not captured
  # log streams whose last event is earlier than this value are not monitored
  stream_event_horizon: 3h
  # defines the refresh frequency of log events for streams
  # AWS API call: GetLogEvents
  stream_event_refresh_frequency: 10s

  # === PROSPECTORS ===
  # Applications whose log groups we'd like to monitor
  prospectors:
    - id: logs-misc
      groupnames:
        - logs

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

output.redis:
  hosts: ["${REDIS_HOST}"]
  password: "${REDIS_AUTH}"
  key: "filebeat"
  db: 0
  timeout: 10
  ssl.enabled: "${REDIS_SSL}"

logging.level: "${logging.level:error}"

Some questions about functionalities

Hi Folks,

We have 2 questions about the project's functionalities:

Can we filter by log_stream?
We have log_streams created in per month basis, so sometimes We'd like to index a specific set of months;
Do we need a logstash to be able to index the message field contents?

Thanks in advance.

A bit confused (config)

I am a bit confused with the configuration.

Totally understand the BEAT parts but do I have to have my Cloudwatch logs go to S3?

So am I specifying the "path" that is listed in the Cloudwatch logs/streams? or do I have to send all of that information to S3 as well. (I was hoping not to )

[ec2-user@ip-10-136-26-235 ~]$ ./cloudwatchlogsbeat -e -d '*' -c cloudwatchlogsbeat.yml 
2018/11/08 20:56:10.793913 beat.go:285: INFO Home path: [/home/ec2-user] Config path: [/home/ec2-user] Data path: [/home/ec2-user/data] Logs path: [/home/ec2-user/logs]
2018/11/08 20:56:10.793945 beat.go:186: INFO Setup Beat: cloudwatchlogsbeat; Version: 5.4.2
2018/11/08 20:56:10.793956 processor.go:44: DBG  Processors: 
2018/11/08 20:56:10.793965 beat.go:192: DBG  Initializing output plugins
2018/11/08 20:56:10.794001 metrics.go:23: INFO Metrics logging every 30s
2018/11/08 20:56:10.793998 outputs.go:108: INFO Activated console as output plugin.
2018/11/08 20:56:10.794016 kafka.go:113: DBG  initialize kafka output
2018/11/08 20:56:10.794136 outputs.go:108: INFO Activated kafka as output plugin.
2018/11/08 20:56:10.794142 publish.go:238: DBG  Create output worker
2018/11/08 20:56:10.794184 publish.go:238: DBG  Create output worker
2018/11/08 20:56:10.794208 publish.go:280: DBG  No output is defined to store the topology. The server fields might not be filled.
2018/11/08 20:56:10.794230 publish.go:295: INFO Publisher name: cloudwatchlogsbeat
2018/11/08 20:56:10.794373 async.go:63: INFO Flush Interval set to: 1s
2018/11/08 20:56:10.794382 async.go:64: INFO Max Bulk Size set to: 2048
2018/11/08 20:56:10.794387 async.go:71: DBG  create bulk processing worker (interval=1s, bulk size=2048)
2018/11/08 20:56:10.794432 async.go:63: INFO Flush Interval set to: 1s
2018/11/08 20:56:10.794438 async.go:64: INFO Max Bulk Size set to: 2048
2018/11/08 20:56:10.794442 async.go:71: DBG  create bulk processing worker (interval=1s, bulk size=2048)
2018/11/08 20:56:10.794622 cloudwatchlogsbeat.go:39: INFO settings: s3_bucket_name=XXXXXXX|s3_key_prefix=|aws_region=eu-west-1|group_refresh_frequency=10s|stream_refresh_frequency=5s|report_frequency=5m0s|stream_event_horizon=3h0m0s|stream_event_refresh_frequency=10s|hot_stream_event_horizon=5m0s|hot_stream_event_refresh_frequency=1s
2018/11/08 20:56:10.794629 cloudwatchlogsbeat.go:43: INFO Hot streams activated
2018/11/08 20:56:10.794710 cloudwatchlogsbeat.go:55: INFO Working with s3 registry in bucket BUCKET
2018/11/08 20:56:10.794833 beat.go:221: INFO cloudwatchlogsbeat start running.
2018/11/08 20:56:10.794841 cloudwatchlogsbeat.go:82: INFO cloudwatchlogsbeat is running! Hit CTRL-C to stop it.
2018/11/08 20:56:40.794144 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/11/08 20:57:10.794135 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/11/08 20:57:40.794138 metrics.go:34: INFO No non-zero metrics in the last 30s

Docker should be based off a linux distribution

Currently the docker-image will result in a image that is based-off scratch - meaning no other binaries are installed.

It is good, if you want to keep the size of the container minimal, but it is bad if you want to add things on-top of it.

Go Modules

Now that Go Modules are becoming more mainstream/out of the box supported, would be nice to migrate off Glide :)

Include AWS CloudWatch Log Group Tags

Feature Request:

It is possible to add tags onto CloudWatch Log Groups: https://docs.aws.amazon.com/cli/latest/reference/logs/tag-log-group.html

Is it possible for these to be included in the log event sent to ES?

Add EventId in order to deduplicate

AWS filter logs api call has eventId per message:

events": [
    {
      "ingestionTime": 1396035394997,
      "timestamp": 1396035378988,
      "message": "ERROR Event 1",
      "logStreamName": "my-log-stream-1",
      "eventId": "31132629274945519779805322857203735586714454643391594505"
    },
    {
      "ingestionTime": 1396035394997,
      "timestamp": 1396035378988,
      "message": "ERROR Event 2",
      "logStreamName": "my-log-stream-2",
      "eventId": "31132629274945519779805322857203735586814454643391594505"
    },
    {

While I appreciate the s3 state storage I would rather just reprocess logs for the last X hours and deduplicate by sending a @metadata._id so that elasticsearch output will overwrite the event.

It would be great if the EventId is added as a field so we can further use this information in an ingest pipeline to set the document id. Unfortunately it seems beats can't set elasticsearch document id yet.

feat req: put s3 registry files at prefix

It'd be nice to not only specify a bucket, but also specify a prefix where the registry files are kept.

ability to add fields?

Adding fields is a great way to allow further processing down the line (e.g. by logstash) - so it would be great if it was possible to define custom fields per prospector.

Update to latest Beats

Will there be support for cloud ID?

Inconsistent behavior and how to scale up

Hi, I recently started using this project for pulling my logs from AWS. There aren't a lot of them, like 4k per 15 minutes.

I was able to get a decent result last night by tweaking a bit the default settings. I manually pushed 5k logs to cw within 1 minute, and was able to see them in another 2~3 minutes. However, when I try again this morning, I have constantly waited for over 8 minutes. And the starting 3 minutes is almost always 0 logs delivered.

Here's my settings

      REPORT_FERQUENCY: 5m
      HOT_STREAM_FREQUENCY: 1s
      HOT_STREAM_TIME_HORIZON: 3m
      STREAM_TIME_HORIZON: 2h
      STREAM_EVENT_REFRESH_FREQUENCY: 3s
      COMPRESSION_LEVEL: 0

Also is there a way scale up the project, I tried to have 2 instances the same time with same settings, that seems gets me duplicate logs

Thanks!

No output to Elasticsearch and nothing in the activity log

Below is my configuration that I'm using for this beat. I got ES 6.5 running on Kubernetes on AWS. This beat also runs in Kubernetes too.
I do get logs going to the S3 bucket rather and nothing on elasticsearch. What makes it worse is there is no error on the logs or any sign that it is even attempting to send something to elasticsearch even though I have log level debug enabled.

Not sure how to troubleshoot this when I get no logs about its activity

name: cloudwatchlogsbeat
cloudwatchlogsbeat:
  s3_bucket_name: com.domain.cloudwatchlogsbeat
  s3_key_prefix: cloudwatch/
  group_refresh_frequency: 10s
  stream_refresh_frequency: 5s
  report_frequency: 5m
  aws_region: us-east-1

  hot_stream_event_horizon: 5m
  hot_stream_event_refresh_frequency: 1s
  stream_event_horizon: 3h
  stream_event_refresh_frequency: 10s
  template.enabled: false

  prospectors:
    - id: aws-events
      groupnames:
        - /aws/cloudtrail-events
      multiline:
        pattern: "^REPORT RequestId.+"
        negate: true
        match: before
output.elasticsearch:
  enabled: true
  hosts: ["elasticsearch-logging:9200"]
  worker: 1
  timeout: 90
  template.versions.2x.enabled: false
  template.versions.6x.enabled: false

logging.level: debug

logging.to_files: true
logging.files:
  # path: /var/log/cwlogs
  name: cwlbeatlog
  rotateeverybytes: 10485760 # = 10MB
  keepfiles: 7

Here are the logs that I see:

2019-01-29T21:09:23Z INFO Hot streams activated
2019-01-29T21:09:23Z INFO Working with s3 registry in bucket com.domain.cloudwatchlogsbeat
2019-01-29T21:09:23Z INFO cloudwatchlogsbeat start running.
2019-01-29T21:09:23Z INFO cloudwatchlogsbeat is running! Hit CTRL-C to stop it.
2019-01-29T21:09:33Z INFO [group] /aws/cloudtrail-events started
2019-01-29T21:09:38Z INFO Start monitoring stream 333544645563_CloudTrail_us-east-1 for group /aws/cloudtrail-events
2019-01-29T21:09:38Z INFO [stream] /aws/cloudtrail-events/333544645563_CloudTrail_us-east-1 started
2019-01-29T21:09:38Z INFO Fetching registry info for cloudwatch//aws/cloudtrail-events/333544645563_CloudTrail_us-east-1
2019-01-29T21:09:53Z INFO No non-zero metrics in the last 30s
2019-01-29T21:10:23Z INFO No non-zero metrics in the last 30s
2019-01-29T21:10:53Z INFO No non-zero metrics in the last 30s
2019-01-29T21:11:23Z INFO No non-zero metrics in the last 30s
2019-01-29T21:11:53Z INFO No non-zero metrics in the last 30s
2019-01-29T21:12:23Z INFO No non-zero metrics in the last 30s
2019-01-29T21:12:53Z INFO No non-zero metrics in the last 30s
2019-01-29T21:13:23Z INFO No non-zero metrics in the last 30s
2019-01-29T21:13:53Z INFO No non-zero metrics in the last 30s
2019-01-29T21:14:23Z INFO No non-zero metrics in the last 30s
2019-01-29T21:14:23Z INFO report[manager] 1 1
2019-01-29T21:14:33Z INFO report[group] 1 1 0 /aws/cloudtrail-events 5m0s

Version of application code running

It would be great to have running version of cloudwatchlogsbeat exposed. Either in application logs when application starts or/and with a command-line parameter.

S3 keys not being created, proper value of prospectors.id?

Hi,

I set the S3 bucket to allow my workers to save stream state, but no state info is being saved. The worker role has full rights to the bucket and to the log groups, but I am getting 403 errors.

2018/11/14 15:33:33.510213 s3.go:36: INFO Fetching registry info for /aws/elasticbeanstalk/foo/var/log/nginx/error.log/i-0f759ca46a734f037

2018/11/14 15:33:33.523337 s3.go:32: WARN s3: failed to read key=/aws/elasticbeanstalk/foo/var/log/nginx/error.log/i-0f759ca46a734f037 [message=AccessDenied: Access Denied
status code: 403

It looks like there should be a state file or key created for each stream, which is not happening, and then it tries to read from the state file and fails because it does not exist.

Any ideas?

In addition, the prospectors section of cloudwatchlogsbeat.yml is unclear. What should the value be?

I set this as log, since the normal types for prospectors are:

One of the following input types:

log: Reads every line of the log file (default).
stdin: Reads the standard in.
redis: Reads slow log entries from redis (experimental).
udp: Reads events over UDP. Also see max_message_sizeedit.
docker: Reads logs from Docker. Also see containersedit (experimental).
The value that you specify here is used as the type for each event published to Logstash and Elasticsearch.

=== PROSPECTORS ===

Applications whose log groups we'd like to monitor

prospectors:
# the id will be used as the _type field
- id: log

Thanks

[High Priority] Hitting the AWS Cloudwatch Logs GetLogEvents Limit

Hi,

the application hit's the limits of Cloudwatch Logs.

Due to this it is almost impossible to watch logs from AWS Console and moving the logs to cloudwatchlogsbeat resulted in a 1h delay of logs, which makes it impossible to monitor.

Suggestion:

AWS recommends to subscribe to logs and send it somewhere else for processing like Kinesis, Lambda etc.

There we could implement a Lambda to send the data wherever we want.

Thanks

Provide Docker image

Any chance of providing a Docker image for this beat?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

e-travel / cloudwatchlogsbeat Goto Github PK

cloudwatchlogsbeat's People

Stargazers

Watchers

Forkers

cloudwatchlogsbeat's Issues

=== PROSPECTORS ===

Applications whose log groups we'd like to monitor

Recommend Projects

Recommend Topics

Recommend Org