I am playing with this plugin with packetbeat, some of the messages send to firehose gets truncated or appended garbage characters. Which fails deliver to ES. Not sure why it is happening

firehose raw data: eyJAdGltZXN0YW1wIjoiMjAxOC0wNS0yMVQyMzoyODo0MC4wMDBaIiwiQG1ldGFkYXRhIjp7ImJlYXQiOiJwYWNrZXRiZWF0IiwidHlwZSI6ImRvYyIsInZlcnNpb24iOiI2LjIuNCJ9LCJmaW5hbCI6ZmFsc2UsImJlYXQiOnsibmFtZSI6ImlwLTEtMTEtMy0yMDIiLCJob3N0bmFtZSI6ImlwLTEtMTEtMy0yMDIiLCJ2ZXJzaW9uIjoiNi4yLjQifSwic291cmNlIjp7ImlwdjYiOiJmZTgwOjo0NDo3YmZmOmZlZDg6YTIyZSIsInBvcnQiOjU0Niwic3RhdHMiOnsibmV0X3BhY2tldHNfdG90YWwiOjEsIm5ldF9ieXRlc190b3RhbCI6MTE2fX0sImxhc3RfdGltZSI6IjIwMTgtMDUtMjFUMjM6Mjg6MDkuMTc5WiIsInN0YXJ0X3RpbWUiOiIyMDE4LTA1LTIxVDIzOjI4OjA5LjE3OVoiLCJ0eXBlIjoiZmxvdyIsImZsb3dfaWQiOiJRQUwvLy8vLy8vOEEvLzhnLy84QUFBSCtnQUFBQUFBQUFBQkVlLy8rMktJdS93SUFBQUFBQUFBQUFBQUFBQUVBQWlJQ0l3SSIsImZpZWxkcyI6eyJsb2dfc291cmNlIjoidGF5bG9yLXBhY2tldGJlYXQifSwibWV0YSI6eyJjbG91ZCI6eyJhdmFpbGFiaWxpdHlfem9uZSI6ImV1LXdlc3QtMWIiLCJwcm92aWRlciI6ImVjMiIsImluc3RhbmNlX2lkIjoiaS0wYWRjZmUxMTg1MmViMmQxNiIsIm1hY2hpbmVfdHlwZSI6InQyLm1lZGl1bSIsInJlZ2lvbiI6ImV1LXdlc3QtMSJ9fSwidHJhbnNwb3J0IjoidWRwIiwiZGVzdCI6eyJpcHY2IjoiZmYwMjo6MToyIiwicG9ydCI6NTQ3fX1BQUFGL0FBQUJmdw

json:
{"@timestamp":"2018-05-21T23:28:40.000Z","@metadata":{"beat":"packetbeat","type":"doc","version":"6.2.4"},"final":false,"beat":{"name":"ip-1-11-3-202","hostname":"ip-1-11-3-202","version":"6.2.4"},"source":{"ipv6":"fe80::44:7bff:fed8:a22e","port":546,"stats":{"net_packets_total":1,"net_bytes_total":116}},"last_time":"2018-05-21T23:28:09.179Z","start_time":"2018-05-21T23:28:09.179Z","type":"flow","flow_id":"QAL///////8A//8g//8AAAH+gAAAAAAAAABEe//+2KIu/wIAAAAAAAAAAAAAAAEAAiICIwI","fields":{"log_source":"packetbeat"},"meta":{"cloud":{"availability_zone":"eu-west-1b","provider":"ec2","instance_id":"i-xxxx","machine_type":"t2.medium","region":"eu-west-1"}},"transport":"udp","dest":{"ipv6":"ff02::1:2","port":547}}AAAF/AAABfw

Can someone help me how to fix this error message?

2019-02-13T05:13:34.287Z        DEBUG   [kinesis]       streams/client.go:114   failed to map event({{2019-02-13 05:13:30.925710489 +0000 UTC m=+0.052119993 null {"beat":{"hostname":"ip-xxxxxx.ec2.internal","name":"ip-xxxxxx.ec2.internal","version":"6.5.4"},"host":{"name":"ip-xxxxxx.ec2.internal"},"message":"Feb 10 03:47:01 ip-xxxxxx systemd: Removed slice User Slice of root.","meta":{"cloud":{"availability_zone":"us-east-1e","instance_id":"i-xxxxxxx","machine_type":"t2.large","provider":"ec2","region":"us-east-1"}},"offset":1215,"source":"/var/log/messages"} { false 0xc000337e10 /var/log/messages 1291 2019-02-13 05:13:30.923972221 +0000 UTC m=+0.050381713 -1ns log map[] 4321741-51713}} 1}): failed to get parititon key: failed to get parition key: key not found

Here's my configuration file:

`filebeat.prospectors:
  - input_type: log
    paths:
    - /var/log/messages
    document_type: syslog

processors:
- add_cloud_metadata:

output:
  streams:
    region: us-east-1
    stream_name: system-log-stream
    partition_key: testing1

queue.mem:
  events: 4096
  flush.min_events: 5
  flush.timeout: 3s

It seems that the backoff/retry configuration only applies to connection errors (e.g. no network), in which case libbeat's backoff policy kicks in and retries. However, since connect and close don't really make sense for Kinesis (and their implementation in the client is just a stub), this results in immediate retry.

So, here are a few examples of cases where the kinesis stream client enters an infinite cpu loop:

1. When the stream's throughput is exceeded, the events are retried immediately and in most cases result in more throughput errors (bcz no backoff).
2. When the stream's IAM permissions are missing, the error from Kinesis is permission denied, and the records are then retried immediately and infinitely, resulting in AWS API rate limiting.
3. When kinesis is hit too frequently (e.g. because of the above) the error is Rate Limit, in which case the client simply retries immediately, exacerbating the problem.

Unclear if this is a problem to solve at:

1. The output's level (add some retrying to put_records.
2. The aws sdk level (it knows how to backoff in certain circumstances)
3. The publisher

to recreate this problem, create a kinesis stream and deny putRecords permission to it to everyone, then feed a single input event to filebeat and see the cpu go to 100% and stay there.

@s12v Hi! Thank you very much for sharing an awesome plugin 👍

I was just looking into the code-base of this plugin, and realized that it supports Kinesis Firehose only.

Would you mind if I contribute a support for Kinesis Data Streams?
I believe it makes sense for this project to contain supports for both, like the fluentd-equivalent of this plugin aws-fluent-plugin-kinesis does.

I migrated multiple projects that used glide to use dep instead. All of them are working fine so far.
It seems to be the defacto standard and even glide itself is suggesting to migrate.

@s12v Would you mind if I contributed a PR for that? 😃

I've started coding ansible-awsbeats to support downloading/installing/configuring/running filebeat with the kinesis.so plugin via ansible.

It's very bare-bones at the moment, and I haven't started on docs, but it's working locally.

I'm currently using Kubernetes to deploy filebeat to listen to my logs. I use this manifest file:
https://github.com/elastic/beats/blob/master/deploy/kubernetes/filebeat-kubernetes.yaml

I would like to send my logs to kinesis via filebeat. How would I go about using awsbeats with my current setup?

I started to see persistent panics like the below in my production deployment:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x7fda4c8ce9b1]

goroutine 83 [running]:
github.com/s12v/awsbeats/streams.collectFailedEvents(0xc420ad1dd0, 0xc420f4b300, 0x32, 0x3b4, 0xc420ad1dd0, 0x0, 0x0)
        /go/src/github.com/s12v/awsbeats/streams/client.go:170 +0xd1
github.com/s12v/awsbeats/streams.(*client).publishEvents(0xc4200da930, 0xc420f4b300, 0x32, 0x3b4, 0x1d14c30, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/s12v/awsbeats/streams/client.go:92 +0x2be
github.com/s12v/awsbeats/streams.(*client).Publish(0xc4200da930, 0x14b9860, 0xc422b27940, 0xc42008ce40, 0xc420f69f78)
        /go/src/github.com/s12v/awsbeats/streams/client.go:64 +0x45
github.com/elastic/beats/libbeat/outputs.(*backoffClient).Publish(0xc42000d8a0, 0x14b9860, 0xc422b27940, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/outputs/backoff.go:43 +0x4b
github.com/elastic/beats/libbeat/publisher/pipeline.(*netClientWorker).run(0xc4200af280)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:90 +0x1a9
created by github.com/elastic/beats/libbeat/publisher/pipeline.makeClientWorker
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:31 +0xf0

This indicates that we're receiving records with unexpected structures from kinesis streams. I have no reference to the concrete specification of kinesis records that I'm unable to "fix" it.

In the meantime, I'd make awsbeats gracefully degrade, that is to give up retrying failed records with unexpected structures but just leave some log messsage to help further investigation.

.

How about renaming to kinesisbeats or even kinesisbeat so that it is clearer that this project covers aws kinesis sub-services only?

I am keep getting this error.
streams/client.go:97 retrying 5 events on error: failed to put records: InvalidParameter: 1 validation error(s) found. minimum field size of 1, PutRecordsInput.Records

Below is my filebeat.yml file. I couldn't figure out what is the error.

filebeat.prospectors:
   input_type: log
    paths:
    /tmp/test_beat/*.log
       json.keys_under_root: true
           json.add_error_key: true
processors:
- add_cloud_metadata:

output:
  streams:
    enabled: true
    region: XX
    stream_name: XX
    shard_count: XX
    partition_key: XX
    access_key_id: XXX
    secret_access_key: XXX

queue.mem:
  events: 100
  flush.min_events: 5
  flush.timeout: 3s

logging.level: info
logging.to_syslog: false
logging.to_files: true
logging.files:
  path: /tmp/
  name: filebeat.log

`

I've submitted the heartbeat chart to the k8s charts repository.
How about adding an example of it and awsbeats?

That is, running any beat with our usual command heartbeat --plugin kinesis.so within a container fails. The docker image is built by running make heartbeat-image, and
the error message is flag redefined: strict.perms.

As far as I remember, this error happens when when you built beat and awsbeats against different version of libbeat. Actually I encountered it before while implementing #17, and thought I had managed to fix it completely. Probably something has changed externally since then, which revived the issue.

I'll submit a fix asap.

Right now the version that the dockerfile uses is 6.5
There have been many upgrades since and the current version should be 7.0

When running go build, I got the following error message:
make variables for this build:
DOCKER_TAG = 0.2.10-v6.5.0-"filebeat"-v6.5.4
GO_PLATFORM = linux-amd64
AWSBEATS_VERSION = 0.2.10-v6.5.0
GO_VERSION = 1.11.4
DOCKER_IMAGE = s12v/awsbeats
BEAT_GITHUB_REPO = github.com/elastic/beats
BEAT_GO_PKG = github.com/elastic/beats/"filebeat"
BEATS_TAG = v6.5.4
BEAT_NAME = "filebeat"
BEAT_DOCKER_IMAGE = docker.elastic.co/beats/"filebeat":v6.5.4
To override, run make like 'make SOME_VAR=some_value'.
Building filebeat:v6.5.4...
Previous HEAD position was 75b5dd3... Close changelog for the 6.1.3 release (#6163)
HEAD is now at 9238415... Update testing env 6.5.4 (#9631)
make[1]: Entering directory '/home/linxlogic/go_projects/src/github.com/elastic/beats/filebeat'
go build -i -ldflags "-X github.com/elastic/beats/libbeat/version.buildTime=2018-12-21T00:16:30Z -X github.com/elastic/beats/libbeat/version.commit=923841526f8c32b10040eddbe9a3ecfc7772da4f"
make[1]: Leaving directory '/home/linxlogic/go_projects/src/github.com/elastic/beats/filebeat'
test -z "$(find . -path ./vendor -prune -type f -o -name '*.go' -exec gofmt -d {} + | tee /dev/stderr)"
go test ./firehose -v -coverprofile=coverage.txt -covermode=atomic

github.com/s12v/awsbeats/firehose [github.com/s12v/awsbeats/firehose.test]

firehose/firehose.go:32: cannot assign *client to client (type outputs.NetworkClient) in multiple assignment:
*client does not implement outputs.NetworkClient (missing String method)
FAIL github.com/s12v/awsbeats/firehose [build failed]
Makefile:26: recipe for target 'test' failed
make: *** [test] Error 2

As per me and @s12v's discussion in #8, probably we can include CloudWatch Logs output plugin within awsbeats.
Anyone has a specific use-case for that? Any concrete feature you'd like to have?

Release https://github.com/s12v/awsbeats/releases/tag/0.2.2-v6.2.4

Expected: kinesis-0.2.2-v6.2.4-go1.10-linux-amd64.so
Actual: kinesis-0.2.2-v6.2.4-go1.10-.so

linux-amd64 is missing

I occasionally see interesting errors like this:

2018-05-27T13:49:36.652Z        DEBUG   [kinesis]       streams/client.go:56    mapped to records: [{
  Data: <binary> len 1191,
  PartitionKey: "/var/lib/docker/containers/cb718fd1316624b898befeb767dd22af9dd3db4fba34c8b8f8f169cd196c5d85/cb718fd1316624b898befeb767dd22af9dd3db4fba34c8b8f8f169cd196c5d85-json.log"
} {
<中略>
} {
  Data: <binary> len 969,
  PartitionKey: "/var/lib/docker/containers/03b7c50f36b202f66f3f08fe267fc6f913f12969da348d4d4e348a9865aaf2a7/03b7c50f36b202f66f3f08fe267fc6f913f12969da348d4d4e348a9865aaf2a7-json.log"
} {
  Data: <binary> len 1131,
  PartitionKey: "/var/lib/docker/containers/6dc39940d6815b9578af296b251ab0f2a53981f318b9703155efd672214f711f/6dc39940d6815b9578af296b251ab0f2a53981f318b9703155efd672214f711f-json.log"
} {
  Data: <binary> len 3367,
  PartitionKey: "/var/lib/docker/containers/beaada804640bfdfd5c5be5cc684d60044b27891e05577b1556af65fb27c805a/beaada804640bfdfd5c5be5cc684d60044b27891e05577b1556af65fb27c805a-json.log"
}]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x7efeeda62cd6]

goroutine 76 [running]:
github.com/s12v/awsbeats/streams.processFailedDeliveries(0xc42128f980, 0x155dbe0, 0xc420554280)
        /go/src/github.com/s12v/awsbeats/streams/client.go:129 +0xe6
github.com/s12v/awsbeats/streams.(*client).Publish(0xc4202bad80, 0x155dbe0, 0xc420554280, 0xc420198e40, 0xc42023bf78)
        /go/src/github.com/s12v/awsbeats/streams/client.go:63 +0x386
github.com/elastic/beats/libbeat/outputs.(*backoffClient).Publish(0xc42011d260, 0x155dbe0, 0xc420554280, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/outputs/backoff.go:43 +0x4b
github.com/elastic/beats/libbeat/publisher/pipeline.(*netClientWorker).run(0xc420082800)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:90 +0x1a9
created by github.com/elastic/beats/libbeat/publisher/pipeline.makeClientWorker
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:31 +0xf0

In theory, this happens only when there is at least one nil inside records in processFailedDeliveries:

		for i, r := range records {
			if *r.ErrorCode != "" {
				failedEvents = append(failedEvents, events[i])
			}
		}

I can hardly believe it there's such a bug in aws-sdk-go, but to gather information regarding the issue, I'd like to handle nil here and emit some logs for our note.

#17 added Dockerfile which is capable of building custom filebeat image, based of elastic.co/filebeat image, along with awsbeat.

I'd love it if any new tag results in a docker image automatically pubished to dockerhub. Probably we can do that with the script deploy provider.

@s12v Would you mind setting it up?
If you're short on time, I can write .traivls.yml. Then all we need is your DOCKER_USER and DOCKER_PASSWORD within travis env for docker login.

Hi,

Just wondering if you have encounter this error for filebeat-kinesis

INFO streams/client.go:97 retrying 1 events on error: failed to put records: InvalidSignatureException: Signature not yet current: 20181017T093750Z is still later than 20181017T093639Z (20181017T093139Z + 5 min.)

We have encounter this in version 0.2.5-v6.3.2 of filebeat and still persist in version 0.2.7-v6.4.0.

Hope you can help us with this. Thank you!

As I have contributed it to the official charts repo 😉

Any reasons why assume role is not supported?

Example build: https://travis-ci.org/s12v/awsbeats/builds/382528626

Error:

Building filebeats:v6.2.4...
error: pathspec 'vv6.2.4' did not match any file(s) known to git.
make: *** [beats] Error 1

vv6.2.4 doesn't look good, something is wrong in Makefile

Hi,

I have been using this plugin for a while and it has worked great!

Is there any plan to support 7.3.0 anytime soon?

Thanks

Elastic APM Server seems to be actually a beat. It should be technically possible to build the APM server along with the plugin like other beats. Sending APM transactions and spans via e.g. the Kinesis Streams output plugin would be nice, isn't it? :)

The build script would diverge from the one for other beats though. It isn't included on the official elastic/beats repository. It has a specific version of libbeats vendored.

Hello

I have configured kinesis plugin to access firehose, but can not find way to pass relevant profile.
Is there additional configuration to set specific user with wright access to the expected stream?

something similar to the following sample
output.firehose:
region: us-east-1
stream_name: my_stream
access_key_id: xxx
secret_access_key: xxx

I have also tried to set default was credentials, but got error
"NoCredentialProviders: no valid providers in chain"

Please advice

Hi, Will winlogbeat be supported in the future?

I'm seeing very interesting bug on awsbeat(filebeat 6.2.4 built w/ golang 1.10.2 for linux-amd64).

In nutshell, most of your records have invalid data.

To reproduce this issue, send 5 distinct records to the stream:

$ for i in $(seq 5); do str="${str}${i}"; echo '{"mykey":"mykey4","myvalue":"'$i'"}' | tee -a logs/foo.log; done

{"mykey":"mykey4","myvalue":"1"}
{"mykey":"mykey4","myvalue":"2"}
{"mykey":"mykey4","myvalue":"3"}
{"mykey":"mykey4","myvalue":"4"}
{"mykey":"mykey4","myvalue":"5"}

Tail the stream:

kinesis-tail -stream kuokatest1

And then, you'll surprisingly see the last record({"mykey":"mykey4","myvalue":"5"}) is repeated 5x times!

ApproximateArrivalTimestamp: 2018-05-22 09:45:51 +0000 UTC
Data: {"@timestamp":"2018-05-22T09:45:50.888Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.4"},"mykey":"mykey4","myvalue":"5","source":"/mnt/log/foo.log","beat":{"name":"bd98d11e2e3b","hostname":"bd98d11e2e3b","version":"6.2.4"},"offset":165
SequenceNumber: 49584698524577875697015278055900189069985746083771318274

ApproximateArrivalTimestamp: 2018-05-22 09:45:51 +0000 UTC
Data: {"@timestamp":"2018-05-22T09:45:50.888Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.4"},"mykey":"mykey4","myvalue":"5","source":"/mnt/log/foo.log","beat":{"name":"bd98d11e2e3b","hostname":"bd98d11e2e3b","version":"6.2.4"},"offset":165
SequenceNumber: 49584698524577875697015278055901397995805360712946024450

ApproximateArrivalTimestamp: 2018-05-22 09:45:51 +0000 UTC
Data: {"@timestamp":"2018-05-22T09:45:50.888Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.4"},"mykey":"mykey4","myvalue":"5","source":"/mnt/log/foo.log","beat":{"name":"bd98d11e2e3b","hostname":"bd98d11e2e3b","version":"6.2.4"},"offset":165
SequenceNumber: 49584698524577875697015278055902606921624975342120730626

ApproximateArrivalTimestamp: 2018-05-22 09:45:51 +0000 UTC
Data: {"@timestamp":"2018-05-22T09:45:50.888Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.4"},"mykey":"mykey4","myvalue":"5","source":"/mnt/log/foo.log","beat":{"name":"bd98d11e2e3b","hostname":"bd98d11e2e3b","version":"6.2.4"},"offset":165}
SequenceNumber: 49584698524577875697015278055903815847444589971295436802

ApproximateArrivalTimestamp: 2018-05-22 09:45:51 +0000 UTC
Data: {"@timestamp":"2018-05-22T09:45:50.888Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.4"},"mykey":"mykey4","myvalue":"5","source":"/mnt/log/foo.log","beat":{"name":"bd98d11e2e3b","hostname":"bd98d11e2e3b","version":"6.2.4"},"offset":165}
SequenceNumber: 49584698524577875697015278055905024773264204600470142978

This is due to the misuse of libbeat/outputs/codec/json here.

You should never do this:

serializedEvent, err := client.encoder.Encode(client.beatName, &event.Content)
// ...
return &kinesis.PutRecordsRequestEntry{Data: serializedEvent, PartitionKey: aws.String(partitionKey)}, nil

Instead you should do:

serializedEvent, err := client.encoder.Encode(client.beatName, &event.Content)
// ...
data := make([]byte, len(serializedEvent))
copy(data, serializedEvent)
return &kinesis.PutRecordsRequestEntry{Data: data, PartitionKey: aws.String(partitionKey)}, nil

The json codec seems to be designed that way, as we can see in the implementation of the official kafka output plugin, and the json codec's usage of bytes.Buffer.Reset().

The same thing applies to awsbeat's firehose plugin, as I have pointed out in #10.

After #28, it seems like filebeat+awsbeats+kinesis data streams occasionally fail while retrying batch send to Kinesis:

2018-05-28T03:35:56.266Z	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":90,"time":97},"total":{"ticks":710,"time":723,"value":710},"user":{"ticks":620,"time":626}},"info":{"ephemeral_id":"6703632b-8559-4104-8e5a-5b9d64489e08","uptime":{"ms":60027}},"memstats":{"gc_next":18773184,"memory_alloc":13742984,"memory_total":136054552,"rss":27189248}},"filebeat":{"events":{"active":4568,"added":4578,"done":10},"harvester":{"open_files":30,"running":30,"started":30}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":2},"output":{"events":{"acked":4146,"batches":83,"total":4146}},"pipeline":{"clients":4,"events":{"active":4067,"filtered":404,"published":4166,"retry":50,"total":4570},"queue":{"acked":100}}},"registrar":{"states":{"current":70,"update":10},"writes":10},"system":{"load":{"1":8.43,"15":7.16,"5":7.46,"norm":{"1":4.215,"15":3.58,"5":3.73}}}}}}
2018-05-28T03:35:58.686Z	INFO	log/harvester.go:216	Harvester started for file: /var/lib/docker/containers/26bbe1f1660ab24647e5d0342d700fe820776312d3fb4603e07831a6eceb1dbf/26bbe1f1660ab24647e5d0342d700fe820776312d3fb4603e07831a6eceb1dbf-json.log
2018-05-28T03:36:12.918Z	INFO	log/harvester.go:216	Harvester started for file: /var/lib/docker/containers/26bbe1f1660ab24647e5d0342d700fe820776312d3fb4603e07831a6eceb1dbf/26bbe1f1660ab24647e5d0342d700fe820776312d3fb4603e07831a6eceb1dbf-json.log
2018-05-28T03:36:17.215Z	ERROR	registrar/registrar.go:246	Writing of registry returned error: rename /usr/share/filebeat/data/registry.new /usr/share/filebeat/data/registry: no such file or directory. Continuing...
2018-05-28T03:36:17.465Z	ERROR	registrar/registrar.go:246	Writing of registry returned error: rename /usr/share/filebeat/data/registry.new /usr/share/filebeat/data/registry: no such file or directory. Continuing...
2018-05-28T03:36:21.155Z	ERROR	registrar/registrar.go:246	Writing of registry returned error: rename /usr/share/filebeat/data/registry.new /usr/share/filebeat/data/registry: no such file or directory. Continuing...
2018-05-28T03:36:22.237Z	WARN	streams/client.go:144	Retrying 50 events
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xd74489]

goroutine 71 [running]:
github.com/elastic/beats/libbeat/publisher/pipeline.(*Batch).Drop(0xc42168a000)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/batch.go:59 +0x29
github.com/elastic/beats/libbeat/publisher/pipeline.(*retryer).loop(0xc420139fc0)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:140 +0x4ff
created by github.com/elastic/beats/libbeat/publisher/pipeline.newRetryer
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:70 +0x138

Hello,

I've been trying to use your firehose output to send to firehose -> s3 -> logstash, but I noticed some missing events. On closer inspection I noticed the contents of the s3 objects have lines with multiple json objects without any delimiter. Logstash doesn't complain about this; it processes the first event, then silently drops the rest of the events on any given line. The aws kinesis agent produces newline-delimited logs, so I'd expect awsbeats to do the same.

I've had a quick look through the code and I noticed this for the streams output:
https://github.com/s12v/awsbeats/blob/master/streams/client.go#L138

But there is no corresponding manipulation for firehose/client.go. I tried an ignorant copy-paste of that line into what looked like the correct place in firehose/client.go but I ended up with broken json for some reason.

Am I missing something, or is that a bug?

Cheers,

Matt