mdoi / fluent-plugin-gcloud-pubsub Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
The following error can occur, stalling fluentd processing and (apparently) requiring a restart.
2016-07-28 14:54:59 +0000 [error]: unexpected error error="Net::OpenTimeout"
2016-07-28 14:54:59 +0000 [error]: /usr/local/lib/ruby/2.3.0/net/http.rb:930:in `connect'
2016-07-28 14:54:59 +0000 [error]: /usr/local/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
2016-07-28 14:54:59 +0000 [error]: /usr/local/lib/ruby/2.3.0/net/http.rb:852:in `start'
2016-07-28 14:54:59 +0000 [error]: /usr/local/lib/ruby/2.3.0/net/http.rb:1398:in `request'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:82:in `perform_request'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:40:in `block in call'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:32:in `call'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/faraday-0.9.2/lib/faraday/response.rb:8:in `call'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/google-api-client-0.8.6/lib/google/api_client/request.rb:163:in `send'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/google-api-client-0.8.6/lib/google/api_client.rb:648:in `block (2 levels) in execute!'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/google-api-client-0.8.6/lib/google/api_client.rb:645:in `block in execute!'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/google-api-client-0.8.6/lib/google/api_client.rb:636:in `execute!'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/google-api-client-0.8.6/lib/google/api_client.rb:679:in `execute'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/gcloud-0.6.3/lib/gcloud/pubsub/connection.rb:220:in `pull'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/gcloud-0.6.3/lib/gcloud/pubsub/subscription.rb:254:in `pull'
2016-07-28 14:54:59 +0000 [error]: /usr/local/bundle/gems/fluent-plugin-gcloud-pubsub-0.0.5/lib/fluent/plugin/in_gcloud_pubsub.rb:65:in `subscribe'
I used PubSub and BigQuery fluentd plugin together, and hit the dependnecy conflict issue.
glcoud 0.6.3 and google-api-client 0.9.3
I solved the conflict by changed the gcloud to the newest version.
2017-02-02 13:25:01 +0900 [error]: unexpected error error="SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A"
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:923:in `connect'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:923:in `block in connect'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/timeout.rb:75:in `timeout'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:923:in `connect'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:852:in `start'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:1375:in `request'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:82:in `perform_request'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:40:in `block in call'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:32:in `call'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.2/lib/faraday/response.rb:8:in `call'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client/request.rb:163:in `send'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client.rb:648:in `block (2 levels) in execute!'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client.rb:645:in `block in execute!'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client.rb:636:in `execute!'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client.rb:679:in `execute'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gcloud-0.6.3/lib/gcloud/pubsub/connection.rb:220:in `pull'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gcloud-0.6.3/lib/gcloud/pubsub/subscription.rb:254:in `pull'
2017-02-02 13:25:04 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-gcloud-pubsub-0.0.5/lib/fluent/plugin/in_gcloud_pubsub.rb:65:in `subscribe'
2017-02-06 16:30:51 +0900 [error]: unexpected error error="Connection reset by peer - SSL_connect"
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:923:in `connect'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:923:in `block in connect'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/timeout.rb:75:in `timeout'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:923:in `connect'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:852:in `start'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:1375:in `request'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:80:in `perform_request'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:38:in `block in call'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:85:in `with_net_http_connection'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:33:in `call'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.11.0/lib/faraday/response.rb:8:in `call'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client/request.rb:163:in `send'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client.rb:648:in `block (2 levels) in execute!'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client.rb:645:in `block in execute!'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client.rb:636:in `execute!'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client.rb:679:in `execute'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gcloud-0.6.3/lib/gcloud/pubsub/connection.rb:220:in `pull'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gcloud-0.6.3/lib/gcloud/pubsub/subscription.rb:254:in `pull'
2017-02-06 16:30:51 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-gcloud-pubsub-0.0.5/lib/fluent/plugin/in_gcloud_pubsub.rb:65:in `subscribe'
2017-02-07 21:14:39 +0900 [error]: unexpected error error="Connection reset by peer - SSL_connect"
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:923:in `connect'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:923:in `block in connect'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/timeout.rb:75:in `timeout'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:923:in `connect'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:852:in `start'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:1375:in `request'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:80:in `perform_request'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:38:in `block in call'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:85:in `with_net_http_connection'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:33:in `call'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.11.0/lib/faraday/response.rb:8:in `call'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client/request.rb:163:in `send'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client.rb:648:in `block (2 levels) in execute!'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client.rb:645:in `block in execute!'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client.rb:636:in `execute!'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.7/lib/google/api_client.rb:679:in `execute'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gcloud-0.6.3/lib/gcloud/pubsub/connection.rb:220:in `pull'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gcloud-0.6.3/lib/gcloud/pubsub/subscription.rb:254:in `pull'
2017-02-07 21:14:39 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-gcloud-pubsub-0.0.5/lib/fluent/plugin/in_gcloud_pubsub.rb:65:in `subscribe'
Mentioned in #8, out_gcloud_pubsub don't work properly in fluentd v0.14.0.
The error's stacktrace is below:
2016-06-14 11:06:54 +0900 [error]: unexpected error error="\"\\xE8\" from ASCII-8BIT to UTF-8"
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/lib/fluent/plugin/out_gcloud_pubsub.rb:49:in `encode'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/lib/fluent/plugin/out_gcloud_pubsub.rb:49:in `to_json'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/lib/fluent/plugin/out_gcloud_pubsub.rb:49:in `block in write'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/vendor/bundle/ruby/2.2.0/gems/fluentd-0.14.0/lib/fluent/event.rb:194:in `each'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/vendor/bundle/ruby/2.2.0/gems/fluentd-0.14.0/lib/fluent/event.rb:194:in `block in each'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/vendor/bundle/ruby/2.2.0/gems/fluentd-0.14.0/lib/fluent/plugin/buffer/memory_chunk.rb:90:in `open'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/vendor/bundle/ruby/2.2.0/gems/fluentd-0.14.0/lib/fluent/plugin/buffer/memory_chunk.rb:90:in `open'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/vendor/bundle/ruby/2.2.0/gems/fluentd-0.14.0/lib/fluent/event.rb:193:in `each'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/lib/fluent/plugin/out_gcloud_pubsub.rb:48:in `write'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/vendor/bundle/ruby/2.2.0/gems/fluentd-0.14.0/lib/fluent/compat/output.rb:118:in `write'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/vendor/bundle/ruby/2.2.0/gems/fluentd-0.14.0/lib/fluent/plugin/output.rb:778:in `try_flush'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/vendor/bundle/ruby/2.2.0/gems/fluentd-0.14.0/lib/fluent/plugin/output.rb:953:in `flush_thread_run'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/vendor/bundle/ruby/2.2.0/gems/fluentd-0.14.0/lib/fluent/plugin/output.rb:360:in `block (2 levels) in start'
2016-06-14 11:06:54 +0900 [error]: /path/to/fluent-plugin-gcloud-pubsub/vendor/bundle/ruby/2.2.0/gems/fluentd-0.14.0/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
My fluent.conf is below:
<source>
@type tail
path /path/to/dummy.log
pos_file /path/to/dummy.log.pos
tag test.testlog
format ltsv
</source>
<match test.**>
@type gcloud_pubsub
@id out_pubsub
project my-project
topic my-topic
autocreate_topic true
key /path/to/keyfile.json
</match>
Sample log:
id:0363 time:[2016-06-13 18:14:42] level:DEBUG method:PUT uri:/api/v1/people reqtime:3.6950025457668514 foobar:UNpTAPOS text:ใ
Hi,
I'm interested in use a Fluent out plugin for Google pub sub.
Searching in github I find 2 plugins:
Please, can you indicate what is the scope of this plugin and motivations to do not use shojika plugin. (Any differences?)
Thanks!
We are currently running into an issue with our Google Compute PubSub where our instances are all using PDT for time but our local Kafka servers are using EST. We would like to be able to specify the timezone of the source logs such that when the data is sent to Kafka the times can be in sync.
Example of file_output plugin here: http://docs.fluentd.org/articles/out_file
See timezone reference in link.
Hi I am using this output plugin to send different logs to different pub topics. following are my td-agent.conf
<match tag1.xxx>
@type copy
<store>
.....
</store>
<store>
type gcloud_pubsub
project my.project
topic my.topic1
key my.key
flush_interval 1
</store>
</match>
<match tag2.xxx>
@type copy
<store>
.....
</store>
<store>
type gcloud_pubsub
project my.project
topic my.topic2
key my.key
flush_interval 1
</store>
</match>
Then fluentd will keep restarting, looks like the configuration is conflict. Any ideas?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.