tuffk / logstash-output-to-azure-blob Goto Github PK
View Code? Open in Web Editor NEWPlugin to send Logstash output to Azure blob
License: Other
Plugin to send Logstash output to Azure blob
License: Other
is there an sample output config
:error=>#<NameError: undefined local variable or method upload_options' for #<LogStash::Outputs::LogstashAzureBlobOutput:0x24e4e3d>>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/azure.rb:248:in
upload_file'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/azure.rb:232:in block in rotate_if_needed'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/blob/file_repository.rb:94:in
block in get_factory'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/blob/file_repository.rb:30:in
block in with_lock'", "org/jruby/ext/thread/Mutex.java:165:in
synchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/blob/file_repository.rb:29:in with_lock'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/blob/file_repository.rb:94:in
get_factory'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/azure.rb:223:in block in rotate_if_needed'", "org/jruby/ext/set/RubySet.java:601:in
each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/azure.rb:219:in rotate_if_needed'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/azure.rb:154:in
multi_receive_encoded'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:87:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in
multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in
block in output_batch'", "org/jruby/RubyHash.java:1419:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in
output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in
block in start_workers'"]
hi,
after install the latest logstash-output-azure
with config like:
output {
stdout {}
azure {
storage_account_name => ""
storage_access_key => ""
contianer_name => ""
}
}
get error as follow:
[ERROR][logstash.outputs.logstashazurebloboutput] Unknown setting 'contianer_name' for azure
I tried to use this plugin with default configuration. It did not work properly, I got the output as follows. After several hours troubleshooting, it turn out that the default codes doesn't work with dynamic fields.
016-03-28T11:33:23.000Z 10.x.x.10 %{message}
2016-03-28T11:33:23.000Z 10.x.x.10 %{message}
2016-03-28T11:33:23.000Z 10.x.x.10 %{message}
2016-03-28T11:33:23.000Z 10.x.x.10 %{message}
2016-03-28T11:33:23.000Z 10.x.x.10 %{message}
After add the codec in the code, it works as my expectation. I noticed that this plugin comes from logstash-output-s3, but very old version. now in that plugin, the default codec has been added.
It would be good to add that one in the code as follows .
default :codec, "line"
or json format
hi,
please add config like "storage_dns_suffix" to support not global azure
blob_client = Azure::Storage::Blob::BlobService.create(
storage_account_name: account_name,
storage_access_key: account_key,
storage_dns_suffix: "core.chinacloudapi.cn"
)
Hi, i'm tried to configure container_name with variable in name for example %{+yyyy-MM-dd}
Unfortunately i got error
logstash_1 | [2020-12-30T09:13:16,012][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<URI::InvalidURIError: bad URI(is not URI?): https://ddd.blob.core.windows.net/logs-%{+yyyy-MM-dd}>, :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/rfc3986_parser.rb:67:in `split'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/rfc3986_parser.rb:73:in `parse'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/common.rb:237:in `parse'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/service.rb:40:in `generate_uri'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-0.7.10/lib/azure/blob/blob_service.rb:1422:in `container_uri'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-0.7.10/lib/azure/blob/blob_service.rb:111:in `create_container'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-2.1.0/lib/logstash/outputs/azure.rb:213:in `blob_container_resource'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-2.1.0/lib/logstash/outputs/azure.rb:129:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:585:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x5f20765e@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:54 run>"}
Is it possible to create container from plugin with variable in name like
container_name => "logs-s%{+YYYY-MM-dd}"
@tuffk I have an issue with the plugin: after the plugin starts, I'm getting an error:
[ERROR][logstash.outputs.logstashazurebloboutput][main] An error occured in the on_complete
uploader {:exception=>Errno::ENOTEMPTY, :message=>"Directory not empty - C:/Users/xxx/AppData/Local/Temp/logstash/37d3800e-f987-4372-9c12-226d33215bc6", :path=>"C:/Users/xxx/AppData/Local/Temp/logstash/37d3800e-f987-4372-9c12-226d33215bc6/2020-07-25T18.46.fef14fc5-cff8-424f-b6f1-ffb8c115c3c2.part0.txt", :backtrace=>["org/jruby/RubyDir.java:493:in rmdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1432:in
block in remove_dir1'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1446:in platform_support'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1431:in
remove_dir1'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1424:in remove'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:763:in
block in remove_entry'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1481:in postorder_traverse'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:761:in
remove_entry'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:681:in remove_entry_secure'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:611:in
block in rm_r'", "org/jruby/RubyArray.java:1809:in each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:609:in
rm_r'", "C:/Users/xxx/Downloads/logstash-7.8.0/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-2.1.0/lib/logstash/outputs/blob/temporary_file.rb:60:in delete!'", "C:/Users/xxx/Downloads/logstash-7.8.0/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-2.1.0/lib/logstash/outputs/azure.rb:267:in
clean_temporary_file'", "org/jruby/RubyMethod.java:131:in call'", "C:/Users/xxx/Downloads/logstash-7.8.0/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-2.1.0/lib/logstash/outputs/blob/uploader.rb:55:in
upload'", "C:/Users/xxx/Downloads/logstash-7.8.0/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-2.1.0/lib/logstash/outputs/blob/uploader.rb:32:in block in upload_async'", "C:/Users/xxx/Downloads/logstash-7.8.0/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79:in
run'"]}
A temp file is actually created under: AppData\Local\Temp\logstash\a4749586-b1e3-4940-bc55-55bb814bff08
and contains records of:
2020-07-25T15:35:56.382Z %{host} %{message}
The config looks like:
input {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "bank"
}
}
output {
file {
path => "./test.txt" // The data is written to this test file
}
azure {
storage_account_name => "XXX"
storage_access_key => "XXX"
container_name => "data"
}
}
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.