tuffk / logstash-output-to-azure-blob Goto Github PK

is there an sample output config

@tuffk I have an issue with the plugin: after the plugin starts, I'm getting an error:
[ERROR][logstash.outputs.logstashazurebloboutput][main] An error occured in the on_complete uploader {:exception=>Errno::ENOTEMPTY, :message=>"Directory not empty - C:/Users/xxx/AppData/Local/Temp/logstash/37d3800e-f987-4372-9c12-226d33215bc6", :path=>"C:/Users/xxx/AppData/Local/Temp/logstash/37d3800e-f987-4372-9c12-226d33215bc6/2020-07-25T18.46.fef14fc5-cff8-424f-b6f1-ffb8c115c3c2.part0.txt", :backtrace=>["org/jruby/RubyDir.java:493:in rmdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1432:in block in remove_dir1'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1446:in platform_support'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1431:in remove_dir1'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1424:in remove'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:763:in block in remove_entry'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1481:in postorder_traverse'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:761:in remove_entry'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:681:in remove_entry_secure'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:611:in block in rm_r'", "org/jruby/RubyArray.java:1809:in each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:609:in rm_r'", "C:/Users/xxx/Downloads/logstash-7.8.0/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-2.1.0/lib/logstash/outputs/blob/temporary_file.rb:60:in delete!'", "C:/Users/xxx/Downloads/logstash-7.8.0/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-2.1.0/lib/logstash/outputs/azure.rb:267:in clean_temporary_file'", "org/jruby/RubyMethod.java:131:in call'", "C:/Users/xxx/Downloads/logstash-7.8.0/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-2.1.0/lib/logstash/outputs/blob/uploader.rb:55:in upload'", "C:/Users/xxx/Downloads/logstash-7.8.0/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-2.1.0/lib/logstash/outputs/blob/uploader.rb:32:in block in upload_async'", "C:/Users/xxx/Downloads/logstash-7.8.0/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79:in run'"]}

A temp file is actually created under: AppData\Local\Temp\logstash\a4749586-b1e3-4940-bc55-55bb814bff08
and contains records of:
2020-07-25T15:35:56.382Z %{host} %{message}

The config looks like:
input {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "bank"
}
}
output {
file {
path => "./test.txt" // The data is written to this test file
}
azure {
storage_account_name => "XXX"
storage_access_key => "XXX"
container_name => "data"
}
}

• When I tested it with a generator input plugin, the same issue occurred (although the data was sent to Azure blob)
  Any ideas?

Thanks

hi,
please add config like "storage_dns_suffix" to support not global azure

blob_client = Azure::Storage::Blob::BlobService.create(
            storage_account_name: account_name,
            storage_access_key: account_key,
        storage_dns_suffix: "core.chinacloudapi.cn"
          )

hi,
after install the latest logstash-output-azure
with config like:

output {
    stdout {}
    azure {
      storage_account_name => ""
      storage_access_key  => ""
      contianer_name  => ""
    }
}

get error as follow:
[ERROR][logstash.outputs.logstashazurebloboutput] Unknown setting 'contianer_name' for azure

Hi, i'm tried to configure container_name with variable in name for example %{+yyyy-MM-dd}
Unfortunately i got error

logstash_1       | [2020-12-30T09:13:16,012][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<URI::InvalidURIError: bad URI(is not URI?): https://ddd.blob.core.windows.net/logs-%{+yyyy-MM-dd}>, :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/rfc3986_parser.rb:67:in `split'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/rfc3986_parser.rb:73:in `parse'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/common.rb:237:in `parse'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/service.rb:40:in `generate_uri'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-0.7.10/lib/azure/blob/blob_service.rb:1422:in `container_uri'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-0.7.10/lib/azure/blob/blob_service.rb:111:in `create_container'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-2.1.0/lib/logstash/outputs/azure.rb:213:in `blob_container_resource'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-2.1.0/lib/logstash/outputs/azure.rb:129:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:585:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x5f20765e@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:54 run>"}

Is it possible to create container from plugin with variable in name like

          container_name => "logs-s%{+YYYY-MM-dd}"

I tried to use this plugin with default configuration. It did not work properly, I got the output as follows. After several hours troubleshooting, it turn out that the default codes doesn't work with dynamic fields.

016-03-28T11:33:23.000Z 10.x.x.10 %{message}
2016-03-28T11:33:23.000Z 10.x.x.10 %{message}
2016-03-28T11:33:23.000Z 10.x.x.10 %{message}
2016-03-28T11:33:23.000Z 10.x.x.10 %{message}
2016-03-28T11:33:23.000Z 10.x.x.10 %{message}

After add the codec in the code, it works as my expectation. I noticed that this plugin comes from logstash-output-s3, but very old version. now in that plugin, the default codec has been added.

It would be good to add that one in the code as follows .
default :codec, "line"
or json format

:error=>#<NameError: undefined local variable or method upload_options' for #<LogStash::Outputs::LogstashAzureBlobOutput:0x24e4e3d>>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/azure.rb:248:in upload_file'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/azure.rb:232:in block in rotate_if_needed'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/blob/file_repository.rb:94:in block in get_factory'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/blob/file_repository.rb:30:in
block in with_lock'", "org/jruby/ext/thread/Mutex.java:165:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/blob/file_repository.rb:29:in with_lock'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/blob/file_repository.rb:94:in get_factory'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/azure.rb:223:in block in rotate_if_needed'", "org/jruby/ext/set/RubySet.java:601:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/azure.rb:219:in rotate_if_needed'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-azure-1.0.0/lib/logstash/outputs/azure.rb:154:in multi_receive_encoded'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:87:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in block in output_batch'", "org/jruby/RubyHash.java:1419:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in block in start_workers'"]