phutchins / logstash-input-mongodb Goto Github PK

MongoDB input plugin for Logstash

License: Other

Ruby 100.00%

logstash-input-mongodb's Introduction

Logstash Plugin

This is a plugin for Logstash.

It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.

Documentation

This is a logstash plugin for pulling data out of mongodb and processing with logstash. It will connect to the database specified in uri, use the collection attribute to find collections to pull documents from, start at the first collection it finds and pull the number of documents specified in batch_size, save it's progress in an sqlite database who's location is specified by placeholder_db_dir and placeholder_db_name and repeat. It will continue this until it no longer finds documents newer than ones that it has processed, sleep for a moment, then continue to loop over the collections.

This was designed for parsing logs that were written into mongodb. This means that it may not re-parse db entries that were changed and already parsed.

Installation

Logstash installed from ZIP | TGZ
- bin/plugin install /path/to/logstash-input-mongodb-0.3.0.gem
Logstash from GIT
- git clone https://github.com/elastic/logstash.git
- cd logstash
- (ensure that the correct jruby is installed for the version of logstash you are installing)
- rake test:install-core
- bin/plugin install /path/to/logstash-input-mongodb-0.3.0.gem
- bin/plugin install --development

Configuration Options

Name                 Type          Description
uri                  [String]      A MongoDB URI for your database or cluster (check the MongoDB documentation for further info on this) [No Default, Required]
placeholder_db_dir   [String]      Path where the place holder database will be stored locally to disk [No Default, Required]
  This gets created by the plugin so the directory needs to be writeable by the user that logstash is running as
placeholder_db_name  [String]      Name of the database file that will be created [Default: logstash_sqlite.db]
collection           [String]      A regex that will be used to find desired collecitons. [No Default, Required]
generateId           [Boolean]     If true, this will add a field '_id' that contains the MongoDB Document id
batch_size           [Int]         Size of the batch of mongo documents to pull at a time [Default: 30]
parse_method         [String]      Built in parsing of the mongodb document object [Default: 'flatten']
dig_fields           [Array]       An array of fields that should employ the dig method
dig_dig_fields       [Array]       This provides a second level of hash flattening after the initial dig has been done

Configuration

Example

input {
  mongodb {
    uri => 'mongodb://10.0.0.30/my-logs?ssl=true'
    placeholder_db_dir => '/opt/logstash-mongodb/'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'events_'
    batch_size => 5000
  }
}

filter {
  date {
    match => [ "logdate", "ISO8601" ]
  }
}

output {
  redis {
    host => "localhost"
    data_type => "list"
    key => "logstash-mylogs"
  }
}

MongoDB URI

The URI parameter is where you would specify all of your mongodb options including things like auth and SSL. You should use a connection string (URI) compatible with the mongodb spec.

For more information on MongoDB URI's please see the MongoDB documentation: https://docs.mongodb.org/v3.0/reference/connection-string/

logstash-input-mongodb's People

Contributors

Stargazers

Watchers

Forkers

vinsidious trywildcard phamthaithinh jatanpatel92 gairom puneethn louis-bompart dominicscimeca inoxth zqzjz0911 nreese wx7614140 bogdangi kurkop gimhaniu 1mdata fnet123 piraxis wompy joniba sallyruthstruik inetufo sulthonzh phirov atulagrawal pigri mglee72 witnesslq nebalceroc tomerdvir fabriciomassula prudvihemanth erettozi marianogabriels sabya1981 belisarus songshuangkk nuvi cyrusmg abhiroyq1 breaktimetaiwan mkameni yangzhaoboyu i-liu stone-wlg gerasim13 shanzhaigege iriveroaks echocxw foxmaik fuzhong1990 sylvain-mazet doytsujin pesang72 zjhiphop biaotang saitejareddy007 hyenaproto pakool jackyken deadwalking7775 cashflow5 asilvis jizhang1 wangahodaye zhangpng hanwei996 hizhangbo paymentworks cristiprg faghihian71 9253761 bgpgfelipe baruch2008 gadueduik fmalykh mikychow mrotan hungbang maksymoryshchuk handl3r carset vincetrumental jekbon ayb3rk adileo oguzozgen usench kin9jon9 wisnubaldas ishfaqhaider setnext fatmaashraframadan zhangyingwei scriptmaster donniexu0918 harsh-ssm

logstash-input-mongodb's Issues

Retain complex JSON structure from Mongo

@phutchins My mongo contains complex nested JSON documents that I am trying to index into Elastic search (with a predefined mapping). By using the plugin in either 'flatten' or 'dig' mode, the complex JSON structure is lost. For example, array of nested json documents are represented as a string so Elastic does apply the mapping against it.

Any thoughts on how to retain the structure from Mongo without any flattening ?

Thanks!

MONGODB | host being removed from the cluster.

Hey, I've got an issue with connection between LogStash and MongoDB.

MongoDB Version 3.3.9
Logstash Version 2.3.4-1

2016-07-21T14:03:54.930913520Z Settings: Default pipeline workers: 2
2016-07-21T14:03:55.346991872Z D, [2016-07-21T14:03:55.346000 #1] DEBUG -- : MONGODB | Adding IP:27018 to the cluster.
2016-07-21T14:03:55.397462102Z D, [2016-07-21T14:03:55.397000 #1] DEBUG -- : MONGODB | Adding 59dab37a6e17:27017 to the cluster.
2016-07-21T14:03:55.582269918Z D, [2016-07-21T14:03:55.581000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:03:55.586333543Z D, [2016-07-21T14:03:55.585000 #1] DEBUG -- : MONGODB | Server IP:27018 discovered as primary in replica set: 'rs0'. Changing topology to replica set.
2016-07-21T14:03:55.681482935Z D, [2016-07-21T14:03:55.680000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:55.690459992Z D, [2016-07-21T14:03:55.687000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.004s
2016-07-21T14:03:55.823540480Z D, [2016-07-21T14:03:55.821000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:55.838810649Z D, [2016-07-21T14:03:55.838000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.013s
2016-07-21T14:03:55.878286624Z D, [2016-07-21T14:03:55.877000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:55.881207073Z D, [2016-07-21T14:03:55.880000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.002s
2016-07-21T14:03:55.934179617Z D, [2016-07-21T14:03:55.933000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:55.936905464Z D, [2016-07-21T14:03:55.936000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.002s
2016-07-21T14:03:55.948433473Z D, [2016-07-21T14:03:55.947000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:55.951456885Z D, [2016-07-21T14:03:55.950000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.002s
2016-07-21T14:03:56.026110027Z D, [2016-07-21T14:03:56.024000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:56.030478215Z D, [2016-07-21T14:03:56.029000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.002s
2016-07-21T14:03:56.047168308Z D, [2016-07-21T14:03:56.046000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:56.051397686Z D, [2016-07-21T14:03:56.049000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.003s
2016-07-21T14:03:56.166734805Z D, [2016-07-21T14:03:56.166000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:56.168845318Z D, [2016-07-21T14:03:56.168000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.002s
2016-07-21T14:03:56.179821637Z D, [2016-07-21T14:03:56.179000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:56.182601771Z D, [2016-07-21T14:03:56.182000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.003s
2016-07-21T14:03:56.360019595Z D, [2016-07-21T14:03:56.357000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:56.363478803Z D, [2016-07-21T14:03:56.362000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.003s
2016-07-21T14:03:56.371330707Z D, [2016-07-21T14:03:56.371000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:56.373536476Z D, [2016-07-21T14:03:56.373000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.002s
2016-07-21T14:03:56.534728827Z Pipeline main started
2016-07-21T14:03:56.734602817Z D, [2016-07-21T14:03:56.734000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:56.737201519Z D, [2016-07-21T14:03:56.737000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.002s
2016-07-21T14:03:56.748718347Z D, [2016-07-21T14:03:56.748000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:56.751771200Z D, [2016-07-21T14:03:56.750000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.001s
2016-07-21T14:03:57.418523856Z D, [2016-07-21T14:03:57.417000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:57.426493735Z D, [2016-07-21T14:03:57.426000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.007s
2016-07-21T14:03:57.435871242Z D, [2016-07-21T14:03:57.435000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:57.445850020Z D, [2016-07-21T14:03:57.441000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.006s
2016-07-21T14:03:58.811576284Z D, [2016-07-21T14:03:58.811000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:58.816773559Z D, [2016-07-21T14:03:58.816000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.004s
2016-07-21T14:03:58.828617310Z D, [2016-07-21T14:03:58.828000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:58.838503594Z D, [2016-07-21T14:03:58.838000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.009999999999999998s
2016-07-21T14:04:01.451048043Z D, [2016-07-21T14:04:01.450000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:04:01.457235672Z D, [2016-07-21T14:04:01.456000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.005s
2016-07-21T14:04:01.476911916Z D, [2016-07-21T14:04:01.476000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:04:01.479310542Z D, [2016-07-21T14:04:01.478000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.002s
2016-07-21T14:04:05.592725429Z D, [2016-07-21T14:04:05.592000 #1] DEBUG -- : MONGODB | IP:27018 being removed from the cluster.
2016-07-21T14:04:10.848395264Z D, [2016-07-21T14:04:10.847000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:04:10.850026777Z D, [2016-07-21T14:04:10.849000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:04:36.680396897Z D, [2016-07-21T14:04:36.679000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:04:36.685525505Z �[33mMongoDB Input threw an exception, restarting {:exception=>#<Mongo::Error::NoServerAvailable: No server is available matching preference: #<Mongo::ServerSelector::Primary:0x206510 tag_sets=[] server_selection_timeout=30 local_threshold=0.015>>, :level=>:warn}�[0m
2016-07-21T14:04:37.180376679Z D, [2016-07-21T14:04:37.179000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:05:07.126564411Z D, [2016-07-21T14:05:07.126000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:05:07.130668424Z �[33mMongoDB Input threw an exception, restarting {:exception=>#<Mongo::Error::NoServerAvailable: No server is available matching preference: #<Mongo::ServerSelector::Primary:0x206510 tag_sets=[] server_selection_timeout=30 local_threshold=0.015>>, :level=>:warn}�[0m
2016-07-21T14:05:07.133406692Z D, [2016-07-21T14:05:07.132000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:05:36.141040717Z D, [2016-07-21T14:05:36.140000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:05:36.640674677Z D, [2016-07-21T14:05:36.640000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:05:37.140718973Z D, [2016-07-21T14:05:37.140000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:05:37.144758436Z �[33mMongoDB Input threw an exception, restarting {:exception=>#<Mongo::Error::NoServerAvailable: No server is available matching preference: #<Mongo::ServerSelector::Primary:0x206510 tag_sets=[] server_selection_timeout=30 local_threshold=0.015>>, :level=>:warn}�[0m
2016-07-21T14:05:37.640441332Z D, [2016-07-21T14:05:37.640000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known


input {
  mongodb {
    uri => 'mongodb://IP:27018/routine_reports?replicaSet=rs0'
    placeholder_db_dir => '/opt/logstash-mongodb/'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => '^reports$'
    batch_size => 5000
  }
}

output {
  stdout { codec => rubydebug }

  file {
      path => "/opt/logstash-mongodb/mongo2es.log"
  }

  elasticsearch {
    index => "routine_reports"
    document_type => "test_report"
    document_id => "%{mongo_id}"
    hosts => ["IP:9201"]
  }
}

I'm not sure what's missing (or it's a bug?). Any help? Thanks in advance!

Regards,
drptbl.

Exception occurred

Hello

How to fix this exception:

D, [2016-04-07T22:37:20.095000 #1] DEBUG -- : MONGODB | my-db:27017 | my.find | SUCCEEDED | 0.006s
MongoDB Input threw an exception, restarting {:exception=>#<TypeError: The field '@timestamp' must be a (LogStash::Timestamp, not a String ("2016-04-03T15:40:39.794Z")>, :level=>:warn}
D, [2016-04-07T22:37:20.099000 #1] DEBUG -- : MONGODB | my-db:27017 | my.find | STARTED | {"find"=>"coll", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5701397834ba8f0b61000001')}}, "limit"=>5000}

How to select or exclude fields in collection!

Thx

Exception: BSON::ObjectId::Invalid: 'Infinity' is an invalid ObjectId

I get this exception in loop (I'm running v0.3.3):
{:timestamp=>"2016-04-21T01:15:09.431000+0000", :message=>"MongoDB Input threw an exception, restarting", :exception=>#<BSON::ObjectId::Invalid: 'Infinity' is an invalid ObjectId.>, :level=>:warn}

and no document are indexed after that.
Tried to search for any invalid ID but can't find one. The IDs are given by Mongo, I'm not setting any custom IDs. Also run query in Mongo: ({"_id":{$not:{$type:7}}}) which return empty.

Anyone else get this exception??
Please help.

Input config - Mongo specific collection

Hi,
Is there any way to config a specific collection instead of the regex?
So if I have these collections: hits, hits-old, hits-1, I want to be able to define "hits" and get docs only from this collection without the other collections.
Thanks!

How to handle multi dimensional objects

I have field in mongodb that contains json object.
How can I save this document structure into elastic?
Example:

in mongodb 
"location" : {
    "lon" : -71.715313,
    "lat" : 42.286992
  }

parse_method => "flatten" returns

 "location": "{"lon"=>-71.715313, "lat"=>42.286992}"

parse_method => "dig" returns

 "location": "{"lon"=>-71.715313, "lat"=>42.286992}"

parse_method => "simple" returns

  "location_lon": "-71.715313",
   "location_lat": "42.286992",

How can I leave same json structure?

"location" : {
    "lon" : -71.715313,
    "lat" : 42.286992
  }

How to bypass the SSL certificate like ssl_cert_reqs=ssl.CERT_NONE

uri => 'mongodb://uu:[email protected]:7397,10.14.5.130:7397,10.14.5.131:7397/shinken_dev2?ssl=true

Error:

D, [2016-11-22T02:16:44.288000 #28422] DEBUG -- : MONGODB | SSL handshake failed. MongoDB may not be configured with SSL support.

Are there any build instructions?

I am trying to build logstash-input-mongodb. Are there any build instructions?

I have been using Elastic's guide. Below are the steps I have followed so far

Set up development envirnoment
clone code base git clone https://github.com/phutchins/logstash-input-mongodb.git
cd logstash-input-mongodb
rm Gemfile.lock
bundle install
Run tests bundle exec rspec
Build plugin gem build logstash-input-mongodb.gemspec

I am unable to successfully run the tests. When I run bundle exec rspec I get the following error

Using Accessor#strict_set for specs
NameError: uninitialized constant LogStash::Inputs::Mongodb
    const_missing at org/jruby/RubyModule.java:3212
            <top> at /Users/nreese/projects/logstash-input-mongodb/spec/inputs/mongodb_spec.rb:10
             load at org/jruby/RubyKernel.java:955
  block in (root) at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1
             each at org/jruby/RubyArray.java:1560
  load_spec_files at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105
  load_spec_files at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105
            setup at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:96
              run at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:84
              run at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:69
            <top> at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:37
             load at org/jruby/RubyKernel.java:955
            <top> at /Users/nreese/projects/jruby-9.0.5.0/bin/rspec:23

Looks like the test file can not find the class LogStash::Inputs::Mongodb. Am I missing a build step?

DB_Auth don't work

Hi.

I've got a problem regarding the authentication with the MongoDB.

The tools I use:

MongoDB: 3.0.6
Elasticsearch: 1.7.2
Logstash: 1.5.4
logstash-input-mongodb: logstash-input-mongodb-0.2.0.gem

This is the output of my log file containing the problem:

D, [2015-09-15T14:13:25.455000 #21908] DEBUG -- : MONGODB | Adding team-db1:27017 to the cluster.
D, [2015-09-15T14:13:25.480000 #21908] DEBUG -- : MONGODB | Adding team-db2:27017 to the cluster.
D, [2015-09-15T14:13:25.523000 #21908] DEBUG -- : MONGODB | Server team-db1:27017 discovered as primary in replica set: 'team_replSet'. Changing topology to replica set.
D, [2015-09-15T14:13:25.558000 #21908] DEBUG -- : MONGODB | team-db1:27017 | team.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
D, [2015-09-15T14:13:25.561000 #21908] DEBUG -- : MONGODB | team-db1:27017 | team.listCollections | FAILED | not authorized on team to execute command { listCollections: 1, cursor: {}, filter: { name: { $not: /system\.|\$/ } } } (13) | 0.002s
The error reported is:
  not authorized on team to execute command { listCollections: 1, cursor: {}, filter: { name: { $not: /system\.|\$/ } } } (13)

My logstash Config:

input {
  mongodb {
    uri => "mongodb://admin:****@team-db1,team-db2/team"
    placeholder_db_dir => '/opt/logstash-mongodb/'
    placeholder_db_name => 'auftraggeber.db'
    collection => 'auftraggeber'
    batch_size => 5000
  }
}

output {
  elasticsearch {
    host => ["team-es2:9300"]
    cluster => 'team-es-cluster'
    index =>  'auftraggebers'
  }
}

The password contains a comma. To escape the comma I have used url-encode (%2C).
The authenticationDatabase I use is admin.

If I remove the username and password in the Logstash config, I get the same error.

Exit loop once all documents are read

@phutchins Currently, the plugin processes all the documents in the collection and infinitely waits/listens for new documents. I am trying to make the process exit once all the documents are read, so I tried this fix within 'get_cursor_for_collection' method -

        public
          def get_cursor_for_collection(mongodb, mongo_collection_name, last_id_object, batch_size)
            collection = mongodb.collection(mongo_collection_name)
            # Need to make this sort by date in object id then get the first of the series
            # db.events_20150320.find().limit(1).sort({ts:1})
            cursors = collection.find({:_id => {:$gt => last_id_object}}).limit(batch_size).to_a #store the array
            if cursors.empty? then exit end #If cursor array empty, then exit
            return collection.find({:_id => {:$gt => last_id_object}}).limit(batch_size)
        end

Although i was able to test this out, when I run it against my actual collection it doesn't work. It doesn't error out but simply exits after a few seconds.

Is there a better way to do this ?

Thanks!

Install fails caused by Gem::Package::FormatError

Still trying to figure out how to actually install your plugin and I simply cant.

Logstash 2.1.1 install by repo

ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash":
  In Gemfile:
    logstash-input-mongodb (>= 0) java depends on
      logstash (< 2.0.0, >= 1.4.0) java
Could not find gem 'logstash (< 2.0.0, >= 1.4.0) java', which is required by gem 'logstash-input-mongodb (>= 0) java', in any of the sources.

Install by file

Validating ./logstash-input-mongodb-0.3.3.gem
Error reading plugin file ./logstash-input-mongodb-0.3.3.gem, caused by Gem::Package::FormatError
ERROR: Installation aborted, verification failed for ./logstash-input-mongodb-0.3.3.gem

That's for 0.3.1 and 0.3.2 as well

CentOS Linux release 7.1.1503 (Core)

deleting mongodb documents doesn't get removed in ElasticSearch(ES)

Hi @phutchins ,

I tried using logstash-input-mongodb-0.2.0 for syncing mongodb data with ES.

However when we remove documents in mongoldb corresponding documents are not getting removed from ES.

Is this known issue or expected behaviour.

Please let us know how to resolve this issue if already fixed.

regards,
chakrapani

Could not fetch specs from https://rubygems.org/ HTTPError Installation Aborted

I am using ELK stack on Windows, with logstash-2.2.0. When I try installing logstash-input-mongodb plugin using the following command in the terminal.
logstash\bin>plugin install <path to logstash-input-mongodb-0.3.3.gem>
I get the following error
Validating D:\........\logstash-input-mongodb-0.3.3.gem Installing logstash-input-mongodb Error Bundler::HTTPError, retrying 1/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 2/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 3/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 4/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 5/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 6/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 7/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 8/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 9/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 10/10 Could not fetch specs from https://rubygems.org/ Too many retries, aborting, caused by Bundler::HTTPError ERROR: Installation Aborted, message: Could not fetch specs from https://rubygems.org/

I am new to ELK. and haven't installed any plugin in the logstash before. Please help me install the mongo db input plugin.

Error on logstash 5.1.2

I'm trying to run the plugin on logstash 5.1.2, and getting the error:
Direct event field references (i.e. event['field'] = 'value') have been disabled in favor of using event get and set methods (e.g. event.set('field', 'value')). Please consult the Logstash 5.0 breaking changes documentation for more details.

19:20:09.456 [LogStash::Runner] INFO logstash.inputs.mongodb - Using version 0.1.x input plugin 'mongodb'. This plugin isn't well supported by the community and likely has no maintainer.
D, [2017-02-09T19:20:09.766000 #7068] DEBUG -- : MONGODB | Adding localhost:27017 to the cluster. | runtime: 0.0000ms
D, [2017-02-09T19:20:12.839000 #7068] DEBUG -- : MONGODB | COMMAND | namespace=admin.$cmd selector={:ismaster=>1} flags=[] limit=-1 skip=0 project=nil | runtime: 3066.0000ms
19:20:12.845 [[main]-pipeline-manager] INFO logstash.inputs.mongodb - Registering MongoDB input
D, [2017-02-09T19:20:12.936000 #7068] DEBUG -- : MONGODB | COMMAND | namespace=test.$cmd selector={:listCollections=>1, :cursor=>{}, :filter=>{:name=>{"$not"=>/system.|$/}}} flags=[:slave_ok] limit=-1 skip=0 project=nil | runtime: 22.0001ms
19:20:13.140 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
19:20:13.151 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
D, [2017-02-09T19:20:13.356000 #7068] DEBUG -- : MONGODB | QUERY | namespace=test.auditcommunications selector={:_id=>{:$gt=>BSON::ObjectId('0761b057fccc5c277c23dfbe')}} flags=[:slave_ok] limit=50 skip=0 project=nil | runtime: 224.9999ms
19:20:13.420 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
19:20:13.647 [[main]<mongodb] ERROR logstash.pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::MongoDB uri=>"mongodb://localhost:27017/test", placeholder_db_dir=>"/elk/logstash-mongodb/", placeholder_db_name=>"logstash_sqlite.db", collection=>"auditcommunications", batch_size=>50, id=>"797b2fd78162084f6d57991a196d656a6c95f190-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_cd856514-8fcc-43ad-becd-9497e98577b9", enable_metric=>true, charset=>"UTF-8">, since_table=>"logstash_since", parse_method=>"flatten", isodate=>false, retry_delay=>3, generateId=>false, unpack_mongo_id=>false, message=>"Default message...", interval=>1>
Error: Direct event field references (i.e. event['field'] = 'value') have been disabled in favor of using event get and set methods (e.g. event.set('field', 'value')). Please consult the Logstash 5.0 breaking changes documentation for more details.
D, [2017-02-09T19:20:14.782000 #7068] DEBUG -- : MONGODB | QUERY | namespace=test.auditcommunications selector={:_id=>{:$gt=>BSON::ObjectId('0761b057fccc5c277c23dfbe')}} flags=[:slave_ok] limit=50 skip=0 project=nil | runtime: 128.9999ms

My configuration is pretty much copy pasted from the instructions:

input {
  mongodb {
    uri => 'mongodb://localhost:27017/test'
    placeholder_db_dir => '/elk/logstash-mongodb/'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'auditcommunications'
    batch_size => 50
  }
}

filter {
  date {
    match => [ "timestamp", "ISO8601" ]
  }
}

output {
    stdout { codec => rubydebug }
}

Thanks in advance

Mixed source of plugins, you can't mix local `.gem` and remote gems

Hi,

When exe command

sudo bin/plugin install –no-verify /home/local/logstash-input-mongodb/logstash-input-mongodb-1.0.0-java.gem
get below error

ERROR: Mixed source of plugins, you can't mix local.gem` and remote gems

See: 'bin/plugin install --help'
`

array not stored to elasticsearch

Hi guys, i have a problem, for stored array from mongodb to elasticsearch (picture above), fields with array type not store to elasticsearch why? have set a config?

Failing on document - Invalid ObjectID

I'm working on using this input gem to pull data from a MongoDB instance (that I don't own or manage) and sync it up with Elasticsearch. This input seems to do just that, but I'm having an issue actually fetching the data.

I get the following error on all objects:

MongoDB Input threw an exception, restarting {:exception=>#<BSON::ObjectId::Invalid: '[domain name or MD5]' is an invalid ObjectId.>, :level=>:warn, :file=>"logstash/inputs/mongodb.rb", :line=>"346", :method=>"run"}

I believe the error is because the IDs in this mongo collection are domains, and another collection uses md5 hashes as the ID. Is there any way to get around this or fix it?

I wanna say I think it's [https://github.com/phutchins/logstash-input-mongodb/blob/master/lib/logstash/inputs/mongodb.rb#L227](Line 227) that's erroring, but I could be wrong. I can post my logstash config if needed. Any help would be greatly appreciated!

feature request: add the collection name to the output

The lack of this information forces us to create separate logstash pipelines for every collection in order to add this information to the output.

ObjectId::Invalid

Hi guys,I have a Invalid Objectid problem in version 0.4.1.
The log looks like:
[2017-06-02T23:34:24,310][WARN ][logstash.inputs.mongodb ] MongoDB Input threw an exception, restarting {:exception=>#<BSON::ObjectId::Invalid: '0000abf12901b36a15fc8354e3328ddc' is an invalid ObjectId.>}

And my data in Mongodb looks like:

{
        "_id" : "0000abf12901b36a15fc8354e3328ddc",
        "data" : {
                "uid" : "C6TJVe2kpTfjosl1hc",
                .........
        },
        "connection_id" : "0000abf12901b36a15fc8354e3328ddc"
}

Logstash config like this:

input {
  mongodb {
    uri => 'mongodb://127.0.0.1:27017/test'
    placeholder_db_dir => '/opt/logstash-mongodb/'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'mycollection_'
    batch_size => 1
    unpack_mongo_id => true
  }
}

output {
  stdout {codec => rubydebug}
  elasticsearch {
    hosts => "127.0.0.1"
    index => "%{collection}"
    template_name => "logstash_1"
    codec => json
  }
}

I want to know is there anyone else has this problem? and how to solve this?

Input from Mongodb to Elasticsearch

I am trying to get mongodb data into Elasticssearch using logstash, with logstash-input-mongodb plugin. I tried to update the plugin to support version 5 of ES.

In terminal it shows version 0.4.1 but in error logs it throws error for version 0.1.X.

Also it's not connecting with mongodb to get data in to ES.

Can someone help with this ?

Error log

[2017-07-10T12:50:31,447][INFO ][logstash.inputs.mongodb  ] Using version 0.1.x input plugin 'mongodb'. This plugin isn't well supported by the community and likely has no maintainer.
[2017-07-10T12:50:32,061][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
[2017-07-10T12:50:32,062][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2017-07-10T12:50:32,225][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<Java::JavaNet::URI:0x4e62230d>}
[2017-07-10T12:50:32,227][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-07-10T12:50:32,289][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-07-10T12:50:32,299][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<Java::JavaNet::URI:0x38dfd049>]}
[2017-07-10T12:50:32,302][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-07-10T12:50:32,644][INFO ][logstash.inputs.mongodb  ] Registering MongoDB input
[2017-07-10T12:50:32,867][ERROR][logstash.pipeline        ] Error registering plugin {:plugin=>"<LogStash::Inputs::MongoDB uri=>\"mongodb://127.0.0.1:27017/jtrade\", placeholder_db_dir=>\"/home/jtrade/logstash\", placeholder_db_name=>\"logstash_sqlite.db\", collection=>\"activities\", batch_size=>5000, id=>\"961874444def3510514cefaccdbeb8ecd95670de-1\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_03e61419-953b-4267-803b-f8550e160cf7\", enable_metric=>true, charset=>\"UTF-8\">, since_table=>\"logstash_since\", since_column=>\"_id\", since_type=>\"id\", parse_method=>\"flatten\", isodate=>false, retry_delay=>3, generateId=>false, unpack_mongo_id=>false, message=>\"Default message...\", interval=>1>", :error=>"Java::JavaSql::SQLException: path to '/home/jtrade/logstash/logstash_sqlite.db': '/home/jtrade' does not exist"}
[2017-07-10T12:50:33,239][ERROR][logstash.agent           ] Pipeline aborted due to error {:exception=>#<Sequel::DatabaseConnectionError: Java::JavaSql::SQLException: path to '/home/jtrade/logstash/logstash_sqlite.db': '/home/jtrade' does not exist>, :backtrace=>["org.sqlite.core.CoreConnection.open(org/sqlite/core/CoreConnection.java:190)", "org.sqlite.core.CoreConnection.<init>(org/sqlite/core/CoreConnection.java:74)", "org.sqlite.jdbc3.JDBC3Connection.<init>(org/sqlite/jdbc3/JDBC3Connection.java:24)", "org.sqlite.jdbc4.JDBC4Connection.<init>(org/sqlite/jdbc4/JDBC4Connection.java:23)", "org.sqlite.SQLiteConnection.<init>(org/sqlite/SQLiteConnection.java:45)", "org.sqlite.JDBC.createConnection(org/sqlite/JDBC.java:114)", "org.sqlite.JDBC.connect(org/sqlite/JDBC.java:88)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "RUBY.connect(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/adapters/jdbc.rb:226)", "RUBY.make_new(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool.rb:116)", "RUBY.make_new(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:228)", "RUBY.available(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:201)", "RUBY._acquire(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:137)", "RUBY.acquire(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:151)", "RUBY.sync(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:282)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:149)", "RUBY.sync(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:282)", "RUBY.acquire(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:150)", "RUBY.hold(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:106)", "RUBY.synchronize(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/database/connecting.rb:306)", "RUBY.execute(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/adapters/jdbc.rb:251)", "RUBY.execute(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/dataset/actions.rb:1081)", "RUBY.fetch_rows(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/adapters/jdbc.rb:763)", "RUBY.with_sql_each(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/dataset/actions.rb:942)", "RUBY.with_sql_first(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/dataset/actions.rb:950)", "RUBY.single_record!(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/dataset/actions.rb:748)", "RUBY.first(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/dataset/actions.rb:248)", "RUBY.[](/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/dataset/actions.rb:39)", "RUBY.get_placeholder(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:112)", "RUBY.update_watched_collections(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:160)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.update_watched_collections(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:158)", "RUBY.register(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:182)", "RUBY.register_plugin(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:281)", "RUBY.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:292)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:292)", "RUBY.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:442)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:336)", "RUBY.run(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:226)", "RUBY.start_pipeline(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:398)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
[2017-07-10T12:50:33,304][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2017-07-10T12:50:36,267][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}

Thanks you so much.

error raised when installing on logstash 1.5.3

when i enter "./plugin install logstash-input-mongodb" , error below raised:
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash":
In Gemfile:
logstash-input-mongodb (>= 0) java depends on
logstash (< 2.0.0, >= 1.4.0) java
Could not find gem 'logstash (< 2.0.0, >= 1.4.0) java', which is required by gem 'logstash-input-mongodb (>= 0) java', in any of the sources.

Field [_id] is a metadata field and cannot be added inside a document

I'm using Elasticsearch 5.4, Logstash 5.4.2, MongoDB 3.4.5.
When I import from MongoDB to Elasticsearch through Logstash, I found the following error:

[2017-07-11T16:57:06,180][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mongodb-2017.07.11", :_type=>"logs", :_routing=>nil}, 2017-07-11T08:57:03.744Z %{message}], :response=>{"index"=>{"_index"=>"mongodb-2017.07.11", "_type"=>"logs", "_id"=>"AV0w3bobj2yUz3xTJdU4", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."}}}}

It mentioned that there was a conflict between the name of [_id] meta field in Kibana and [_id] meta column of MongoDB. How should I solve this conflict in my logstash pipeline config? I think this is a critical problem between ES and Mongo, how did this plugin work before?

Thank you!

Work around for error raised when installing?

(EDIT) Error has changed after mucking with it - first issue was my fault.

bin/plugin install logstash-input-mongodb

LoadError: no such file to load -- file-dependencies/gem
require at org/jruby/RubyKernel.java:1072
require at /local/logstash-1.5.3/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:54
(root) at /local/logstash/lib/pluginmanager/install.rb:4
require at org/jruby/RubyKernel.java:1072
(root) at /local/logstash-1.5.3/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
require at /local/logstash-1.5.3/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:54

Authentication failure

Am trying to configure this plugin to connect to my mongo replica set and am getting the following error: "not authorized for query on uilogs.system.namespace (13)"

From the mongo logs, reports something similar: "assertion 13 not authorized for query on uilogs.system.namespaces ns:uilogs.system.namespaces query: { name: $not: /system.|$/ } }"

My conf file looks like this:
...
mongodb {
uri => 'mongodb://uilogsAdmin:@host-abc123:27017/uilogs'
placeholder_db_dir => '/home/somewhere/logstash-1.5.3/logstash-mongodb'
placeholder_db_name => 'logstatsh_sqlite.db'
collection => 'daily.apps'
}

am using logstash 1.5.3, the most recent version of this plugin 0.2.0, and mongodb 2.6.4

the login id 'uilogsAdmin' currently has the following roles: dbOwner, clusterAdmin, dbAdminAnyDatabase.

If I log into the primary with the above id and run uilogs.system.namespaces.find() I do not get the auth error.

Please advise what could be wrong here.

bug: flatten parse_method ignores Object IDs

I think it's self explanatory. The "simple" parse_method converts them to strings, which is what the flatten method should do as well.

How to enable SSL pull request for Mongo

How to pass SSL mongo connector URI and truststore

Getting data from system collections

Hello,

I've been trying out this plugin and it's great!
I'd like to get data from a system.profile collection in order to create an operation analysis dashboard in Kibana.
When I try to do that, logstash gives me this message:
D, [2015-08-16T12:17:14.914000 #30642] DEBUG -- : MONGODB | COMMAND | namespace=collection1.$cmd selector={:listCollections=>1, :cursor=>{}, :filter=>{:name=>{"$not"=>/system.|$/}}} flags=[] limit=-1 skip=0 project=nil | runtime: 32.0001ms

So system collections are filtered out. The code responsible for that is in the ruby mongo driver:
https://github.com/mongodb/mongo-ruby-driver/blob/15f5707794e852a8124d8d9d89019223e0183ebe/lib/mongo/operation/commands/list_collections.rb#L49

Is there any way to get around that in your plugin?

Thanks.

Infinite loop adding mongoDB documents into ElasticSearch

We use this logstash configuration:

input {
mongodb {
uri => 'mongodb://127.0.0.1/test'
placeholder_db_dir => '/users/sth00/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'Users'
batch_size => 5000
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
host => "127.0.0.1"
protocol => "http"
index => "mongodb-%{+YYYY.MM.dd}"
}
}

in mongoDB the collection "Users" only have 4 fields: _id, name, email, age
we have only 7 records and in elasticsearch these 7 records have been duplicate every second and it doesn't stop. why?

Among other things, how I can delete the fileds: logdate and logentry, we only need dump the mongodb data (Fields) into ES and keep synced these data between mongoDB and ES thanks to logstash and the plugin.

cordial regards

How to select collection from database

I have several databases and collections in them.

How do you specify this in config?

Ie. want to read from myExampleDb.collectionABC

Tail Mongo Oplog

Hi,
Is there any plans to implement tailing the Mongo oplog in order to pickup changes to existing docs?

"comparison of Fixnum with nil failed"

Hi,

I am trying out logstash and I was very happy to find this plugin.

I installed the plugin but when I use to pull the database i get the following error:

D, [2016-01-07T03:59:54.337000 #9574] DEBUG -- : MONGODB | Adding localhost:27017 to the cluster.
D, [2016-01-07T03:59:54.501000 #9574] DEBUG -- : MONGODB | localhost:27017 | umbrella.find | STARTED | {"find"=>"system.namespaces", "filter"=>{:name=>{"$not"=>/system.|$/}}}
D, [2016-01-07T03:59:54.509000 #9574] DEBUG -- : MONGODB | localhost:27017 | umbrella.find | FAILED | comparison of Fixnum with nil failed | 0.006s
The error reported is:
comparison of Fixnum with nil failed

This is my config:

input {
mongodb {
uri => 'mongodb://localhost:27017/umbrella'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'users'
}
}

I have the umrella db and users collection.
I am using logstash version 2.1.1

Any idea what is causing this issue?

Thank you in advance

Feature request: Add support for mapping embedded mongoDB docs to Elastic 'Object' datatype

ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html
It would be cool if the flatten and dig parse methods can format the event names using the bracket format v.s. underscores. logstash uses this format to translate to elastic's Object datatype. If done correctly, embedded mongodb docs can be mapped to elastic objects.

E.g.
change: new_hash[k1.to_s+"_"+k2.to_s] = v2
to: new_hash["[#{k1.to_s}][#{k2.to_s}]"] = v2

thus =>
Mongo doc:
{
some_field: "some value",
foo: { bar: "yay"}
}

mongo logstash:
"some_field" = "some value"
"[foo][bar]" = "yay"

elastic:
{
"some_field": "some value",
"foo": { bar: "yay"}
}

and of course, the property needs to be mapped as an object type before applying the import:
{
"some_mapping": {
"properties": {
"some_field": {"type": "text"},
"foo": {"type": "object"}
}
}
}

get collection of document

My collection configuration looks like this: collection => '.*'
Is there a way i can find out the collection of which the document is coming from?
By looking into the source code i could not find anything like that.

What is the vision of this plugin?

Hey, I would like to input my mongodb data into elasticsearch via logstash. So I've searched for a logstash input plugin and this repo popped up.

@phutchins What is your vision for this plugin?

Logstash 5.0 support

Hello,

I'm trying to install this plugin on a logstash 5.0.0 installation, but Gemfile.lock limits logstash-core (>= 1.4.0, < 2.0.0).

~ # /opt/logstash/current/bin/logstash-plugin install --no-verify  logstash-output-mongodb
Installing logstash-output-mongodb
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-core-plugin-api":
  In snapshot (Gemfile.lock):
    logstash-core-plugin-api (= 2.1.16)

  In Gemfile:
    logstash-devutils (~> 1.1) java depends on
      logstash-core-plugin-api (~> 2.0) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-output-file (>= 0) java depends on
      logstash-core-plugin-api (< 2.99, >= 2.0.0) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-output-stdout (>= 0) java depends on
      logstash-core-plugin-api (< 2.99, >= 1.60.1) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

    logstash-output-mongodb (>= 0) java depends on
      logstash-core-plugin-api (~> 1.0) java

    logstash-core-plugin-api (>= 0) java

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Bundler could not find compatible versions for gem "logstash-core":
  In snapshot (Gemfile.lock):
    logstash-core (= 5.0.0)

  In Gemfile:
    logstash-core-plugin-api (>= 0) java depends on
      logstash-core (= 5.0.0) java

    logstash-output-mongodb (>= 0) java depends on
      logstash-core (< 2.0.0, >= 1.4.0) java

    logstash-core (>= 0) java

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Bundler could not find compatible versions for gem "logstash":
  In Gemfile:
    logstash-output-mongodb (>= 0) java depends on
      logstash (< 2.0.0, >= 1.4.0) java
Could not find gem 'logstash (< 2.0.0, >= 1.4.0) java', which is required by gem 'logstash-output-mongodb (>= 0) java', in any of the sources.

Question: is there any way to change the frequency of the pull from mongodb?

hi there:

i have one separate job updating a single document in mongodb multiple times a minute, but i'd like to have the logstash mongodb input only pull documents from mongodb, say, once every 2 minutes or so. this would essentially 'consolidate' all of those updates to the same document into one. i looked through the code briefly for something that might support that, but didn't find such an option. is this something you've considered before? thank you for your time and thanks for your excellent work on this plugin -- works very nicely for me.

logstash_sqlite.db problems

I have a problem when i configure the plugin. I've created a config file :

input {
mongodb {
uri => 'mongodb://mongourl/mng?ssl=false'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'pages_feed'
batch_size => 5000
}
}

filter {

}

output {
elasticsearch {
hosts => ["elasticurl:9200"]
index => "pagesfeed"
document_type => "feed"
document_id => "%{id}"
}
stdout { codec => rubydebug }
}

When logstash executes the file i get the error :

{
: timestamp=>"2016-09-06T09:10:32.424000+0000",
: message=>"An unexpected error occurred!",
: error=>#<Sequel: : DatabaseConnectionError: Java: : JavaSql: : SQLException: pathto'/opt/logstash-mongodb/logstash_sqlite.db': '/opt/logstash-mongodb'doesnotexist>,
: class=>"Sequel::DatabaseConnectionError",
: backtrace=>["org.sqlite.core.CoreConnection.open(org/sqlite/core/CoreConnection.java:190)",
"org.sqlite.core.CoreConnection.(org/sqlite/core/CoreConnection.java:74)",
"org.sqlite.jdbc3.JDBC3Connection.(org/sqlite/jdbc3/JDBC3Connection.java:24)",
"org.sqlite.jdbc4.JDBC4Connection.(org/sqlite/jdbc4/JDBC4Connection.java:23)",
"org.sqlite.SQLiteConnection.(org/sqlite/SQLiteConnection.java:45)",
"org.sqlite.JDBC.createConnection(org/sqlite/JDBC.java:114)",
"org.sqlite.JDBC.connect(org/sqlite/JDBC.java:88)",
"java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)",
"RUBY.connect(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/adapters/jdbc.rb:222)",
"RUBY.make_new(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool.rb:103)",
"RUBY.make_new(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:224)",
"RUBY.available(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:197)",
"RUBY._acquire(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:133)",
"RUBY.acquire(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:147)",
"RUBY.sync(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:265)",
"org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:149)",
"RUBY.sync(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:265)",
"RUBY.acquire(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:146)",
"RUBY.hold(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:104)",
"RUBY.synchronize(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/database/connecting.rb:256)",
"RUBY.execute(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/adapters/jdbc.rb:247)",
"RUBY.execute(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/dataset/actions.rb:952)",
"RUBY.fetch_rows(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/adapters/jdbc.rb:738)",
"RUBY.with_sql_each(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/dataset/actions.rb:835)",
"RUBY.with_sql_first(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/dataset/actions.rb:843)",
"RUBY.single_record!(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/dataset/actions.rb:661)",
"RUBY.single_record(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/dataset/actions.rb:649)",
"RUBY.first(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/dataset/actions.rb:200)",
"RUBY.",
"RUBY.get_placeholder(/opt/logstash/vendor/local_gems/9a1942b2/logstash-input-mongodb-0.3.3/lib/logstash/inputs/mongodb.rb:98)",
"RUBY.update_watched_collections(/opt/logstash/vendor/local_gems/9a1942b2/logstash-input-mongodb-0.3.3/lib/logstash/inputs/mongodb.rb:146)",
"org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)",
"RUBY.update_watched_collections(/opt/logstash/vendor/local_gems/9a1942b2/logstash-input-mongodb-0.3.3/lib/logstash/inputs/mongodb.rb:144)",
"RUBY.register(/opt/logstash/vendor/local_gems/9a1942b2/logstash-input-mongodb-0.3.3/lib/logstash/inputs/mongodb.rb:168)",
"RUBY.start_inputs(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:322)",
"org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)",
"RUBY.start_inputs(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:321)",
"RUBY.start_workers(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:172)",
"RUBY.run(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:126)",
"RUBY.execute(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/agent.rb:210)",
"RUBY.run(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:90)",
"org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)",
"RUBY.run(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:95)",
"org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)",
"RUBY.initialize(/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24)",
"java.lang.Thread.run(java/lang/Thread.java:745)"],
: level=>: warn
}

My problem is with logstash_sqlite.db. The database should exist?

Milliseconds get lost when ingesting datetimes

First of all thanks for the implementation of a MongoDB-input for Logstash. It seems to be very reasonable as there might be many use-cases out there demanding a connection of Elasticsearch to MongoDB.

The same demand is present in the use-cases I am currently working on.

I am quite new to the ELK-stack.
Thus, in case you believe the error is related to issues on my side, please let me know.

Problem:
Milliseconds of date-times are cut off when ingesting data from MongoDB to Elasticsearch.
Thus, aggregations over large amounts of data-points might lead to skewed results.

Description of the problem:

Example data in MongoDB:

{ "_id" : ObjectId("5721d60a1c4a9e0fc06dc0ba"), "State" : <State>, "thingId" : <another id>, "Name" : <Name>, "StartTime" : ISODate("2016-03-16T16:30:22.000Z"), "EndTime" : ISODate("2016-03-16T16:37:17.453Z"), "Duration" : 0.115403681777778 }

Logstash configuration:

`input {
mongodb {
uri =>
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'AggregatedData'
generateId => true
batch_size => 5000
}
}

filter {
date {
match => [ "StartTime", "yyyy-MM-dd HH:mm:ss,SSS" ]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "aggregated_data"
}
}`

Error Message:

The error message occurs for all data-points that are to be imported into ES.
Failed parsing date from field {:field=>"StartTime", :value=>"2016-03-21 03:03:15 UTC", :exception=>"Invalid format: \"2016-03-21 03:03:15 UTC\" is malformed at \" UTC\"", :config_parsers=>"ISO8601", :config_locale=>"default=en_US", :level=>:warn}

In my eyes, the following part of the error-message indicates, that the information is cut off by the input-plugin before reaching the filter in the logstash config-file.
Also I have no clue, where the "UTC"-string is coming from.
:field=>"StartTime", :value=>"2016-03-21 03:03:15 UTC",

Do you have an idea about what is going wrong here?
Do you have a schedule for a possible fix, in case this is directly related to your plugin and you want to fix it.

Cheers and thanks in advance,
StetoGraph

logs events in loop

I sucessfully installed logstash-input-mongodb plugin in my logstash instance but probably there's something missing somewhere in my configuration. Just for test, I added a single entry in my mongodb collection and started up logstash with the following configuration:

input{

    mongodb {
    uri => 'mongodb://127.0.0.1:27017/syslog?ssl=no'
     placeholder_db_dir => '/opt/logstash-mongodb/'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'messages'
    batch_size => 5000

}}

output {
file{
path => "/var/log/mongoteste"
}}

When checking "/var/log/mongoteste" I realize that, in fact, it's writing in in the file, but at each 5 seconds (checked in timestamp) , the entry is replicated and keeps in this loop everytime, just like this:

{"host":"localhost.localdomain","@Version":"1","@timestamp":"2015-09-05T00:58:31.632Z","logdate":"2015-09-04T17:31:13+00:00","log_entry":"{"_id"=>BSON::ObjectId('55e9d5611adb62fa4d000001'), "TAGS"=>".source.s_tcp", "SOURCEIP"=>"172.16.102.148", "SEQNUM"=>"1", "PROGRAM"=>"ola", "PRIORITY"=>"notice", "LEGACY_MSGHDR"=>"ola", "HOST_FROM"=>"172.16.102.148", "HOST"=>"172.16.102.148", "FACILITY"=>"user", "DATE"=>"Sep 4 18:31:13"}","mongo_id":"55e9d5611adb62fa4d000001","TAGS":".source.s_tcp","SOURCEIP":"172.16.102.148","SEQNUM":1,"PROGRAM":"ola","PRIORITY":"notice","LEGACY_MSGHDR":"ola","HOST_FROM":"172.16.102.148","HOST":"172.16.102.148","FACILITY":"user","DATE":"Sep 4 18:31:13"}

{"host":"localhost.localdomain","@Version":"1","@timestamp":"2015-09-05T00:58:36.675Z","logdate":"2015-09-04T17:31:13+00:00","log_entry":"{"_id"=>BSON::ObjectId('55e9d5611adb62fa4d000001'), "TAGS"=>".source.s_tcp", "SOURCEIP"=>"172.16.102.148", "SEQNUM"=>"1", "PROGRAM"=>"ola", "PRIORITY"=>"notice", "LEGACY_MSGHDR"=>"ola", "HOST_FROM"=>"172.16.102.148", "HOST"=>"172.16.102.148", "FACILITY"=>"user", "DATE"=>"Sep 4 18:31:13"}","mongo_id":"55e9d5611adb62fa4d000001","TAGS":".source.s_tcp","SOURCEIP":"172.16.102.148","SEQNUM":1,"PROGRAM":"ola","PRIORITY":"notice","LEGACY_MSGHDR":"ola","HOST_FROM":"172.16.102.148","HOST":"172.16.102.148","FACILITY":"user","DATE":"Sep 4 18:31:13"}

{"host":"localhost.localdomain","@Version":"1","@timestamp":"2015-09-05T00:58:41.714Z","logdate":"2015-09-04T17:31:13+00:00","log_entry":"{"_id"=>BSON::ObjectId('55e9d5611adb62fa4d000001'), "TAGS"=>".source.s_tcp", "SOURCEIP"=>"172.16.102.148", "SEQNUM"=>"1", "PROGRAM"=>"ola", "PRIORITY"=>"notice", "LEGACY_MSGHDR"=>"ola", "HOST_FROM"=>"172.16.102.148", "HOST"=>"172.16.102.148", "FACILITY"=>"user", "DATE"=>"Sep 4 18:31:13"}","mongo_id":"55e9d5611adb62fa4d000001","TAGS":".source.s_tcp","SOURCEIP":"172.16.102.148","SEQNUM":1,"PROGRAM":"ola","PRIORITY":"notice","LEGACY_MSGHDR":"ola","HOST_FROM":"172.16.102.148","HOST":"172.16.102.148","FACILITY":"user","DATE":"Sep 4 18:31:13"}

If I add a new entry in my collection, the plugin will consume new data but just like before, will write it in file constantly each 5 seconds.

Has anyone run into the same problem? My conf is logstash 1.5.4 + logstash-input-mongodb-0.2.0 .

Best Regards

Logstash & ES 2.0 support

Hi,
Is anyone tested this input with the new Logstash 2.0 version?
I want to upgrade soon...
Thanks.

Pull the data from the collection based on time stamp

Hi Team ,

I want to pull the data from Collection with in a specific interval time . Once the data is available it should pull the data automatically from where it left off initially .

Can we do this with this plugin

error about install logstash-input-mongodb-0.3.3 for logstash-2.4.0

Command:

bin/logstash-plugin install --no-verify plugins/logstash-input-mongodb/logstash-input-mongodb-0.3.3.gem
Installing logstash-input-mongodb
WARNING: SSLSocket#session= is not supported
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "jdbc-sqlite3":
In snapshot (Gemfile.lock):
jdbc-sqlite3 (= 3.8.11.2)

In Gemfile:
jdbc-sqlite3 (>= 0) java

logstash-input-mongodb (= 0.3.3) java depends on
  jdbc-sqlite3 (= 3.8.10.1) java

Running bundle update will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

Plugin for logstash>2.0.0

Hi,

I want used logstash with the plugin, but I understand that is not avalaible for logstash>2.0.0. Are you planning to upgrade?

elastic/logstash#4169

plugin install logstash-input-mongodb
 ---> Running in 062b87f7ae18
Validating logstash-input-mongodb
Installing logstash-input-mongodb
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash":
  In Gemfile:
    logstash-input-mongodb (>= 0) java depends on
      logstash (< 2.0.0, >= 1.4.0) java
Could not find gem 'logstash (< 2.0.0, >= 1.4.0) java', which is required by gem 'logstash-input-mongodb (>= 0) java', in any of the sources.
The command '/bin/sh -c ${LOGSTASH_SRC_DIR}/bin/plugin install logstash-input-mongodb' returned a non-zero code: 1

No connection string specified Error

The error i get is-

My config file is

and what does the PATH specify in conf file. can you please explain it.

Nested fields

Thank you for the plugin, so far it works well and was quick to set up.

I have many fields that contain other fields and are nested to many levels. For example a field may contain the following:

"[{\"Field_1\"=>\"Field_1_Value\", \"Field_2\"=>\"Field_2_Value\", \"Field_3\"=>\"Field_3_Value\", \"Field_4\"=>\"Field_4_Value\", \"Field_5\"=>\"Field_5_Value\", \"Parent_Field\"=>[{\"Child_Field_1\"=>\"Child_Field_1_Value\", \"Child_Field_2\"=>nil, \"Child_Field_3\"=>\"Child_Field_3_Value\", \"Child_Field_4\"=>nil}]}]"

Is there a way to either have the plugin parse the contents as children of the parent field, or if there are existing filters that can parse the data properly?

I'm currently looking into using the ruby filter to iterate through the contents, but thought maybe I missed something and maybe there is something out there that can already do this.

Thanks

How to add Mongodb query?

I want to copy only data for last month to elasticsearch.
How can I specify some query to mongodb?

i/o console not supported tty will not be manipulated in logstash error in windows

i am a new user to elk i am getting the error please suggest me

Inifinite loops due to "invalid byte sequence in UTF-8"

When the input MongoDB document has a non UTF-8 (or something like that), the input plugin gets into infinite loop, as execption is raised and process is restarted, just to hit the same case again.

logstash.stdout:
D, [2016-09-17T21:14:03.445000 #22826] DEBUG -- : MONGODB | X.X.X.X:27017 | logs.find | STARTED | {"find"=>"logs", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('57d948b1995d5258198b457c')}}, "limit"=>500}
D, [2016-09-17T21:14:03.458000 #22826] DEBUG -- : MONGODB | X.X.X.X:27017 | logs.find | SUCCEEDED | 0.013s

logstash.
{:timestamp=>"2016-09-17T21:14:03.423000+0000", :message=>"MongoDB Input threw an exception, restarting", :exception=>#<ArgumentError: invalid byte sequence in UTF-8>, :level=>:warn}

Proposed solution:
Skip error record, by adding a fake record to ES, just to make sure that when Max is requestied, the failed will not be counted

Lost only one data when to elasticsearch

my logstash config:

input {
    mongodb {
        uri => 'mongodb://127.0.0.1:27017/wiki'
        collection => 'wiki'
        placeholder_db_dir => "E:/mongo2es/data"
        placeholder_db_name => "datalogstash_sqlite_wiki.db"
        batch_size => 1000
    }
}

filter {
    mutate {
        remove_field => [ "host", "@version", "@timestamp", "logdate", "log_entry" ]
    }
}

output {
    stdout { codec => rubydebug }

    file {
        path => "E:/mongo2es/logs/mongo2es-wiki.log"
    }

    elasticsearch {
        index => "wiki"
        document_type => "wiki"
        document_id => "%{mongo_id}"
        hosts => ["127.0.0.1:9200"]
    }
}

2767278 in mongodb, but 2767277 in elasticsearch.

Any thought?
Thanks in Advance

Input Config - exit loop after end of collection

Currently this plugin sleeps and continues to look for new data once collection is finished. This behavior should be configurable.

Please add flag in input config to decide the behavior.

We can also provide pull request if needed.