mikowals / batch-insert Goto Github PK

I'm using mikowals:[email protected] on a Meteor/OrionJS app.
On the local dev environment the insertion of millions of documents with batch-insert works and that takes only a few seconds.
On our staging server (running on EC2 nginx), batch-insert fails when dealing with more than approximately ~100k documents to insert.

Here's the code that get executed on the server:

var stuffToInsert = [];
_.each(thing, function(element){
  stuffToInsert.push({
    // ...
  });
});

// tag-manager.js:63:9
Collection.batchInsert(stuffToInsert);

And here's the error stack provided by PM2:

dashboard-19 (out):     at packages/meteor/helpers.js:119:1
dashboard-19 (out):     at _batchInsert (packages/mikowals:batch-insert/batch-insert-server.js:112:1)
dashboard-19 (out):     at [object Object].Mongo.Collection.batchInsert (packages/mikowals:batch-insert/batch-insert-common.js:89:1)
dashboard-19 (out):     at [object Object].Meteor.methods.generateTags (app/server/tag-manager.js:63:9)
dashboard-19 (out):     at maybeAuditArgumentChecks (packages/ddp/livedata_server.js:1617:1)
dashboard-19 (out):     at packages/ddp/livedata_server.js:648:1
dashboard-19 (out):     at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1)
dashboard-19 (out):     at packages/ddp/livedata_server.js:647:1
dashboard-19 (out):     at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1)
dashboard-19 (out):     - - - - -
dashboard-19 (out):     at [object Object].InsertCommand.toBinary (/var/www/viveat/dashboard/bundle/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/commands/insert_command.js:86:11)
dashboard-19 (out):     at [object Object].Connection.write (/var/www/viveat/dashboard/bundle/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/connection/connection.js:252:42)
dashboard-19 (out):     at __executeInsertCommand (/var/www/viveat/dashboard/bundle/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/db.js:1953:14)
dashboard-19 (out):     at Db._executeInsertCommand (/var/www/viveat/dashboard/bundle/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/db.js:2026:5)
dashboard-19 (out):     at insertAll (/var/www/viveat/dashboard/bundle/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/collection/core.js:249:13)
dashboard-19 (out):     at Collection.insert (/var/www/viveat/dashboard/bundle/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/collection/core.js:36:3)
dashboard-19 (out):     at packages/meteor/helpers.js:118:1
dashboard-19 (out):     at _batchInsert (packages/mikowals:batch-insert/batch-insert-server.js:112:1)
dashboard-19 (out):     at [object Object].Mongo.Collection.batchInsert (packages/mikowals:batch-insert/batch-insert-common.js:89:1)
dashboard-19 (out):     at [object Object].Meteor.methods.generateTags (app/server/tag-manager.js:63:9)

I'm running into an interesting issue (not sure if you'd call it a bug). I'm using collection2 to define a schema for documents, but it looks like batch insert is not being validated against the collection (like you might expect with a plain Collection.insert()). Is this expected?

The specific issue revolves around inserting an object with an incorrect property name via batch insert. For example, my schema has a field like industryId, however, if I insert an object with industry using Batch Insert, the object is still inserted. With a vanilla insert, I'd expect the field to get chopped off by collection2 or throw an error but that's not the case.

Any thoughts?

I've lately seen that I started out having problems with my tests on the client-side. It shows, that every time I try to create a stub of the collection Meteor.users by using the package meteor-stub-collections, I get the following error:

Error: use "new" to construct a Mongo.Collection
    at ns.Collection.Mongo.Collection (packages/mongo.js?hash=90f037f47abee1e74ba80360e6b3f3dbaa792260:113:50)
    at new ns.Collection (packages/matb33_collection-hooks.js?hash=d44fd0eb02806747e1bb0bdc3938463e415c63ec:284:27)
    at packages/hwillson_stub-collections.js?hash=bef58d9f36ab39205e53f7e1fffa813805568f78:80:28
    at Array.forEach (native)
    at Object.publicApi.stub (packages/hwillson_stub-collections.js?hash=bef58d9f36ab39205e53f7e1fffa813805568f78:74:35)
    at Context.<anonymous> (app/app.js?hash=6db509560fea1b6fa4bd95b4897829e3a9148f6c:9987:48)
    at callFn (packages/practicalmeteor_mocha-core.js?hash=6ff455cf7e5c128f88197d5d8c9244271a433d12:4346:21)
    at Hook.Runnable.run (packages/practicalmeteor_mocha-core.js?hash=6ff455cf7e5c128f88197d5d8c9244271a433d12:4339:7)
    at next (packages/practicalmeteor_mocha-core.js?hash=6ff455cf7e5c128f88197d5d8c9244271a433d12:4685:10)
    at packages/practicalmeteor_mocha-core.js?hash=6ff455cf7e5c128f88197d5d8c9244271a433d12:4707:5

Have you seen this before? Do you know where this is originated from?

After installing this package all of my tests that check that parameter is an instance of a Mongo.Collection now fail.

Not really an issue rather a question. First of all: I really love this package! Boosted my import methods by 3-4x which is a blast!

What's the state of this merging to meteor core? Is there anything to mention regarding mongo 5 drivers?

Thanks so much!

Changing to mocha reports test results in the console and would allow GitHub action to automate testing.

I really like this package and it's working fine locally. But as soon as the app is built and is deployed I run into this error whenever invoking SomeCollection.batchInsert(). Any hints would be really appreciated :o

Hi,
I am using simple schema in my collections.
Does this bulk insert support simple schema when doing bulk insert please?

I have a meteor app where the server gets real-time streaming data feeds .. the first batch would be an insert, but from then on, it's all updates. Is there an equivalent for that? Also -- it's not exactly clear the advantage of this versus using a _.each() type loop to insert the documents .. just curious. I am hoping to stop LiveQuery from going berserk with client updates (a bit at a time) and be more efficient.

Cheers!

Thanks for this package! I've been happily using it up until now when I updated to Meteor 1.4.
It seems there is a dependency on an old version of npm-mongo, so the Meteor update fails.

I've added the package but it doesn't work. The only error report I get is that batchInsert is not defined. Last version is 1.1.13, that's what I have on my versions file.

I've logged some collections and batchInsert function isn't defined.

Hi, I faced with the following error while running tests, it this mean something to you?

W20151225-19:50:19.558(6)? (STDERR)           
W20151225-19:50:19.559(6)? (STDERR) /home/sohaam/.meteor/packages/meteor-tool/.1.1.10.gjfrq3++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:245
W20151225-19:50:19.560(6)? (STDERR)                                             throw(ex);
W20151225-19:50:19.560(6)? (STDERR)                                                   ^
W20151225-19:50:19.653(6)? (STDERR) Error: A method named '/Products/insert' is already defined
W20151225-19:50:19.653(6)? (STDERR)     at meteor://💻app/livedata_server.js:1542:15
W20151225-19:50:19.653(6)? (STDERR)     at Function._.each._.forEach (meteor://💻app/packages/underscore/underscore.js:113:1)
W20151225-19:50:19.653(6)? (STDERR)     at [object Object]._.extend.methods (meteor://💻app/livedata_server.js:1538:7)
W20151225-19:50:19.653(6)? (STDERR)     at [object Object].Mongo.Collection._defineMutationMethods (meteor://💻app/packages/mongo/collection.js:927:1)
W20151225-19:50:19.653(6)? (STDERR)     at [object Object].Mongo.Collection (meteor://💻app/packages/mongo/collection.js:218:1)
W20151225-19:50:19.653(6)? (STDERR)     at [object Object].ns.Collection (meteor://💻app/packages/matb33_collection-hooks/collection-hooks.js:244:1)
W20151225-19:50:19.653(6)? (STDERR)     at new Mongo.Collection (meteor://💻app/packages/mikowals_batch-insert/batch-insert-common.js:101:1)
W20151225-19:50:19.654(6)? (STDERR)     at allTrue.insert (meteor://💻app/packages/local-test_aldeed_collection2/tests/collections.js:91:1)
W20151225-19:50:19.654(6)? (STDERR)     at /tmp/meteor-test-runlg927q/.meteor/local/build/programs/server/packages/local-test_aldeed_collection2.js:415:4
W20151225-19:50:19.654(6)? (STDERR)     at /tmp/meteor-test-runlg927q/.meteor/local/build/programs/server/packages/local-test_aldeed_collection2.js:1579:3
W20151225-19:50:19.654(6)? (STDERR)     at /tmp/meteor-test-runlg927q/.meteor/local/build/programs/server/boot.js:242:10
W20151225-19:50:19.654(6)? (STDERR)     at Array.forEach (native)
W20151225-19:50:19.654(6)? (STDERR)     at Function._.each._.forEach (/home/sohaam/.meteor/packages/meteor-tool/.1.1.10.gjfrq3++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/server-lib/node_modules/underscore/underscore.js:79:11)
W20151225-19:50:19.654(6)? (STDERR)     at /tmp/meteor-test-runlg927q/.meteor/local/build/programs/server/boot.js:137:5

Hi,

Just tried to install this package in my Meteor project today ([email protected]).

It apparently clashes with ground:db. Here's the error stack:

W20160206-23:11:27.055(0)? (STDERR) 
W20160206-23:11:27.057(0)? (STDERR) /Users/joaopedreira/.meteor/packages/meteor-tool/.1.1.10.1r5n12p++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:245
W20160206-23:11:27.057(0)? (STDERR)                         throw(ex);
W20160206-23:11:27.057(0)? (STDERR)                               ^
W20160206-23:11:27.057(0)? (STDERR) Error: use "new" to construct a Mongo.Collection
W20160206-23:11:27.057(0)? (STDERR)     at new Mongo.Collection (packages/mongo/collection.js:29:1)
W20160206-23:11:27.057(0)? (STDERR)     at new Ground.Collection (packages/ground_db/groundDB.server.js:81:1)
W20160206-23:11:27.057(0)? (STDERR)     at lib/collections.js:12:1
W20160206-23:11:27.057(0)? (STDERR)     at lib/collections.js:177:1
W20160206-23:11:27.057(0)? (STDERR)     at /Users/joaopedreira/WebstormProjects/RoutesDirectMobile/.meteor/local/build/programs/server/boot.js:242:10
W20160206-23:11:27.057(0)? (STDERR)     at Array.forEach (native)
W20160206-23:11:27.057(0)? (STDERR)     at Function._.each._.forEach (/Users/joaopedreira/.meteor/packages/meteor-tool/.1.1.10.1r5n12p++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/server-lib/node_modules/underscore/underscore.js:79:11)
W20160206-23:11:27.057(0)? (STDERR)     at /Users/joaopedreira/WebstormProjects/RoutesDirectMobile/.meteor/local/build/programs/server/boot.js:137:5

Any ideas? Thanks

Hi! I'm working on integrating bulk/batch inserts into Meteor core:
meteor/meteor#11222

Would appreciate if you could provide feedback to my efforts there. Thanks a lot!

Hi, is there any reason why batch-insert would fail when using the new mongodb engine WiredTiger?

I'm getting this exception:

I20160515-09:00:34.389(-4)? Exception in callback of async function: TypeError: Object Error: no open connections has no method 'toJSON'
I20160515-09:00:34.390(-4)?     at _batchInsert (packages/mikowals_batch-insert/batch-insert-server.js:140:1)

I'm doing insert-batches ranging from 20k ~ 150k record at once.

Using the mongodb default storage engine fix the issue, but I do need to use WiredTiger.

Any clues?

Thanks

as title

https://github.com/matb33/meteor-collection-hooks

I have a batch insert operation that's just inserting two items into a collection:

const newIds = Records.batchInsert(oldItems);

Yet the value of newIds is:

[ undefined, undefined, undefined, undefined ]

Hi, I find your package really useful for a project I'm working on, so thanks a lot!

Here's the problem I've been having:

I usually deny all operations on the client, and expose server methods that clients may call, inside the server method I'm allowed to do any insert/update/remove. This works pretty well with insert/update/remove.

So I wanted to try the same approach with your package, deny all operations in the client, so they can't do MyCollection.batchInsert(), but define a server method where the server is in charge of executing the batch insert. this is currently not working since allow/deny rules are being checked even for server methods.

If I allow inserts, my server method works, but clients are now able to do MyCollection.batchInsert() now.

I've noticed this at batch-insert.js line 135:

if ( this.connection ) {
  //server method called by client so check allow / deny rules.
}

I believe removing the allow/deny checks inside this function should fix the problem, and thus this package would behave as collection operations in meteor.

Kind regards,
Paolo.

Thanks for the great work on this functionality!

Im using ensureUnique on the collection, to prevent duplicate documents being inserted, rather than checking against every doc in the db.

Is there a way to specify { ordered: false } as shown in mongo bulk write docs so that the bulk insert will continue even if one insert fails?

I hope that was clear explanation. Looking forward to your response. thanks.

Hi, I'm using the package as follows:

Items = new Meteor.Collection('items');
Items.allow({insert: function() {return true;}});
Items.batchInsert([{item: 1},{item: 2}]);

And I 'm getting the following error:

Exception while invoking method 'procesarCsvVentas' TypeError: Object [object Object] has no method 'rawCollection'
at [object Object].m.(anonymous function) (packages/mikowals:batch-insert/batch-insert.js:161:1)
at maybeAuditArgumentChecks (packages/ddp/livedata_server.js:1599:1)
at packages/ddp/livedata_server.js:1513:1....

Can you help me?
Thanks!
Lionel.

Hey there,
As meteor/meteor#2668 seems to be fixed already, you could remove the warning in your README 😉

/Users/mg/.meteor/packages/meteor-tool/.1.1.8.tvnipv++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:245
throw(ex);
^
Error: The global definition of Mongo.Collection has changed since the job-collection package was loaded. Please ensure that any packages that redefine Mongo.Collection are loaded before job-collection.
at JobCollectionBase (packages/vsivsi_job-collection/packages/vsivsi_job-collection.js:1520:1)
at new JobCollection (packages/vsivsi_job-collection/packages/vsivsi_job-collection.js:3034:1)
at JobCollection (packages/vsivsi_job-collection/packages/vsivsi_job-collection.js:3032:1)
at insocl_crawler.js:59:14
at /Users/mg/www/insocl_crawler/.meteor/local/build/programs/server/app/insocl_crawler.js:324:4
at /Users/mg/www/insocl_crawler/.meteor/local/build/programs/server/boot.js:242:10
at Array.forEach (native)
at Function..each..forEach (/Users/mg/.meteor/packages/meteor-tool/.1.1.8.tvnipv++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/server-lib/node_modules/underscore/underscore.js:79:11)
at /Users/mg/www/insocl_crawler/.meteor/local/build/programs/server/boot.js:137:5

Hey,

After I added this package and tried performing a batch insert on a method call inside a "Meteor.isServer" block, I get an error saying that insert failed because of documents having duplicate _id. I'm not assigning _id's anywhere and currently use SimpleSchema.

I tried using this without attaching a schema to my collection and still have the issue. I have a deny rule for insert on the client and no allow insert because I want to modify the collection only on the server.

It seems the library isn't compatible with Meteor 1.7. I'll try to fork and test.

meteor: 1.10.2
batch-insert: 1.2.0

Can't use，error:
Exception while invoking method 'database.restore.version' TypeError: Cannot read property 'batchInsert' of undefined

Would be super helpful to also have batch upsert functionality. Should be doable. http://docs.mongodb.org/v3.0/reference/method/Bulk.find.upsert/

Thanks for the great work on this plugin!

I have used aldeed:collection2 to define schema for my Meteor collections. After calling batch insert on the collection, the createdAt field (auto field with Date.now() value) was null. Moreover, the after insert hook as also not called. Could you please let me know the reason for the issue and how can I resolve it?

Since you already have some tests ... why not set up automated tests using Travis CI? This way these tests would automatically run if someone (like me 😉) sends in a pull-request.

Here's a manual of how to set it up for a meteor-extension repository: http://practicalmeteor.com/testing-your-meteor-packages-on-travis-cis-faster-docker-based-infrastructure/

Does this package use Mongo's Bulk.insert()
http://docs.mongodb.org/manual/reference/method/Bulk.insert/#Bulk.insert
or does it loop through and make separate inserts for each doc?

I see CPU hangs when looping/inserting, so I'm looking for a better solution that does a bulk insert all at once instead of multiple insert operations.

Thanks
Don

I would like to use this package to upload huge csv file to mongodb. Whats the maximum count of the document for batch-insert?

When using the callback version, what is inside the error object? I seem to be getting always an array of objects, so is it always failing? However my DB doesn seem to chage.

I am also checking if the return value length matches the length of the array I am inserting, but it never seems to match either.

Maybe I am doing something wrong?

Hello,

when I use batchInsert in asynchronous inside a method of validateMethod, I get back the callback only if there was an error.

https://github.com/meteor/validated-method

How I can use it in asynchronous with always a callback in return, please?

thank you and good day

Answer to this is probably extremely obvious, but would batch inserts improve performance?

If so, is it substantial? Thanks !

Hi @mikowals

Thanks for making this package. Just what I need. Just one question. Will a batch-insert amount to (1) a total of one DDP update message or (2) as many DDP update messages as docs inserted into mongo?

I ask because if it's the latter I'd have to figure out a way to temporarily disable reactivity for connected clients or they'd probably crash.

/ Karl

Would it be possible to add an option to ignore duplicate inserts?

Not sure if something changed recently or it always was like this, but how do you differentiate between the error or result that gets returned.

Seems like the version where there isn't an error returns -> {"result":{"ok":<number of items inserted successfully>,"n":<total number of items tried to insert>},"ops":[{ thing inserted }, ... ], ,"insertedCount":<number of items inserted>,"insertedIds":["ids of items inserted", ... ]}

What's the easiest way to distinguish between that and an actual error ? Also why not set it up so that it returns the standard callback combo of (error, result) where if the error is undefined there wasn't any error and then return the result. Whereas if there is an error it's an object and result contains the same info as it does now.