alejandrodevs / mass_insert Goto Github PK

Mass database insertion in Rails

License: MIT License

Ruby 98.88% Shell 1.12%

insertion database mysql ruby activerecord rails single-database-insertions

mass_insert's Introduction

MassInsert

This gem aims to provide an easy and faster way to do single database insertions in Rails. Support Mysql, PostgreSQL and SQLite3 adapters. It depends on ActiveRecord.

Installation

Add this line to your application's Gemfile:

gem 'mass_insert'

And then execute:

$ bundle

Or install it yourself as:

$ gem install mass_insert

Usage

To use MassInsert gem you need to call mass_insert method from your ActiveRecord model and pass it an array with the values that you want to persist into the database.

values = [
  {
    name:   'Jay',
    email:  '[email protected]',
    age:    15
  },
  {
    name:   'Beverly',
    email:  '[email protected]',
    age:    24
  }
]

User.mass_insert(values)

Allow primary key

Sometimes you can need to insert records forcing primary keys. Just pass the primary_key option with true. Example...

values = [
  {
    id:     1000, # Force primary key.
    name:   'Jay',
    email:  '[email protected]',
    age:    15
  }
]

User.mass_insert(values, primary_key: true)

Insertion per batches

Due you can get a database timeout error you can specify that the insertion will be in batches. Just pass the per_batch option with the records per batch. Example...

User.mass_insert(values, per_batch: 1000)

Handle unique index on MySQL

Sometimes we want to ignore errors when adding duplicated records. MySQL has the ability to do that with ON DUPLICATE KEY UPDATE. By using the option handle_duplication we will ignore the new values by doing:

User.mass_insert(values, handle_duplication: true)

INSERT INTO table (a,b,c) VALUES (1,2,3)
  ON DUPLICATE KEY UPDATE a=a,b=b,c=c;

Advantages

Faster. It depends on the computer and the data but these are some results...

PostgreSQL - Saving 10,000 records in 0.49s

Attention

Since this is a single database insertion your model validations will be ignored, then if you use this gem you need to be sure that information is OK to be persisted.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/alejandrodevs/mass_insert. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

The gem is available as open source under the terms of the MIT License.

Code of Conduct

Everyone interacting in the MassInsert project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.

mass_insert's People

Contributors

Stargazers

Watchers

Forkers

ignoch joseluistorres agilekoding marcelocajueiro jairovm

mass_insert's Issues

Interoperability with Rails 6.0 `insert_all`?

Hi @alejandrodevs - thanks for this gem! I found it by looking for a backport of the Rails 6 bulk insert function insert_all.

It seems like it uses the same syntax, is that correct? I ask because i'm wondering, if I use it in a Rails 5.2 project that one day gets updated to Rails 6, are there any changes i should think about?

Documentation when running tests

Hi @AlejandroGutierrez I'm sure you are aware of this, perhaps you should include in the README how to run migrations first in order to run tests

bundle exec rake db:create RAILS_ENV=mysql2
bundle exec rake db:create RAILS_ENV=postgresql
bundle exec rake db:migrate RAILS_ENV=mysql2
bundle exec rake db:migrate RAILS_ENV=postgresql

And update username and password in database.yml I hope this helps!

Cheers

Implicit conversion from data type varchar to varbinary(max) is not allowed

It show an error when trying to insert an varchar value into varbinary data type. ActiveRecord assigns varbinary data type in boolean values.

This is the error:
TinyTds::Error: Implicit conversion from data type varchar to varbinary(max) is not allowed. Use the CONVERT function to run this query.

License missing from gemspec

RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can image, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.

There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!

Appendix:

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies

p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post aobut this project for more information.