outoforder / multidb Goto Github PK

View Code? Open in Web Editor NEW

57.0 57.0 28.0 87 KB

An ActiveRecord extension to switch between multiple databases, such as in a master/slave setup

License: MIT License

Ruby 100.00%

multidb's People

Contributors

Stargazers

Watchers

multidb's Issues

Use Multidb and establish_connection in same application

I have been using Multidb to switch b/w my master-slaves psql databases for various part of code.

However I also want to have a model level 'establish_connection' like functionality to connect a model to a different ms sql database.

In presence of multidb -> establish_connection is not working to switch to a different database. Any ways to bypass multidb completely for a single model.

Suggestion to invoke ActiveRecord::Relation if returned from a block

If you were to perform the below code:

@posts = Multidb.use(:slave) do
  Post.where(is_active: true)
end
@posts.each {|p| ... }

the query won't actually be invoked until the .each call, which means it won't happen while the slave is selected. This is because Post.where(is_active: true) is merely an ActiveRecord::Relation. I've found this annoying, and easy to mess up so we monkey patched multidb to invoke .all on any ActiveRecord::Relation returned from use. I could issue a PR for this if you want.

Also, I highly recommend including some form of logging so that a user can see in the debug output which host/db the query ran against. I also have some code for this ripped from another sharding gem.

Is the timeout setting working with this gem?

I have this setting:

default: &default
  adapter: postgresql
  encoding: unicode
  pool: 5
  host: <%= ENV['DB_HOST'] || 'localhost' %>
  port: <%= ENV['DB_PORT'] %>
  database: <%= ENV['DB_NAME'] %>
  username: <%= ENV['DB_USERNAME'] %>
  password: <%= ENV['DB_PASSWORD'] %>

staging:
  <<: *default
  pool: 1
  timeout: 5
  multidb:
    databases:
      master:
        - host: <%= ENV['DB_HOST'] %>
          port: <%= ENV['DB_PORT'] %>
      slave:
        - host: <%= ENV['REPLICA_DB_HOST'] %>
          port: <%= ENV['REPLICA_DB_PORT'] %>

with a timeout of only 5sec

On my staging machine I add an iptables rule:

iptables -A OUTPUT -d 10.10.12.120 -j DROP

Now I am visiting a page of a controller that uses Multidb.use(:slave, &block)
The browser hang until I get a 504
The rails log just does not show much:

Started GET "/j?q=java&l=&button=&sp=search" for .... at 2017-06-28 05:39:18 +0000
Processing by SearchController#index as HTML
  Parameters: {"q"=>"java", "l"=>"", "button"=>"", "sp"=>"search"}

ActiveRecord::Base::ConnectionSpecification doesn't exist in AR 4.0.0

uninitialized constant ActiveRecord::ConnectionAdapters::ConnectionSpecification (NameError)

hey, getting this weird error:

gems/ar-multidb-0.1.11/lib/multidb/balancer.rb:13:in `initialize': uninitialized constant ActiveRecord::ConnectionAdapters::ConnectionSpecification (NameError)
    from /Users/znouza/.rvm/gems/ruby-1.9.3-p448@rp2/gems/ar-multidb-0.1.11/lib/multidb/balancer.rb:39:in `new'
    from /Users/znouza/.rvm/gems/ruby-1.9.3-p448@rp2/gems/ar-multidb-0.1.11/lib/multidb/balancer.rb:39:in `block (2 levels) in initialize'
    from /Users/znouza/.rvm/gems/ruby-1.9.3-p448@rp2/gems/ar-multidb-0.1.11/lib/multidb/balancer.rb:38:in `each'
    from /Users/znouza/.rvm/gems/ruby-1.9.3-p448@rp2/gems/ar-multidb-0.1.11/lib/multidb/balancer.rb:38:in `block in initialize'
    from /Users/znouza/.rvm/gems/ruby-1.9.3-p448@rp2/gems/ar-multidb-0.1.11/lib/multidb/balancer.rb:36:in `each_pair'
    from /Users/znouza/.rvm/gems/ruby-1.9.3-p448@rp2/gems/ar-multidb-0.1.11/lib/multidb/balancer.rb:36:in `initialize'
    from /Users/znouza/.rvm/gems/ruby-1.9.3-p448@rp2/gems/ar-multidb-0.1.11/lib/multidb/configuration.rb:50:in `new'
    from /Users/znouza/.rvm/gems/ruby-1.9.3-p448@rp2/gems/ar-multidb-0.1.11/lib/multidb/configuration.rb:50:in `create_balancer'
    from /Users/znouza/.rvm/gems/ruby-1.9.3-p448@rp2/gems/ar-multidb-0.1.11/lib/multidb/configuration.rb:10:in `balancer'
    from /Users/znouza/.rvm/gems/ruby-1.9.3-p448@rp2/gems/ar-multidb-0.1.11/lib/multidb/model_extensions.rb:14:in `connection_with_multidb'

my database.yml looks like:

production:
 ....
  #socket: /var/run/mysqld/mysqld.sock
  timeout: 5000
  multidb:
    databases:
      slave:
        database: my_slave_db
        host: localhost

running rails 3.2.17. any idea?
Thanks

support postgres and sqlite

hi - is it possible to support switching between a 'production' postgres db and a :memory: sqlite db? How would this be expressed in the databases.yml file?

Does this work on Heroku

I have the following database.yml but it does not work

staging:
  url: <%= ENV['DATABASE_URL'] %>
  pool: 7
  multidb:
    databases:
      test_users:
        url: <%= ENV['DATABASE_URL_TEST_USERS'] %>
        pool: 7

I get this error at the time of deployment to Heroku -

remote:        Bundle completed (64.73s)
remote:        Cleaning up the bundler cache.
remote: -----> Preparing app for Rails asset pipeline
remote:        Running: rake assets:precompile
remote:        rake aborted!
remote:        NameError: uninitialized constant ActiveRecord::SessionStore
remote:        /tmp/build_df2ad8d7803b52702b8cf85917e61c7b/vendor/bundle/ruby/2.2.0/gems/multidb-4.0.1/lib/multi_db/after_initialize_patches.rb:1:in `<top (required)>'
remote:        /tmp/build_df2ad8d7803b52702b8cf85917e61c7b/vendor/bundle/ruby/2.2.0/gems/multidb-4.0.1/lib/multi_db/engine.rb:17:in `block in <class:Engine>'
remote:        /tmp/build_df2ad8d7803b52702b8cf85917e61c7b/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/lazy_load_hooks.rb:38:in `instance_eval'
remote:        /tmp/build_df2ad8d7803b52702b8cf85917e61c7b/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/lazy_load_hooks.rb:38:in `execute_hook'
remote:        /tmp/build_df2ad8d7803b52702b8cf85917e61c7b/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/lazy_load_hooks.rb:45:in `block in run_load_hooks'
remote:        /tmp/build_df2ad8d7803b52702b8cf85917e61c7b/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/lazy_load_hooks.rb:44:in `each'

ConnectionHandler

Hi,
Just a question here.
Rails 4(.2?) introduced a threadsafe runtime registry to store the connection handler, which you don't use.
Wouldn't that be an interested addition?

ActiveRecord log subscriber debug broken

The signature of Multidb::LogSubscriberExtension#debug has a bug that's breaking strict loading violation reports in our Rails 7.0 app. At least in current Rails versions, the param to debug is supposed to be optional (https://github.com/rails/rails/blob/v7.0.8/activerecord/lib/active_record/log_subscriber.rb#L127). You can see it being called with no args in ActiveRecord::LogSubscriber#strict_loading_violation.

I'm happy to submit a PR (unless you care to). Should be a one-liner after all. Just checking to make sure this is still actively maintained first.

Dont fallback on Rails development

I am testing multidb on my dev environment (rails), and I dont want to use fallback DB.

But looks like fallback: false is not working -- the app is always connecting to default db??

development:
  url: postgres://my_user@localhost:5432/my_db_dev
  multidb:
    fallback: false
    databases:
      db_test:
        url: postgres://my_user@localhost:5432/my_db_dev_test

UPDATE:
Also this does not work on Heroku (our production env). So I tried the following on heroku run rails c

Multidb.use(:db_test) { User.count } ...it always falls back to default prod db.

MultiDB does not work in Rails 5.2 with Phusion Passenger

I am upgrading an application from Rails 4.2 to Rails 5.2 and we use Multidb to handle our master/slave DB access. However in testing our application in our pre-staging ENV the application fails in AR's connection pool.

We receive an undefined method []' for nil:NilClassinactiverecord-5.2.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:382`

Debugging has been a pain as it's hard to debug passenger when it is running in forking mode. (when running passenger in non-forking mode (so we can debug) the issue does not occur).

Right now this is happening 100% of the time. However, if I remove only the multidb gem then things work fine.

From looking through the code in passenger and in MultiDB the best thing I can figure is that

passenger forks
passenger calls ActiveRecord::Base.clear_all_connections! after spawning a new fork
And at this point the connection pool is discarded!

From walking through the code (and lots of manual log lines)
What happens is then when clear_all_connections! is called, the connection pool tries to be "smart" and realizes that the PID is different (through their @owner_to_pool hash) which upon accessing a new PID discards the connection pool (which is the one that MultiDB is keeping hold of).

Based on looking through this AR code it appears that Multidb should instead hold a reference to the ConnectionHandler and NOT the ConnectionPool!

Snyk complains of license issues when using multidb in a project

Snyk was complaining in a project we tried to incorporate ar-multidb. I opened #31 to fix the issue. Feel free to make changes or comments, thanks!

ActiveRecord::ConnectionNotEstablished with rails 3.2

I try to use multidb in production environment with rails 3.2 but got following error for any request:

ActiveRecord::ConnectionNotEstablished (ActiveRecord::ConnectionNotEstablished):
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:323:in `checkout_new_connection'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:247:in `block (2 levels) in checkout'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:242:in `loop'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:242:in `block in checkout'
  /usr/lib/rbenv/versions/1.9.3-p327/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:239:in `checkout'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:102:in `block in connection'
  /usr/lib/rbenv/versions/1.9.3-p327/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:101:in `connection'
  /home/avaxhome/app.com/shared/bundle/ruby/1.9.1/bundler/gems/multidb-2a9ad2f341fd/lib/multidb/balancer.rb:23:in `connection'
  /home/avaxhome/app.com/shared/bundle/ruby/1.9.1/bundler/gems/multidb-2a9ad2f341fd/lib/multidb/balancer.rb:87:in `current_connection'
  /home/avaxhome/app.com/shared/bundle/ruby/1.9.1/bundler/gems/multidb-2a9ad2f341fd/lib/multidb/model_extensions.rb:18:in `connection_with_multidb'
  activerecord (3.2.11) lib/active_record/query_cache.rb:67:in `rescue in call'
  activerecord (3.2.11) lib/active_record/query_cache.rb:61:in `call'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
  actionpack (3.2.11) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  activesupport (3.2.11) lib/active_support/callbacks.rb:405:in `_run__2689738203702554028__call__2455757002539370000__callbacks'
  activesupport (3.2.11) lib/active_support/callbacks.rb:405:in `__run_callback'
  activesupport (3.2.11) lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  activesupport (3.2.11) lib/active_support/callbacks.rb:81:in `run_callbacks'
  actionpack (3.2.11) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  rack (1.4.3) lib/rack/sendfile.rb:102:in `call'
  actionpack (3.2.11) lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  actionpack (3.2.11) lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  actionpack (3.2.11) lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  railties (3.2.11) lib/rails/rack/logger.rb:32:in `call_app'
  railties (3.2.11) lib/rails/rack/logger.rb:16:in `block in call'
  activesupport (3.2.11) lib/active_support/tagged_logging.rb:22:in `tagged'
  railties (3.2.11) lib/rails/rack/logger.rb:16:in `call'
  actionpack (3.2.11) lib/action_dispatch/middleware/request_id.rb:22:in `call'
  rack (1.4.3) lib/rack/methodoverride.rb:21:in `call'
  rack (1.4.3) lib/rack/runtime.rb:17:in `call'
  rack (1.4.3) lib/rack/lock.rb:15:in `call'
  rack-cache (1.2) lib/rack/cache/context.rb:136:in `forward'
  rack-cache (1.2) lib/rack/cache/context.rb:245:in `fetch'
  rack-cache (1.2) lib/rack/cache/context.rb:185:in `lookup'
  rack-cache (1.2) lib/rack/cache/context.rb:66:in `call!'
  rack-cache (1.2) lib/rack/cache/context.rb:51:in `call'
  airbrake (3.1.5) lib/airbrake/rack.rb:41:in `call'
  airbrake (3.1.5) lib/airbrake/user_informer.rb:12:in `call'
  railties (3.2.11) lib/rails/engine.rb:479:in `call'
  railties (3.2.11) lib/rails/application.rb:223:in `call'
  railties (3.2.11) lib/rails/railtie/configurable.rb:30:in `method_missing'
  unicorn (4.3.0) lib/unicorn/http_server.rb:530:in `process_client'
  unicorn (4.3.0) lib/unicorn/http_server.rb:605:in `worker_loop'
  newrelic_rpm (3.5.3.25) lib/new_relic/agent/instrumentation/unicorn_instrumentation.rb:18:in `call'
  newrelic_rpm (3.5.3.25) lib/new_relic/agent/instrumentation/unicorn_instrumentation.rb:18:in `block (4 levels) in <top (required)>'
  unicorn (4.3.0) lib/unicorn/http_server.rb:487:in `spawn_missing_workers'
  unicorn (4.3.0) lib/unicorn/http_server.rb:137:in `start'
  unicorn (4.3.0) bin/unicorn:121:in `<top (required)>'
  /home/avaxhome/app.com/shared/bundle/ruby/1.9.1/bin/unicorn:23:in `load'
  /home/avaxhome/app.com/shared/bundle/ruby/1.9.1/bin/unicorn:23:in `<main>'

I use Multidb.use(:slave, &block) in around_filter.

database.yml:

login: &login
  adapter: postgresql
  host: aaa.bbb.ccc.ddd
  username: app
  password: ***
  encoding: UTF8

development:
  database: app
  <<: *login

# Warning: The database defined as 'test' will be erased and
# re-generated from your development database when you run 'rake'.
# Do not set this db to the same as development or production.
test:
  database: app_test
  <<: *login

production:
  database: app
  <<: *login
  multidb:
    databases:
      slave:
        host: aaa.bbb.ccc.eee

I changed password and host fields.

I have no idea what's happening.

Error message "Balancer not initialized. You need to run Multidb.setup first" is misleading

I believe the intent is to inform the user to run Multidb.init, not Multidb.setup (which does not exist).

ActiveRecord::Base::ConnectionSpecification doesn't exist in AR 4.0.0

You need something like

if ActiveRecord::VERSION::STRING >= "4.0.0"
  ConnectionSpecification =  ActiveRecord::ConnectionAdapters::ConnectionSpecification
else
  ConnectionSpecification = ActiveRecord::Base::ConnectionSpecification
end

Source: zendesk/active_record_shards#5

Can it fallback only when connection fail

Let suppose I have a master db and slave in different availability zones.
I would like to use the slave but if slave becomes unavailable I would like to fallback to master.
Is that what the fallback option doing? I looked at the code it does not seem so.

outoforder / multidb Goto Github PK

multidb's People

Contributors

Stargazers

Watchers

Forkers

multidb's Issues

Recommend Projects

Recommend Topics

Recommend Org