Giter Club home page Giter Club logo

table_sync's Introduction

TableSync · Gem Version Build Status Coverage Status

DB Table synchronization between microservices based on Model's event system and RabbitMQ messaging

Installation

gem 'table_sync'
$ bundle install
# --- or ---
$ gem install table_sync
require 'table_sync'

Usage

Usage information available in the Wiki.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/umbrellio/table_sync.

License

Released under MIT License.

Authors

Team Umbrellio


Supported by Umbrellio

table_sync's People

Contributors

0exp avatar 7dr1v3 avatar anotherregulardude avatar as-alstar avatar deniskem avatar dependabot[bot] avatar enthrops avatar eugeneyak avatar firstsano avatar justsnow avatar loadkpi avatar stefkin avatar tycooon avatar vegetableprophet avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

table_sync's Issues

Ошибка "ArgumentError: Cache expiration time is invalid, cannot be negative: -0.000102"

Во время выполнения кода

TableSync::Publishing::Single
  .new(object_class: 'Player::FraudStatus', original_attributes: data, debounce_time: 0)
  .publish_later

Возникает ошибка ArgumentError: Cache expiration time is invalid, cannot be negative: -0.000102
Судя по всему ошибка возникает тут - https://github.com/umbrellio/table_sync/blob/master/lib/table_sync/publishing/helpers/debounce.rb#L130 , потому что ActiveSupport cache не может переварить Time.current .

image

rename

discuss the idea of renaming table_sync -> rofl_sync

Feature: ignore_fields (an ability to ignore a list of attributes that should not be synced)

It is suitable for "temporary" ignore some fields in order to make some technical works with some fields during the regular working sync) for example: email delivery over the big amount of clients with temporary changed email field during the active updates of the client table (but we want the email field to remain non-changeable)

At the moment, to stimulate this, we need to build complex combinations of wrap_receiving, additional_data, mapping_overrides, only and etc.

It would be nice to have the new option with both dynamic and static behavior (in order to calculate ingorance list in runtime or in load time)

[Recevier] Support for "except" (in "only" manner)

We have only:

class Rabbit::Handler::MyProject::TableSync < TableSync::ReceivingHandler
   receive "User", to_table: :users do
      only %w[project_id user_name]
   end
end

It would be convenient to have except too:

class Rabbit::Handler::MyProject::TableSync < TableSync::ReceivingHandler
   receive "User", to_table: :users do
      except %w[user_age]
   end
end

TableSync doesn't work if hook was used in model

TableSync defines hooks directly in model without including or prepending code and if you define similar hooks, tablesync's hooks are shadowed and don't work

class Test < Sequel::Model
  TableSync.sync(self)

  def after_create
    super
    p "AFTER CREATE"
  end
end

Test.create

You will see "AFTER CREATE" in console but data won't be synced

Проблемы с потенциальными дедлоками для ActiveRecord и отсуствием блокировок для Sequel

  1. Не понимаю, что проверяют спеки avoid dead locks, так как там переопределяется upsert и мы там не берем вообще никаких локов. К тому же эти спеки отжирают больше 3/4 времени выполнения вообще всех спеков из-за sleep 2.
  2. В Актив Рекорде берется куча локов под транзой FOR NO KEY UPDATE и как раз там вероятность дедлока крайне высока?
  3. В Сиквеле не берется никаких локов, соотвественно, победит та транза, которая коммитнется последней и перетрет изменения предыдущей транзы
  4. Кажется, вместо кучи локов в актив рекорде и отсутсвии этих локов в сиквеле имеет смысл изменить уровень изоляции транзы на какую-нибудь сериализацию 🤔

Unclear additional_data behavior of different event's types.

Current implementation intends that update and destroy events must have same attribute fields.
As consequence, dirty additional_data appears.
I think it will be useful if it is possible to create additional_data for each type of event separately.

Below case that shows dirty additional_data.
Let's imagine that receiver get update-event with attributes:

   {
     id: 12,
     name: "kek"
     ...,
   }

and destroy_event with:

   {
     id: 12,
   }

receiver is:

receive "Model", to_table: :models do
  additional_data do |current_row:|
    next {} if current_row.keys.count == 1 # destroy condition
    super_name = current_row[:name].upcase
    {
      name: super_name
    }
  end
  on_destroy do |attributes:, target_keys:| # rubocop:disable Lint/UnusedBlockArgument
    Model[attributes[:id]].update
  end
end

delete TableSync::Plugins

TableSync::Plugins it is not plugins, It is API for monkey patching which makes monkey patching difficult and does not resolve problems with monkey patching

absolutely useless

Undefined behavior when using the if and unless predicates

Now:
If I setup synchronization with if predicate like
TableSync.sync(self, if: -> (*) { syncable? })
model will send updates if predicate is truthy, but destroy will always send.

What I want:
The model must check predicates when sending both types of updates.

couple ideas

  1. add message versions and a counter that will send errors for message omissions
  2. add feature that allows to send more than one model in batches. it resolve some consistency issues.
  3. add information to readme, why should we use batches and what problems does it solve.

destroy with versions

at now destroy event is ignoring version_key but it is wrong behavior for obvious reasons

Separate logging for different receivers

We want to have an ability to define/configure an individual logger for each receiver separately (incoming messages, malformed messages, etc). this is a good way to disable logging for unnecessary (or very active) receivers on staging systems (you can disable some super-active logging operations, but not the hole logging system).

Problems with routing_metadata_callable

  1. Maybe we should rename it to something like routing_headers_callable? Right now it's not very clear that it only affects RabbitMQ headers.
  2. Currently it yields only model class name and raw attributes, but I might have attributes_for_sync method defined so I need to do stuff like model = model_class.constantize.new(attrs).try(:attributes_for_sync) which is slow and inefficient. Would be cool to be able to get processed attributes there as well (or instead of raw attributes).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.