Giter Club home page Giter Club logo

django-shard-library's Introduction

Django-Shard-Library

Build Status

Introduction

  • Python library for supporting sharding in Django ORM.

Requirement

  • Django 2.0.0 or higher
  • Python 3.6 or higher
  • MySQL or Mariadb only

Dependency

Usage

If you want to use only shard

# in settings.py

INSTALLED_APPS = [
    # ...
    'shard',
]

DATABASES = ConfigHelper.generate_database_configs(
    unshard = {
        'default': {
            'conn_max_age': 0,
            'master': {
                'url': 'mysql://user:pwd@host/metadata',
                'conn_max_age': 1000,  # Optional
            },
            'slaves': [{
                'url': 'mysql://user:pwd@host/metadata?sql_mode=STRICT_TRANS_TABLE&charset=utf8',
                'conn_max_age': 500,  # Optional
            }]
        },
    },
    shard = {
        'SHARD_GROUP_A': {
            'database_name': 'product',
            'logical_count': 4,
            'conn_max_age': 0,  # Optional
            'options': {  # Optional
                'sql_mode': 'STRICT_TRANS_TABLE',
                'charset': 'utf8',
            },
            'shards': [
                {
                    'master': {
                        'url': 'mysql://user:pwd@host/',
                        'conn_max_age: 400,  # Optional
                        'options': {  # Optional
                            'charset': 'latin',
                        },
                    },
                    'slaves': [{
                        'url': 'mysql://user:pwd@host/',
                        'conn_max_age: 600,  # Optional
                    }, {
                        'url': 'mysql://user:pwd@host/',
                        'options': {  # Optional
                            'charset': 'latin',
                        },
                    },]
                },
                {
                    'master': { 'url': 'mysql://user:pwd@host/', },
                    'slaves': [{ 'url': 'mysql://user:pwd@host/' }, { 'url': 'mysql://user:pwd@host/' },]
                },
            ],
        },
    },
)
DATABASE_ROUTERS = ['shard.routers.specific.SpecificRouter', 'shard.routers.shard.ShardRouter']
# in models.py
class ShardModel(ShardMixin, models.Model):
    user_id = models.IntegerField(null=False, verbose_name='User Idx')

    shard_group = 'SHARD_GROUP_A'
    shard_key_name = 'user_id'
  • shard_key is supported only integer type.

If you want to use shard_static together.

# in settings.py

INSTALLED_APPS = [
    # ...
    'shard',
    'shard_static',
]

DATABASES = ConfigHelper.generate_database_configs(
    # ...
)

DATABASE_ROUTERS = ['shard.routers.specific.SpecificRouter', 'shard_static.routers.ShardStaticRouter']
# in models.py
class ExampleStaticModel(BaseShardStaticModel):
    field1 = models.CharField(max_length=64, null=False)
    field2 = models.CharField(max_length=64, null=False)
  • Must includes shard_static to INSTALLED_APPS
  • Must implement transmitter and BaseStaticTransmitStatus.
  • Must use shard_static.routers.ShardStaticRouter
    • ShardStaticRouter is extended feature that all of the ShardRouter.

If you want to change replica count

# in settings.py

SHARD_REPLICA_COUNT_SETTING = {
    'SHARD_GROUP_A': 1024,
    'SHARD_GROUP_B': 128,
}
  • Default is 512.

How to run test

  • Step 1: Run database for testing.
    make run-test-db

  • Step 2: Run test.
    make test

  • Step 3: When end of test, you stop to database.
    make stop-test-db

Warning

  • If you want to use this with django 2.1.0, Be careful missing shard key when getting queryset by raw query.

Prior art

django-shard-library's People

Contributors

junglekim avatar namenu avatar reduxionist avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.