Giter Club home page Giter Club logo

drift-config's Introduction

Build Status codecov Latest version on PyPi

Drift Config

Drift Multi-Tenant Configuration Database - Drift-MTC.

This library enables multi-tenant software deployment and operation for local and cloud based development.

Installation

For developer setup pipenv is used to set up virtual environment and install dependencies.

pip install --user pipenv
pipenv install --dev -e ".[s3-backend,redis-backend,trigger]"

This installs drift-config in editable mode with S3 and Redis backend support and lambda trigger support.

Using the library

The config can be accessed directly from S3 or Redis cache but for this tutorial we will pull the config and store it on a local hard drive. (Note that the following S3 url only worksfor members of Directive Games organization):

pipenv shell
driftconfig init s3://relib-test/directive-games

# To get future updates of the config just run following command:
driftconfig pull dgnorth

Now the config DB is stored locally. Run the following command and note the Local file path:

dconf list configs

To use the Python library start an interpreter:

pipenv run python

First let's list out some fun stuff. Run only one command at a time otherwise you will just get a big wall of text with no context.

# Specify which Drift config to use. Only needed if there are more than one stored locally.
# Note that this environment variable is always set on EC2 instances during launch and typically
# points to a Redis cache.
import os
os.environ["DRIFT_CONFIG_URL"] = "dgnorth"

# Now grab the config DB in its entirety
from driftconfig.util import get_default_drift_config
c = get_default_drift_config()

# Show list of table names
c.tables.keys()

# A table has a get() and find() function to fetch a row specifying the primary key and to select
# multiple rows based on a search criteria respectively.
tiers = c.get_table("tiers")
tiers.find()  # No search criteria returns all rows.

# Get the row for DEVNORTH
tiers.get({"tier_name": "DEVNORTH"})

# Now let's get some tenant info. This here will return config for all deployables associated with
# the tenant 'dg-oasis' on DEVNORTH:
tenants = c.get_table("tenants")
tenants.find({"tier_name": "DEVNORTH", "tenant_name": "dg-oasis"})

# It's possible to cheat a little bit and do some introspection. Here we can find the primary key
# definition for tenants:
tenants._pk_fields

# The primary in 'tenants' table is composed of three fields: tier_name, deployable_name and tenant_name.
# We can thus get a specific record using all three fields:
tenants.get({"tier_name": "DEVNORTH", "tenant_name": "dg-oasis", "deployable_name": "drift-base"})

# Every bit of info in Drift Config DB should be easily queryable using simple commands as shows above. There
# is however also another helper function that goes one step further.
from driftconfig.util import get_drift_config
c = get_drift_config(tier_name="DEVNORTH")  # We must specify the tier here, usually found in DRIFT_TIER env var.

# str-ifying 'c' yields a lot of info:
c

# Normally this function is used in conjunction with a specific tenant and a deployable/app. A request comes
# through some endpoint and we need the configuration context for it:
c = get_drift_config(tier_name="DEVNORTH", deployable_name="drift-base", tenant_name="dg-oasis")

# Now we have everything at our fingertips using a dot-notation syntax:
c.tier
c.deployable
c.tenant
c.product
c.organization
c.domain.get()
c.source

That's all!

Running unittests

pipenv run pytest

Python 2/3 compatibility

Set up the virtualenv by adding --three or --two to the pipenv install command line. The syntax to run the unittests for either version of python is the same.

When switching between version, you need to remove the Pipfile that gets created from the setup.py file since it contains the python version and isn't automatically owerwritten when a new environment is created. You may also need to remove .pyc files, e.g. with a command such as:

find . -name "*.pyc" --exec rm "{}" ";"

The steps to switch between python versions are encapsulated in the files scripts/init_py2.sh and scripts/init_py3.sh. To switch between versions, cd to the root folder and source the scripts. E.g.:

cd drift-config
. scripts/init_py3.sh

Initialize from url

If you already have a config db, initialize it for local development. Example:

driftconfig init s3://some-bucket/config-folder

WARNING - OLD STUFF BELOW: Usage

Run driftconfig --help for help on usage.

Errata: Run dconf --help as well.

Install Cache trigger

Drift config with an S3 based origin can be cached in a Redis DB with very high concurrency. An AWS lambda will monitor the S3 bucket and update the cache when there is an update.

The lambda is set up using Zappa. The Zappa config file is generated from a template which sets the S3 bucket name (origin), subnet id's and security group id's of the selected drift tier.

To update (or deploy) the trigger on AWS, run the following commands:

python scripts/update-trigger.py

Releasing drift-config

Drift-Config is a Pypi package which is released here: https://pypi.org/project/python-driftconfig/

To create a new release simply add a git tag with a semantic version. The Travis-CI process will create a new build and upload it to Pypi.

The following command will tag the latest commit on the current branch and create a new pypi package.

git tag 1.0.1
git push --tags

After a few minutes the package should be built and uploaded. You can now run the following command in services which depend on drift-config:

pipenv update

This will update the python-drift-config package to the latest version in that service. You can verify that this was successful by checking the Pipfile.lock file in the project folder.

drift-config's People

Contributors

aripall avatar judgeaxl avatar kristjanvalur avatar mattig avatar nonnib avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.