Giter Club home page Giter Club logo

london's Introduction

Project London

Project London is an experimental implementation of etcd api on top of Azure Storage Table. The implementation is scoped to supporting kubernetes clusters. A full implementation of etcd api is not on the scope of this project. A full implementation of etcd api on top azure storage is also not possible due to way azure table perform transactions.

While the functional correctness has been validated, the project is still in alpha status and we are tracking a set of known issues.

How To Use

# clone the repo
git clone https://github.com/khenidak/london.git

cd london
# build. drop will be ./_output
make binary-build 


# run
./_output/london run \
--use-tls \
--cert-file=<path> \
--key-file=<path> \
--trusted-ca-file=<path> \
--account-name=<storage account name> \
--table-name=<storage table name> \
--primary-account-key=<storage account key>

## kubernetes api-server can now connect to https://localhost:2379 or alternatively use --listen-address argument to change listening address

Testing and Validation

the repo contains a set of tests as the following

  1. Unit tests make unit-tests validates functionality of data access layer.
  2. Integration tests make integration-tests validates the functionality using etcd client apis.
  3. E2E tests make e2e-test validates the functionality using an in-proc kubernets api-server backed by London.

london's People

Contributors

khenidak avatar aramase avatar paulgmiller avatar

Stargazers

jstoker avatar Xin Yan avatar Lawrence Gripper avatar Heba avatar Jordan Olshevski avatar David Justice avatar avatar Dario avatar Sertaç Özercan avatar Tarun Pothulapati avatar

Watchers

Joni Collinge avatar avatar William Zhang avatar Heba avatar avatar

Forkers

paulgmiller aramase

london's Issues

Optimize logging

We have way too many log lines that are either unnecessary or is using the wrong log level. Optimize by removing the unneeded ones and adjust log level for the remaining. And add log level knop in configuration

github workflow

We should re-arrange actions as the following

  • ...
  • run unit-tests
  • run integration-tests
  • get kubernetes release
  • run e2e-tests

instead of current order (which is also missing e2e).

List-Manager: Memory and Load optimization

Currently list manager performs a full-load of all events on it is first run and and accumulates events until Kubernetes performs a compact against the store. This leads to:

  1. A large events held in memory upon load (note: Watchers really need events after a List() call i.e. most recent).
  2. Unneeded memory consumption by London
  3. A race condition where watchers may not get events fast enough if London was still in the process

To fix that:

  1. in NewListManager() run event loop once. During this run
  • Don't maintain more than X number of events per key (watch out for events for deleted keys). (fixes item 1 above)
  • etcd listener should not start until this has finished (fixes item 3 above)
  1. add a maintenance step in the mgmt loop to trim events to keep them X per key. This step will run side by side with the current compact maintenance step

Too many open files and too many connections

randomly "too many open" or "can not assign address" errors will be logged. This is due to leaked connections between the binary and Azure storage table. The indicates a leak somewhere in Azure SDK. The error will not show up only in e2e tests.

Ignoring compact versioned when watch

Watch just perform a list with revision > target-revision. Watch should notice if revision < last-compacted-revision and return errors accordingly.

Pre Beta Tasks

  • Core
    • Split data into multiple records to allow max of 2MB key-values
    • Watch single key
    • review all filters for reading the smallest of comment
    • FIX THE PATCH UPDATE BUG
  • Configuration / cli
    • Add watch interval and default it to 250s
    • Validate user configuration
    • Add london version command and read tags/commits
    • Add msi integration
  • Release
    • Add binary packaging
    • Add docs
    • Add sample sytstemd unit file

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.