Giter Club home page Giter Club logo

appengine-sharded-counters-python's Introduction

Sharded Counters

A simple application that demonstrates two approaches to sharding counters.

To achieve higher throughput across different instances, counters can be sharded to avoid concurrency issues. This application performs this in two different ways:

  • Simple Sharding: Uses a constant to define the number of shards and randomly picks an index up to this number when incrementing the counter. Each shard is stored in the datastore using one of these indices as ID.

  • General Sharding: Stores the number of shards in the datastore and randomly picks an index up to this number when incrementing the counter. Each shard is stored in the datastore using one of these indices as ID, but also using a counter name as an ancestor.

Products

Language

APIs

Dependencies

appengine-sharded-counters-python's People

Contributors

danholevoet avatar dhermes avatar fredsa avatar tmatsuo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

appengine-sharded-counters-python's Issues

GeneralShardedCounter and memcache mismatch

If a GeneralShardedCounter is accessed often enough, the memcache representation will get out of sync.

Running 100-1000 parallel operations incrementing the counter (and occasionally reading its value) results in memcache reaching a level between 0.1% and 1% higher than the actual sharded counter value. I've tested this numerous times on 10,000 operations; the counter reaches 10,000 as the operations (task queue tasks) are spawned, but the count of operations finished (each operation increments the counter on exit) reaches anywhere from 10,001 to 10,100 or thereabout.

If the memcache value expires, and any number of operations has just incremented a shard but not yet incremented memcache, a call to get_count will count the just-incremented shards, store the value in memcache, and then memcache will be incremented, leaving those operations counted twice, with a resulting higher count in memcache.

My workaround is to invalidate the cache if I want to get a more "accurate" number, but I think the memcache results here should be taken as no more than a hint.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.