ubervu / mongo-pool Goto Github PK

Need

As a developer
I want to be able to use as much code as possible from an open-source project instead of rewriting it
So that I use it at its full potential, avoid reimplementing things and copy-paste code

Deliverables

Make getattr method easier to be extended

Solution

Break getattr logic in two parts. One that gets the connection to the cluster based on the name of the database to be accessed and one that returns the database from the cluster. This way, both can be modified to add functionality.

TODO

• write _get_connection_by_db_name method
• write _init_database method
• change getattr to use them
• write test to check that a connection is created only once for a cluster

Notes

This will enable us to use proper inheritance in private MongoPool

#ETA 0.5p

• Create a USP (unique selling proposition), something catchy that can be attached to the MongoPool name if anyone wants to share it. Something like “All your mongos in one place.”
• Think about what MongoPool fixes and put that before everything. Why is it needed, what frustrations does it fix that occurred in the past. Describe that shitty past and compare it with using MongoPool.
• Take advantage of "The Sinatra test" (aka if you can do it in XXX you can do it anywhere.) uberVU (now Hootsuite) is renown in big data tech. Present how MongoPool powers X servers and Y volumes of data, serving Z clients, etc.
• A logo or even some sort of schema would be great. A server-side project is hard to demo, so having a catchy visual might make a big difference in attracting an initial audience. It doesn't have to be as elaborate as a real-life product logo. Can be hard to scale and childish, as long as it conveys an image. Maybe @MihneaVU can help. Here are two examples
  • 
  • 
• Split README into more sections. The usage part is kind of bloated. This is an opportunity to highlight use-cases. Have subtitles like Single cluster, Multiple clusters, Dynamic paths, etc.
• Make examples as brief as possible, otherwise the lib doesn't seem as easy to use. E.g. isn't

>>> blogs = mongopool.blogs
>>> blogs
Database(MongoClient('127.0.0.1', 27017), u'blogs')
>>> posts = mongopool.posts
>>> posts
Database(MongoClient('127.0.0.1', 27017), u'posts')

the same as?

>>> mongopool.blogs
Database(MongoClient('127.0.0.1', 27017), u'blogs')
>>> mongopool.posts
Database(MongoClient('127.0.0.1', 27017), u'posts')

• Don't be sloppy with indenting
  

/cc @claudiu-coman

Need

As a developer
I want my packages to follow the naming convention
So that I can import them as expected

Deliverables

• change package name to mongo_pool

Solution

Prerequisites

• [required dependency for identifying the problem, scope and then solution for this issue]

TODO

• rename repository to mongo_pool https://help.github.com/articles/renaming-a-repository
• change the name of the package in setup.py
• change the url of the package
• rename mongopool folder to mongo_pool
• rename mongopool.py to mongo_pool.py
• change imports in init.py to point to mongo_pool instead of mongopool
• review documentation and replace mongopool with mongo_pool where needed

Notes

PR #26

#ETA [0-2]p

Passing optional arguments based on their position produces confusion because you might forget what does a certain value stand for and can lead to hard to find bugs if the method signature is changed.

To avoid these situations, it is better to pass arguments as keyword arguments.

def foo(a, b=None)
Wrong

foo(2, 3)

Better

foo(2, b=3)

Since the code already has nice, formatted docstrings, we should take advantage and host the docs on http://readthedocs.org/, as most serious python projects do.

cc @bgdnc @claudiu-coman

Need

As a developer
I want an easy way to get a connection to a cluster based on its label
So that I can reuse connections

Deliverables

• get_cluster method that gets a cluster label and returns a connection to the cluster
• tests for the method

#ETA 0p

• implement method to share the same version number in the entire project
• convert README.md to README.rst to be properly randerd on PyPi
• add important data files in MANIFEST.in (start_instances, clean_instances, LICENSE.txt, README.rst)

Need

As a developer
I want to be prompted as early as possible if the configurations provided are not valid
So that potential problems are discovered at instantiation rather than later when trying to access a database

Deliverables

Scenario: Providing faulty configurations
Given I pass configurations that are not valid
When  I try to instantiate MongoPool
Then  an exception should be raised

Solution

Write 2 private methods that validate mandatory configs parameters and optional ones

Need

As a developer
I want a mechanism to manage multiple mongo connections to different clusters and map databases to connections 
So that I won't be forced to remember the host and the port and create a connection each time I want to work with a database on a different cluster and write those details in a single place

Deliverables

Scenario: Working with databases on multiple clusters
Given I have to work with some databases located on different clusters
  And I write connection information in a configuration file
When  I want to access a database
Then   I can access the wanted database based solely on it's name

Solution

A class MongoPool that keeps internally a list with database information (host, port, read preference, database name).
When the user wants to access a database, all he has to do is mongopool.db_name. Behind the scene, it will create a connection to the correct cluster and return the database. To make it accessible as a simple attribute, the __getattr__ method will be implemented.

TODO

• write test to check that the returned database is from the correct connection (check the host and port with the ones from the configuration)
• write test to check that "No such database" Exception is raised when trying to access a database that was not configured
• implement the first version in which the user can access databases via dot notation (it will work only with precise database names)
• replace the database name with a regexp pattern

Notes

Need

As a developer
I want the timeout set for each each mongo connection to have an adequate value
So that it kills lagging connections, but doesn't affect connections that take a little longer

## Deliverables
- [ ] have the timeout set in seconds instead of miliseconds

Need

As a developer
I want to provide the 'host' value of a cluster in the config as a list of hosts
So that I can connect to replicaSets

Deliverables

Given I have an entry in the configuration file that sets 'host' to a list of hosts
When  I instantiate MongoPool
Then MongoPool will accept the config as valid

TODO

• In _validate_config method if replicaSet is defined, assert check that host is a list.
• Write test for this situation

#ETA 0p

Write a sample config file to serve as a starting point for using mongopool.

Need

As a developer
I want to be able to pass a custom connection class to mongopool to use for connecting to the mongo instance
So that I can have better control over the connection (for example, create a self reconnecting connection)

Solution

Add 2 private attributes to MongoPool that hold the classes to create a simple connection and a connection to a replicaSet

Set the version of mongopool to 0.2 on setup.py