apavlo / py-tpcc Goto Github PK
View Code? Open in Web Editor NEWPython implementation of TPC-C
Python implementation of TPC-C
+ ----------------------------------------------- + + Python TPC-C + + ----------------------------------------------- + The basic idea is that you will need to create a new driver file that implements the functions defined in "abstractdriver.py". One function will load in the tuples into your database for a given table. Then there are five separate functions that execute the given transaction based on a set of input parameters. All the work for generating the tuples and the input parameters for the transactions has been done for you. Here's what you need to do to get started: (1) Download the source code from Github: https://github.com/apavlo/py-tpcc/tree/master/pytpcc (2) Create a new file in the 'drivers' directory for your system that follows the proper naming convention. For example, if your system is 'MongoDB', then your new file will be called 'mongodbdriver.py' and that file will contain a new class called 'MongodbDriver' (note the capitalization). (3) Inside your class you will need to implement the required functions of defined in AbstractDriver. There is documentation on what these need to do also available on Github: https://github.com/apavlo/py-tpcc/wiki (3) Try running your system. I would start by defining the configuration file that gets returned with by the 'makeDefaultConfig' function in your driver and then implement the data loading part first, since that will guide how you actually execute the transactions. Using 'MongoDB' as an example again, you can print out the driver's configuration dict to a file: $ python ./tpcc.py --print-config mongodb > mongodb.config Make any changes you need to 'mongodb.config' (e.g., passwords, hostnames). Then test the loader: $ python ./tpcc.py --no-execute --config=mongodb.config mongodb You can use the CSV driver if you want to see what the data or transaction input parameters will look like. The following command will dump out just the input to the driver's functions to files in /tmp/tpcc-* $ python ./tpcc.py csv You can also look at my SqliteDriver implementation to get an idea of what your transaction implementation functions need to do: https://github.com/apavlo/py-tpcc/blob/master/pytpcc/drivers/sqlitedriver.py
There is an error when trying to use mongodb driver:
AttributeError: 'module' object has no attribute 'Connection'
You have to use MongoClient instead of Connection (pymongo changed api).
You have to change line 236 of file drivers/mongodbdriver.py to:
self.conn = pymongo.MongoClient(config['host'], int(config['port']))
When I try to run the benchmark against with Python 3.9, I'm getting a syntax error on tpcc.py line 117.
$ python ./tpcc.py --print-config mongodb
File "D:\projects\tpcc\apavlo-py-tpcc\pytpcc\tpcc.py", line 117
except (Exception, AssertionError), ex:
^
SyntaxError: invalid syntax
Am I using the wrong version of Python for this project?
Hi, I'm interesting in benchmarking MongoDB Transaction performances, and I found your project seems to fit my requirement.
But I found that there is no api that support transaction in the document of pymongo 2.7.2,and I cannot find any similar function calls like the following (taken from https://docs.mongodb.com/manual/core/transactions/) in your code:
# For a replica set, include the replica set name and a seedlist of the members in the URI string; e.g.
# uriString = 'mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017/?replicaSet=myRepl'
# For a sharded cluster, connect to the mongos instances; e.g.
# uriString = 'mongodb://mongos0.example.com:27017,mongos1.example.com:27017/'
client = MongoClient(uriString)
wc_majority = WriteConcern("majority", wtimeout=1000)
# Prereq: Create collections.
client.get_database(
"mydb1", write_concern=wc_majority).foo.insert_one({'abc': 0})
client.get_database(
"mydb2", write_concern=wc_majority).bar.insert_one({'xyz': 0})
# Step 1: Define the callback that specifies the sequence of operations to perform inside the transactions.
def callback(session):
collection_one = session.client.mydb1.foo
collection_two = session.client.mydb2.bar
# Important:: You must pass the session to the operations.
collection_one.insert_one({'abc': 1}, session=session)
collection_two.insert_one({'xyz': 999}, session=session)
# Step 2: Start a client session.
with client.start_session() as session:
# Step 3: Use with_transaction to start a transaction, execute the callback, and commit (or abort on error).
session.with_transaction(
callback, read_concern=ReadConcern('local'),
write_concern=wc_majority,
read_preference=ReadPreference.PRIMARY)
Did I misunderstand anything ? How do you implement the atomic transaction operations for MongoDB?
Thanks~
Hi there,
Could you please tell us what the license under which you release this code? I.e is it ok to fork it and heavily modify it?
Maybe the simplest would be for you to add a LICENSE file among those proposed by GitHub.
Thanks in advance!
DIdier
If i have more than one client, (e.g. 4) i see that i can submit more new orders in same time, than if have only on client. For example.:
Result for one client on my machine (after 120s):
NEW_ORDER 1646 41954704.7615 39.23 txn/s
Result for four clients on my machine (after 120s):
Execution Results after 120 seconds
NEW_ORDER 4657 229757157.326 20.27 txn/s
Obviously 4657 is greater than 1646, but the calculated rate for the second run is smaller than the rate for the first. I think the way it is calculated is wrong. You cannot simply add the execution time in Results.stopTransaction(), because the client are executing in parallel. The right way would be to calculate tx/s per client and then add the results.
I want to know wether py-tpcc has supported h-store. In wiki, all the databases are NoSQL.:)
Loding data aborts with an AssertionError.
The Problem is, that rand.NURand() doesn't allow a None object as nurand, but nurand is never set to something different.
The solution is to import nurand and call nurand.makeForLoad()
On the Redis and Cassandra driver, when loading the ORDERS table, it creates the 'O_C_ID' atribute with the 4th position of the tuple it receives (row[3]).
However, in the generateOrder function of the loader.py, the 'o_c_id' attribute is in the 2nd position of the tuple, so it seems to be inserting the values incorrectly.
Note that I have not been able to test its correctness, as I found this when debuging my own driver based on the redis implementation, so take it with a grain of salt, but it seems to be an issue.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.