Comments (5)
thomasdziedzic
commented on May 29, 2024
Just an fyi, I increased the speed from 3000 rows / 5 minutes to 5000 rows / 5 minutes by not autocommitting after each insert.
from fake2db.
thomasdziedzic
commented on May 29, 2024
Another update.
I wrote my own program [0] to populate tables pretty quickly using a combination of batching and executemany [1].
Using batches of 20000 records at a time, I was able to achieve about 2500 rows per second.
[0] - https://gist.githubusercontent.com/gostrc/87b73c6bac916ca6de50/raw/86d475f1f7f1c7adb1b13e8cf1948098e686eace/load.py
[1] - http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-executemany.html
from fake2db.
emirozer
commented on May 29, 2024
Thank you very much for your detailed input on the speed! I have to improve it...
from fake2db.
emirozer
commented on May 29, 2024
Hey!
I took your advice about executemany, thank you again!
from fake2db.
emirozer
commented on May 29, 2024
Applied for mysql and sqlite, i will also improve other adapters..
from fake2db.
Related Issues (20)
- Install shouldn't fail if I don't have pg_config HOT 3
- Python socket package throws error in fake2db_logger under helpers.py HOT 3
- Adding more nosql db HOT 1
- Error when running cli HOT 5
- Missing faker_options_container HOT 4
- Not sure how to start with. HOT 2
- name not taken in consideration for mysql HOT 3
- mongodb hanging on 1 million rows HOT 5
- Misprint in column name 'address' in sqlite_handler.data_filler_detailed_registration HOT 1
- Populate an existing database + schema? HOT 1
- How to get auto-increment ID and specify custom column names? HOT 1
- custom table name and number
- Missing tags HOT 1
- database already exists HOT 1
- Allow multiple columns with same faker key OR allow column naming
- --database option
- MySql connector is always required
- AttributeError: 'NoneType' object has no attribute 'execute' HOT 3
- Companies Data Not Working? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fake2db.