nerdwalletoss / savage Goto Github PK
View Code? Open in Web Editor NEWAutomatic version tracking for SQLAlchemy + PostgreSQL (based on versionalchemy)
License: MIT License
Automatic version tracking for SQLAlchemy + PostgreSQL (based on versionalchemy)
License: MIT License
Issue by luislew
Wednesday Mar 07, 2018 at 09:07 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/9
Compare data attribute between different archive versions of a record and generate a (deep) JSON diff.
Issue by luislew
Monday Nov 20, 2017 at 06:21 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/3
There is currently a caveat with using savage
(see https://github.com/NerdWallet/nwpy-savage#caveats), which is that multiple commits with changes on the same object inside a nested transaction will result in IntegrityError
.
Unfortunately, there's currently no way to get a sub-transaction ID in Postgres. However, if the end user doesn't care about anything but the last commit within the nested transaction, then this issue could be solved by adding an ON CONFLICT
handler which would overwrite records with identical version IDs. This should be a safe change, because version ID should only be duplicable within a single transaction, but this could also be gated with a configuration setting (e.g. ALLOW_ARCHIVE_OVERWRITES
).
Issue by luislew
Monday Mar 12, 2018 at 20:46 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/11
Example data
value:
{
"created_at": "2018-02-23T21:36:44.325409",
"data_hash": "be6ebf73-2e2c-ff27-1b3c-9f05d6031340",
"details": "{\"wp_product_id\": null, \"apy\": 0.01, \"fee_paper_statements\": null, \"cd_deposit_term\": 0, \"display_status\": \"INACTIVE\", \"fee_atm\": null, \"bonus_details\": null, \"account_details\": null, \"offer_logo_url\": null, \"fee_monthly\": null, \"direct_deposit_waive\": null, \"benefits\": null, \"account_holder\": null, \"fee_debit_replacement\": null, \"fee_stop_payment\": null, \"maximum_balance\": 49999, \"manual_entry\": false, \"display_name\": \"INTCK2.5K\", \"related_content\": null, \"fee_overdraft\": null, \"display_tags\": null, \"offer_url\": \"https://www.bannkofamerica.com\", \"bonus_amount\": null, \"fee_early_closure\": null, \"offer_url_monetizing\": null, \"minimum_balance\": 1, \"apy_annotation\": null}",
"details_override": "{}",
"ends_at": null,
"external_id": "14A02086BA5BDCD91AE49F50DAA195302B7761094FAAB59A240EF684CC8C869F",
"id": "a4b47ea8-18e1-11e8-bbf4-9b03e1f2357f",
"name": "INTCK2.5K",
"product_id": "62c84e16-18e1-11e8-b389-8359e5572f80",
"product_type_name": "banking",
"starts_at": null,
"status": "ACTIVE",
"tags": [],
"updated_at": "2018-02-23T21:35:38.681501"
}
The details
and details_override
columns are JSONB
columns and could be properly stored as nested JSONB, but instead are stored as serialized JSON strings.
Issue by luislew
Monday Nov 20, 2017 at 21:32 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/4
Currently, every archive row insert is executed individually. However, when inserting multiple archive rows for a single table, this could be sped up by batching up writes by table. This will only yield performance gains when writing a large batch of objects, so the work was de-prioritized and I'm logging the issue as a nice to have.
Issue by luislew
Wednesday Feb 21, 2018 at 21:37 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/7
Currently, SavageModelMixin.register(archive_table, engine)
calls the _validate
methods of both the model and archive model. These model class validations rely on an existing DB engine, which may not be desirable if registration is done during application startup. Adding a validate
kwarg to the register
method would allow end users to skip this validation (and thus avoid needing an active DB connection).
Issue by luislew
Friday Feb 16, 2018 at 20:57 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/6
Hello, please check this Nextpertise/versionalchemy version and also check this pull request to the NerdWalletOSS/versionalchemy. These new features are only available for versionalchemy. But I transform those new features for savage module. Please take a look at my fork kushan-gunasekera/savage and let me know, I'm happy to merge my pull request.
Issue by luislew
Wednesday Feb 21, 2018 at 21:38 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/8
To enable faster indexed searches on archived data, we should create a GIN index on the archive table's data
column.
Issue by luislew
Monday Mar 12, 2018 at 20:35 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/10
Currently, version_id
is generated from txid_current()
, which is not portable between database installations. As a result, it's not possible to simply dump data from one server to another, because the version ID generation may conflict.
One way to around this issue would be to add a database_version_id
, which would be incremented upon a database migration. SavageLogMixin
would need to add a database_version_id
column, and there would need to be a separate DatabaseVersionId
table with a single value (which defaults to 0/1).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.