nerdwalletoss / savage Goto Github PK

Issue by luislew
Monday Mar 12, 2018 at 20:35 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/10

Currently, version_id is generated from txid_current(), which is not portable between database installations. As a result, it's not possible to simply dump data from one server to another, because the version ID generation may conflict.

One way to around this issue would be to add a database_version_id, which would be incremented upon a database migration. SavageLogMixin would need to add a database_version_id column, and there would need to be a separate DatabaseVersionId table with a single value (which defaults to 0/1).

Hello, please check this Nextpertise/versionalchemy version and also check this pull request to the NerdWalletOSS/versionalchemy. These new features are only available for versionalchemy. But I transform those new features for savage module. Please take a look at my fork kushan-gunasekera/savage and let me know, I'm happy to merge my pull request.

Issue by luislew
Wednesday Feb 21, 2018 at 21:37 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/7

Currently, SavageModelMixin.register(archive_table, engine) calls the _validate methods of both the model and archive model. These model class validations rely on an existing DB engine, which may not be desirable if registration is done during application startup. Adding a validate kwarg to the register method would allow end users to skip this validation (and thus avoid needing an active DB connection).

Issue by luislew
Monday Nov 20, 2017 at 06:21 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/3

There is currently a caveat with using savage (see https://github.com/NerdWallet/nwpy-savage#caveats), which is that multiple commits with changes on the same object inside a nested transaction will result in IntegrityError.

Unfortunately, there's currently no way to get a sub-transaction ID in Postgres. However, if the end user doesn't care about anything but the last commit within the nested transaction, then this issue could be solved by adding an ON CONFLICT handler which would overwrite records with identical version IDs. This should be a safe change, because version ID should only be duplicable within a single transaction, but this could also be gated with a configuration setting (e.g. ALLOW_ARCHIVE_OVERWRITES).

Issue by luislew
Wednesday Feb 21, 2018 at 21:38 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/8

To enable faster indexed searches on archived data, we should create a GIN index on the archive table's data column.

Issue by luislew
Friday Feb 16, 2018 at 20:57 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/6

Relevant code: https://github.com/NerdWallet/nwpy-savage/blob/df31007f2a0d01335756891c70bb5c0c3b460ab7/src/nerdwallet/savage/models/__init__.py#L66-L91

Issue by luislew
Monday Mar 12, 2018 at 20:46 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/11

Example data value:

{
  "created_at": "2018-02-23T21:36:44.325409",
  "data_hash": "be6ebf73-2e2c-ff27-1b3c-9f05d6031340",
  "details": "{\"wp_product_id\": null, \"apy\": 0.01, \"fee_paper_statements\": null, \"cd_deposit_term\": 0, \"display_status\": \"INACTIVE\", \"fee_atm\": null, \"bonus_details\": null, \"account_details\": null, \"offer_logo_url\": null, \"fee_monthly\": null, \"direct_deposit_waive\": null, \"benefits\": null, \"account_holder\": null, \"fee_debit_replacement\": null, \"fee_stop_payment\": null, \"maximum_balance\": 49999, \"manual_entry\": false, \"display_name\": \"INTCK2.5K\", \"related_content\": null, \"fee_overdraft\": null, \"display_tags\": null, \"offer_url\": \"https://www.bannkofamerica.com\", \"bonus_amount\": null, \"fee_early_closure\": null, \"offer_url_monetizing\": null, \"minimum_balance\": 1, \"apy_annotation\": null}",
  "details_override": "{}",
  "ends_at": null,
  "external_id": "14A02086BA5BDCD91AE49F50DAA195302B7761094FAAB59A240EF684CC8C869F",
  "id": "a4b47ea8-18e1-11e8-bbf4-9b03e1f2357f",
  "name": "INTCK2.5K",
  "product_id": "62c84e16-18e1-11e8-b389-8359e5572f80",
  "product_type_name": "banking",
  "starts_at": null,
  "status": "ACTIVE",
  "tags": [],
  "updated_at": "2018-02-23T21:35:38.681501"
}

The details and details_override columns are JSONB columns and could be properly stored as nested JSONB, but instead are stored as serialized JSON strings.

Issue by luislew
Monday Nov 20, 2017 at 21:32 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/4

Currently, every archive row insert is executed individually. However, when inserting multiple archive rows for a single table, this could be sped up by batching up writes by table. This will only yield performance gains when writing a large batch of objects, so the work was de-prioritized and I'm logging the issue as a nice to have.

Issue by luislew
Wednesday Mar 07, 2018 at 09:07 GMT
Originally opened as https://github.com/NerdWallet/nwpy-savage/issues/9

Compare data attribute between different archive versions of a record and generate a (deep) JSON diff.