jantman / biweeklybudget Goto Github PK

refactor all modal forms so we can use multiple on the same page - i.e. the same modal div but can use multiple modal JS on one page, i.e. Pay Period view can have modals on that page to edit ScheduledTransactions, Transactions, etc. This also means passing dataTableObj in from the link.

There are currently no docs for the db module or many of the other submodules.

• in settings.py, give all setting constants default values and docstrings, even if default values are None.
• document all biweeklybudget submodules; ensure docs build correctly.

For properly determining pay period balances, we should have specific "income" budgets. Pay period balances can be calculated by subtracting the sum of all transaction amounts on non-income budgets from the sum of all transaction amounts on income budgets.

Index/landing page (/):

• displays table with standing budgets and their current balances
• add versionfinder; display version at bottom of nav menu

Pay Periods, using data from #19

• Status (balance, etc.) of current period with link to it (possibly in one of those big dashboard widgets?); this should be a reusable component that can be put on the index page too
• Balances of the next N periods with links to them
• shows all recurring budgets, and their balances for the current and next 2 pay periods

Right now, there's no way to add Accounts through the UI. Add that.

• modal to add and edit accounts
• change /accounts/N from a static view to pre-load the modal
• update all /accounts/N links to display the modal instead

Add a waffle.io link to the README

Need a way to do a real round-trip acceptance test (i.e. make a change in the DB from the UI, commit it, ensure it's reflected in the DB or elsewhere in the UI) and then roll it back. Don't know if we can do some sort of inner transaction for this, or need to make use of savepoints.

This should probably be done at the class-level. The tests will get too ugly if we have to do this much in one method.

References:

• http://stackoverflow.com/questions/1294372/savepoint-commit-rollback-in-mysql
• https://dev.mysql.com/doc/refman/5.7/en/savepoint.html
• http://stackoverflow.com/questions/28526781/isolating-py-test-db-sessions-in-flask-sqlalchemy
• http://stackoverflow.com/questions/26555125/rollback-transactions-not-working-with-py-test-and-flask/26624146#26624146
• http://alexmic.net/flask-sqlalchemy-pytest/
• http://koo.fi/blog/2015/10/22/flask-sqlalchemy-and-postgresql-unit-testing-with-transaction-savepoints/

• Test new settings module functionality
• Document how to use it

• Status (balance, etc.) of current period with link to it (possibly in one of those big dashboard widgets?); this should be a reusable component that can be put on the index page too
• Balances of the next N periods with links to them
• Calendar widget to find a budget for a specific date; redirects to the budget page for that date

Integration Tests:

• Alex's Tech Rants: Unit testing SQLAlchemy apps
• Delightful testing with pytest and Flask-SQLAlchemy | Alex Michael
• Transactions and Connection Management — SQLAlchemy 1.2 Documentation

Unit Tests:

• The Flask Mega-Tutorial, Part VII: Unit Testing - miguelgrinberg.com

• Testing Flask Applications — Flask Documentation (0.12)

• verify acceptance tests for all current UI functionality

• identify current areas of code that will most benefit from unit tests; create empty tests for them, mark as skip (not implemented) for now?

• Default to current period
• Previous/next links
• Space for transactions table
• Shows table of all recurring budgets for this period, along with budget amount, allocated amount, spent amount, remaining amount (and totals for all of those)
• Show standing budgets and their balances, as well as any adjustments (transactions) this period
• show balances for next N pay periods

Around the time this issue was opened (just after TravisCI built 45) I was seeing a lot of intermittent acceptance test failures, where one of the selenium.get() calls would run forever. I fixed this by putting a timeout both on selenium calls, and on each test.

If this happens again, look into running Travis builds locally using their docs on running their Docker containers and travis-build.

Need a cutoff date for transactions - i.e. don't show anything as unreconciled before this date. Should add a hybrid property or something on OFXTransaction and Transaction i.e. .unreconciled() that generates the appropriate query, including the cutoff date.

• Income transactions should be negative amounts, expenses are positive
• The amounts on Income budgets are completely ignored, only Transaction or ScheduledTransaction amounts are considered ( #19 )

Need to archive all build artifacts. I guess I'll need S3 for this. Be sure to include the selenium HTML reports.

There doesn't seem to be a built-in way to archive results/reports from failed tests. So, I think the plan is as follows:

1. Have all test result output go to a results/ directory.
2. Include S3 credentials in the secure project env vars (either in .travis.yml or via the UI)
3. Add a script to after_success and after_failure that generates an index.html for everything in results/, then uses s3cmd or boto or something to sync that directory to a S3 bucket under $TRAVIS_REPO_SLUG/$TRAVIS_JOB_NUMBER/

• view or modal popup for a single OFXTransaction
• link-able (i.e. by URL)
• /ofx table should link to these

Support for loading and saving cookies from a cookie file is broken in PhantomJS 2.x (see ariya/phantomjs#13115 ). We need a workaround for this.

• fix the ScreenScraper example in the docs to include actually getting a page - right now it's just doing Ajax stuff
• instead of self.browser.get(), use a self.get_page() to allow some wrapping logic
• make sure we're not calling browser close anywhere, or if we are, it's wrapped
• every time we get a page, if the browser already had a page loaded, call get_cookies() and save them to disk.
• When we first create the browser object, read the saved cookies from disk (if they exist) and pass them to the browser via add_cookie().

The unreconciled transactions notification is currently just a placeholder. Instead, use the .count() value of OFXTransaction.unreconciled

See https://groups.google.com/forum/m/#!topic/sqlalchemy-alembic/DekaORufgMY

I think the basic idea is to generate a DB and fill it with sample data, as of the sampledata and schema when Alembic was first introduced. Dump that to a SQL file. Then we init the db once just using create_all, dump that (A), load the static SQL from before migrations, run all the migrations, dump that (B) and compare A and B.

See if I can run at least some of these tests on a mobile browser, or the closest I can get (or maybe SauceLabs does free for OSS? Maybe just run them manually pre-release?)

There should be a way to mark a Transaction as reconciled without a matching OFXTransaction.

• link in the Transaction div on the reconcile view, that uses a modal to allow a message for reconciling without OFX.
• add one like this into sampledata, especially for the reconciled modal
• test reconciled modal with this
• add tooltips to "(no OFX)" and "(make trans)" links

Implement OFXTransaction post-processing using the re_* attributes on the Account model.

• OfxUpdater should take the Account object. After setting up the transaction, it should call a postprocess() method on the OFXTransaction:
  • zero out all is_* attributes
  • use the regex fields on the Account to figure out how to set the is_* fields, and set them.
• Account, when loaded, should compile those re's and cache them as object properties. Or even better, lazily compile and store them on first use.
• entrypoint or administrative command to re-process all transactions for a given account or for all accounts

• add a Dockerfile, and add Docker as a supported install method
  • app installed via pip, via a built distribution (tox env will put the dist at .tox/dist/biweeklybudget-VERSION.zip; tox code for making sdist which is put in the tox config distdir and found with self.config.distdir.listdir()[0]; example of parsing tox config )
  • this should use a real web server; leaning towards gunicorn (docs); some Dockerfile examples
  • Document how to run in Docker, with either Dockerized MySQL or local MySQL and how to specify config file.
• add a tox environment to automate building the image, tagging it properly, and then running some basic tests against it
• add Docker image build and push (to Hub) to release checklist
• Mention the screenshots, and the Docker installation method, in README.md
• Cut a 0.1.0 release and upload to pypi and Docker Hub

See #22

• need a way to craft a transaction to put money in standing budgets; some sort of pseudo-transaction. i.e., how do we allocate funds to a standing budget?

• Show transactions for the current period, both actual and scheduled (future, but only if not converted to actual) sorted by date
• Show totals for the pay period: allocated (budgets plus any transactions not in a budget), spent and remaining (unbudgeted/unspent income)
• Link to mark a scheduled transaction as having been made
  • brings up transaction input modal that converts from scheduled txn to real txn
  • links real txn to scheduled txn ID - hybrid attributes and setters; this should also be do-able directly from reconcile
  • When this is done, trans table should refresh with actual transaction.
• Link to bring up transaction input modal or scheduled transaction modal

Need a name for what I personally call "budgets", but are really pay periods. The term "Budget" already has another specific meaning.

• Reconcile View
  • logic to get Transactions and OFXTransactions for this view is reusable; use it in the notifications count bar ( #44 ).
  • 2 columns, Transactions on left and OFXTransactions on right; each is a div/box with appropriate information, link to modal, and other links as needed
  • Need to be able to add Transactions dynamically; maybe best is to store a mapping of reconciles (i.e. OFXTransaction to Transaction) as a JS var; then we can re-populate the columns dynamically and call a method to match up as necessary (which would also be used for pre-population)
  • columns are individually scrollable
  • OFX boxes have a link to create a Transaction (modal for this, pre-populated with as much as we can) for the OFXTransaction and reconcile them; this updates the view
  • Transactions have a link to trigger modal to mark them as reconciled without a matching OFXTransaction, but with a note; this updates the view
  • Transactions have a link to their edit modal; this updates the view.
  • Dropping an OFX on a Transaction disables the edit modal link
• Actual Reconciling
  • User drags an OFXTransaction onto a Transaction; Transaction is no longer droppable. This reconciles.
  • OFXTransaction can be dragged back to OFX column to "unreconcile"
  • Drop triggers Ajax validation; if it fails, OFXTransaction moves back to its column and a warning message is displayed.
• Form Submission
  • Displays a confirmation screen; table of matched transactions to reconcile, and a "reconcile" check box for each. Submission sends to form handler, only reconciles the checked ones, updates DB then redirects back to /reconcile.
• Reconcile Validation
  • Amounts must match
  • Accounts must match
  • Dates must be within 7 days

Drag and drop:

• jsfiddle
• cards tutorial
• w3schools
• explanation

Transactions:

• Transactions can be associated with a Budget; link to that budget
• View to list all Transactions
  • should be able to filter and sort very similarly to /ofx
• Modal to view/edit a Transaction, with URL like #26 ( see #20 )
• Modal to add a Transaction
  • This should be available from other views as well; it will be used in reconcile ( #20 ).
  • table updates when a Transaction is added
  • Account dropdown defaults to a default account (set by name in settings).

• run all tests locally, confirm that everything still works
• run app locally, with settings from my private repo; document how to run this way

Start a basic docs layout

• Introduction/overview
• Getting Started
  • Requirements (Vault, MySQL only)
  • Installation
  • Flask app use
  • CLI use
• Transaction Downloads
  • ofxgetter
  • Selenium
• Third-party dependencies and credits
• Development

• Reconcile modal - for a given reconcile ID, show the reconcile id, date, notes (if any), and links to the Transaction and OFXTransaction modals, as appropriate. Or maybe should just include all the info for all of them...
• Accounts need to have a boolean flag for whether or not their OFXTransaction amounts should be negated when comparing to Transactions.
• Using the above, OFXTransaction should have a hybrid property account_amount, that is generated using this boolean.
• PayPeriod view - reconcile column in transaction table, with link for reconcile modal.
• OFX view - reconcile column in transaction table, with link for reconcile modal.
• Transaction view - reconcile column in transaction table, with link for reconcile modal.
• Reusable logic to craft a query to get all unreconciled Transactions or OFXTransactions. For OFX, this should respect a RECONCILE_BEGIN_DATE setting and only return ones with dates after that (this should probably be a hybrid property with an expression).

• Status (balance, etc.) of current period with link to it (possibly in one of those big dashboard widgets?); this should be a reusable component that can be put on the index page too
• Balances of the next N periods with links to them
• shows all recurring budgets, and their balances for the current and next 2 pay periods
• notification if a budget balance goes negative (how to handle this per-pay-period / recurring? only current and next N?)

To enable the rest of the PayPeriod-related issues ( #34 #35 #36 #37 ), the PayPeriod object should have the following methods which take a db_session (a sqlalchemy.orm.scoping.scoped_session or a sqlalchemy.orm.session.Session) and return a sqlalchemy.orm.query.Query (we assume that in most cases we'll just be calling .all() on this Query, but we should return the Query itself for flexibility):

• All active scheduled transactions for the period
• All actual transactions for the period

And the following methods which take a db_session and return a dict or object to be used by something downstream:

• each active recurring budget, along with the budget amount, allocated amount (amount of scheduled transactions or actual transactions this period), the spent amount and the remaining amount.
• Total allocated (budgets plus transactions), total spent, total remaining
• All transactions for the period; this should combine ScheduledTransactions and Transactions into one list, ordered by date, and return them as either objects or dicts in a common format so they can be displayed in one table. This is not going to be a DataTable, or at least is not going to be searchable/filterable/orderable; it will be a fixed representation.
  • Acceptance test for BiweeklyPayPeriod._dict_for_sched_trans() - set a budget start date at the beginning of the second week of a month, so we have a block of 3 consecutive pay periods, the first and last of which span month changes and the middle of which does not. Generate ~56 Monthly ScheduledTransactions on consecutive days, so we have transactions for at least one week on either side of the block of 3 consecutive pay periods. Ensure that each of the 3 pay periods contain the right transactions in the right order.
• Need to exempt any ScheduledTransactions that have been converted to Transactions
• How to handle scheduled income? See comment below

Also:

• Given that a lot of the above use the same queries, we should explicitly cache the query results in the object.
• For testing PayPeriod-related things (acceptance tests), biweeklybudget.settings_example.PAY_PERIOD_START_DATE should be set to the closest Friday to the current date.

• implement module/class name based Selenium downloader loading
• document how to use this and include an example
• migrate my personal downloader to a private package, test it using this method

Also:

• document how to setup accounts for ofxgetter (direct OFX downloads); write script for json-ization of ini file

Look into autodoc for the javascript stuff, or at least generated docs for the JS stuff.

Possibilities:

• https://github.com/erikrose/sphinx-js
• https://www.safaribooksonline.com/blog/2013/04/23/documenting-multiple-programming-language-apis-with-sphinx/
• https://github.com/Nuulogic/sphinx-jsapidoc
• https://github.com/debrouwere/jsdoc-for-sphinx

Script to load sample data, run selenium to take screenshots. This should be added to the release checklist, to update screenshots in the docs.

Also figure out how to get selenium to do a screenshot for a demo of drag-and-drop reconciling.

Header or footer on all pages:

• app name and link
• fork button
• support button/link
• AGPL notice with versionfinder

autodoc imports the modules. They include DB access attempts, which fails because there's no DB. Fix this.

Ideas:

1. Use sqlite, assume it'll work acceptably since we're only building docs (test locally).
2. Figure out how to not even try connecting to the DB if we're running on RTD. Maybe guard all of the DB connecting with an env var?
3. Ugh... have to build docs locally and push them.

• verify acceptance tests for all current UI functionality
• identify current areas of code that will most benefit from unit tests

Each Pay Period should be represented by a PayPeriod subclass (i.e. BiweeklyPayPeriod).

• start and end date properties
• properties to return the next and previous period
• class method to get the period object for a given date
• method to filter a SQLAlchemy query to this pay period, given the query object and a reference to the date property (i.e. .filter_query(query, Transaction.date) would filter query so Transaction.date is in the pay period.

Dates for pay periods will be based off of a PAY_START_DATE setting (the starting date of one arbitrary budget period).

These may be useful in the future:

• Assuming flask is run as shown above, the full example pages from the SB Admin 2 theme/template can be seen at http://localhost:5000/static/startbootstrap-sb-admin-2/pages/index.html
• Flask message flashing
• Flask views
• Flask sessions
• WTForms validation
• Nice full datatables demo
• Possibly-useful icons:
  • fa-check-square-o
  • fa-edit
  • fa-shopping-cart
  • fa-credit-card
  • fa-bar-chart-o
  • fa-dollar
  • fa-history
  • glyphicon-piggy-bank
• Selenium Python docs - Locating Elements

Document how to use data loaders; include custom package in getting started docs.

This requires #20.

Implement rules-based automatic reconciling of transactions.

• The following will not reconcile transactions without manual confirmation. It will be implemented as a hook on the reconcile view added in #20, and will simply pre-populate the state of the reconcile, but can still be manually overridden.
• Model for reconciling rules:
  • Each rule should have fields for each pertinent transaction field (account, name/memo, amount, etc.)
  • name/memo and other strings will be either exact match, contains, or regex
  • Amount will be either exact match, greater than, less than, or within X of Y
  • Should also have a date range field (i.e. the Transaction and OFXTransaction dates are within X days of each other; default to 7)
  • "unique" field, where this rule will only apply if it matches exactly one unreconciled Transaction and exactly one unreconciled OFXTransaction
• view to list reconciling rules; buttons/links to add, edit, delete and make inactive.
• properly generate add/edit modal
• user can add reconciling rules using any combination of fields specified above; list view updates and DB is updated correctly
• user can edit reconciling rules using any combination of fields specified above; list view updates and DB is updated correctly
• user can delete reconciling rules; list view updates and DB is updated correctly
• user can toggle active flag on reconciling rules; list view updates and DB is updated correctly

• view of all budgets
  • modal to add and edit budgets
  • shows balances of all standing budgets
  • shows the starting balances of all recurring budgets
  • direct URL to display budget modal
• notification if sum of all budget balances exceeds sum of all budget source account balances (Account.is_budget_source)
• notification if a budget balance goes negative
• Index page - shows all standing budgets and their balances

Sometimes we pull OFX data for an account, but don't want to reconcile the transactions - such as investment accounts, or credit accounts that we're just paying off.

Add a do_not_reconcile boolean field to Account, and then update the OFXTransaction.unreconciled() filter to include checking that field.

Reconcile needs a way to convert ofx trans to internal trans (select budget) and a way to set internal trans as cash or otherwise mark as non-ofx reconcile.

Wherever we show account balances, we should show the ledger balance, and then that minus any unreconciled transactions against the account.

This should be for the index and accounts views.

• Make the search box on the OFX view actually work
  • it should search Name, Memo, Description; may need to add indices for these fields

See: https://github.com/orf/datatables/blob/master/tests/test_basic.py#L208

• Transactions can be associated with a Budget; link to that budget
• scheduled with specific date, day of month, number per pay period, or "any date" per pay period
• edit modal for these
• add modal for these
• refactor all modal forms so we can use multiple on the same page - i.e. the same modal div but can use multiple modal JS on one page, i.e. so the Pay Period view can have modals on that page to edit ScheduledTransactions, Transactions, etc. This also means passing dataTableObj in from the link.
• account and budget; all other fields from model
• view that shows all scheduled transactions, grouped by type (date/day of month/num per period)

Ability to set a default Account for new transactions. This should ideally be a DB field, a unique boolean.