elaske / tsp-data Goto Github PK

[WIP] Just some scripts that help with gathering and projecting data for the Thrift Savings Plan.

License: GNU General Public License v3.0

Python 99.99% HTML 0.01%

tsp-data's Introduction

TSP Data

This project is kind of a catch-all for different scripts I think might be useful for retirement planning with the Thrift Savings Plan (US federal employee retirement plan).

Intro

The one problem with the Thrift Savings Plan for federal employees is that it's very simplistic. Very limited choices help keep costs down. Unfortunately, though, the lack of features that people might be interested in don't exist as a result of these cost-savings measures.

One of the features I feel like the TSP lacked was the ability to easily collect and chart historical prices. Luckily, the TSP does publish the share prices for the funds at least since June, 2003. However, they're in a table on the website that is only accessible 30 days at a time.

Current Development

Step 1

The first goal of this project is to create a script / class that takes care of gathering this data from the website and making it available to scripts that might be run by the user (future parts of this project). Getting the basic funtionality of gathering the data, saving the data to a local file, and updating this data file automatically when run will be the first milestone.

Step 2

Once the data is available locally, the next step will be to wrap this into a class that packages this functionality so that it takes care of it itself without management from an external script. This will allow another script to easily use this data, and essentially query it as needed.

Step 3

The last preliminary goal of this project will be to link up this data with a viewer of some kind. The hope is to have something similar to Google Finance for all of the data.

The Future

There are some features that I believe might be useful to add to this toolset once the preliminary project is completed:

A separate financial calculator to project cleanly into retirement
A modification to this calculator that will use historical prices to modify the "ideal" retirement calculator
Statistical display of performance curves over different time-frames

Eventually, this might be useful as a free-to-use web-service - and I have already been thinking about this possibility throughout my current development.

I'm sure I will think of many more features on my own as the time goes by. However, I am but one man; this will most likely benefit from other's contributions in feature requests or direct development contributions.

tsp-data's People

Contributors

Stargazers

Watchers

Forkers

fesp21

tsp-data's Issues

Retrieval Termination Logic Breaks

On 12/26/14, the share price history page only showed 11/18 to 12/24, or 26 lines. This means that the logic that is checking when to terminate the retrieval of new data is fooled: it's looking for 30 items.

We should really compare the date information to what is already in the data structure.

Append New Data

Complete the appending of new data that is found from the TSP site.

Compare CSV and HTML data gathering methods.

The most obvious thing is what is going to be faster. The CSV will have gains in not having to parse as much, but the HTML version might get posted back to us quicker (less server processing time?).

Need to measure this to be sure which is the best option. This will become the default option. The other will stay in the code as a user option.

Sorted Data

Sort the data in at least the output files. A change over to an OrderedDict might be warranted:
http://stackoverflow.com/questions/18871217/python-how-to-custom-sort-a-list-of-dict-to-use-in-json-dumps

Remove Data Storage From Repository

Should this really be stored in the repository? It might be a good baseline to include with each revision so that less data has to be downloaded on everyone's end. However, it's not really that useful...

Add command line parameters

Create script interface to new class

Retrieve TSP Prices Directly From CSV Request

It seems that there is a new way to retrieve data from the TSP share price history site in that it allows you to request a CSV by dates. We should be able to retrieve all of the data as one download instead of having to scrape their site.

Add Old TSP Statement Parser

The only way to really get all of the historical data from your TSP account is to download the PDFs. On the website, it is only possible to get the data from the last statement.

There are two versions of the TSP statements that have been used. This is the current version.

Add Logic for Checking for New Data

Currently, it always looks at the current page to check for new data. It shouldn't check if it doesn't expect there to be new data from what it's already loaded.

This means that if it's already got data for today in the dictionary, don't look.
If it doesn't have data for today:

Make sure it's not a weekend
Make sure it's not a federal holiday
Check to see if it's after 7PM EST (when new data is posted)

This should make it easier to download the data, too.