In the old version, I hard coded in the cashout amount, but I don't actually remember how I got it in the first place. Need to read the pension documentation.

Current inflation is symmetric Gaussian since it was based on the code that made stock returns. Actual inflation follows a more positively skewed distribution as shown in the image below. Need to change code to make more representative inflation.

Ex: time_ls -> date_ls, FI Quarter -> FI Date

Seems like clearer language

average withdrawal rate @ FI date and median final value

A single place for all constants that might be referred to by multiple scripts like return characteristics and gov_params.

Since I'm targeting similar annualized returns, would I get equally accurate information by just shuffling a single set of generated returns 5000 different ways rather than generating 5000 sets of returns? Would this be faster?

returnGenerator.py is using constants for generating returns, but should be taking in params from the simulator.py so return parameters can be adjusted

Goal

Looking to create algorithm for optimizing adjustable parameters. This would help find high-performing parameter combinations faster than manually trying out combinations.

Requirements

• Existing Monte Carlo simulator would provide the judgement result of the parameter combination.
• Should be able to define unique range for each parameter and interval qty/spacing

Considerations

• Some parameters won't cause any changes depending on the state of a different parameter (useless genes). For example, if the 'Life Cycle' allocation mode is not chosen, then altering 'Equity Target' parameter would cause no changes.

Options

Genetic Algorithm

A genetic algorithm treats the parameters as genes that can be mutated in a hyper-volume of possibilities. Different algorithms implement different approaches with evolutionary traits such as cross-breeding, parenting, and mutation.

DIY

Create my own algorithm specialized for this project

• Full control of evolutionary traits
• May not need to modify parameter management

PyGAD

Open source algorithm with many options for evolution

• Documentation is not clean on how 'genes' need to be fed to the library
• Unclear if I'll be able to clearly provide max, min, and intervals of each parameter
• Unlikely to be able to prevent running useless gene combinations.

Follow tutorial here for making a test distribution

There's a partial correlation between inflation and RE & bond returns that would help success rates. Would be more accurate, but less conservative, to incorporate it

To be fair, we should build in all correlations, including those between Equities/Bonds, Equities/RE, etc. It may also be worth researching correlation in good times vs bad to see if it's consistent.

Do the same work you did for inflation to have real annualized limits for stock/bond returns

More accurate, but less conservative. Should check and see how well SS has tracked inflation in the past. Perhaps a cap on the increase since SS might not track really high inflation years

Seems to be around +/-2% at 5000 runs, would be valuable to see how that accuracy changes for smaller and larger numbers of runs

https://www.ssa.gov/benefits/retirement/planner/applying7.html

A little convoluted, but seems to say spouse gets the greater of 50% of partner's benefit or 100% their own benefit.
Need to dig into the pension rules

A method that decides whether to withdraw SS/Pension depending on portfolio size

perhaps by checking latest modification data on simulator.py and params.json

Rather than working with a fixed FI date, start with the date from params, then at each successful param combination (>95% success), reduce the FI date by 0.25 and start again.

perhaps also seed each updated date with the previous dates success

Running 5000x takes ~15sec (not including generating returns). Improving this would allow far more combinations to be tried.

Should we randomize life expectancy within it's actual variation range? Seems risky to always assume the same life span.

Concerns:
How do you deal with spouses?
Would have to move the time_ls inside the monte carlo loop

Currently using fresh returns for each set of children. Could consider only refreshing returns for each generation (when parent_is_best_qty == 0) Trading off variation for speed.

Right now, we're running the entire Simulation.main() for every child creation. We should be able to save a little time by skipping the static lists (Time and Job Income). Would just need to separate that part into a different function and deal with retrieving all variables needed.

Had issues trying to modify the global INVESTIGATE_MODE variable to False if the user wants to exit investigation mode.

Complete method _back_estimate() so that if no historic earnings given to socialSecurity.py, it'll work backward from user's age to estimate previous earnings

Should have folders to organize MVC code. Just not sure how to import files in different folders. Need to test

Solutions:

1. Do what was done in issue #42 and .gitignore the file with try/except statements to manage first time implementation
2. Put SS_earnings.csv into params.json

Improve _add_to_earnings_record() in socialSecurity.py to avoid adding fractional earnings for the first year

Diagnostics notebook shows there's a difference between averaging yearly results converted from quarterly results versus averaging quarterly results and then converting to yearly.

This shows very high average results for the worst_failure tested in this instance (12.7% average returns) and may be an issue elsewhere in the simulator. Need to investigate further!

I'd like to be able to bring up a bar chart (or histogram) for every mutable parameter and update it with each generation with which option was used by the best child. Would give a sense of which option from each parameter works the best with most combinations.

Consider increasing the number of gene shifts possible with each additional generation of 'No Better Children'

Major blocker here is the View. There's no ability in the view to add to a list

First step towards enabling others to use the package. Possible implementation would be creating a sub-folder in the data folder that's not synced to Git. This folder would have a params.json that's specific to the user. The original params.json would used as the default and copied when the user first runs.

Allocation needs to be tracked in a list like the other variables so we can look at specific instances in Diagnostics. Might need a general method to do this for any variable that's created in an isolated loop.

Make an automatic check whether data/params.json has different keys than data/default_params/params.json

Could save/overwrite each best parameter combo

https://www.geeksforgeeks.org/what-is-the-difference-between-yaml-and-json/

The ability to add comments and sequences would be valuable

Should I stop putting all currency numbers in thousands? It was originally done to improve readability in Google Sheets. It also is more readable in the Tk GUI.

Changing it to regular numbers would make it easier to read though in the CSV file since decimal places aren't limited to 3. I may also be able to change everything over to Ints instead since I don't care about pennies. Rates would still be floats, but all dollar values could be rounded to ints.

Would this reduce time spent trying combinations? Would the list search eventually get too long? Since we're generally running at low resolution, is there value in giving a given parameter combination another chance?

Would need a way to reset the list whenever parameter changes are made

A method to dynamically purchase annuities as portfolio reaches higher values

I suspect you could have a situation where you are stuck in a local max trying to beat the TARGET_SUCCESS_RATE and run out of options (2^8 is only 256 combinations). In that case, you would get stuck in a loop making more parameter sets that have all been used.

Possible solutions:

1. skip the child if it is a repeat. This would cause you to hit the ITER_LIMIT and finally do a random mutation. Risk is you'd have slightly less children tested around each local max
2. Set a limit to the len(prev_used_params) that triggers a reset

The genetic algorithm sometimes stagnates in a state of pretty high success rates at low monte carlo counts that can't hit the threshold when the monte carlo count is increased from it's regular 250 to 5000.

At the lower counts, random noise can push a parameter set above the 95% target, but it'll fall just below when tested at higher values. That parameter set is then used as a parent and it's likely one of the children will also luck about the 95% and start the cycle over again.

Possible solutions:

1. Gradually increase the monte carlo count as you get higher levels of success rate. You may use 250 when you're in the 50s%, then gradually creep up to 1-2000 when you're in the high 80s - low 90s%.
2. Retest the parameter set at a higher monte carlo count every time none of it's children defeat it. This would ensure we're not pushing forward parameter sets that just got lucky once.
3. Require a slightly higher success rate to trigger the higher judgement (say 97% for a 95% target)

Add user selectable parameter for trust level in pensions. If user enters 0.5, pensions and social security payments would be cut in half

and how many generations? Would be interesting to build a study that tries different combinations of children/generation limits and tracks success growth rate...basically an optimization algorithm for an optimization algorithm!

directories in the constants.py need to be changed to use the OS module to avoid failures on MacOS

Create a function for different models of spending.

Currently limited to inflation-driven spending. Other methods could be something like portfolio percentage with or without ceilings/floors as advocated by Vanguard.

DEBUG_LVL 2 enables saving individual runs. Each time the simulator runs, it should delete any existing files in the /Investigation folder before creating new ones to avoid confusion between different instances