This is to implement an example and a test with the A2C algorithm in the bestest_hydronic_heatpump case. This is the synchronous, deterministic variant of Asynchronous Advantage Actor Critic (A3C), as described in https://arxiv.org/pdf/1602.01783.pdf, and documented in https://stable-baselines.readthedocs.io/en/master/modules/a2c.html.

This is to allow time to be a feature of the environment.

Add unittests for DiscretizedObservationWrapper and DiscretizedActionWrapper

This is to make excluding_periods an attribute of BoptestGymEnv instead of an argument of the reset method. This way, the periods will not be used either when using external algorithms as those from Stable Baselines.

This is to implement examples that use behavior cloning to accelerate training.

Currently the length of episodes is fixed. It may help a lot for learning to terminate the episodes upon certain conditions are met, e.g. when reward lowers a certain value. This is to implement a callback method that can be customized by the user to define those conditions. Notice that the episode_length argument would become max_episode_length to define the maximum possible episode length instead of a fixed length. This is still needed for the definition of the end_year_margin and for those cases where a fixed episode length is desired.

Edit gitignore to ignore .pyc files

This is to modify the default reward in

The current train_A2C example mistakes the model.save call with the model.load call.

This is to change the simulation step keyword. Currently Ts is being used, although it'd be more intuitive to use step, which is also used in BOPTEST.

This is to prepare a first documentation draft. This may include a Binder demo.

Slight update in references is needed when switching BOPTEST version. This version change is not reflected in the repo because CI is not yet implemented.

This is to add an example and test with reward clipping.

Right now two separate arguments are being used to define the starting time of an episode: random_start_time and start_time, with the latter being used when random_start_time=False. It'd be clearer and cleaner to use only one argument: start_time that can have either a fixed value or be assigned None in which case a random time is used.

This is to add an environment.yml file specifying the conda environment required to run the examples of this repo.

This is to add a few custom environments to the main BOPTEST Gym script.

This issue is to not use the weight factor as an attribute of the BOPTEST Gym environment. It should rather be a local variable within the compute_reward method.

This is to enable saving KPI results to file upon calling test_and_plot.

This is to train the A2C agents of the examples with 1e6 interaction steps.

This is to substitute the keyword jan keyword (of January) to feb (of February) in all examples, since they actually run in February, not January.

This is to clean up testing by e.g. avoiding duplicate code or generalizing test_agent method to allow models from stable baselines.

This is to define a callback function that saves the agent model every n iterations. An example implementation can be found here:
https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/monitor_training.ipynb#scrollTo=pUWGZp3i9wyf

This is to create the first unittests for this repo

Registration of new custom Gym environments is easy using gym.envs.registration.register. Registering BOPTEST environments would make it extremely easy for users to access them. However, this issue may need to wait until scenarios are defined better and until BOPTEST is hosted as a web service, so that docker is not required in the installation.

This issue is to use the SaveOnBestTrainingRewardCallback to store the model every fixed number of interaction steps so that the saved models can be loaded later to assess the training progress.

This is to add the possibility to specify excluding periods when resetting the environment such that episodes do not overlap with these periods when initializing from a random start time. This is useful to ensure that testing data is different from training data.

There is a typo on the implementation of the PPO2 algorithm:

That line should load and not save the model. Because of this, the pre-trained model was probably overwriten and lost and therefore the agent needs to be re-trained and references for this example need to be updated.

This is to enable changing the BOPTEST scenario from the gym interface.

The readme example is using the BoptestGymEnvRewardWeightCost while the BoptestGymEnv is imported instead.

The keyword used in the RL community for "measurements" is "observations". Still the word "measurements" (or "meas") appears sometimes interchangeably with "observations" (or "obs"), which may be misleading. This issue is to change all those occurrences: from "measurements" to "observations".

Right now tensorboard logs are also generated during testing. This issue is to avoid these logs as they are not needed for testing purposes.

This is to update the references to run with the latest BOPTEST version. It may require some edits to adapt to the newest version of the interface where each data point of the results needs to be requested independently. Additionally, the references may show differences because the bestest_hydronic_heat_pump case, which is the buidling emulator model used in the examples and tests, has been modified, as well as the pricing scenarios. Hence, the trained agents used in the examples may behave poorly. However, the agents should not be retrained for this issue, but for a new issue that will be opened after this one that will also update for the new scenario periods defined in BOPTEST.

This is to implement an example and a test with the PPO2 algorithm in the bestest_hydronic_heatpump case. This is the proximal policy optimization algorithm as described in in https://arxiv.org/abs/1707.06347, and documented in https://stable-baselines.readthedocs.io/en/master/modules/ppo2.html.

Normalizing observation and action space can help with convergence speed. This issue is to create a NormalizedObservationWrapper and a NormalizedActionWrapper to do so. Use wrappers as in DiscretizedObservationWrapper and DiscretizedActionWrapper.
See: https://stable-baselines.readthedocs.io/en/master/guide/rl_tips.html.

The number of forecasting steps should be of the number of look-ahead prediction steps PLUS1 for the actual time. This is to correct this line:

This is to add a .gitignore file to the repository.

This is to implement examples that include forecast in the agent observations to prove the possibility of training predictive RL agents.

Resolve bug that prevents to return start_time when excluding periods are not defined.

It relevant for diagnosis to see what's actually happening within the environment. This issue is to implement a render method to display such information.

This is to use the BOPTEST peak and typical periods for the examples and testing, instead of the current periods in February and November, that were chosen without any rigorous criteria. This issue should re-train the agents used in the examples as well, following what is discussed in #83.

This is to update the DQN and SAC example models when trained with 1e6 steps.

This is to add the possibility to extend the state espace with past observations. This allows the agent to extract hidden states time features in the partially observable building system. An approach would be to add arguments for the desired regressive variables and regression period.

Add readme file explaining basic features and functionality.

This is to setup tests in GitHub Actions for CI/CD.

Add examples to illustrate how the default compute_reward method works and how it can be overridden with a custom reward function. Add unittests associated to these examples.

This is to make the normalized action wrapper a bit more clear.

This pull request was merged without adding unittests associated to the examples. This issue is to complete that pull request by adding unittests using pre-trained agents.

Rescaling rewards may be useful to increase sampling efficiency. This can be done using previous rewards, but without shifting mean as that may affect agent's will to live.

So far all tests are performed with a one-dimensional action space. Potential issues may arise from the implementation of the interface with a multi-dimensional action space. It'd be convenient to perform tests in a multi-dimensional action space to prevent those issues.

Avoid printing in __init__ and use a more formal method to display the environment.