nismod / energy_demand Goto Github PK

The current documentation is excellent if you wish to understand how the model works. Unfortunately, it doesn't help to understand how to run the model!

Perhaps you could add a section to the docs which describes how to install the package, where to put the data (and what data is required), what command to enter to run the model and what and where the results are placed.

Newscastle will provide most likely only base year floor area for the service sector. Future floor area of the service sector however needs to be projected (e.g. by establishing a link between GVA & floor area for the base year for the different industries across uk and then using future GVA to project grow in area)

Testing under smif, the population check fails the assertion in https://github.com/nismod/energy_demand/blob/master/energy_demand/building_stock_generator.py#L181

Note that round rounds to a number of decimal places, not significant figures, so

52000000 and 52000000.00364002

are rounding to

52000000 and 52000000.004

and comparing as not equal, which doesn't seem to be what's intended.

The readthedoc is really genious! Thanks for the tipps...I'm no enthusiastically using all the features...:-)

Before leaving the whole of sepetember, it would be cool to have

• a chat about memory and speed
• running enegery demand with smif

I think we can make it even easier to setup energy demand.

I have been running into issues with conflicts between the version of data and code I am using causing errors. One solution would be to link to a version of the energy demand data - this could be a zipped archive hosted on the ftp which is called energy_demand_data_v0.45.15

We could follow semantic versioning approaches - minor updates increment the rightmost digits, major updates increment the middle digits, non backwards-compatible changes to the data increment the leftmost digit.

The codebase could then reference the minimum version of data required in a configuration file?

I get the following error when trying to install version 0.2.0 on Linux:

==> default:     error: can't copy 'config_data/01-validation_datasets/03_subnational_gas/Regional': doesn't exist or not a regular file

I'm using the following command to install the package:

VERSION=0.2.0
pip3 install git+https://github.com/nismod/energy_demand.git@$VERSION#egg=energy_demand

Just that we don't forget that we said that we want to have a look at what is loaded beofre executing the main code... maybe we can doe this sometime after your break! :-)

A note on code style. Generally code that does something to an argument that is passed in and then returns that is confusing to debug. For example, this:

data['assumptions'] = self.load_smif_parameters(data_handle, data['assumptions'])

calls this:

def load_smif_parameters(self, data_handle, assumptions):

    <<<does stuff>>

    # Add to assumptions
    assumptions['strategy_variables'] = strategy_variables
    return assumptions

This can be written more clearly by dropping the argument.

new_assumptions = self.new_load_smif_parameters(data_handle)
data['assumptions'].update('strategy_variables', new_assumptions)

where this side effect is removed.

def new_load_smif_parameters(data_handle):
    <<<does stuff>>>
    return strategy_variables

This seems to be a bug in setuptools, but python setup.py install breaks due to the space in this filename:
~/nismod/models/energy_demand/data/submodel_residential/_EXCELFILES/__ECUK TABLES.xlsx

It's about 50MB, which is getting on the large side for a source code repository. There seem to be a wide variety of files, including pdfs and xlsx - could these be removed from the repository and stored elsewhere with the main data pack?

config_data $ du -h 
 40K	./07-switches
124K	./04-census_data
5.6M	./05-dwelling_stock_related/_orig/08-Building dataset
4.5M	./05-dwelling_stock_related/_orig/07-Energy fact sheet
 10M	./05-dwelling_stock_related/_orig
 10M	./05-dwelling_stock_related
 16K	./02-spatial_index
912K	./09-emission_factors
1004K	./01-validation_datasets/03_subnational_gas/_orig
2.6M	./01-validation_datasets/03_subnational_gas
 10M	./01-validation_datasets/02_subnational_elec/_orig
 10M	./01-validation_datasets/02_subnational_elec
2.1M	./01-validation_datasets/01_national_elec_2015
 15M	./01-validation_datasets
 64K	./10-SMIF_config_files/config/sector_models
 16K	./10-SMIF_config_files/config/sos_models
 12K	./10-SMIF_config_files/config/sos_model_runs
 96K	./10-SMIF_config_files/config
100K	./10-SMIF_config_files
2.4M	./08-technologies/_orig
2.4M	./08-technologies
188K	./06-fuels/_alternative_fuels_tables
2.8M	./06-fuels/_orig
3.0M	./06-fuels
 80K	./03-load_profiles/ss_submodel/_orig
 88K	./03-load_profiles/ss_submodel
952K	./03-load_profiles/rs_submodel/_orig/_checked
 88K	./03-load_profiles/rs_submodel/_orig/lp_bossmann_storage_heater
 15M	./03-load_profiles/rs_submodel/_orig
 15M	./03-load_profiles/rs_submodel
 15M	./03-load_profiles
 47M	.

Install NISMOD from github.com/nismod/nismod#v2 onto Sven's development computer

Running v0.7.4 on linux vm using vagrant.

When I run energy_demand minimal_setup -d /vagrant/data/energy_demand, the following error is raised:

Traceback (most recent call last):
  File "/usr/lib/python3.5/configparser.py", line 1135, in _unify_values
    sectiondict = self._sections[section]
KeyError: 'PATHS'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/energy_demand", line 11, in <module>
    load_entry_point('energy-demand', 'console_scripts', 'energy_demand')()
  File "/vagrant/energy_demand/energy_demand/cli/__init__.py", line 71, in main
    args.func(args)
  File "/vagrant/energy_demand/energy_demand/scripts/s_write_dummy_data.py", line 90, in post_install_setup_minimum
    config = basic_functions.convert_config_to_correct_type(config)
  File "/vagrant/energy_demand/energy_demand/basic/basic_functions.py", line 15, in convert_config_to_correct_type
    out_dict['PATHS']['path_local_data'] = config.get('PATHS', 'path_local_data')
  File "/usr/lib/python3.5/configparser.py", line 778, in get
    d = self._unify_values(section, vars)
  File "/usr/lib/python3.5/configparser.py", line 1138, in _unify_values
    raise NoSectionError(section)
configparser.NoSectionError: No section: 'PATHS'

I think that this is because in s_write_dummy_data.py on line 83 the assumed location of the wrapper config file is hard coded?

    path_config_file = os.path.abspath(
        os.path.join(
            os.path.dirname(__file__), '..', '..', '..', 'config', 'wrapperconfig.ini'))

    # Get config in dict and get correct type
    config = configparser.ConfigParser()
    config.read(path_config_file)
    config = basic_functions.convert_config_to_correct_type(config)

Perhaps the location of the wrapper config file could be passed in as a command line argument?

• The memory keeps increasing whilst iterating over the years

Rewrite parameters that employement statistics from shine can be laoded for each LAD

• update download file with newest data version
• include README file

When logging debug lever, the energy demand model reports hundreds of rows of:
DEBUG The HLC could not be calculated of a dwelling
Could they be removed?

add that absolute peak load per different fueltypes are read out for:

• different seasons
• full year

Currently hardcoded, should be passed in at runtime.

Implement sensitivity analysis to test e.g.

• thermal comfort (which includes temp assumptions)
• efficiencies
• GVA
• Population
• ...

write function to sho load facotrs (e.g. seasonal) over time for all regions with an averaged fatter line. This chart should be generated for every fueltype

See e.g. https://travis-ci.org/nismod/energy_demand/jobs/562541094

Ends with

PyPI upload failed.
failed to deploy

Deep in "Deploying application" logs, more informative error:

HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information. for url: https://upload.pypi.org/legacy/

When trying to run the energy demand model using smif, I get the following error caused by the matplotlib requirement of the energy demand model.

Traceback (most recent call last):
  File "/Users/will2/miniconda3/envs/smif/bin/smif", line 11, in <module>
    load_entry_point('smif', 'console_scripts', 'smif')()
  File "/Users/will2/repository/nismod2/smif/src/smif/cli/__init__.py", line 461, in main
    args.func(args)
  File "/Users/will2/repository/nismod2/smif/src/smif/cli/__init__.py", line 225, in run_model_runs
    execute_model_run(model_run_ids, store, args.warm)
  File "/Users/will2/repository/nismod2/smif/src/smif/controller/execute.py", line 19, in execute_model_run
    model_run_definitions.append(get_model_run_definition(store, model_run))
  File "/Users/will2/repository/nismod2/smif/src/smif/controller/build.py", line 40, in get_model_run_definition
    sector_models = get_sector_models(sos_model_config['sector_models'], store)
  File "/Users/will2/repository/nismod2/smif/src/smif/controller/build.py", line 107, in get_sector_models
    sector_model = loader.load(sector_model_config)
  File "/Users/will2/repository/nismod2/smif/src/smif/data_layer/model_loader.py", line 42, in load
    klass = self.load_model_class(config['name'], config['path'], config['classname'])
  File "/Users/will2/repository/nismod2/smif/src/smif/data_layer/model_loader.py", line 80, in load_model_class
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "models/energy_demand/run.py", line 10, in <module>
    from energy_demand import wrapper_model
  File "/Users/will2/repository/nismod2/energy_demand/energy_demand/wrapper_model.py", line 6, in <module>
    from energy_demand.read_write import data_loader
  File "/Users/will2/repository/nismod2/energy_demand/energy_demand/read_write/data_loader.py", line 13, in <module>
    import matplotlib.pyplot as plt
  File "/Users/will2/miniconda3/envs/smif/lib/python3.7/site-packages/matplotlib/pyplot.py", line 2372, in <module>
    switch_backend(rcParams["backend"])
  File "/Users/will2/miniconda3/envs/smif/lib/python3.7/site-packages/matplotlib/pyplot.py", line 207, in switch_backend
    backend_mod = importlib.import_module(backend_name)
  File "/Users/will2/miniconda3/envs/smif/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/will2/miniconda3/envs/smif/lib/python3.7/site-packages/matplotlib/backends/backend_macosx.py", line 14, in <module>
    from matplotlib.backends import _macosx
ImportError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are using (Ana)Conda please install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib on OSX' in the Matplotlib FAQ for more information.

Model run as configured in nismod energy_demand_unconstrained with Arc scenario data may be underestimating electricity and gas demand for 2015:

• Gas 180 TWh - National Grid reports 880 TWh total; or 545 TWh with exports and power generation excluded
• Electricity 248 TWh - National Grid reports 334 TWh

In unconstrained mode, heat demand is output from the model as heat directly - the above calculations assume this is supplied by: 80% gas boilers with 80% efficiency and 20% heat pumps with 1.5 COP (coefficient of performance).

Possible sources of error or misconfiguration:

• model runs in constrained mode will output gas/electricity demands for heating end uses; these might give a different picture to the boiler/heat pump assumptions used above (though such a large difference is surprising)
• floor area inputs to the model have been recently configured; these might drive demand differently to the within-model virtual stock; test against previous configurations

The smif run energy_demand_test run looks for /vagrant/data/energy_demand/_raw_data/K-floor_area/floor_area_2018_03_27.csv while the energy_demand minimal_setup has created this file in /vagrant/data/energy_demand/_raw_data_minimal/K-floor_area/floor_area_2018_03_27.csv

Traceback (most recent call last):
  File "/usr/local/bin/smif", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.5/dist-packages/smif/cli/__init__.py", line 623, in main
    args.func(args)
  File "/usr/local/lib/python3.5/dist-packages/smif/cli/__init__.py", line 435, in execute_model_run
    modelrun.run(store)
  File "/usr/local/lib/python3.5/dist-packages/smif/modelrun.py", line 118, in run
    modelrunner.solve_model(self, store)
  File "/usr/local/lib/python3.5/dist-packages/smif/modelrun.py", line 147, in solve_model
    model_run.sos_model.before_model_run(data_handle)
  File "/usr/local/lib/python3.5/dist-packages/smif/model/sos_model.py", line 114, in before_model_run
    model.before_model_run(model_data_handle)
  File "./models/energy_demand/run.py", line 180, in before_model_run
    f_mixed_floorarea=assumptions.f_mixed_floorarea)
  File "/usr/local/lib/python3.5/dist-packages/energy_demand/read_write/data_loader.py", line 162, in floor_area_virtual_dw
    f_mixed_floorarea=f_mixed_floorarea)
  File "/usr/local/lib/python3.5/dist-packages/energy_demand/read_write/read_data.py", line 1183, in read_floor_area_virtual_stock
    with open(path_to_csv, 'r') as csvfile:
FileNotFoundError: [Errno 2] No such file or directory: '/vagrant/data/energy_demand/_raw_data/K-floor_area/floor_area_2018_03_27.csv'
vagrant@vagrant:/vagrant$ 

This involves changing the expected resolution of temperature inputs to match one-to-one with the model regions.

Impacts:

• could involve some rewriting to remove spatial matching from within the model
• may simplify code - or initially may be able to avoid any code changes ("weather station" locations simply identical with region centroids)
• may reduce data volume
• should be simpler to plug in alternative data sources in the future