iiasa / aneris Goto Github PK

@rgieseke I am migrating issues from my remote to iiasa's. here is your issue gidden/aneris#10

Hi @gidden

I just helped a colleague to install this in a Conda environment.
We ended with a local install from a Git clone.

I think https://github.com/gidden/aneris/blob/dcf6679334f44067f376f2fe696e55bf689f2cfb/requirements.txt#L1 should not mention the aneris package -- it is unrelated, isn't it?

Is this Python2 only? PyPI only has a Python2 version it seems.

Is this available in some conda forge?

was gidden/aneris#11

It looks like this was an intended feature, but never made it all the way through to implementation. See https://github.com/iiasa/aneris/blob/master/aneris/methods.py#L296 for where it looks like it should have been applied

When working with pyam.IamDataFrame, there is an extra column 'Meta' compared to the pandas.DataFrame that is passed around in aneris.

While aneris does do anything with this column, it would be nice to align the aneris dataformat with the pyam.IamDataFrame.

was gidden/aneris#9

currently all categories in utils.py are hard coded..

Following on from #42, a description of how to move unit handling and other changes into the body of aneris.

convenience

I would keep convenience.harmonise_all where it is i.e. not merge it into the main body: it's a useful interface that can just wrap around other stuff and provides an interface that doesn't currently exist. There might be a better way to implement it rather than this groupby stuff...

convenience._harmonise_single is not that well written tbh (well, it's too big). The steps are basically:

1. work out what history timeseries you're harmonising to (explode if there's nothing that matches or harmonisation year isn't there/is nan)
2. convert the units of history to the timeseries being harmonised
3. determine whether to override the default tree or not. I basically added a new way of handling overrides, that behave a bit more like pyam and scmdata's filter methods. I'm not sure if the new implementation is the best, but it did what I wanted and was more intuitive to me cause I could use the variable/region names as I saw them in the input data rather than having to know about how aneris chops variable names into sectors etc. (My implementation does also require a check to make sure that the user hasn't put in an ambiguous choice over override)
4. harmonise (fairly simple now as we know the timeseries to harmonise, history, harmonisation year and method)

convenience.convert_units is pretty simple so could just stay (and I think is much faster because it's all pandas and pint based). Otherwise make pyam or scmdata a dependency and use their unit conversion (might come with a performance hit for large sets of timeseries).

convenience._harmonise_aligned and convenience._get_delta are just wrappers to make harmonisation slightly clearer. The pattern of having the harmonisation methods as a class method does seem a bit weird to me, but it's also not really an issue. You could merge this into the main code base. Alternately, you could just rethink the way the main code base works (could be a lot of work and there are probably good reasons it's written like it is). Finally, we could just leave these mini wrappers as they are (more stuff to maintain, but provides an extra handy interface for those who don't need the full power of the current implementation like me).

errors

Just added an errors module. Nice and easy, probably can just stay as is.

overall

In terms of migrating it all, you could put some of the unit handling stuff deep in the guts. I think the more fundamental question is whether you want to maintain one interface (the existing one or the new one) or multiple. The old one has the advantage that it's probably better when you have lots of sectors, want people to put config in csvs etc. The new one has the advantage that it's a bit more what you see is what you get, you start with some timeseries, you use the names of the timeseries as they appear to set any overrides, all unit conversions are handled for you.

I think the fundamental difference between the two is how they cut the data. My implementation just does one by one harmonisation, ignoring any need for internal consistency between the timeseries. The existing implementation is definitely much better at ensuring that regional sums are maintained sensibly, sectoral breakdowns are maintained etc. You could probably change the internal logic and config to make it more what you see is what you get (rather than breaking variable names down into gas and sector in the guts of the processing) and use some of pyam's internal summing and consistency checking to simplify the code base, that might be where I would start.

As hinted at in #12 and #5, the current prefix/suffix behaviour is not desireable.

The current case is that for harmonization to be triggered, both history and model values must have the exact same variable name. This includes the specific prefix and suffix that are used, and specified in the RunControl file.

Two points should be addressed here:

1. The current error message that is thrown when prefix/suffix does not match is unclear. Checking the data before running the harmonization routine to see if there are values in either dataset matching the expected prefix/suffix, could be a way to make this more user-friendly.

  File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 40, in <module>
    call_harmonization(hist_file)
  File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 32, in call_harmonization
    harmonize(model, history=history, regions=regions, rc=rc, output_path=output_path, output_prefix=output_prefix)
  File "c:\users\kikstra\documents\github\aneris\aneris\cli.py", line 64, in harmonize
    driver.harmonize(scenario)
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 476, in harmonize
    scenario).results()
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 353, in process
    self._downselect_var()  # only prefix|*|suffix
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 297, in _downselect_var
    assert(len(self.hist) > 0)
AssertionError```

2. A more long-term solution would be to create more flexibility in how aneris deals with variable names. I.e., the prefix/suffix construction could be removed to allow for more flexibility in data input regarding the naming of variables.

Even when for instance HFC|HFC125 is fed into aneris as kt HFC125/yr, current code in utils.py (notably unit_gas_names = {...}, def gases(...) and def units(...)) force all gases in one sector to have the same output string (while actually no conversion of the numbers happens).

I am now trying to harmony SSP emissions to the new version CEDS v_2021_04_21 historical emissions with a base year of 2019. When I used the test data, the aneris run smoothly. But when I replaced the test data by real annual BC emission over U.S. from CEDS and SSP245 (aneris ssp245_EU_BC.xls --history history_EU_BC.xls --regions regions_mapping_unused.csv --rc aneris_regions_sectors.yaml), the following error occurred:

INFO:root:Downselecting prefix|suffix variables
INFO:root:Translating to standard format
INFO:root:Aggregating historical values to native regions
WARNING:root:Removing regions without direct mapping: ['EU']
INFO:root:Harmonizing (with example methods):
INFO:root:region gas sector units
World BC prefix|sector1|suffix kt reduce_ratio_2080
Name: method, dtype: object
INFO:root:Harmonizing with reduce_ratio_2080
Traceback (most recent call last):
File "/home/b/b381315/.conda/envs/hjia/bin/aneris", line 33, in
sys.exit(load_entry_point('aneris-iamc==0.3.0', 'console_scripts', 'aneris')())
File "/home/b/b381315/.conda/envs/hjia/lib/python3.9/site-packages/aneris/cli.py", line 90, in main
harmonize(args.input_file, args.history, args.regions,
File "/home/b/b381315/.conda/envs/hjia/lib/python3.9/site-packages/aneris/cli.py", line 64, in harmonize
driver.harmonize(scenario)
File "/home/b/b381315/.conda/envs/hjia/lib/python3.9/site-packages/aneris/harmonize.py", line 489, in harmonize
self._model, self._meta = _harmonize_regions(
File "/home/b/b381315/.conda/envs/hjia/lib/python3.9/site-packages/aneris/harmonize.py", line 619, in _harmonize_regions
model = harmonizer.harmonize(overrides=overrides)
File "/home/b/b381315/.conda/envs/hjia/lib/python3.9/site-packages/aneris/harmonize.py", line 261, in harmonize
raise ValueError(msg.format(method, report))
ValueError: Harmonization failed with method reduce_ratio_2080 harmonized values != historical values. This is likely due to an override in the following variables:

region gas sector units 2019
0 World BC prefix|sector1|suffix kt 53369.230769

I searched on the offical website and github of aneris, but couldn't find a solution. The input data are also attached (data_aneris.zip). Many thanks!

data_aneris.zip

The call at

Which defaults to reduce_offset_2050 pbb.

• pypi
• [ ] conda

was gidden/aneris#8

A number of things look to be required out of the box, they should be optional or have smart defaults. These include

• prefix/suffix (used in run control)
  • this is actually a larger problem in that it is assumed these exist, thus it also searches for '|'.join(...). more work here is actually needed
• regions, ie region mapping, one should be able to just harmonize without needing explicit region mappings
• overrides this looks to be required, but can be empty. it shouldn't be required or should default to empty.

When replicating the Getting Started Tutorial on my pc, everything works as expected.
See this folder for the files used, if you'd like to reproduce.

Now, with the same script, but different minimal input data, aneris seems to break at _downselect_var().
To reproduce, use the files in this folder.

When history data has suffix |History with model data suffix being |Unharmonized, I get the error:

Traceback (most recent call last):
  File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 40, in <module>
    call_harmonization(hist_file)
  File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 32, in call_harmonization
    harmonize(model, history=history, regions=regions, rc=rc, output_path=output_path, output_prefix=output_prefix)
  File "c:\users\kikstra\documents\github\aneris\aneris\cli.py", line 64, in harmonize
    driver.harmonize(scenario)
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 476, in harmonize
    scenario).results()
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 353, in process
    self._downselect_var()  # only prefix|*|suffix
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 297, in _downselect_var
    assert(len(self.hist) > 0)
AssertionError

--> see #14 for further specification of the prefix/suffix issue addressed above. Below, in this thread (#14) we focus further on down_select_var() behaviour wrt aggregates.

When history data has suffix |Unharmonized, to more closely mimic the Getting Started Tutorial (with model data suffix being |Unharmonized), I get:

Traceback (most recent call last):
  File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 41, in <module>
    call_harmonization(hist_file)
  File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 31, in call_harmonization
    harmonize(model, history=history, regions=regions, rc=rc, output_path=output_path, output_prefix=output_prefix)
  File "c:\users\kikstra\documents\github\aneris\aneris\cli.py", line 64, in harmonize
    driver.harmonize(scenario)
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 490, in harmonize
    self.config['harmonize_year'], self.add_5regions
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 605, in _harmonize_regions
    'Model is empty after downselecting regional values')
RuntimeError: Model is empty after downselecting regional values