openscm / pymagicc Goto Github PK
View Code? Open in Web Editor NEWPython wrapper for the simple climate model MAGICC
Home Page: https://pymagicc.readthedocs.io/
License: BSD 3-Clause "New" or "Revised" License
Python wrapper for the simple climate model MAGICC
Home Page: https://pymagicc.readthedocs.io/
License: BSD 3-Clause "New" or "Revised" License
The current image in README.md shows the old scenario names:
When running the code mentioned to create this plot, I get the following:
Obviously there is some style differences. Perhaps @rgieseke can regenerate the plot with the correct size and style.
Goes together with thinking about config module #27
Same as used in MAGICC7, easier to loop around ...
We haven't tested this case and it's not clear what happens
Is your feature request related to a problem? Please describe.
At the moment our notebooks aren't run automatically under CI, I don't think.
Describe the solution you'd like
Use scons
or something else to run all our notebooks under CI to make sure our examples actually work.
The config file writing currently done in pymagicc.run
should become part of the API
Version when installed from PyPI is broken '0+unknown'
Is your feature request related to a problem? Please describe.
One of the most annoying things is having to do all your unit conversions yourself before you run your model. I'd love for us to find a way that users can just throw dataframes with whatever (well-defined) units and magicc will run properly.
Describe the solution you'd like
I think my ideal solution would have two components:
units
module which defines all of the expected MAGICC units and can convert any input dataframes to the required units before writing files and running MAGICCDescribe alternatives you've considered
Continue to do this stuff by hand because it's tricky. That is an option but I think we could produce something really nice if we tried.
A general question, with Numpy dropping Python 2 support in 2019 (https://docs.scipy.org/doc/numpy/neps/dropping-python2.7-proposal.html) and Pandas (https://pandas.pydata.org/pandas-docs/stable/install.html)
The final release before December 31, 2018 will be the last release to support Python 2. The released package will continue to be available on PyPI and through conda.
Starting January 1, 2019, all releases will be Python 3 only.
Since there are very few users of Pymagicc right now I propose we drop Python 2 support now.
Any project starting now will likely be in Python3. People who need Python2 for some reason can still install an old Pymagicc version.
This would allow us to use e.g. (the very nice) pathlib
or unittest.mock
without having to import/install compatibility backports.
Thoughts, @lewisjared @znicholls @matthiasmengel?
Makefile
Some strings in PARAMETERS.OUT are fixed width strings padded with \0 (which aren't rendered by GitHub...). For example the last item in the array below is 7 * an empty string.
FGAS_FILES_CONC="HISTCMIP6_CF4_CONC_ENDOFYEAR.IN ","HISTCMIP6_C2F6_CONC_ENDOFYEAR.IN ","HISTCMIP6_C3F8_CONC_ENDOFYEAR.IN ","HISTCMIP6_C4F10_CONC_ENDOFYEAR.IN ","HISTCMIP6_C5F12_CONC_ENDOFYEAR.IN ",
"HISTCMIP6_C6F14_CONC_ENDOFYEAR.IN ","HISTCMIP6_C7F16_CONC_ENDOFYEAR.IN ","HISTCMIP6_C8F18_CONC_ENDOFYEAR.IN ","HISTCMIP6_cC4F8_CONC_ENDOFYEAR.IN ","HISTCMIP6_HFC23_CONC_ENDOFYEAR.IN ","HISTCMIP6_HFC32_CONC_ENDOFYEAR.IN ",
"HISTCMIP6_HFC4310mee_CONC_ENDOFYEAR.IN ","HISTCMIP6_HFC125_CONC_ENDOFYEAR.IN ","HISTCMIP6_HFC134a_CONC_ENDOFYEAR.IN ","HISTCMIP6mod_HFC143a_CONC_ENDOFYEAR.IN ","HISTCMIP6_HFC152a_CONC_ENDOFYEAR.IN ","HISTCMIP6_HFC227ea_CONC_ENDOFYEAR.IN ",
"HISTCMIP6_HFC236fa_CONC_ENDOFYEAR.IN ","HISTCMIP6mod_HFC245fa_CONC_ENDOFYEAR.IN ","HISTCMIP6mod_HFC365mfc_CONC_ENDOFYEAR.IN ","HISTCMIP6_NF3_CONC_ENDOFYEAR.IN ","HISTCMIP6_SF6_CONC_ENDOFYEAR.IN ","HISTCMIP6_SO2F2_CONC_ENDOFYEAR.IN ",
7*" ",
Which is serialised to:
"fgas_files_conc": [
"HISTCMIP6_CF4_CONC_ENDOFYEAR.IN",
"HISTCMIP6_C2F6_CONC_ENDOFYEAR.IN",
"HISTCMIP6_C3F8_CONC_ENDOFYEAR.IN",
"HISTCMIP6_C4F10_CONC_ENDOFYEAR.IN",
"HISTCMIP6_C5F12_CONC_ENDOFYEAR.IN",
"HISTCMIP6_C6F14_CONC_ENDOFYEAR.IN",
"HISTCMIP6_C7F16_CONC_ENDOFYEAR.IN",
"HISTCMIP6_C8F18_CONC_ENDOFYEAR.IN",
"HISTCMIP6_cC4F8_CONC_ENDOFYEAR.IN",
"HISTCMIP6_HFC23_CONC_ENDOFYEAR.IN",
"HISTCMIP6_HFC32_CONC_ENDOFYEAR.IN",
"HISTCMIP6_HFC4310mee_CONC_ENDOFYEAR.IN",
"HISTCMIP6_HFC125_CONC_ENDOFYEAR.IN",
"HISTCMIP6_HFC134a_CONC_ENDOFYEAR.IN",
"HISTCMIP6mod_HFC143a_CONC_ENDOFYEAR.IN",
"HISTCMIP6_HFC152a_CONC_ENDOFYEAR.IN",
"HISTCMIP6_HFC227ea_CONC_ENDOFYEAR.IN",
"HISTCMIP6_HFC236fa_CONC_ENDOFYEAR.IN",
"HISTCMIP6mod_HFC245fa_CONC_ENDOFYEAR.IN",
"HISTCMIP6mod_HFC365mfc_CONC_ENDOFYEAR.IN",
"HISTCMIP6_NF3_CONC_ENDOFYEAR.IN",
"HISTCMIP6_SF6_CONC_ENDOFYEAR.IN",
"HISTCMIP6_SO2F2_CONC_ENDOFYEAR.IN",
"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
]
I am proposing right stripping the null chars in parameters.out to make them empty strings instead of fixed width null chars.
In the docs we have,
Robert and Jared I'm pretty sure this isn't true anymore?
The _magiccpath
and _magiccbinary
can be changed to point to different MAGICC
development versions.
Is this still true?
How would you feel about starting to refactor the tests to make them scale slightly more easily? In my MATLAB code to write files for CMIP6 stuff I ended up having something like:
tests
|--- writer_tests
|--- reader_tests
|--- plotter_tests
etc.
I felt this made it much easier to keep track of what was going on rather than just having one big test_everything
file.
Is it MAGICC_EXECUTABLE
or MAGICC_EXECUTABLE_7
Is your feature request related to a problem? Please describe.
Enabling carbon cycle output in pymagicc does not return carbon cycle output variables. It seems that the output file name (CARBONCYCLE.OUT) may need to be standardised by placing DAT at the start of the filename in the same format as other optional output.
Describe the solution you'd like
It would be good to return the carbon cycle output variables (currently in CARBONCYCLE.OUT) when the flag is on (i.e. out_carboncycle=1).
Describe alternatives you've considered
I could write my own python script to handle this. But, the infrastructure is already in place with pymagicc so it "should be" trivial to add this feature in.
Additional context
Note that the carbon cycle output file differs from others such that the columns are different variables (e.g. CURRENT_GPP), not different regions (e.g. GLOBAL, NH-OCEAN). However, I don't think this should change how it is handled.
We should add testing with Python 3.7 and consider dropping 3.5 from the CI. Is anyone still stuck on 3.5? @lewisjared @znicholls
As a user of pymagicc I would like to be able to run both MAGICC6 and MAGICC7 scenarios in the same interpreter
As a magicc developer I want to be able to dynamically specify a local version of MAGICC which is used with pymagicc
This would currently be possible using our MAGICC_EXECUTABLE environment variable, but it isn't a nice API. You would have to update os.environ
between simulations.
Perhaps we actually need two configuration parameters MAGICC6_EXECUTABLE
and MAGICC7_EXECUTABLE
. The low level API then chooses the appropriate executable and as a byproduct already knows which version of MAGICC it is using! This suggests that we might need a more complex config
handling methods sooner than I anticipated
Is your feature request related to a problem? Please describe.
At the moment you can't read or write HIST_XXX_EMIS.IN
files or HIST_XXX_CONC.IN
files from pymagicc
. This is annoying because you can't see the input to MAGICC
easily nor modify it to see what effect it has.
I think this issue should largely focus on a discussion of how we keep the metadata with the data, given that pandas
does not yet have a good way of doing this.
Describe the solution you'd like
An api which offers the ability to write and read these files. After conversations with @lewisjared and @rgieseke it's still not clear what the best way to deal with the metadata is. In my ideal world the api would be something like
MData = MAGICCInput()
MData.read('HIST_EMIS_CO2I.IN')
MData.print_data()
This would print a dataframe in wide format by default, same as human readable data files, however the data would actually be stored in long format internally.
MData.df
This would be the long format data frame
I'm not sure what the best way to access data is given that our data is effectively multi-dimensional, perhaps
http://pandas.pydata.org/pandas-docs/version/0.22/generated/pandas.DataFrame.xs.html
MData.df.xs(('CO2I_EMIS', 'WORLD'), level=['Variable', 'Region'])
https://pandas.pydata.org/pandas-docs/stable/advanced.html#basic-indexing-on-axis-with-multiindex
From this link: 'Note that df.loc['bar', 'two'] would also work in this example, but this shorthand notation can lead to ambiguity in general'
MData.df.loc[('CO2I_EMIS', 'WORLD'),] #
Custom options
MData['Gas']['Region']
MData.df['Gas']['Region']
MData.filter(gas='Gas', region='Region') # copying pyam
I'm not sure what this should return, maybe a Pandas DataSeries with year as the index? We could also include units in the index..
MData['Gas']
I think something like this should return a dataframe with all of the available regions (subject to same access question as above)
Re metadata, perhaps our class should have a metadata attribute which is a dictionary
MAGICCInput.metadata
MAGICCInput.metadata['source']
Where the units should sit is tricky. I'm leaning towards having them as an index, despite the extra cost of duplicating them however many times, to ensure they stay with the data. Another option may be to have something like
MData.metadata['units']['CO2I_EMIS']
i.e. the units are simply stored in the class's metadata attribute and can be looked up when necessary. We could then have a decorator on data setting which requires a 'units' argument and raises a warning/error if they're not provided.
Describe alternatives you've considered
Making changes by hand or only using files which are made elsewhere. These seem like bad options.
Is your feature request related to a problem? Please describe.
I'd like to read and probably change (e.g. for sensitivity analyis) more of the input files that ship with MAGICC. Currently only parameters and scenarios are exposed.
Describe the solution you'd like
Not sure about nicest API yet.
Describe alternatives you've considered
It's possible to read the files manually.
Could it make it more obvious which config is coming from where whilst debugging?
At the moment MAGICC input files use a mix of region names. In the API I figure we may as well make them all come out the same for the regions which are equivalent. Hence how do we feel about always using:
'GLOBAL' rather than 'WORLD'
'RASIA', 'RREF' etc. rather than 'R5ASIA', 'R6ASIA', 'R5REF', 'R6REF' (or should we actually keep these separate because their definitions are, in a very minor way, different and instead just allow people to access both 'R5ASIA' and 'R6ASIA' with 'RASIA' depending on what is in the dataframe)
The release process needs to be written down and, ideally, automated more:
Like CARBONCYCLE.OUT
output discussed in #84, #85 there are a few other outputs not named DATSOMETHING.OUT
-- should we read them all or add them as we go when someone actually needs them?
Thought @znicholls @NortonAlex ?
Ensure that pymagicc also runs with wine
installed from installer package available from
https://dl.winehq.org/wine-builds/macosx/download.html
With wine
on your path in should in principle work โฆ The 32-bit version is required, but it seems this is the default for all downloads.
which wine
/Applications/Wine Stable.app/Contents/Resources/wine/bin/wine
Following up from openjournals/joss-reviews#516 (comment)
@scollis Can you give some more error details? For example
wine --version
Any error message that is shown from Python?
import pymagicc
pymagicc.run(pymagicc.rcp3pd)
Is your feature request related to a problem? Please describe.
At the moment the native variable names are all over the place and not always that descriptive e.g. CO2B_EMIS
doesn't really tell you anything.
Describe the solution you'd like
Defining a set of mapping dictionaries to change MAGICC's raw output to nicer names. I'm not sure how exactly this would look. I don't know if it's better to have trees so you access things with e.g. output['emissions']['co2']['AFOLULUC']
or delimited variable names e.g. output['emissions-co2-AFOLULUC']
. In both cases having standardisation across variables should make life easier.
Describe alternatives you've considered
Sticking with the native variable names. I don't like this because they're not that descriptive.
Describe the bug
live.magicc.org provides tunings for the other bench marked atmospheric-ocean general circulation models mentioned in the main MAGICC reference paper (http://www.atmos-chem-phys.net/11/1417/2011/acp-11-1417-2011.html). These tunings appear to be defined in specific files named MAGTUNE_tuningname.CFG and MAGTUNE_FULLTUNE_tuningname.cfg. They don't seem to exist in the kit delivered by pymagicc or even the windows zip from magicc.org. These files are also used to build up the data used in bulk runs.
The names of the tunings are as follows, and taken from bulk runs from live.magicc.org:
file_tuningmodel_1 = list[
'c4mip_bern',
'c4mip_ccsm1',
'c4mip_climber',
'c4mip_frcgc',
'c4mip_hadley',
'c4mip_ipsal',
'c4mip_llnl',
'c4mip_mpi',
'c4mip_umd2',
'c4mip_uvic']
file_tuningmodel_2_1 = list[
'fulltune_cccma_cgcm3_1_t47',
'fulltune_cnrm_cm3',
'fulltune_csiro_mk3_0',
'fulltune_gfdl_cm2_0',
'fulltune_gfdl_cm2_1',
'fulltune_giss_model_e_h',
'fulltune_giss_model_e_r',
'fulltune_iap_fgoals1_0_g',
'fulltune_inmcm3_0',
'fulltune_ipsl_cm4',
'fulltune_miroc3_2_hires',
'fulltune_miroc3_2_medres',
'fulltune_miub_echo_g',
'fulltune_mpi_echam5',
'fulltune_mri_cgcm2_3_2a',
'fulltune_ncar_ccsm3_0',
'fulltune_ncar_pcm1',
'fulltune_ukmo_hadcm3',
'fulltune_ukmo_hadgem1']
System (please complete the following information):
macOS 10.13.6, pymagicc 1.3.1
(and additionally the native Windows: windows zip file: Magicc6.801 Beta)
Some settings are in the appendix tables for the paper mentioned above, but don't appear to match the default settings.
This API can be improved by being able to use this package in a with
statement which automatically initialises and cleans the package i.e.
with Package() as p:
for i in range(10):
p.run()
Explicitly defined root dirs should not be automatically initialised
Need to add an argument for initialising Package in the initialiser
I tried to "use a different MAGICC version" following
"""
#!/bin/bash
export MAGICC_EXECUTABLE_7=tmp/magicc/run/magicc.exe
python run_tests.py
python generate_plots.py
"""
from Readme.md
However, a run_tests.py
file does not exist, nor does a generate_plots.py
.
Could you update this section?
We need a way of accessing configuration from anywhere within pymagicc
in a consistent manner. This state should not be stored in the root __init__.py
as that would cause lots of circular dependencies (Module A imports Module B which imports Module A). There is a need to define default config, long-lived somewhat static config and to be able to override these values dynamically. Having a separate module with a simple interface will hide the complexity of this.
Here is how I propose that we lookup configuration variables in other parts of the library. The config object looks like a dict
, but it knows how to find the appropriate parameter value.
from pymagicc.config import config
executable = config['MAGICC6_EXECUTABLE']
Any feedback on the API?
Is your feature request related to a problem? Please describe.
Some development copies of MAGICC end up being very large due to large untracked files such as temporary build output. Using pymagicc with these directories can take several minutes to copy the large number of files which limits pymagicc utility.
Describe the solution you'd like
Only copy the top-level files in the run directory and create the out directory rather than copying it. The configuration in the run directory is currently flat and anything not in the root run directory are build artifacts.
Describe alternatives you've considered
Currently Pymagicc always re-writes the Scenario from a DataFrame -- it should have the ability to directly to run a SCEN file if given as parameter. (Similar to what Pandas does with inputs.)
v0.22 might have broken something.
Is your feature request related to a problem? Please describe.
At the moment I don't know what a sensible way to provide access to dataframe data is. Using a multi-indexed frame can make things pretty nasty.
Describe the solution you'd like
It'd be good to have some sort of wrapper included in #49 which makes it easy for users to access data through our api. I'm not sure what the best way for that to look is. It seems like now is a good time to define how we want it to look/behave, given it's not built yet.
I'm guessing we probably want to be able to do things like:
minput.df.loc['ASIA', 'CO2_CONCS']
and have sensible values returned (in this case a dataframe, the values of which could then be extracted with .values
). At the moment you have to specify every 'dimension' (e.g. minput.df.loc['ASIA', 'CO2_CONCS', 'SET', 'ppm', slice(2015,2100)]
to get data and I don't think that should be the case).
Describe alternatives you've considered
We could just force all users to learn how to use multi-indexed dataframes but that seems contrary to the spirit of the project.
When this happens, can we ship with a minimal run
directory. The tests being so slow because we're copying so many files for MAGICC7 is driving me insane.
Will running MAGICC with only three input gases work as shown in the examples?
This points to the same value and _magiccbinary
is not used anymore I believe.
# default parameters and cannot be changed after module load
_magiccpath, _magiccbinary = MAGICC6().original_dir, MAGICC6().original_dir
Can we get rid of this?
_magiccpath
could likely be replaced when reading scenario data.
General question, @lewisjared @znicholls -- might be helpful for decisions on how to make design decisions.
Do we need to maintain MAGICC6 compatibility once MAGICC7 is publicly available?
Will there be a need to run both from one installation? Do you need this to run regressions tests/comparisons for MAGICC7?
One could always install version 1.x and run MAGICC6, might mean separate Virtualenvs if you actually do need both but I'd think that would be fairly rare.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.