Giter Club home page Giter Club logo

pymagicc's Introduction

Pymagicc

Repository health

CI CD

Coverage

Docs
Pypi

PyPI Install

PyPI

PyPI Version

Conda conda install conda platforms conda version
Paper JOSS Zenodo
Binder Launch Binder

Other info

Contributors Last Commit License

Pymagicc is a Python wrapper around the reduced complexity climate model MAGICC6. It wraps the CC-BY-NC-SA licensed MAGICC6 binary. Pymagicc itself is BSD-3 licensed.

MAGICC (Model for the Assessment of Greenhouse Gas Induced Climate Change) is widely used in the assessment of future emissions pathways in climate policy analyses, e.g. in the Fifth Assessment Report of the Intergovernmental Panel on Climate Change or to model the physical aspects of climate change in Integrated Assessment Models (IAMs).

Pymagicc makes the MAGICC model easily installable and usable from Python and allows for the easy modification of all MAGICC model parameters and emissions scenarios directly from Python. In climate research it can, for example, be used in the analysis of mitigation scenarios, in Integrated Assessment Models, complex climate model emulation, and uncertainty analyses, as well as in climate science education and communication.

See www.magicc.org and Meinshausen et al. 2011 for further information.

Basic Usage

import matplotlib.pyplot as plt

import pymagicc
import scmdata
from pymagicc import rcps

results = []
for scen in rcps.groupby("scenario"):
    results_scen = pymagicc.run(scen)
    results.append(results_scen)

results = scmdata.run_append(results)

temperature_rel_to_1850_1900 = (
    results
    .filter(variable="Surface Temperature", region="World")
    .relative_to_ref_period_mean(year=range(1850, 1900 + 1))
)

temperature_rel_to_1850_1900.lineplot()
plt.title("Global Mean Temperature Projection")
plt.ylabel("°C over pre-industrial (1850-1900 mean)");
# Run `plt.show()` to display the plot when running this example
# interactively or add `%matplotlib inline` on top when in a Jupyter Notebook.

image

For more example usage see this Jupyter Notebook. Thanks to the Binder project the Notebook can be run and modified without installing anything locally.

Installation

pip install pymagicc

On Linux and OS X the original compiled Windows binary available on http://www.magicc.org/ and included in Pymagicc can run using Wine.

On modern 64-bit systems one needs to use the 32-bit version of Wine

sudo dpkg --add-architecture i386
sudo apt-get install wine32

On 32-bit systems Debian/Ubuntu-based systems wine can be installed with

sudo apt-get install wine

On OS X wine is available in the Homebrew package manager:

brew install wine

It should also be available in other package managers, as well as directly from the Wine project.

Note that after the first install the first run of Pymagicc might be slow due to setting up of the wine configuration and be accompanied by pop-ups or debug output.

To run an example session using Jupyter Notebook and Python 3 you can run the following commands to create a virtual environment venv and install an editable version for local development:

git clone https://github.com/openscm/pymagicc.git

cd pymagicc
make venv
./venv/bin/pip install --editable .
./venv/bin/jupyter-notebook notebooks/Example.ipynb

Development

Setup

For local development, install dependencies and an editable version of Pymagicc from a clone or download of the Pymagicc repository with

make venv
./venv/bin/pip install --editable .

Running the tests

To run the tests run

./venv/bin/pytest tests --verbose

To skip tests which run MAGICC and take longer use

./venv/bin/pytest tests --skip-slow

To get a test coverage report, run

./venv/bin/pytest --cov

Conventions

Style

To unify coding style, allowing us to focus more on writing useful code and less time worrying about formatting, black is used.

To format the files in pymagicc and tests as well as setup.py run

make black

Csvs

In our miscellaneous csv's, for example the definitional csv's, we follow the following conventions to make our lives easier:

  • column names are all lower case, with underscores as separators (i.e. no spaces)

Dependencies

A user of pymagicc should be able to pip install and run all of our notebooks. This means that all of the libraries for running notebooks should be explicit dependencies, rather than being included in an extras requirement. Whilst this means that we have more dependencies, it makes it easier for end users and avoids extremely cryptic import errors.

Building the documentation

The docs use Sphinx and can be rebuilt locally in docs/builds/html/ with

make docs

More usage examples

Use an included scenario

from pymagicc.scenarios import rcp26

rcp26.head()

Read a MAGICC scenario file

from pymagicc.scenarios import read_scen_file

scenario = read_scen_file("PATHWAY.SCEN")

Run MAGICC for a scenario

import pymagicc
from pymagicc.scenarios import read_scen_file

scenario = read_scen_file("PATHWAY.SCEN")

results = pymagicc.run(scenario)

temperature_rel_to_1850_1900 = (
    results
    .filter(variable="Surface Temperature")
    .relative_to_ref_period_mean(year=range(1850, 1900 + 1))
)

Using a different MAGICC version

A custom version of MAGICC may be used with pymagicc using the MAGICC_EXECUTABLE_6 and MAGICC_EXECUTABLE_7 environment variables for MAGICC6 and MAGICC7 respectively. These environment variables should be set to the location of the magicc executable (either magicc for linux/mac or magicc.exe for Windows). For example, a custom MAGICC7 folder located at /tmp/magicc can be used on under Linux by setting MAGICC_EXECUTABLE_7 to /tmp/magicc/run/magicc.

Example usage in Bash:

MAGICC_EXECUTABLE_7=/tmp/magicc/run/magicc.exe make test

Or in a script:

#!/bin/bash
export MAGICC_EXECUTABLE_7=tmp/magicc/run/magicc.exe
make test

Contributing

Please report issues or discuss feature requests on Pymagicc's issue tracker.

You can also contact the pymagicc authors via email: mailto:[email protected], [email protected]

License

The compiled MAGICC binary by Tom Wigley, Sarah Raper, and Malte Meinshausen included in this package is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

See also the MAGICC website and Wiki for further information.

The pymagicc wrapper itself is released under a BSD-3 license. For details, see LICENSE.

Citation

If you make any use of MAGICC, its license requires citing of:

M. Meinshausen, S. C. B. Raper and T. M. L. Wigley (2011). "Emulating coupled atmosphere-ocean and carbon cycle models with a simpler model, MAGICC6: Part I "Model Description and Calibration." Atmospheric Chemistry and Physics 11: 1417-1456. https://doi.org/10.5194/acp-11-1417-2011

If you use Pymagicc in your research, please additionally cite

R. Gieseke, S. N. Willner and M. Mengel, (2018). Pymagicc: A Python wrapper for the simple climate model MAGICC. Journal of Open Source Software, 3(22), 516, https://doi.org/10.21105/joss.00516

For proper reproducibility please reference the version of Pymagicc used. In Python it can be printed with

import pymagicc
print(pymagicc.__version__)

Pymagicc releases are archived at Zenodo and the version used should also be cited. See https://doi.org/10.5281/zenodo.1111815.

pymagicc's People

Contributors

gabriel-abrahao avatar lewisjared avatar matthiasmengel avatar phackstock avatar rgieseke avatar swillner avatar znicholls avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pymagicc's Issues

Expose more MAGICC input files

Is your feature request related to a problem? Please describe.

I'd like to read and probably change (e.g. for sensitivity analyis) more of the input files that ship with MAGICC. Currently only parameters and scenarios are exposed.

Describe the solution you'd like

Not sure about nicest API yet.

Describe alternatives you've considered

It's possible to read the files manually.

Add carbon cycle variables to output

Is your feature request related to a problem? Please describe.

Enabling carbon cycle output in pymagicc does not return carbon cycle output variables. It seems that the output file name (CARBONCYCLE.OUT) may need to be standardised by placing DAT at the start of the filename in the same format as other optional output.

Describe the solution you'd like

It would be good to return the carbon cycle output variables (currently in CARBONCYCLE.OUT) when the flag is on (i.e. out_carboncycle=1).

Describe alternatives you've considered

I could write my own python script to handle this. But, the infrastructure is already in place with pymagicc so it "should be" trivial to add this feature in.

Additional context

Note that the carbon cycle output file differs from others such that the columns are different variables (e.g. CURRENT_GPP), not different regions (e.g. GLOBAL, NH-OCEAN). However, I don't think this should change how it is handled.

Add instructions for Wine.app installed `wine`

Ensure that pymagicc also runs with wine installed from installer package available from
https://dl.winehq.org/wine-builds/macosx/download.html

With wine on your path in should in principle work … The 32-bit version is required, but it seems this is the default for all downloads.

which wine
/Applications/Wine Stable.app/Contents/Resources/wine/bin/wine

Following up from openjournals/joss-reviews#516 (comment)
@scollis Can you give some more error details? For example

wine --version

Any error message that is shown from Python?

import pymagicc
pymagicc.run(pymagicc.rcp3pd)

Missing Output files

Like CARBONCYCLE.OUT output discussed in #84, #85 there are a few other outputs not named DATSOMETHING.OUT -- should we read them all or add them as we go when someone actually needs them?

Thought @znicholls @NortonAlex ?

Trailing \0 in parameters.out

Some strings in PARAMETERS.OUT are fixed width strings padded with \0 (which aren't rendered by GitHub...). For example the last item in the array below is 7 * an empty string.

 FGAS_FILES_CONC="HISTCMIP6_CF4_CONC_ENDOFYEAR.IN                                            ","HISTCMIP6_C2F6_CONC_ENDOFYEAR.IN                                           ","HISTCMIP6_C3F8_CONC_ENDOFYEAR.IN                                           ","HISTCMIP6_C4F10_CONC_ENDOFYEAR.IN                                          ","HISTCMIP6_C5F12_CONC_ENDOFYEAR.IN                                          ",
 "HISTCMIP6_C6F14_CONC_ENDOFYEAR.IN                                          ","HISTCMIP6_C7F16_CONC_ENDOFYEAR.IN                                          ","HISTCMIP6_C8F18_CONC_ENDOFYEAR.IN                                          ","HISTCMIP6_cC4F8_CONC_ENDOFYEAR.IN                                          ","HISTCMIP6_HFC23_CONC_ENDOFYEAR.IN                                          ","HISTCMIP6_HFC32_CONC_ENDOFYEAR.IN                                          ",
 "HISTCMIP6_HFC4310mee_CONC_ENDOFYEAR.IN                                     ","HISTCMIP6_HFC125_CONC_ENDOFYEAR.IN                                         ","HISTCMIP6_HFC134a_CONC_ENDOFYEAR.IN                                        ","HISTCMIP6mod_HFC143a_CONC_ENDOFYEAR.IN                                     ","HISTCMIP6_HFC152a_CONC_ENDOFYEAR.IN                                        ","HISTCMIP6_HFC227ea_CONC_ENDOFYEAR.IN                                       ",
 "HISTCMIP6_HFC236fa_CONC_ENDOFYEAR.IN                                       ","HISTCMIP6mod_HFC245fa_CONC_ENDOFYEAR.IN                                    ","HISTCMIP6mod_HFC365mfc_CONC_ENDOFYEAR.IN                                   ","HISTCMIP6_NF3_CONC_ENDOFYEAR.IN                                            ","HISTCMIP6_SF6_CONC_ENDOFYEAR.IN                                            ","HISTCMIP6_SO2F2_CONC_ENDOFYEAR.IN                                          ",
  7*"                                                                            ",

Which is serialised to:

"fgas_files_conc": [
            "HISTCMIP6_CF4_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_C2F6_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_C3F8_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_C4F10_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_C5F12_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_C6F14_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_C7F16_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_C8F18_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_cC4F8_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_HFC23_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_HFC32_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_HFC4310mee_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_HFC125_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_HFC134a_CONC_ENDOFYEAR.IN",
            "HISTCMIP6mod_HFC143a_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_HFC152a_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_HFC227ea_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_HFC236fa_CONC_ENDOFYEAR.IN",
            "HISTCMIP6mod_HFC245fa_CONC_ENDOFYEAR.IN",
            "HISTCMIP6mod_HFC365mfc_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_NF3_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_SF6_CONC_ENDOFYEAR.IN",
            "HISTCMIP6_SO2F2_CONC_ENDOFYEAR.IN",
            "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
            "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
            "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
            "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
            "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
            "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
            "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
        ]

I am proposing right stripping the null chars in parameters.out to make them empty strings instead of fixed width null chars.

Automatic testing of notebooks with CI

Is your feature request related to a problem? Please describe.

At the moment our notebooks aren't run automatically under CI, I don't think.

Describe the solution you'd like

Use scons or something else to run all our notebooks under CI to make sure our examples actually work.

Become units agnostic

Is your feature request related to a problem? Please describe.

One of the most annoying things is having to do all your unit conversions yourself before you run your model. I'd love for us to find a way that users can just throw dataframes with whatever (well-defined) units and magicc will run properly.

Describe the solution you'd like

I think my ideal solution would have two components:

  • find a library which can actually do unit conversions for us or build out https://github.com/znicholls/pysciunits to achieve this
  • have a units module which defines all of the expected MAGICC units and can convert any input dataframes to the required units before writing files and running MAGICC

Describe alternatives you've considered

Continue to do this stuff by hand because it's tricky. That is an option but I think we could produce something really nice if we tried.

Dealing with two versions of MAGICC

As a user of pymagicc I would like to be able to run both MAGICC6 and MAGICC7 scenarios in the same interpreter
As a magicc developer I want to be able to dynamically specify a local version of MAGICC which is used with pymagicc

This would currently be possible using our MAGICC_EXECUTABLE environment variable, but it isn't a nice API. You would have to update os.environ between simulations.

Perhaps we actually need two configuration parameters MAGICC6_EXECUTABLE and MAGICC7_EXECUTABLE. The low level API then chooses the appropriate executable and as a byproduct already knows which version of MAGICC it is using! This suggests that we might need a more complex config handling methods sooner than I anticipated

Improve API docs

  • review doc strings
  • check config module
  • maybe split in separate files and link to them from Readme

no run_tests.py available as indicated in Readme.

I tried to "use a different MAGICC version" following

"""
#!/bin/bash
export MAGICC_EXECUTABLE_7=tmp/magicc/run/magicc.exe
python run_tests.py
python generate_plots.py
"""
from Readme.md

However, a run_tests.py file does not exist, nor does a generate_plots.py.

Could you update this section?

Refactor tests

@rgieseke @lewisjared

How would you feel about starting to refactor the tests to make them scale slightly more easily? In my MATLAB code to write files for CMIP6 stuff I ended up having something like:

tests
|--- writer_tests
|--- reader_tests
|--- plotter_tests
etc.

I felt this made it much easier to keep track of what was going on rather than just having one big test_everything file.

Provide access to AOGCM tunings besides the default C4MIP_BERN

Describe the bug
live.magicc.org provides tunings for the other bench marked atmospheric-ocean general circulation models mentioned in the main MAGICC reference paper (http://www.atmos-chem-phys.net/11/1417/2011/acp-11-1417-2011.html). These tunings appear to be defined in specific files named MAGTUNE_tuningname.CFG and MAGTUNE_FULLTUNE_tuningname.cfg. They don't seem to exist in the kit delivered by pymagicc or even the windows zip from magicc.org. These files are also used to build up the data used in bulk runs.

The names of the tunings are as follows, and taken from bulk runs from live.magicc.org:

file_tuningmodel_1 = list[
    'c4mip_bern',
    'c4mip_ccsm1',
    'c4mip_climber',
    'c4mip_frcgc',
    'c4mip_hadley',
    'c4mip_ipsal',
    'c4mip_llnl',
    'c4mip_mpi',
    'c4mip_umd2',
    'c4mip_uvic']

file_tuningmodel_2_1 = list[
    'fulltune_cccma_cgcm3_1_t47', 
    'fulltune_cnrm_cm3', 
    'fulltune_csiro_mk3_0',
    'fulltune_gfdl_cm2_0', 
    'fulltune_gfdl_cm2_1',
    'fulltune_giss_model_e_h',
    'fulltune_giss_model_e_r', 
    'fulltune_iap_fgoals1_0_g',
    'fulltune_inmcm3_0',
    'fulltune_ipsl_cm4',
    'fulltune_miroc3_2_hires',
    'fulltune_miroc3_2_medres',
    'fulltune_miub_echo_g',
    'fulltune_mpi_echam5',
    'fulltune_mri_cgcm2_3_2a',
    'fulltune_ncar_ccsm3_0',
    'fulltune_ncar_pcm1',
    'fulltune_ukmo_hadcm3',
    'fulltune_ukmo_hadgem1']

runparam

System (please complete the following information):
macOS 10.13.6, pymagicc 1.3.1
(and additionally the native Windows: windows zip file: Magicc6.801 Beta)

Additional context
tunings

Some settings are in the appendix tables for the paper mentioned above, but don't appear to match the default settings.

Release v1.2

  • update changelog
  • re-run generated API docs (maybe move to separate file)
  • ensure examples are working

_magiccbinary variable is broken

This points to the same value and _magiccbinary is not used anymore I believe.

# default parameters and cannot be changed after module load
_magiccpath, _magiccbinary = MAGICC6().original_dir, MAGICC6().original_dir

Can we get rid of this?

_magiccpath could likely be replaced when reading scenario data.

Assigning different MAGICC versions

In the docs we have,

Using a different MAGICC version

Robert and Jared I'm pretty sure this isn't true anymore?

The _magiccpath and _magiccbinary can be changed to point to different MAGICC
development versions.

Is this still true?

MAGICC7 compatibility

General question, @lewisjared @znicholls -- might be helpful for decisions on how to make design decisions.

Do we need to maintain MAGICC6 compatibility once MAGICC7 is publicly available?

Will there be a need to run both from one installation? Do you need this to run regressions tests/comparisons for MAGICC7?

One could always install version 1.x and run MAGICC6, might mean separate Virtualenvs if you actually do need both but I'd think that would be fairly rare.

Plot in README.md

The current image in README.md shows the old scenario names:
image

When running the code mentioned to create this plot, I get the following:
image

Obviously there is some style differences. Perhaps @rgieseke can regenerate the plot with the correct size and style.

Config Module

We need a way of accessing configuration from anywhere within pymagicc in a consistent manner. This state should not be stored in the root __init__.py as that would cause lots of circular dependencies (Module A imports Module B which imports Module A). There is a need to define default config, long-lived somewhat static config and to be able to override these values dynamically. Having a separate module with a simple interface will hide the complexity of this.

Here is how I propose that we lookup configuration variables in other parts of the library. The config object looks like a dict, but it knows how to find the appropriate parameter value.

from  pymagicc.config import config
executable = config['MAGICC6_EXECUTABLE']

Any feedback on the API?

Opps

@rgieseke I was in the zone and accidentally pushed to master...

Single commit with no breaking changes
8706760

Standardising variable names

Is your feature request related to a problem? Please describe.

At the moment the native variable names are all over the place and not always that descriptive e.g. CO2B_EMIS doesn't really tell you anything.

Describe the solution you'd like

Defining a set of mapping dictionaries to change MAGICC's raw output to nicer names. I'm not sure how exactly this would look. I don't know if it's better to have trees so you access things with e.g. output['emissions']['co2']['AFOLULUC'] or delimited variable names e.g. output['emissions-co2-AFOLULUC']. In both cases having standardisation across variables should make life easier.

Describe alternatives you've considered

Sticking with the native variable names. I don't like this because they're not that descriptive.

Drop Python 2 support

A general question, with Numpy dropping Python 2 support in 2019 (https://docs.scipy.org/doc/numpy/neps/dropping-python2.7-proposal.html) and Pandas (https://pandas.pydata.org/pandas-docs/stable/install.html)

The final release before December 31, 2018 will be the last release to support Python 2. The released package will continue to be available on PyPI and through conda.

Starting January 1, 2019, all releases will be Python 3 only.

Since there are very few users of Pymagicc right now I propose we drop Python 2 support now.
Any project starting now will likely be in Python3. People who need Python2 for some reason can still install an old Pymagicc version.

This would allow us to use e.g. (the very nice) pathlib or unittest.mock without having to import/install compatibility backports.

Thoughts, @lewisjared @znicholls @matthiasmengel?

Run existing Scenario files

Currently Pymagicc always re-writes the Scenario from a DataFrame -- it should have the ability to directly to run a SCEN file if given as parameter. (Similar to what Pandas does with inputs.)

MAGICC7 shipping

When this happens, can we ship with a minimal run directory. The tests being so slow because we're copying so many files for MAGICC7 is driving me insane.

Discussion: API to add read and write for HIST EMIS and CONC input files

Is your feature request related to a problem? Please describe.

At the moment you can't read or write HIST_XXX_EMIS.IN files or HIST_XXX_CONC.IN files from pymagicc. This is annoying because you can't see the input to MAGICC easily nor modify it to see what effect it has.

I think this issue should largely focus on a discussion of how we keep the metadata with the data, given that pandas does not yet have a good way of doing this.

Describe the solution you'd like

An api which offers the ability to write and read these files. After conversations with @lewisjared and @rgieseke it's still not clear what the best way to deal with the metadata is. In my ideal world the api would be something like

MData = MAGICCInput()
MData.read('HIST_EMIS_CO2I.IN')
  • reads in the data
  • fails if it can't find the file
  • fails if the file isn't in the expected format
  • warning if the file has some metadata inconsistencies i.e. was written misleadingly/in an unexpected way (this one can be for stage 2 but I'm writing it down so I don't forget)
MData.print_data()

This would print a dataframe in wide format by default, same as human readable data files, however the data would actually be stored in long format internally.

MData.df

This would be the long format data frame

I'm not sure what the best way to access data is given that our data is effectively multi-dimensional, perhaps

http://pandas.pydata.org/pandas-docs/version/0.22/generated/pandas.DataFrame.xs.html

MData.df.xs(('CO2I_EMIS', 'WORLD'), level=['Variable', 'Region'])

https://pandas.pydata.org/pandas-docs/stable/advanced.html#basic-indexing-on-axis-with-multiindex
From this link: 'Note that df.loc['bar', 'two'] would also work in this example, but this shorthand notation can lead to ambiguity in general'

MData.df.loc[('CO2I_EMIS', 'WORLD'),] # 

Custom options

MData['Gas']['Region'] 
MData.df['Gas']['Region']
MData.filter(gas='Gas', region='Region') # copying pyam

I'm not sure what this should return, maybe a Pandas DataSeries with year as the index? We could also include units in the index..

MData['Gas']

I think something like this should return a dataframe with all of the available regions (subject to same access question as above)

Re metadata, perhaps our class should have a metadata attribute which is a dictionary

MAGICCInput.metadata
  • prints dictionary of metadata
  • warns you where metadata is missing
MAGICCInput.metadata['source']
  • returns a string with the source

Where the units should sit is tricky. I'm leaning towards having them as an index, despite the extra cost of duplicating them however many times, to ensure they stay with the data. Another option may be to have something like

MData.metadata['units']['CO2I_EMIS']

i.e. the units are simply stored in the class's metadata attribute and can be looked up when necessary. We could then have a decorator on data setting which requires a 'units' argument and raises a warning/error if they're not provided.

Describe alternatives you've considered

Making changes by hand or only using files which are made elsewhere. These seem like bad options.

Region names

At the moment MAGICC input files use a mix of region names. In the API I figure we may as well make them all come out the same for the regions which are equivalent. Hence how do we feel about always using:

'GLOBAL' rather than 'WORLD'
'RASIA', 'RREF' etc. rather than 'R5ASIA', 'R6ASIA', 'R5REF', 'R6REF' (or should we actually keep these separate because their definitions are, in a very minor way, different and instead just allow people to access both 'R5ASIA' and 'R6ASIA' with 'RASIA' depending on what is in the dataframe)

Package with statement

This API can be improved by being able to use this package in a with statement which automatically initialises and cleans the package i.e.

with Package() as p:
    for i in range(10):
        p.run()

Explicitly defined root dirs should not be automatically initialised
Need to add an argument for initialising Package in the initialiser

Release automation

The release process needs to be written down and, ideally, automated more:

  • check notebooks
  • check plot
  • cut release for Zenodo tagging

Check examples

Will running MAGICC with only three input gases work as shown in the examples?

Slow copying of large MAGICC distributions

Is your feature request related to a problem? Please describe.

Some development copies of MAGICC end up being very large due to large untracked files such as temporary build output. Using pymagicc with these directories can take several minutes to copy the large number of files which limits pymagicc utility.

Describe the solution you'd like

Only copy the top-level files in the run directory and create the out directory rather than copying it. The configuration in the run directory is currently flat and anything not in the root run directory are build artifacts.

Describe alternatives you've considered

  • Whitelisting files to copy - this seems like it could cause headaches to maintain a list of white lists which may differ between MAGICC6 and MAGICC7
  • Blacklisting - That causes too much coupling between the development processes of magicc and pymagicc

Accessing MAGICC input data

Is your feature request related to a problem? Please describe.

At the moment I don't know what a sensible way to provide access to dataframe data is. Using a multi-indexed frame can make things pretty nasty.

Describe the solution you'd like

It'd be good to have some sort of wrapper included in #49 which makes it easy for users to access data through our api. I'm not sure what the best way for that to look is. It seems like now is a good time to define how we want it to look/behave, given it's not built yet.

I'm guessing we probably want to be able to do things like:

minput.df.loc['ASIA', 'CO2_CONCS']

and have sensible values returned (in this case a dataframe, the values of which could then be extracted with .values). At the moment you have to specify every 'dimension' (e.g. minput.df.loc['ASIA', 'CO2_CONCS', 'SET', 'ppm', slice(2015,2100)] to get data and I don't think that should be the case).

Describe alternatives you've considered

We could just force all users to learn how to use multi-indexed dataframes but that seems contrary to the spirit of the project.

Automatic linting

  • automatic linting (PEP8 and docstyle) would be useful
  • either automatic on PRs or at least documented and callable from the Makefile

Config writing

The config file writing currently done in pymagicc.run should become part of the API

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.