On-site fixer script for cmorized data
Python code to fix the incorrect longitude eastward shift of half a grid cell by ece2cmor
, see ece2cmor issue 553.
The procedure to use the cmor-fixer
is described at the EC-Earth portal wiki page Correct cmorised data with the cmor-fixer. That wiki page referes for some description to this README. A pdf print of this page is uploaded in this repository and can be found here.
Issue 731 at the EC-Earth portal is dedicated to this subject.
- netCDF4
- numpy
The Miniconda python distribution should be installed. With miniconda all the packages can be installed within one go by the package manager conda
. This applies also to systems where one is not allowed to install complementary python packages to the default python distribution.
Download miniconda (e.g. take the latest miniconda version for python 3) by using wget
and install with bash
:
mkdir -p Downloads; cd Downloads/
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -u -p /$HOME/miniconda3 # for instance on your laptop
bash Miniconda3-latest-Linux-x86_64.sh -b -u -p /$PERM/miniconda3 # for instance on cca, because on $PERM the disk space is sufficient, which is on cca not the case for $HOME.
One could consider to add the following variable definition and alias in the .bashrc
file:
miniconda3path=${HOME}/miniconda3/ # for instance on your laptop
miniconda3path=${PERM}/miniconda3/ # for instance on cca
alias activateminiconda3='source ${miniconda3path}/etc/profile.d/conda.sh'
For example we create the directoy ${HOME}/cmorize/ for the cmor-fixer:
git clone https://github.com/EC-Earth/cmor-fixer.git
In the cmor-fixer git checkout directory, type
activateminiconda3 # The alias as defined above
conda update -y -n base -c defaults conda # for updating conda itself
cd cmor-fixer
conda env create -f environment.yml # for linux & mac os
# Activate the cmor-fixer conda environment:
activateminiconda3 # The alias as defined above
conda activate cmorfixer
# The help of cmor-fixer list its argument options:
./cmor-fixer.py -h
# A dry-run example which, due to the --forceid option, will give a new tracking id to any file
# encountered including files with correct lon data:
./cmor-fixer.py --verbose --forceid --olist --npp 1 --dry CMIP6/
conda deactivate
# Activate the cmor-fixer conda environment:
activateminiconda3 # The alias as defined above
conda activate cmorfixer
# An example in which cmor fixer is applying the lon & lon_bnds fix to the data:
./cmor-fixer.py --verbose --forceid --olist --npp 1 CMIP6/
conda deactivate
The cmor-fixer-save-mode-wrapper.sh
script will only apply changes if at least one file with one error is detected in the entire dataset. In case an error is detected and the script will continue to apply changes, additional checks will be applied to check for interuptions during running the script.
# Activate the cmor-fixer conda environment:
activateminiconda3 # The alias as defined above
conda activate cmorfixer
./cmor-fixer-save-mode-wrapper.sh 1 CMIP6/
conda deactivate
The submit-cmor-fixer.sh
script is an sbatch
template for a submit script which needs adjustments of all the paths and possibly adjustent of the cmor-fixer arguments depending on the preferences of the user. The script can be called (it activates the cmor environment itself at the compute node) by:
sbatch submit-cmor-fixer.sh
# Activate the cmor-fixer conda environment:
activateminiconda3 # The alias as defined above
conda activate cmorfixer
# An example in which cmor fixer is applying the lon & lon_bnds fix to the data:
./cmor-fixer.py --verbose --keepid --olist --npp 1 CMIP6/
conda deactivate
First run the version.sh
script, for its help use:
./versions.sh -h
In order to see whether there is more than one version in your data:
./versions.sh -l CMIP6/
If so (this happens when during running the script the date changed because you crossed midnight), you need to set one version (you can choose what you want, but it should be a different and later date than the one which was previously published). One can set the date for instance to January 20 2020 by:
./versions.sh -v v20200120 -m CMIP6/
./files-per-year.sh 1850 2014 CMIP6
conda deactivate # Leaving the cmorfixer environment due to trouble with installing nctime
activateminiconda3 # Only needed when not done before (doesn't harm to do it twice)
cd nctime/
./run-nctime-for-cmip6.sh CMIP6 # This will install nctime directly in the miniconda3 environment en run it
With
ncdump -v lon some-cmorised-file.nc
ncdump -v lon_bnds some-cmorised-file.nc
one can easily see the lon
& lon_bnds
values.
Incorrect lon
values start at 0.3515625
like:
lon = 0.3515625, 1.0546875, 1.7578125, 2.4609375, 3.1640625, 3.8671875,
while correct lon
values start at 0
like:
lon = 0, 0.703125, 1.40625, 2.109375, 2.8125, 3.515625, 4.21875, 4.921875,
Incorrect lon_bnds
values start at 0, 0.703125
like:
lon_bnds =
0, 0.703125,
0.703125, 1.40625,
1.40625, 2.109375,
2.109375, 2.8125,
2.8125, 3.515625,
3.515625, 4.21875,
4.21875, 4.921875,
4.921875, 5.625,
5.625, 6.328125,
while correct lon_bnds
values start at -0.3515625, 0.3515625
like:
lon_bnds =
-0.3515625, 0.3515625,
0.3515625, 1.0546875,
1.0546875, 1.7578125,
1.7578125, 2.4609375,
2.4609375, 3.1640625,
3.1640625, 3.8671875,
3.8671875, 4.5703125,
4.5703125, 5.2734375,
5.2734375, 5.9765625,
5.9765625, 6.6796875,
Because the error and the fix are in the coordinate values itself, diffing the files well is slightly complicated. The best way to inspect the changes in a file after fixing is to make a ncdump
of the file before fixing and a ncdump
of the file after fixing and thereafter use meld
to compare them (it takes a while before meld manages to load). You will then see that the lon
and/or lon_bnds
values are shifted half a gird cell westwards (lower), that the tracking_id
is changed and that a message is added to the netcdf history attribute telling that @cmor-fixer@ has modified the file.