linkedearth / pyleoclim_util Goto Github PK

It would be nice to add a lowpass filtering capability that does not distort phase.

Splines are pretty awesome for that.
I'd start with:
http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.spline_filter1d.html#scipy.ndimage.spline_filter1d
or
http://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.UnivariateSpline.html#scipy.interpolate.UnivariateSpline

On no account should a running mean ever be used!!!
http://mres.uni-potsdam.de/index.php/2017/02/10/better-avoid-running-means/

Installation should be through pip but if want to use RBchron, needs a little tweaking.

R needs to be present. Add to documentation.

And integrate to Timeseries module

Integrate querying and downloading from the wiki

Mean spacing: mean of the median for a realization
start_time: max(min)
end_time: min(max)

Proposed change:

1. Wrapper module called workflow.py that will handle LiPD files directly and automate some of the decisions (as much as possible).

Workflows would either allow for a LiPD object (dictionary) or for a user dictionary containing appropriate information as specified by function (e.g., time, value for Series).

2. Upper module api.py which would contain the functions that are most likely being used by a user to return either results, plots or text. Object-driven.

Can be used to create personal workflows. See Jupyter Notebook as example. These methods will be re-used in workflow.py

Classes to consider:

• Series -> plot, hist, clean, correlation, causality, spectral, wavelet, wavelet coherence, generate surrogate of the series, filter, PCA, SSA, stats, interpolation (interp/bin), standardize, annualize (with lots of error checks),
• Output of spectral becomes a PSD object -> plot
• Output of wavelet becomes a scalogram object-> plot
• Output of coherence becomes Coherence object->plot
• Ens class -> takes a list of series objects, plot
• Chron obj -> generate ensembles
• LiPD object -> map, summaryplots, then send to other objects
• MultipleSeries -> list of series/ens objects; correlation with FDR, PCA, plot (stack plot)

3. Lower modules, which contains base methods. Need to concatenate as follows:

• examples.py - example data
• lipdutils.py
• map + plot + summaryplots -> visualization and deal with issue #66
• rbchron -> age model
• spectral, stats, timeseries -> analysis.py

Change the import to fortran code to be within the function so the package doesn't fail on import. Put a warning that the method is slow and we strongly suggest installing said compiler.

Feng, Julien wanted to integrate your notebook into Pyleoclim. He said using statsmodel. More details would be nice.

It's important for our plots to have a distinct look and feel.
To me that involves two things:

• using a Matplotlib style (either existing or custom) throughout, instead of doing so on a function by function basis.
• despining most (all?) plots.

Currently our style is a bit of a mish-mash, so we need to get coherent. What style do you prefer? I'm pretty happy with the new matplotlib default, with ggplot a close second.

Another consideration is customizability: it's essential to have sensible defaults, but it should be easy for users to override that if they have a preferred style. I think this can be done easily by explaining in the documentation that the package sets its own Matplotlib style, but explain how users can override that in one line of code.

Switch Lists to NumPy arrays.

Currently, Pyleoclim does not interpret the variable "distance_from_top" as "depth". Thus for example, in the summaryTs function, age model plot can not be generated.

Several functions, like corrsig and rednoise, look like they are not used and should therefore be canned to avoid confusion.

Some of the functions have while loop asking for the right term. raise error instead.

Hello! Just a heads up:

Users will get an error for your basemap requirement when following the install guide.

Can recreate this on a clean conda environment on OSX or Linux:

conda create --name spacemonkey
source activate spacemonkey
conda install python=3.6 pip
pip install 

The last pip command gives:

 Could not find a version that satisfies the requirement basemap>=1.0.7 (from pyleoclim) (from versions: )
No matching distribution found for basemap>=1.0.7 (from pyleoclim)

See here: https://www.statsmodels.org/stable/generated/statsmodels.tsa.stattools.grangercausalitytests.html

*ready for api.py

I remember we investigated this at some point. What's the status on this? If not possible, comment on why and close the issue

Propose to replace it with nitime.

Installation issues are real (particularly for Windows users, for whom Anaconda is hit or miss, and Basemap is still a major issue). One idea to circumvent them is to run our notebooks on
JetStream (https://jetstream-cloud.org). First, we need to check that it is feasible. Can someone give it a try?

Use this: https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.interp2d.html instead of the bumpy version

If ensembles are available, show the envelope on the plot

Add this feature for evenly-spaced timeseries

1. Provide notebook example
2. Integrate into Pyleoclim

Using clustering

Class in in timeseries.py.

PCA algorithm here -> https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html

--finished

Needs to show only the dates (circles), together with their uncertainties (error bars). No line connecting them unless there is a ChronData table or EnsembleTable in the lipd file.

Should we create an upper workflow with all the spectral methods in? In this case, we need to make sure that inputs/ouputs are dictionaries to allow for the variability in outputs for each functions

Add examples to selected functions.

Pass the code to insert ensembles into LiPD files in lipdutils

enabled SSA as a method for Series in ui.py
enable MSSA as a method for MultipleSeries in ui.py

Hello again!

If I work around Issue #21 by installing basemap from conda, I can open a python shell or ipython or what have you and run:

import pyleoclim as pyleo

At this point I'm super excited for the wave of paleo awesomeness that is about to hit me. But instead I get:

~/miniconda/envs/spacemonkey/lib/python3.6/site-packages/pyleoclim/__init__.py in <module>()
     29 from pyleoclim import SummaryPlots
     30 from pyleoclim import Plot
---> 31 from pyleoclim import Spectral
     32 from pyleoclim import Stats
     33 from pyleoclim import Timeseries

~/miniconda/envs/spacemonkey/lib/python3.6/site-packages/pyleoclim/Spectral.py in <module>()
     34 
     35 if sys.platform.startswith('darwin'):
---> 36     from . import f2py_wwz as f2py
     37 
     38 '''

ImportError: dlopen(/Users/notarealuser/miniconda/envs/spacemonkey/lib/python3.6/site-packages/pyleoclim/f2py_wwz.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libiomp5.dylib
  Referenced from: /Users/notarealuser/miniconda/envs/spacemonkey/lib/python3.6/site-packages/pyleoclim/f2py_wwz.cpython-36m-darwin.so
  Reason: image not found

And the library is not imported. Looks like it might be an issue with f2py needing an Intel lib.... I thought it could be something with MKL or GDAL...

Installing MKL from conda and then reloading the shell and importing pyleoclim gives me:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-2828cd9a80d8> in <module>()
----> 1 import pyleoclim as pyleo

~/miniconda/envs/spacemonkey/lib/python3.6/site-packages/pyleoclim/__init__.py in <module>()
     32 from pyleoclim import Stats
     33 from pyleoclim import Timeseries
---> 34 from pyleoclim import RBchron
     35 
     36 

~/miniconda/envs/spacemonkey/lib/python3.6/site-packages/pyleoclim/RBchron.py in <module>()
     13 import pandas as pd
     14 import datetime
---> 15 import rpy2.robjects as robjects
     16 from itertools import chain
     17 from rpy2.robjects.packages import importr

~/miniconda/envs/spacemonkey/lib/python3.6/site-packages/rpy2/robjects/__init__.py in <module>()
     14 import itertools
     15 from datetime import datetime
---> 16 import rpy2.rinterface as rinterface
     17 import rpy2.rlike.container as rlc
     18 

~/miniconda/envs/spacemonkey/lib/python3.6/site-packages/rpy2/rinterface/__init__.py in <module>()
     49 del(os)
     50 
---> 51 from rpy2.rinterface._rinterface import (baseenv,
     52                                          emptyenv,
     53                                          endr,

ImportError: dlopen(/Users/notarealuser/miniconda/envs/spacemonkey/lib/python3.6/site-packages/rpy2/rinterface/_rinterface.cpython-36m-darwin.so, 2): Library not loaded: /usr/local/opt/xz/lib/liblzma.5.dylib
  Referenced from: /Users/notarealuser/miniconda/envs/spacemonkey/lib/python3.6/site-packages/rpy2/rinterface/_rinterface.cpython-36m-darwin.so
  Reason: Incompatible library version: _rinterface.cpython-36m-darwin.so requires version 8.0.0 or later, but liblzma.5.dylib provides version 6.0.0

I wasn't able to get around this by updating with packages on the conda-forge repos, either. So, there is still a dependency that not being taken care of?

Is there a plan to implement unit tests or continuous integration checks (e.g. TravisCI)? This will definitely ease development as the project grows.

I may be able to help with this if wanted.

Should we also group them so a method is chosen.

We need to check the package for issues that may pop up with the numpy upgrade, specifically the fact that it doesn't accept floats for indices.

We need a function like this one http://www.clidyn.ethz.ch/imputation/standardize.m

Notice that it returns 3 things: the z-score, the original mean and original standard deviation.

Among other things, it can be used to scale a record to a match a given variance.

https://github.com/nickmckay/GeoChronR/blob/master/vignettes/PlotTimeseriesStack.Rmd

PCA available for MultipleSeries in ui.pu

@fzhu2e and @CommonClimate Feng proposed to do this for the functions that are workflows but don't use a LiPD file for the data.

I agree with him that it will make things cleaner for a user, who won't have to mess with modules if they don't want LiPD.

My only suggestion would be to not name this api.py

Since the update to v0.11, statsmodels.tsa.AR has been deprecated in favor of statsmodels.tsa.AutoReg and statsmodels.tsa.SARIMAX.

the tsmodel.py module, and the various functions herein (e.g. ar1_sim), need to be updated accordingly.

asked by @CommonClimate Good time to be doing this.

Warning: all notebooks not using the main api for Pyleoclim would have to be updated.