cci-tools / cate Goto Github PK
View Code? Open in Web Editor NEWESA CCI Toolbox (Cate)
License: MIT License
ESA CCI Toolbox (Cate)
License: MIT License
It should be possible to calculate Pearson correlation on pixel by pixel basis on time/lat/lon datasets
The CLI shall provide a command which displays the names of dataset variables. For example
ect ds info DS_ID -v
Output shall include the netCDF variable name and, if available, also the units, CF standard name and description (i.e. long name).
This is essential because otherwise there is no chance for users to pass variable references which may be required by setting up a workflow JSON or some other CLI commands such as ect run
.
Currently, users using the CLI are forced to explicitly "synchronize" a data source so that it is available for computation, processing and analysis:
ect ds sync DS --time START[,END]
It would be way more user-friendly if the ECT CLI (and later the GUI) would implicitly download any required data from the CCI ODP. A cancellable progress monitor shall be displayed in this case. Ideally, the user would be asked before ECT automatically initiates the data download. When prompted, it would also be very helpful if ECT could given an indication how long the download will take.
When listing available data sources e.g.
ect ds list
it would be very helpful for users if also the temporal coverage of the available data sources is displayed.
See also #14.
ECT shall make direct use of the ESA CCI ODP web services and the provided datasets shall serve as ECT's primary data source. The current ECT implementation is based on the ESA CCI ODP FTP server where we create a local index from scanning all files in the FTP tree (slow!).
ECT shall use the ESA CCI ODP web services, namely the ESGF Search RESTful API, to update meta-information about available (CCI) datasets. That is, the CLI shall print index info from ESGF index when users type ect ds list
and ect ds info DS_ID
(DS_ID = dataset_id
of the index). File download required by the ect ds sync DS_ID
command shall be from a suitable url
provided by the index for a given DS_ID.
Unfortunately, the ESGF index does not provide any time coverage information about its datasets. Using FTP, we extracted time info from filenames.
Selected ECT operations should be able to correctly recognize auxiliary variables associated with a measurement variable. For example
Users shall be able to perform the UC 9 workflow solely using the CLI. An exemplary scenario can be found here: Use case 9 using the CLI.
Although we Sphinx' sphinx_autodoc_annotation
option, the generated type information is not linked with the type's API docs. For example, if a type is xarray.Dataset
the type annotation shall be linked like so: xarray.Dataset.
This is very important as we use xarray.Dataset as our common data model.
tseries_mean should also calculate std.
We should use gridtools for the coregistration operation.
Progress monitors may prompt users before progress observation begins whether they want to continue. For example, some process may detect that large amounts of data would need to be downloaded before some task can be completed. In such a situation it would be nice if the ECT asks the user if it is ok to do so.
This new API feature is a consequence of issue #21.
Hint: We'll could enhance the ConsoleMonitor
implementationso that it conditionally halts in its start()
method and displays a command-line prompt do be answered with Do you really want ... bla ... [Y]es/No?
, if answered with No
, an InterruptedError
may be raised.
So far, ECT has no installer which would include a deeply freezed Conda Python environment and provides Desktop/Startmenu shortcuts for the CLI (and later include the frontend binaries and shortcuts).
See also wiki page Software Installation Approach.
Provide a useful state of the User Manual section in the ECT documentation on ReadTheDocs.
Pearson correlation now uses abs() in its calculation, always resulting in positive correlation. This is because the correlation at some point for some data:
esacci.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.NOAA-15.AVHRR_NOAA.1-0.r1 2007-01-01 2007-12-30
esacci.OZONE.mon.L3.NP.multi-sensor.multi-platform.MERGED.fv0002.r1 2007-01-01 2007-12-30
ect res set cc_tot select_var ds=cl07 var=cc_total
ect res set oz_tot select_var ds=oz07 var=O3_du_tot
results in sqrt(-x).
It has to be found out, how to calculate the correlation properly in this situation. E,g, if abs can be used.
The CLI already validates operation arguments when passed to the ect run OP ...
command. It must also validate operation arguments when calling the
ect res set OP ...
even though actual OP
execution is deferred until some data is written using ect res write FILE
. Otherwise users create an invalid workspace that is later on very hard to debug.
Currently our LICENSE file contains the GPL v3.
However GPL is restrictive regarding redistribution of source code, hard to understand (~10 pages of text), incompatible with a number of other, more liberal OSS licenses, and often creates problems when building commercial software.
The proposal is too move to a simple license that is compatible with the majority of other OSS licenses, e.g. the MIT license.
For comparison:
CLI must offer a command or option that lists all supported input/output formats. Otherwise users don't know what values to pass to the various CLI commands that offer a format name option.
select_variables takes a list, this is not ideal for CLI operation. It would be better to use a single regex string and always use regex for searching the dataset. Then, one could select two or more precisely named variables by simply passing the following string that uses the regex OR operator:
'variable1|variable2'
The CLI should offer some basic, interactive plotting so that users can inspect intermediate results when they work with workspace resources:
ect res set RES1 OP1 ...
ect res set RES2 OP1 ...
ect res plot RES1 RES2 --var X1,X2
Some firewall configurations may prevent user computers to directly access the CCI ODP services.
In such cases the ECT shall allow for configuration of a proxy server. User manual shall comprise a section how to do the proxy configuration.
$HOME/.cate/conf.py
, see also cate.conf
package.isinstance hiccups on validating types from the typing package when using the cli interface. See #45
All CCI data is provided remotely by the ESA ODP. The ECT stores data downloaded from ESA ODP in a cache directory in the user's local file system. Users need a CLI command to identify the locally available datasets so that they can perform operations that won't imply fetching new data from the ESA ODP.
For example, the command
ect ds info --local
would list all locally cached datasets with their temporal coverage.
Opening datasets via OPeNDAP would free ECT (software and users) from downloading individual data files from ESA's CCI ODP.
The real power of OPeNDAP lies in its ability to access data subsets.
When using the ODP data store, it can't be printed/shown using API (python)
Implement at least rudimentary:
repr goal is to be unambiguous
str goal is to be readable
Container’s str uses contained objects’ repr
We are currently using the ESGF service at CEDA of the ESA CCI ODP.
Its responses do not include any temporal dataset coverage as it is urgently required when we e.g. perform CLI commands such as
ect ds sync DATASET_ID START_DATE STOP_DATE
This information is also urgently required for the GUI, e.g. for displaying a dashboard that shows availability of data in time.
By default, we currently put data files synchronized by individual data stores into the directory ~/.ect/data_stores
.
There should be a user configuration setting to change this location.
The following CLI command sequence represents a typical ECT session:
ect ws init
ect res read X input.nc
ect res set Y func1 ds=X p1=53.2
ect res set Z func2 ds=Y p2='nearest'
ect res write Z output.nc
While the ect res read
and ect res set
commands just alter the workspace workflow, ect res write
will finally execute the workflow and write the results to disk.
It is highly desirable to also be able to inspect intermediate results such as X and Y in the example above, e.g. by plotting and printing them. However, this means that the workflow would be executed for each such inspection command, possibly taking a lot of data retrieval and processing time. To get around this limitation, intermediate resources values should some somehow be retained between CLI calls so that they don't need to be recomputed.
This feature is directly related to #25 because a cached state would be needed to validate against cached, intermediate resource values whose types would otherwise remain unknown.
Issue #27 is a part of this issue.
All ect res
commands could be executed by delegating them to a running ECT WebAPI service that could hold intermediate resource values in a two-level cache: The first cache maps a resource name to a resource value, a Python in-memory object. If it does not exist, a second cache could map a resource name to a file which has been previously written by a ect res write
command. The resource value would be read from the file and then put into the first cache. Cached values would be removed as soon as any of the providing workflow steps are modified or removed.
One problem remains: the ECT WebAPI service should start transparently once it is required by a ect
command. However, it should also stop if it is no longer needed. Otherwise a (Python) process would hang in the background and presumably allocating lots of RAM. How can we tell that the service is no longer needed? Here are some suggestions:
ect ws init
and a new ect ws open
would start the WebAPI service if its not already running. A new ect ws close
would close a workspace while ect ws closeall
(or ect ws close --all
) would close them all at once.ect ws exit
would close all workspaces directly stop the service.Note, the ECT WebAPI service is an essential, building block also of the ECT GUI. See also here.
It should be possible to carry out Pearson correlation on time by time basis for time/lat/lon datasets.
A Workspace shall be a collection of named operation executions and related data(set) files. A workspace preserves the order of operations a user has executed, where the first operations are usually loading of datasets from given data sources and the last ones are storing data to files using a given format or mime-type. Read more in Workspaces.
In addition to plotting is also very desireable to have an interactive command allowing for inspecting a datasets scheme and content (similar to IPython).
ect res read X data.nc
ect res print X
The runtime performance of subsequently executed workflows could be drastically increased if a value cache could be used so that some steps don't need to be executed again. Subsequent workflow execution will occur in the ECT WebAPI service, which is an implementation detail of issue #32.
Caching shall occur only if allowed by a certain workflow step. Actually, caching shall only occur for steps, whose operation can be said to be a pure function - the function output depends solely on the function input, with no side-effects, and no random values generated.
Therefore, an operation's meta-info header should include information of whether its output values are allowed to be cached or not.
After
ect ws init
we currently have no means to
ect ws clean
which would delete ./.ect-workspace/workflow.json
, while
ect ws delete
would delete the entire ./.ect-workspace
directory.
Currently ect res open
allows for opening datasets from a registered data store only. It should also allow for accessing a dataset comprising multiple local files. Alternatively, we could also introduce an additional command, say ect res mfopen
(multi-file open).
Another idea is to add a new data source globally to ECT, in order to make it permanently available:
ect ds add [-r, --recursive] DS_NAME FILE_OR_DIR...
The new data source DS_NAME can then be used like any other (remote) data source. FILE_OR_DIR...
is a list of file names and/or directory names which may include wildcards (on Windows).
This issue is a blocker as long as issue #40 is not solved.
Change the uc9 demo notebook such that it uses the new datastore and ozone dataset instead of ftp and aerosol.
Currently coregistration works well only if both datasets to coregister have lat/lon/time dimensions. This should not be the case, as it should be possible to bring two datasets on the same spatial lat/lon grid irrespective of how many other dimensions there are.
For example, having two datasets with such dimensions:
lat/lon/time
lat/lon/time/layer/pressure,
it should be possible to resample the second dataset to the lat/lon resolution of the first one such that it still preserves the other dimensions. E.g., instead of only 'group_by' time, but somehow other dimensions as well.
Users performing e.g. an analysis as defined by use case #9 would like to a visually appealing report that can be displayed in a browser or written as PDF. The report should include the ECVs used, the operations performed and shall contain any analysis results, such as plots and statistics tables. The need comes from users that are unsatisfied by the plot image output and who want to be supported in publishing their result.
Currently we just print an error message. For debugging and tracing errors reported by users we need a CLI option (e.g. --traceback
) to display full Python stack dumps.
Suddenly
$ ect ds list
No data sources found
See #38. Quick investigation showed that the index of the ESA CCI ODP service at CEDA seems empty, datasets can no longer be queried.
This definitely blocks the ECT release.
Make sure all public ECT operations are tagged, have type annotations and comprehensive usage and background information and clear input/output descriptions.
Verify by listing all operations by
ect op list
and then printing information like so
ect op info OP
Issues #25, #27, and finally #32 led to a software design, which requires a running ECT WebAPI service running as a second process for all workspace and resource commands submited by the ECT CLI.
Currently all commands are delegated from the ECT CLI to the ECT WebAPI service, which executes requested tasks on the main thread of the (Tornado) web service, hence they are executed synchronously. This means that long running tasks will cause the CLI client to timeout. Furthermore the service cannot respond to new requests, e.g. from another shell, or later, the GUI.
Therefore all CLI workspace and resource commands must be executed asynchronously by the WebAPI service.
The names of ECT operations shall be catchy and generally understandable. Ideally, we would only use verbs because every operation does something with its inputs and returns a the result of the operation , which is ideally a pure function.
Different ECT operations that have a parameter for a similar purpose should all use the same parameter name, same data type, and same description, and ideally use the same argument position.
Examples:
xarray.Dataset
it should always be the first parameter and should always be called ds
. Other parameters of type xarray.Dataset
should have names using a ds_
prefix.var
and its value may be either None
, a str
, or a sequence of str
. Other rules may apply, e.g. a str
may contain comma-separated names or use wildcards ect., that's fine, but this shall apply to all the operations that have the var
parameter.file
and its value may be either None
, a str
(= file path), or Python file object.Use wildcards in ect.ops.select.select_var to be able to select auxiliary vars.
(ect_env)ccitbx@ccitbx-VirtualBox:~/Development/ect-core$ ect res set cc_tot ect.ops.select.select_var ds=cl07 var=cc_total
Using FSWorkspaceManager
ect res: error: Unions cannot be used with isinstance().
Apparently ect res does not like my new select_var implementation with
var: Union[None, str, List[str]] = None
When ECT opens a dataset, e.g. from the CCI ODP, it shall make sure that the resulting (xarray.Dataset
) in-memory representation conforms to a unique schema that we'd can expect in all our data operations. This is also what we refer to as Common Data Model (CDM).
Harmonization shall address:
missing_value
, _FillValue
, NaN
When the CLI is performing operations within a workspace (ect ws init
, ect res set
, ...) it should be locked so that another CLI instance cannot accidentally modify files and resources currently in use.
EDIT:
Same applies to WebAPI, there the lock must be used in class Workspace
.
In the CLI, and when writing own Workflow JSON files, users must currently use fully qualified type names to identify an operation. A fully qualified type name comprises the (Python) package path and the actual function name. Instead of
$ ect op info ect.ops.coregistration.coregister
users should be able to simply type
$ ect op info coregister
For the standard ECT operations - the ones defined within the ect-core
repository - we should define aliases or allow for using the function name only (without the package path prefix).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.