cdat / cdat_info Goto Github PK

Global infos and functions for CDAT

Shell 1.67% Python 92.15% Jupyter Notebook 0.93% Makefile 5.24%

cdat_info's Introduction

cdat

⚠️ WARNING: Maintenance-only mode until around the end of 2023.
The CDAT library is now in maintenance-only mode, with plans for deprecation and cease of support around the end of calendar year 2023. Until this time, the dependencies for specific CDAT packages (`cdms2`, `cdat_info`, `cdutil`, `cdtime`, `genutil`, `libcdms`) will be monitored to ensure they build and install in Conda environments. We currently support Python versions 3.7, 3.8, 3.9, and 3.10. Unfortunately, feature requests and bug fixes will no longer be addressed.
If you are interested in an alternative solution, please check out the xarray and xCDAT - Xarray Extended With Climate Data Analysis Tools projects.

⚠️ WARNING: Maintenance-only mode until around the end of 2023.

The CDAT library is now in maintenance-only mode, with plans for deprecation and cease of support around the end of calendar year 2023. Until this time, the dependencies for specific CDAT packages (cdms2, cdat_info, cdutil, cdtime, genutil, libcdms) will be monitored to ensure they build and install in Conda environments. We currently support Python versions 3.7, 3.8, 3.9, and 3.10. Unfortunately, feature requests and bug fixes will no longer be addressed.

If you are interested in an alternative solution, please check out the xarray and xCDAT - Xarray Extended With Climate Data Analysis Tools projects.

CDAT builds on the following key technologies:

Python and its ecosystem (e.g. NumPy, Matplotlib);
Jupyter Notebooks and iPython;
A toolset developed at LLNL for the analysis, visualization, and management of large-scale distributed climate data;
VTK, the Visualization Toolkit, which is open source software for manipulating and displaying scientific data.

These combined tools, along with others such as the R open-source statistical analysis and plotting software and custom packages (e.g. DV3D), form CDAT and provide a synergistic approach to climate modeling, allowing researchers to advance scientific visualization of large-scale climate data sets. The CDAT framework couples powerful software infrastructures through two primary means:

Tightly coupled integration of the CDAT Core with the VTK infrastructure to provide high-performance, parallel-streaming data analysis and visualization of massive climate-data sets (other tighly coupled tools include VCS, DV3D, and ESMF/ESMP);
Loosely coupled integration to provide the flexibility of using tools quickly in the infrastructure such as ViSUS or R for data analysis and visualization as well as to apply customized data analysis applications within an integrated environment.

Within both paradigms, CDAT will provide data-provenance capture and mechanisms to support data analysis.

CDAT is licensed under the [BSD-3][bds3] license.

We'd love to get contributions from you! Please take a look at the Contribution Documents to see how to get your changes merged in.

cdat_info's People

Contributors

Stargazers

Watchers

Forkers

xylar commanderpho acordonez

cdat_info's Issues

Problems with version reporting in CDAT82

The function cdat_info.get_version() is reporting the wrong CDAT version - it should be 8.2, but it's currently showing 8.1:

(base) bash-4.2$ conda activate cdat82MesaPy3
(cdat82MesaPy3) bash-4.2$ which python
~anaconda3/envs/cdat82MesaPy3/bin/python
(cdat82MesaPy3) bash-4.2$ python
Python 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 21:52:21) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cdat_info
>>> print(cdat_info.get_version())
8.1
>>>

@downiec @muryanto1 @jasonb5

need to turn off logging

-------------------- >> begin captured logging << --------------------
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: https://uvcdat-usage.llnl.gov:443 "POST /log/add/ HTTP/1.1" 503 412
urllib3.connectionpool: DEBUG: https://uvcdat-usage.llnl.gov:443 "POST /log/add/ HTTP/1.1" 503 412
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): uvcdat-usage.llnl.gov
--------------------- >> end captured logging << ---------------------

Need mecanism to track cdat packages installed and othe rprovenance

Thread failure "too many open files"

Exception in thread Thread-10423:
Traceback (most recent call last):
  File "/usr/local/anaconda2/envs/2.10/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/anaconda2/envs/2.10/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/anaconda2/envs/2.10/lib/python2.7/site-packages/cdat_info/cdat_info.py", line 277, in submitPing
    clean_cache()
  File "/usr/local/anaconda2/envs/2.10/lib/python2.7/site-packages/cdat_info/cdat_info.py", line 272, in clean_cache
    with bz2.BZ2File(cache_file,"w") as f:
IOError: [Errno 24] Too many open files: '/work/zelinka1/.uvcdat/.cdat_cache'

python3 bootstrap

When importing genutil with Python3 There is something rong with cache file and utf-8 or str in the cache file.

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/software/anaconda3/envs/cdms3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/software/anaconda3/envs/cdms3/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/software/anaconda3/envs/cdms3/lib/python3.6/site-packages/cdat_info/cdat_info.py", line 317, in submitPing
    cache_data(data)
  File "/software/anaconda3/envs/cdms3/lib/python3.6/site-packages/cdat_info/cdat_info.py", line 254, in cache_data
    f.write(repr(cache))
  File "/software/anaconda3/envs/cdms3/lib/python3.6/bz2.py", line 245, in write
    compressed = self._compressor.compress(data)
TypeError: a bytes-like object is required, not 'str'

cdat_info version misleading when pure tag (not passed it)

see: CDAT/cdat#2229

py2/py3 switch

it looks like the anonymous question freaks out when switching from a py2 version to a py3

some user report conda command not found in provenance function

see:
CDAT/jupyter-vcdat#44

exceptions in ESGF publisher

Frequently get the following after otherwise successful completion of esgpublish. It does not seem to affect operation of the publisher or the return value. Nonetheless, having "harmless" exception appear in the output on a regular basis will make it harder to spot any that really matter, so it is still a problem, so for the time being we are running the publisher with UVCDAT_ANONYMOUS_LOG=no.

Exception in thread Thread-2 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/threading.py", line 801, in __bootstrap_inner
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/threading.py", line 754, in run
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/site-packages/cdat_info/cdat_info.py", line 284, in submitPing
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/site-packages/cdat_info/cdat_info.py", line 275, in clean_cache
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/site-packages/cdat_info/cdat_info.py", line 234, in post_data
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/site-packages/requests/api.py", line 112, in post
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/site-packages/requests/api.py", line 58, in request
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/site-packages/requests/sessions.py", line 403, in __exit__
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/site-packages/requests/sessions.py", line 712, in close
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/site-packages/requests/adapters.py", line 324, in close
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/site-packages/urllib3/poolmanager.py", line 204, in clear
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/site-packages/urllib3/_collections.py", line 91, in clear
  File "/usr/local/conda/envs/esgf-pub-20180927/lib/python2.7/threading.py", line 168, in acquire
<type 'exceptions.TypeError'>: 'NoneType' object is not callable
Unhandled exception in thread started by 
sys.excepthook is missing
lost sys.stderr

The lost sys.stderr error seems to be when python is writing to a pipe which nothing is reading from, and can be reproduced e.g. with python -c "print 'hello'" | true

cdat / cdat_info Goto Github PK

cdat_info's Introduction

cdat

cdat_info's People

Contributors

Stargazers

Watchers

Forkers

cdat_info's Issues

Problems with version reporting in CDAT82

need to turn off logging

Need mecanism to track cdat packages installed and othe rprovenance

Thread failure "too many open files"

python3 bootstrap

cdat_info version misleading when pure tag (not passed it)

py2/py3 switch

some user report conda command not found in provenance function

exceptions in ESGF publisher

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent