piyush11007 / esg-publisher Goto Github PK

#*****************************************************************
# Copyright (c) 2009  Larence Livermore National Security (LLNS)
# Produced at the Lawrence Livermore National Laboratory
# All rights reserved.
#*****************************************************************
#
#   Description: Basic description of the project(s) and layout of the
#   		project.
#
#*****************************************************************

README:

ESGF Publisher

https://esgf.github.io/esg-publisher 

Requirements:  most requirements are listed in requirements.txt.  Other requirements must be deployed via conda, include cdms2, cdtime and cmor (for CMIP6, other projects coming soon).  

Installation:  the most convenient method to install is via pip.  To capture all dependencies, consider installation of the predetermined conda environment: https://github.com/ESGF/esgf-devOps/blob/master/esgf-pub_env.yml

29 October 2019 version 3.7.3

* Bugfixes and version compatibility updates

22 March 2019 version 3.7.0

* Remove calls to esgfetchini and the esgprep requirement.  Install esgprep separately with pip.

13 March 2019 version 3.6.2

* Skip THREDDS reinit during esgsetup - solves installation issue for successful status code return
* PrePARE to run on each input file rather than first in dataset
* Fix some package versions, eg. requests, SQLAlchemy
* Version information from esgpublish

18 December 2018  version 3.5.6

* process files in sorted order
* bugfix for certs location config
* new config option to skip validation of standard names

26 June 2018 - version 3.5.2

* REST API is default for both publish and unpublish ; support to use the ESGF CA bundle
* Switching data specs versions during operations
* esgtest_publish supports download test and skip-unpublish modes of operation
* new processing modes:
   - --skip-aggregation  
   - --commit-every  allows for more frequent database commits to aid in reducing memory footprints

Note - logging controlled now by the [DEFAULT] section of esg.ini. If you have upgraded the publisher with a previous version of the file, you may notice that the old setting is WARNING, in which case, INFO level messages may not appear.  Thus, the publisher may run successfully without any output.  



16 August 2017 - version 3.2.6

* Support for CMIP6 
- integration of PrePARE
- PID creation via RabbitMQ
- cdf2cim included
- handler for non-netCDF and skip variable scan for multiple cases

29 August 2016 - version 3.1.0

* Changes to support CMIP6
  - Configuration option to extract global attributes
  - Check for CMOR version in datafiles
  - CF checker integration
  - CMIP6-CV check integration (requires CMOR install)
* Included documentation (requires sphinx, sphinx_rtd_theme)

03 August 2016 - version 3.0.1

* Setup.py inclusion of esgprep utility - see http://esgf-prepare.readthedocs.io/ for more information
* Use of project-specific esg.ini files read from the default location
* Support for multiple mapfiles within a single directory
* support for comma or space-delimited facets


Previous versions:

06 July 2015 - version 2.13.0

 * Support for multithreaded checksumming
 * Version pick up in esgscan and use in esgpublish
 * esgfind_excludes to find files in the thredds exclude property
 * Add optional facet values with esgadd_facetvalues for published datasets
 * ACME file handler to support publication of netCDF + other formats in single data set

14 January 2015 - version 2.12.2

 * Warning message for esgscan_directory failues
 * --nodbwrite flag to enable "dry run" for fileO validation without writing to postgreSQL database
 * Additional utility scripts: 
      meta_synchro.py - metadata syncronization checks for PGSQL, Thredds and SOLR
     gen_versions.py, add_checksums_to_map.sh  - used for replica publication (at LLNL/PCMDI)