Afternoon,
currently if a user wants to call an URL like the following from erddapy;

http://erddap.sensors.ioos.us/erddap/tabledap/uk_gov_metoffice_62103.htmlTable?time,latitude,longitude,wind_speed,wind_from_direction&time=max(time)+0minutes

It will fail on line 50 - erddapy.py

parse_date_time = parse_time_string(date_time)[0]

Even though this can be done from within pandas, these types of server-side functions greatly reduce the amount of data transmitted.

As a starter, having a check on the value to see if it starts with max/min? I am not totally sure on the full list of functions...

If agreed, i am happy to pick this up :)

Hello erddapy developers,

Is it possible to get the attributes of a dataset in a list or dictionary or JSON?

In the examples, I only find information on how to get the attributes with the 'get_info_url' method and then display it on screen with an iframe. I would need the attributes information in a variable.

Thanks!

There are a few spots (e.g. at the top, in the examples, etc) where you could direct users to your Sphinx documentation for more in-depth usage information. I think this would help direct people who may not know to look for the documentation link in the top of the GitHub page.

It sucks that we here we find a dataset with the requested standard_name, but then when we try to get the variable associated with that standard_name, we fail:

The problem is that in this dataset, the standard_name is not lower case: It's Sea_Water_Pressure instead of sea_water_pressure:

so our test against sea_water_pressure fails.

I'm not sure how to best solve this in ERDDAPY. We could just do string.lower() but that would probably cause failures for attributes other than standard_name.

Or @BobSimons, perhaps we could fix this on the ERDDAP end?
Maybe ERDDAP could make all standard_name attributes lower case?

Notebook here:
https://gist.github.com/rsignell-usgs/c6647e8debdce4654a90f8321389bd18

Hi,
It seems that erddapy is available under 1.2 with conda but only with 1.1 with pip.
This trigger some inconsistencies in dependencies check under different CI systems (eg here which fails argopy feedstock)
Is this done on purpose ?
thanks
g

PR #17 changed from netcdftime to a .timestamp() method call but that can lead to wrong timestamps b/c 1970-1-1 will be 0 only when UTC is specified. Naive datetime objects will lead to wrong timestamp.

datetime(1970, 1, 1, 0, 0).timestamp()
10800.0

datetime(1970, 1, 1, 0, 0, tzinfo=pytz.utc).timestamp()
0.0

See https://github.com/IrishMarineInstitute/awesome-erddap/pull/5/files

In 0.7.0 or 0.7.1 I'm getting a TypeError: a bytes-like object is required, not '_io.BytesIO' when calling .to_xarray() in some cases from _tempnc()

I'm trying to figure out a minimal example, but so far I have a gulfofmaine/buoy_barn test that throws.

I initially thought that it was because I'm using VCRpy to mock out traffic to various servers, but I'm having it occur outside of testing also.

Here's a Sentry issue with the exception.

Some users are looking for tools to help them assemble ERDDAP urls for use in their own workflows, while others would prefer to work at a higher, more opinionated level. I believe we can more cleanly separate functionality to help support the spectrum of erddapy users better.

Originally errdapy was meant to be a url builder only. We added the main class later and stop in between trying to support many different usage patterns via the single primary class ERDDAP.

Issues to address

• Users in interactive workflows are required to transform ERDDAP objects as they wish to connect to new datasets, for example when moving from searching a server to visiting the datasets.
• Adding constraints to a ERDDAP object are stateful in place changes, where as most interactive users are used to Numpy/Pandas/xarray style workflows where you can return or chain together changes.
• Switching out IO is currently non-trivial due to URL generation and data transformations being tightly coupled to IO.

Proposed Solution

I proposed that we separate erddapy into more functional layers, roughly following the SQLAlchemy core/ORM model.

Core Layer

The core layer would contain two primary components of functionality: url generation & data transformation. This layer makes no choices or assumptions about IO allowing it to be reused easily.

• URL generation - Functions to generate valid URLs from bare components, such as a dataset name, format, and dictionary of constraints to tabledap/M01_met_all.csv?time%2Cair_temperature%26air_temperature_qc%3D0%26time>%3D"2020-12-09T15%3A25%3A00.000Z"
• Data transformation - Functions to convert a raw response (.csv, .nc, ...) into Pandas DataFrames, xarray Datasets...

Object (or opinionated) Layer

The object (or opinionated) layer would present higher level objects for searching servers and accessing datasets with a Pandas or xarray like returning or chainable API compared to the transformational API of the current ERDDAP class.
This layer uses much of the core functionality and presents it in easy to use ways with an opinion as to the access method.

Additionally if possible these objects should be serializable, so they can be pickled and passed to other processes/machines (Dask/Dagster/Prefect).

• class ERDDAPConnection
  • While most ERDDAP servers allow connections via a bare url, some servers may require authentication to access data.
  • .get(url_part: str) -> bytes or str
    • Method actually request data.
    • Uses requests by default similar to most of the current erddapy data fetching functionality.
    • Can be overridden to use httpx, and potentially aiohttp or other async functionality, which could hopefully make anything else async compatible. (investigate await_me_maybe)
  • .open(url_part: str) -> fp
    • Yields a file-like object for access (probably use fsspec.open under the hood) for file types/tools that don't enjoy getting passed a string.
  • @property(server) -> ERDDAPConnection
    • Return a new ERDDAPConnection if trying to set a new server, or change other attributes rather than changing it in place.

For all of the remaining classes, either an ERDDAPConnection or a bare ERDDAP server url that will be transformed into an ERDDAPConnection can be passed in.

• class ERDDAPServer

  • .__init__(connection: str | ERDDAPConnection)
  • .full_text_search(query: str) -> dict[str, ERDDAPDataset]
    • Use the native ERDDAP full text search capabilities
    • Returns a dictionary of search results with dataset ids as keys and ERDDAPDataset values.
  • .search(query: str) -> dict[str, ERDDAPDataset]
    • Points to .full_text_search
  • advanced_search(**kwargs) -> dict[str, ERDDAPDataset]
    • Uses ERDDAPs advanced search capabilities (may return pre-filtered datasets)

• class ERDDAPDataset

  Base class for more focused table or grid datasets.

  • @property(connection)
    • Underlying ERDDAPConnection
  • .get(file_type: str) -> bytes or str
    • Requests the data using the .connection.get() method.
  • .open(file_type: str) -> fp
    • Yields a file-like object for access.
  • .get_meta()
    • Pulls the dataset info and caches it on the _meta attribute.
  • ._meta
    • Set by .get_meta()
    • Passed when a setter returns a subclass.
    • .attrs -> pd.DataFrame- Dataframe of dataset attributes.
    • .variables -> dict - Dictionary of variables as keys, and maximum extent of constraints as values.
  • @property(meta)
    • Returns the ._meta values, and will call .get_meta() if they are not already cached.
  • @property(variables)
    • List current variables the dataset requested from the dataset.
    • Setting variables returns a new ERDDAPDataset subclass.
    • If _meta is cached and an invalid variable is set, throw a KeyError instead of returning.
  • @property(constraints)
    • Returns the current constraints on the dataset.
    • Setting contraints returns a new ERDDAPDataset subclass.
    • If _meta is cached and an invalid constraint is set, throw a KeyError instead of returning.
  • .url_segment(file_type: str) -> str
    • Everything but the base section of the url (http://neracoos.org/erddap/), so tabledap/A01_met.csv....
  • .url(file_type: str) -> str
    • Returns a URL constructed using the underlying ERDDAPConnection base class server info, the dataset ID, access method (tabledap/griddap), file type, variables, and constraints.
    • This allows ERDDAPDataset subclasses to be used as more opinionated URL constructors while still not tying the users to an specific IO method.
    • Not guaranteed to capture all the specifics of formatting a request, such as if a server requires specific auth or headers.
  • .to_dataset() - Open the dataset as an xarray dataset by downloading a subset NetCDF.
  • .opendap_dataset() - Open the full dataset in xarray via OpenDAP.

• class TableDataset(ERDDAPDataset)

  • .to_dataframe() - Open the dataset as a Pandas DataFrame.

• class GridDataset(ERDDAPDataset)

In Practice

So how do these work in practice?
Let's look at a few different scenarios.

Interactive Search

Lets say that a user wants to find and query all datasets on a server that contain sea_water_temperature data?

First they initialize their server object.
This can be done by passing in the server URL, the short name of the server, or an ERDDAPConnection object if authentication or IO methods need to be overridden.

[1] from erddapy import ERDDAPServer

[2] server = ERDDAPServer("neracoos")

Then they can use the native ERDDAP full text search to find datasets.

[3] water_temp_datasets = server.search("sea_water_temperature")
    water_temp_datasets

[3] {"nefsc_emolt_erddap": <TableDataset ...>, "UCONN_ARTG_WQ_BTM": <TableDataset...>, ...}

From there the user can access datasets a variety of ways depending on their needs.

[4] for dataset_id, dataset in water_temp_datasets:
        df = dataset.to_dataframe()
        # Whatever esoteric things fisheries people do with their dataframes

We need to quote strings when used in ERDDAP expressions.

This code:

    kwargs = {
        'minTime<=': stop_time.value,
        'maxTime>=': start_time.value,
        'cdm_data_type=': cdm_data_type
    }

    variables = ['datasetID', 'minLongitude', 'minLatitude']
    url = e.get_download_url(dataset_id='allDatasets', variables=variables, response='csv', **kwargs)

produces this URL
https://erddap-uncabled.oceanobservatories.org/uncabled/erddap/tabledap/allDatasets.csv?datasetID,minLongitude,minLatitude&minTime%3C=2018-01-23T12:52:10Z&maxTime%3E=2018-01-16T12:52:10Z&cdm_data_type=TimeSeries

which gives the response:

Query error: For constraints of String variables, the right-hand-side value must be surrounded by double quotes. Bad constraint: cdm_data_type=TimeSeries

We need to add it to the extras and write a small import/catch/report if-clause.

... so erddapy can run without pandas.

I had a hard time figuring out that the documentation page erddapy was actually the API page. In the left "contents" menu, I assumed it was a section header rather than a clickable page. I think this would be clearer if it were called "erddapy API" here: https://github.com/ioos/erddapy/blob/master/docs/source/erddapy.rst

what do you think?

erddapy doesn't currently support ERDDAP "Server-side functions". The only such functions I'm aware of are distinct() and various ordering functions. The latter don't really seem necessary in the context of erddapy and pandas, b/c sorting can be done client-side via Pandas. But distinct() does make a difference, as it can make the response much smaller, depending on the data.

It's actually very easy to use distinct with erddapy as-is. Once I have the request ready to go, I pass this url to pandas read_csv:

e.get_download_url() + '&distinct()'

But figuring out how to do this takes a fair bit of work. It'd save users some effort if this was built into erddapy. I could be persuaded to try to add it myself and submit a PR, but not this month ...

@mwengren looks like I don't have full rights to the repo b/c I cannot edit the URL in:

Can you update that to https://ioos.github.io/erddapy/ please?

The quick / longer introduction are both quite useful demonstrations of erddapy! It would be helpful to include a link to the notebooks at the top of the example, e.g. something like

"This example is written in a Jupyter Notebook - click here to download the notebook so you can run it locally"

(another option would be to use something like sphinx gallery to generate the examples, but I think that's a longer fix that isn't strictly necessary

See pypi/warehouse#4079 (comment)

@ocefpaf , the purpose of parse_dates is to convert to seconds since 1970, right?

Why not just: timestamp() instead of this code:
https://github.com/pyoceans/erddapy/blob/master/erddapy/utilities.py#L24-L34

Bob brought this to my attention via e-mail. While those who use erddapy as a URL constructor may need that, those who use it as a data-access library are fine waiting for the failure until the data is requested.

In my ERDDAP time series explorer, I used the existing erddapy method for advanced search , but also needed to use an allDatasets query to return the lon/lat values of all stations. I did this with an ugly erddap URL string formatting:
https://github.com/reproducible-notebooks/ERDDAP_timeseries_explorer/blob/master/ERDDAP_timeseries_explorer.py#L197-L200

Seems like the allDatasets query might warrant a method, since it's a bit different than a regular datasets query

I know griddap isn't officially supported but it previously had worked for the purposes I needed... seems the implementation of .ncCF response may be throwing it off (url building it forces .ncCF as the response type now but there isn't a .ncCF type for griddap)

if I change the url to response nc - i get what I expect as returned data

gist of a simple example showing the failure in trying to load - https://gist.github.com/shaunwbell/b11e4ae63638010866b6ae9d64261004

and my adhoc solution (which is purpose built, not holistic or i would push a pull request)

    
  def to_xarray(self, **kw):
        """Load the data request into a xarray.Dataset.

        Accepts any `xr.open_dataset` keyword arguments.
        """
        import xarray as xr

        url = self.get_download_url(response="nc")
        nc = _nc_dataset(url, auth=self.auth, **self.requests_kwargs)
        return xr.open_dataset(xr.backends.NetCDF4DataStore(nc), **kw)

am i missing a part of the API to set this explicitly or is it hardcoded into the script?

thanks

erddapy = 0.7.0
python 3.8.5

What is erddapy?

erddapy takes advantage of ERDDAP's RESTful web services to create the ERDDAP URLs.
The users can create virtually any request like,
searching for datasets, acquiring metadata, downloading data, etc.

What is ERDDAP?

ERDDAP is a data server that provides a consistent way to download subsets of scientific datasets.

There are many scientific data server available,
like OPeNDAP, WCS, SOS, OBIS, etc.
They all have their advantages and disadvantages,
ERDDAP goal is to fill the gaps and unify most of the advantages in a single service.
The main advantages of ERDDAP are:

• offers an easy-to-use, consistent way to request data via the OPeNDAP;
• returns data in the common file format of your choice,
  .html table, ESRI .asc and .csv, Google Earth .kml, OPeNDAP binary, .mat, .nc, ODV .txt, .csv, .tsv, .json, and .xhtml,
  and even some image formats (.png and .pdf);
• standardizes the date-times where string times are always ISO 8601:2004(E) and
  numeric times are always "seconds since 1970-01-01T00:00:00Z";
• acts as a middleman by reformatting the request into the format required by the remote server,
  and reformats the data into the format that you requested.

See https://coastwatch.pfeg.noaa.gov/erddap/index.html for more information.

Ideas for GSoC

• idea 0: Support ERDDAP's griddap

erddapy supports only tabledap protocol but ERDDAP also provides a griddap protocol that servers gridded data like models and satellite images. One could extend erddapy's API to support griddap and expand the amount of data Python users can obtain with this library.

Issue: #32

• Idea 1: Unify constraints

The library has two types of constraints, "regular ones" that are parsed as expected by Python users, and the relative_constraints, which are passed directly to ERDDAP without any pre-processing. Ideally the API should be unified to avoid confusing the users.

Issue: #164

• Idea 2: High level data queries

The current API is a bit too verbose, for example:

e.dataset_id = "whoi_406-20160902T1700"

e.variables = [
    "depth",
    "latitude",
    "longitude",
    "salinity",
    "temperature",
    "time",
]

e.constraints = {
    "time>=": "2016-09-03T00:00:00Z",
    "time<=": "2016-09-04T00:00:00Z",
    "latitude>=": 38.0,
    "latitude<=": 41.0,
    "longitude>=": -72.0,
    "longitude<=": -69.0,
}

while that makes a good base for other libraries to be built on top of erddapy, like argopy and gliderpy, it makes it hard to use high level objects as constraints. For example, a shapefile are any GIS WKT-like object.

The idea would be to create a higher level API that could consume these object and output a similar query as the one described above.

Issue: #96

Hey there,

I'm hoping to use your nifty package but am running into an issue. You should note I don't have a climate science background.

When I run the following I receive an error, which, it seems like the generated URL doesn't conform to the griddap documentation but instead to tabledap.

from erddapy import ERDDAP

e = ERDDAP(
    server='http://oos.soest.hawaii.edu/erddap',
    protocol='griddap',
    response='csv',
    dataset_id='NCEP_Global_Best',
    constraints={
        'time>=': '2018-02-08',
    },
    variables=['time', 'longitude', 'latitude'],
)

# http://oos.soest.hawaii.edu/erddap/griddap/NCEP_Global_Best.csv?time,longitude,latitude&time%3E=1518048000.0
print(e.get_download_url())

Outputs:

Traceback (most recent call last):
  File "/tmp/a.py", line 14, in <module>
    print(e.get_download_url())
  File "/home/danny/.virtualenvs/cwwed-env/lib/python3.6/site-packages/erddapy/erddapy.py", line 143, in get_download_url
    constraints=constraints,
  File "/home/danny/.virtualenvs/cwwed-env/lib/python3.6/site-packages/erddapy/url_builder.py", line 183, in download_url
    return _check_url_response(url)
  File "/home/danny/.virtualenvs/cwwed-env/lib/python3.6/site-packages/erddapy/utilities.py", line 101, in _check_url_response
    r.raise_for_status()
  File "/home/danny/.virtualenvs/cwwed-env/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://oos.soest.hawaii.edu/erddap/griddap/NCEP_Global_Best.csv?time,longitude,latitude&time%3E=1518048000.0

If I use the Data Access Form and only add the time constraint, it generates a url like:

http://oos.soest.hawaii.edu/erddap/griddap/NCEP_Global_Best.csv?time[(2018-02-01):1:(2018-02-13T12:00:00Z)] which works.

If you browse to the malformed URL it warns about "graphical" params.

I must be doing something wrong but I can't identify what it is. Any help would be appreciated.

So I use the CF DSG datatypes (point, profile, timeseries, trajectory) as my dataset id identifiers with my erddap datasets (which i then point to and set as my cf_role) - these variables are usually populated with string names.

For example: i concatenate multiple instruments on a fixed mooring within a single erddap dataset, each instrument gets a timeseries_id that is fixed for the instrument... say mymooring_myinstrument_morerelevantdetails. And if I wanted to retrieve all data from just a single instrument I can constrain on that value...

See example of description in screen shot:

However and herein is my problem...
erddapy looks for any constraint to start with 'time' and then assumes the content is a datetime object, but there could be cases where this is not truly the case

the issue is the following code for reference

    385             for k, v in _constraints.items():
    386                 if k.startswith("time"):
    387                     _constraints.update({k: parse_dates(v)})
    388             _constraints = _quote_string_constraints(_constraints)
    389             _constraints_url = _format_constraints_url(_constraints)

clearly the simple answer is for me to rename my variable id to not start with time... but this may just obscure an issue that another user may have should the word "time" reference data that is not easily translated into a datetime object.

seems a little something has changed and is throwing an error - I've not looked into the erddapy code but here is what I did (both on the swfsc erddap server and my own):

from erddapy import ERDDAP
import pandas as pd

e = ERDDAP(server="https://coastwatch.pfeg.noaa.gov/erddap")

print(url)

url = e.get_search_url(search_for="whoi", response="csv")

df = pd.read_csv(url)
print(
    f'We have {len(set(df["tabledap"].dropna()))} '
    f'tabledap, {len(set(df["griddap"].dropna()))} '
    f'griddap, and {len(set(df["wms"].dropna()))} wms endpoints.'
)

Throws a http 500 error and following the generated link gives the following explicit error:

Error {
    code=500;
    message="Internal Server Error: ERROR in parseISODateTime: for first character of dateTime='(ANY)' isn't a digit!";
}

We could do an async search on a pre-determined list or maybe wrap an interface for [1].

@callumrollo is that something you would like to tackle?

[1] https://coastwatch.pfeg.noaa.gov/erddap/download/SearchMultipleERDDAPs.html

When trying to build a search using the time constraint with erddap's now functionality:

from datetime import date

from erddapy import ERDDAP

server = "http://osmc.noaa.gov/erddap"
e = ERDDAP(server=server, protocol="tabledap")

e.dataset_id = "ioos_obs_counts"
e.variables = ["time", "locationID", "region", "sponsor", "met", "wave"]
e.constraints = {
    "time>=": "now-18months",
}
print(e.get_download_url())

I receive the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
pandas\_libs\tslibs\parsing.pyx in pandas._libs.tslibs.parsing.parse_datetime_string_with_reso()

pandas\_libs\tslibs\parsing.pyx in pandas._libs.tslibs.parsing.dateutil_parse()

ValueError: Unknown datetime string format, unable to parse: now-18months

During handling of the above exception, another exception occurred:

DateParseError                            Traceback (most recent call last)
<ipython-input-16-a4bef5750bff> in <module>
     11     "time>=": "now-18months",
     12 }
---> 13 print(e.get_download_url())

~\programs\Anaconda3\envs\IOOS\lib\site-packages\erddapy\erddapy.py in get_download_url(self, dataset_id, protocol, variables, response, constraints, **kwargs)
    330             for k, v in _constraints.items():
    331                 if k.startswith("time"):
--> 332                     _constraints.update({k: parse_dates(v)})
    333             _constraints = quote_string_constraints(_constraints)
    334             _constraints = "".join([f"&{k}{v}" for k, v in _constraints.items()])

~\programs\Anaconda3\envs\IOOS\lib\site-packages\erddapy\utilities.py in parse_dates(date_time)
    100         # pandas returns a tuple with datetime, dateutil, and string representation.
    101         # we want only the datetime obj.
--> 102         parse_date_time = parse_time_string(date_time)[0]
    103     else:
    104         parse_date_time = date_time

pandas\_libs\tslibs\parsing.pyx in pandas._libs.tslibs.parsing.parse_time_string()

pandas\_libs\tslibs\parsing.pyx in pandas._libs.tslibs.parsing.parse_datetime_string_with_reso()

DateParseError: Unknown datetime string format, unable to parse: now-18months

It looks like pandas is trying to parse the dates a user supplies in the constraints field. Even before the url gets built. Is there a way to bypass that? Or was this intentional?

I'd like to use erddapy with an ERDDAP endpoint that is protected with basic authentication (username/password): https://cgoms.coas.oregonstate.edu/erddap

If we were just using requests, we could use this, right?
http://docs.python-requests.org/en/master/user/authentication/

But is that right approach here?

Or should I just give up on expecting we could access protected endpoints with erddapy?

Given a valid ERDDAP url and:

e = ERDDAP(
             server=url,
             protocol='tabledap',
             response='csv'
             )

kw = {'search_for': 'all',
 'min_lon': -125.32011299999999,
 'max_lon': -122.320202,
 'min_lat': 49.042069,
 'max_lat': 49.96947800000001,
 'min_time': '2019-06-19T00:00:00Z',
 'max_time': '2019-12-31T00:00:00Z',
 'standard_name': 'sea_water_practical_salinity'}

df = pd.read_csv("{}".format(e.get_search_url(**kw), headers=headers))

I would expect the search URL to specify protocol=tabledap, but this is what we get instead:

{url}/advanced.csv?page=1&itemsPerPage=1000&protocol=(ANY)&cdm_data_type=(ANY)&institution=(ANY)&ioos_category=(ANY)&keywords=(ANY)&long_name=(ANY)&standard_name=sea_water_practical_salinity&variableName=(ANY)&minLon=-125.32011299999999&maxLon=-122.320202&minLat=49.042069&maxLat=49.96947800000001&minTime=1560902400.0&maxTime=1577750400.0&searchFor=all

I may be reading the docs wrong, but if this isn't a bug, I think it should be made clearer in the docs how to generate the protocol=tabledap filter in a search request. I see it's mentioned as a parameter in there, but I would expect the higher-level parameter value set on e would carry over to the get_search_url() call.

Expected behaviour:

I can make an advanced dataset search without supplying time bounds

Actual behaviour

If time bounds are not provided to get_search_url, erddapy generates the URI string &minTime=(ANY)&maxTime=(ANY) which returns an error from the server.

Reproduceable example

from erddapy import ERDDAP
import pandas as pd

e = ERDDAP(server="https://gliders.ioos.us/erddap")

kw = {
    "standard_name": "sea_water_temperature",
    "min_lon": -72.0,
    "max_lon": -69.0,
    "min_lat": 38.0,
    "max_lat": 41.0,
    "cdm_data_type": "trajectoryprofile"
}

search_url = e.get_search_url(response="html", **kw)
pd.read_csv(search_url)

Suggested fix

If time bounds are empty, instead of generating &minTime=(ANY)&maxTime=(ANY) erddapy should generate &minTime=&maxTime=

Versions

erddapy == 1.0.0
python == 3.6.10

While trying to connect to a ERDDAP server that had basic authentication set up I found that _check_url_response() was not passing any previously supplied requests_kwargs which are needed to pass long the authentication information.

This is already being done for calls to urlopen()

I just built a conda environment today with erddapy and pandas, among other things. No version pinning at all. Python 3.8.5, pandas 1.1.0, erddapy 0.5.3. Just importing erddapy produces the pandas error below.

Same issue when I built the conda env while pinning Python to 3.7.

Any suggestions for pinning a specific version of Pandas, erddapy or both to get past this issue? Thanks!

~/miniconda/envs/nvsgliderapp/lib/python3.8/site-packages/erddapy/__init__.py in <module>
----> 1 from erddapy.erddapy import ERDDAP, servers
      2 
      3 __all__ = ["ERDDAP", "servers"]
      4 
      5 from ._version import get_versions

~/miniconda/envs/nvsgliderapp/lib/python3.8/site-packages/erddapy/erddapy.py in <module>
     13 import pandas as pd
     14 
---> 15 from erddapy.utilities import (
     16     _check_url_response,
     17     _tempnc,

~/miniconda/envs/nvsgliderapp/lib/python3.8/site-packages/erddapy/utilities.py in <module>
     11 import pytz
     12 import requests
---> 13 from pandas.core.tools.datetimes import parse_time_string
     14 
     15 _server = namedtuple("server", ["description", "url"])

ImportError: cannot import name 'parse_time_string' from 'pandas.core.tools.datetimes' (/home/mayorga/miniconda/envs/nvsgliderapp/lib/python3.8/site-packages/pandas/core/tools/datetimes.py)

Note that not all libnetcdf are compiled with the in-memory support, so we will need the file fallback.

Currently I'm unable to install errdapy for my Apple Silicon based Mac. Please add support for these systems.

Before a13ee24 ERDDAP.requests_kwargs is a dict that can be set for any keyword arguments to be passed to requests.get() when retrieving data. It is still documented as an attribute.

Now attempting to set ERDDAP.requests_kwargs throws an AttributeError.

For example this allows setting a timeout argument that causes requests to throw a requests.Timeout if it is taking too long to retrieve information (without using something like signals to do the same).

I'm a newbie. Installed erddapy as per "pip install erddapy" instructions.
Get a full page of exceptions starting with:
AttributeError Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 from erddapy import ERDDAP
4 e = ERDDAP(
5 server="UAF", # NOAA UAF (Unified Access Framework)
6 protocol="tabledap",
7 response="csv",
8 )

File ~\Anaconda3\envs\tide\lib\site-packages\erddapy_init_.py:3, in
1 """Easier access to scientific data."""
----> 3 from erddapy.erddapy import ERDDAP
4 from erddapy.servers import servers
6 all = ["ERDDAP", "servers"]
.
.
.

Ending in AttributeError: module 'brotli' has no attribute 'error' along with a full page of exceptions.

Are there release requirements? I've always updated Anaconda etc.

It seems that newer versions of iris (installed using pip3 install scitools-iris) no longer supports the method extract_strict() for the CubeList object.

The latest documentation for iris CubeList can be found here: https://scitools-iris.readthedocs.io/en/stable/generated/api/iris/cube.html#iris.cube.CubeList.extract

Documentation for an older version with extract_strict: https://scitools.org.uk/iris/docs/v2.4.0/iris/iris/cube.html?highlight=cubelist#iris.cube.CubeList.extract_strict

This leads to the following error when using pytest:

―――――――――――――――――――――――――――――――――――――――――――――――― test_to_iris_tabledap ―――――――――――――――――――――――――――――――――――――――――――――――――

taodata = <erddapy.erddapy.ERDDAP object at 0x7fc80e26f400>

    @pytest.mark.web
    @pytest.mark.vcr()
    def test_to_iris_tabledap(taodata):
        cubes = taodata.to_iris()

        assert isinstance(cubes, iris.cube.CubeList)
>       assert isinstance(cubes.extract_strict("depth"), iris.cube.Cube)
E       AttributeError: 'CubeList' object has no attribute 'extract_strict'

tests/test_to_objects.py:80: AttributeError

Will there be any issue with just using extract() method?

• add the kwargs options to the __init__ constructor;
• validate the URL
• add .to_pandas() and .to_xarray() methods
• pass requests options along so we can use private server

Right now we can pass the dataset_id, constraints, and variables to the constructor or later as properties, that can be confusing b/c most of the time the users does not know that in advance to pass it to the constructor and some search or data crunching is to figure out those.

@rsignell-usgs and @kwilcox suggested to remove those from the constructor and enforce them to be passed as properties only.

Loading erddapy when being offline raise an error (see below) because it can init the servers list.

A direct use of erddapy when being offline may not be relevant, but in the case where erddapy is part of a larger package that does not entirely rely on online activities, should we expect loading erddapy to fail silently ?

Where should we document the griddap capability? The "longer introduction" notebook? A notebook of its own?

A common task is how to fetch data masked by an arbitrary region of interest (RoI) polygon.
Ideally the OPeNDAP standard & ERDDAP might allow for submission of a WKT, but masking could also be handled on the client side.

@ocefpaf : what are the first steps towards implementing RoI masking in erddapy?

Additional thoughts:

1. Should efforts to implement RoI queries be focused on the server or is this client workaround worthwhile?
2. Is this feature creep? Perhaps masking is a secondary task which should be kept separate from the core functionality of the API.

Hi @ocefpaf
I tried to create an API for url_search in griddap. Can you please have a look and suggest the changes. please refer to #176

Thanks

Also, its implementation is odd b/c it requires two sets of constraints instead of one. We should consolidate that and improve the user experience here.

@ocefpaf, I tried running ERDDAPY intro notebook for the OOI endpoint, and I think
there is a bug in get_var_by_attr.

In this notebook
in cell [16] the names that come back include density and
practical_salinity, which do not exist in the dataset, which you can confirm here:

https://erddap-uncabled.oceanobservatories.org/uncabled/erddap/info/CP05MOAS-GL336-02-FLORTM000-flort_m_glider_instrument-telemetered-deployment0005-tabledap/index.html

We need to check what would be a sensible default. Maybe get all of the data?

Hi, when importing erddapy, I get this:

from erddapy import ERDDAP
...
...
ImportError: No module named tslibs.parsing

Is this a pandas problem? Here's the complete traceback:

ImportErrorTraceback (most recent call last)
<ipython-input-3-20c6c8f35979> in <module>()
----> 1 from erddapy import ERDDAP

/home/bblanton/.local/lib/python2.7/site-packages/erddapy/__init__.py in <module>()
      1 from __future__ import (absolute_import, division, print_function)
      2 
----> 3 from erddapy.erddapy import ERDDAP, servers
      4 
      5 __all__ = [

/home/bblanton/.local/lib/python2.7/site-packages/erddapy/erddapy.py in <module>()
      8 from __future__ import (absolute_import, division, print_function)
      9 
---> 10 from erddapy.url_builder import (
     11     download_url,
     12     info_url,

/home/bblanton/.local/lib/python2.7/site-packages/erddapy/url_builder.py in <module>()
     18     from urllib import quote_plus
     19 
---> 20 from erddapy.utilities import (_check_url_response, parse_dates, quote_string_constraints)
     21 
     22 

/home/bblanton/.local/lib/python2.7/site-packages/erddapy/utilities.py in <module>()
     10 from collections import namedtuple
     11 
---> 12 from pandas._libs.tslibs.parsing import parse_time_string
     13 
     14 import pytz

ImportError: No module named tslibs.parsing

While doing pip3 install -r requirements-dev.txt I get error in cartopy installation.

To overcome the error while installing the cartopy I have included the cython library in the requirements file.
and have to add several commands before installing the requirements.txt to resolve the error while installing dependencies

For Linux Users

1-> sudo apt-get install libproj-dev proj-data proj-bin
2-> sudo apt-get install libgeos-dev
3-> pip3 install -r requirments.txt

I have attached the updated requirements file

For details refer to pr #173