secoora / gutils Goto Github PK

@kerfoot can you point me to the updated find_yo_extrema method you modified?

from https://github.com/kerfoot/slocum

just find this is wrong place ask to question, so i close this

Continued from PR cf-convention/vocabularies#186

• Use pytest test_slocum.py to exercise echometrics/pseudogram code to produce desired netCDF results
• Always assign echometrics variables with extras dimension even if pseudogram is missing
• Wire acoustic sensor configuration through deployment.json using extra_kwargs
• apply extra_kwargs to ascii to nc conversion
• tests/test_slocum.py::TestEcoMetricsThree::test_pseudogram produces three netCDF files that need to be consistent
• Manual running of ascii/netCDF produces one file; pytest produces three files [differences in json config files]

Deferred:

• apply extra_kwargs to dbd to ascii conversion (no current hooks to grab extra_kwargs from deployment.json)

I found two bugs in the GUTILS assign_profile() function.
For line 77 in assign_profiles.
inflections = np.where(np.diff(delta_depth) != 0)[0]

and I guess the sampling algorithm, between line 57-66 you get sampling data depending on the time interval(tsint), which cases the offset issues in the graph below

I think sampling data can find extreme points, but can’t get a very precisely extreme point.

For practical purposes, this sensor list is very rarely included in each file due to it's large size relative to the actual sensor data (as an example, the file binary file I'm working with is 78680 bytes and the ascii header takes up 78481 bytes). The operator can choose to either 1) Include it in every file, 2) include it in no files if there is already a copy of the sensor list on shore, 3) Include it in just the first file of the mission.

For our real-time operations, we generate this file (called a cache or cac file) on shore and then set up the glider to NEVER include it in the data files.

Once the files are on shore, the dbd2asc executable takes one or more binary data files and, using the sensor list either contained in the file or the location specified via dbd2asc -c /PATH/TO/SENSORLILST.cac, converts the binary data to ascii. As far as I can tell, GUTILS provides no option to specify an external cac file. My guess is that the vast majority of users who would potentially use this repo will also NOT transmit this sensor list in every binary data file, but can't say for sure.

There are several steps involved in migration to parquet for intermediate processing of slocum glider data.

• ensure dbdreader reproduces similar results to slocum binaries (smerckel/dbdreader#18)
• replacement of convertDbds.sh with dbdreader/parquet
• desired storage pattern for parquet (just using tables for now)

Is there a particular storage pattern or design desired for the parquet data structures?
REF: https://arrow.apache.org/docs/python/parquet.html#parquet-file-writing-options

• Enforce version 2.4? 2.6?
• Ensure the structure is queryable by time for speedy subsetting?
• If enforcing 2.6, timestamp units become less an issue
• Partitioning? Glider ID, Deployment ID, Process method (rt vs. delayed), QC'd (Level 0, 1, ...)
  • local file system or plug into directly to duckdb (https://duckdb.org/docs/guides/python/filesystems.html) or allow both

Have to nail down a potential dbdreader issue first.

From Kerfoot:

I found a bug in the GUTILS get_decimal_degrees() when trying to convert
small (near the equator) GPS positions from NMEA coordinates to decimal
degrees:

https://github.com/SECOORA/GUTILS/blob/master/gutils/gbdr/methods.py#L281

I rewrote the function to do a straight mathematical conversion instead
of converting to a string, parsing, etc. Here's the code:

def get_decimal_degrees(lat_lon):
     """Converts glider gps coordinate ddmm.mmm to decimal degrees dd.ddd

     Arguments:
     lat_lon - A floating point latitude or longitude in the format ddmm.mmm
         where dd's are degrees and mm.mmm is decimal minutes.

     Returns decimal degrees float
     """
     # Absolute value of the coordinate
     try:
         pos_lat_lon = abs(lat_lon)
     except (TypeError, ValueError) as e:
         return

     # Calculate NMEA degrees as an integer
     nmea_degrees = int(pos_lat_lon/100)*100

     # Subtract the NMEA degrees from the absolute value of lat_lon and divide by 60
     # to get the minutes in decimal format
     gps_decimal_minutes = (pos_lat_lon - nmea_degrees)/60.0

     # Divide NMEA degrees by 100 and add the decimal minutes
     decimal_degrees = (nmea_degrees/100) + gps_decimal_minutes

     if lat_lon < 0:
         return -decimal_degrees

     return decimal_degrees

I saw in the push.yml workflow (which runs on push and PRs) that the image is built and uploaded to axiom/gutils:latest as part of testing. This might cause some confusion since latest in other projects usually refers to the latest stable version, rather than testing.

I'd recommend building locally for testing, or using the git SHA as tag.

The publish workflow (tagging with the version name in publish.yml) is correct though, so should be used instead for production.

Not a big issue, just an FYI for the community.

Suggestions

• Wait on #22 (to avoid merge conflicts)
• Add pre-commit.ci to repo Pre-commit is already in the push.yml GHA workflow
  • Pros: Pre commit hooks run in CI for pull requests (independent of dev install). No associated cost for public repos.
• Modify .pre-commit-config.yaml to add pyupgrade
  • Pros: Automatically remove outdated Python syntax, and convert code to take advantage of new features (e.g., f-strings, remove encoding comments)
  • Cons: Requires explicitly dropping support for Python versions as per arg in pyupgrade workflow

Addition to .pre-commit-config.yaml

- repo: https://github.com/asottile/pyupgrade
  rev: v3.15.0
  hooks:
    - id: pyupgrade
      args: [--py36-plus]

Hey there. I was very happy to find a FOSS package to interact with data output from Teledyne Webb Slocum Gliders.

Following the setup instructions in the readme, I found that gutils isn't on Conda forge. I assume this method of installation doesn't work (yet?)

It would be great to know:

• how feature complete this package is
• whether this package is in active development/what the current focusses are

If contributions are welcome, I'm happy to open some PRs tackling different areas (assuming my supervisor gives me the go-ahead).