secoora / gutils Goto Github PK
View Code? Open in Web Editor NEWThis project forked from usf-cot/glider_utils
π π€ Python utilities for reading, merging, and post processing Teledyne Webb Slocum Glider data
License: MIT License
This project forked from usf-cot/glider_utils
π π€ Python utilities for reading, merging, and post processing Teledyne Webb Slocum Glider data
License: MIT License
@kerfoot can you point me to the updated find_yo_extrema
method you modified?
just find this is wrong place ask to question, so i close this
Continued from PR cf-convention/vocabularies#186
test_slocum.py
to exercise echometrics/pseudogram code to produce desired netCDF resultsDeferred:
I found two bugs in the GUTILS assign_profile() function.
For line 77 in assign_profiles.
inflections = np.where(np.diff(delta_depth) != 0)[0]
Line 77 in 0879678
and I guess the sampling algorithm, between line 57-66 you get sampling data depending on the time interval(tsint), which cases the offset issues in the graph below
I think sampling data can find extreme points, but canβt get a very precisely extreme point.
For practical purposes, this sensor list is very rarely included in each file due to it's large size relative to the actual sensor data (as an example, the file binary file I'm working with is 78680 bytes and the ascii header takes up 78481 bytes). The operator can choose to either 1) Include it in every file, 2) include it in no files if there is already a copy of the sensor list on shore, 3) Include it in just the first file of the mission.
For our real-time operations, we generate this file (called a cache or cac file) on shore and then set up the glider to NEVER include it in the data files.
Once the files are on shore, the dbd2asc executable takes one or more binary data files and, using the sensor list either contained in the file or the location specified via dbd2asc -c /PATH/TO/SENSORLILST.cac, converts the binary data to ascii. As far as I can tell, GUTILS provides no option to specify an external cac file. My guess is that the vast majority of users who would potentially use this repo will also NOT transmit this sensor list in every binary data file, but can't say for sure.
There are several steps involved in migration to parquet for intermediate processing of slocum glider data.
dbdreader
reproduces similar results to slocum binaries (smerckel/dbdreader#18)convertDbds.sh
with dbdreader
/parquet
parquet
(just using tables for now)Is there a particular storage pattern or design desired for the parquet
data structures?
REF: https://arrow.apache.org/docs/python/parquet.html#parquet-file-writing-options
Have to nail down a potential dbdreader issue first.
From Kerfoot:
I found a bug in the GUTILS get_decimal_degrees() when trying to convert
small (near the equator) GPS positions from NMEA coordinates to decimal
degrees:
https://github.com/SECOORA/GUTILS/blob/master/gutils/gbdr/methods.py#L281
I rewrote the function to do a straight mathematical conversion instead
of converting to a string, parsing, etc. Here's the code:
def get_decimal_degrees(lat_lon):
"""Converts glider gps coordinate ddmm.mmm to decimal degrees dd.ddd
Arguments:
lat_lon - A floating point latitude or longitude in the format ddmm.mmm
where dd's are degrees and mm.mmm is decimal minutes.
Returns decimal degrees float
"""
# Absolute value of the coordinate
try:
pos_lat_lon = abs(lat_lon)
except (TypeError, ValueError) as e:
return
# Calculate NMEA degrees as an integer
nmea_degrees = int(pos_lat_lon/100)*100
# Subtract the NMEA degrees from the absolute value of lat_lon and divide by 60
# to get the minutes in decimal format
gps_decimal_minutes = (pos_lat_lon - nmea_degrees)/60.0
# Divide NMEA degrees by 100 and add the decimal minutes
decimal_degrees = (nmea_degrees/100) + gps_decimal_minutes
if lat_lon < 0:
return -decimal_degrees
return decimal_degrees
I saw in the push.yml
workflow (which runs on push and PRs) that the image is built and uploaded to axiom/gutils:latest
as part of testing. This might cause some confusion since latest
in other projects usually refers to the latest stable version, rather than testing.
GUTILS/.github/workflows/push.yml
Lines 116 to 123 in 2969e89
GUTILS/.github/workflows/push.yml
Lines 156 to 162 in 2969e89
I'd recommend building locally for testing, or using the git SHA as tag.
The publish workflow (tagging with the version name in publish.yml
) is correct though, so should be used instead for production.
Not a big issue, just an FYI for the community.
Suggestions
push.yml
GHA workflow
.pre-commit-config.yaml
to add pyupgrade
Addition to .pre-commit-config.yaml
- repo: https://github.com/asottile/pyupgrade
rev: v3.15.0
hooks:
- id: pyupgrade
args: [--py36-plus]
Hey there. I was very happy to find a FOSS package to interact with data output from Teledyne Webb Slocum Gliders.
Following the setup instructions in the readme, I found that gutils
isn't on Conda forge. I assume this method of installation doesn't work (yet?)
It would be great to know:
If contributions are welcome, I'm happy to open some PRs tackling different areas (assuming my supervisor gives me the go-ahead).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.