hydroffice / hyo2_soundspeed Goto Github PK

@zxcvbnmike15 after #24, you should definitely update the author list with your name.

The database interaction becomes painfully slow with the increase of the number of profiles collected.

The current workaround is to create a project for each cruise.

Issue:

SIS5 listener thread crashes when trying to read the second #MRZ datagram from a split #MRZ datagram record.
Issue was first noticed on the R/V Revelle. Sound Speed Manager was setup to appropriately to listen via multicast. however the listener information in the bottom right of the screen was not updating:

I requested recently collected data from the ship and ran it through the PUReplay application provided by SIS5. It was observed that the #MRZ datagrams were split into two UDP datagram packets before being sent.

Sound Speed Manager tries to read the second #MRZ datagram of the series and it appears there is a byte misalignment.

Causes:

The #MRZ datagram is comprised in the following format:

The current #MRZ version is 2. This version is present in the R/V Revelle datafiles. The original kmall reader was written for version 0. Some changes have been made to the #MRZ datagram, however the issue does not correspond with the changes.

When a #MRZ datagram is read into the buffer, the header and partition structures are always correct across files tested (kmall v0 and v1). However, when #MRZ datagram is associated with partition 2 of 2, the cmnPart structure contains invalid values which propagate to proceeding structures, eventually crashing the thread when trying to read the rx_info structure.

kmall_v0 files tested do not split the #MRZ datagrams, so error could not be replicated.

It is unclear where in the Kmall reader logic this could be going wrong.

The current support for gsw library is partial and uses the obsolete API

This is just an preliminary enhancement idea.

For those sound speed profilers that don't have integrated GNSS sensors, the geographic position must still be entered manually. This is still a cumbersome step that could probably be solved by associating the hydrographic survey ship's current position with the imported profile. Indeed, the import into SSM mostly occurs within minutes after the cast completion. An option within the Setup tab would also need to be added to allow the user to enable or disable this feature.

With the current SIS integration, that is certainly possible. When not running SIS, I was wondering if a new network listener could be setup that listens for NMEA $$GGA strings.

Hello Folks!

Today I tried an install from source and it failed.

My setup:
OS-X El Capitan
Anaconda python distribution
python 3 environment

My execution:

python setup.py install

Failures:

1. basemap package could not be found

• Work-around with manual installation from anaconda:
  conda install -c anaconda basemap

2. gdal package compile failure. (setup finds v2.2.3)

• Work-around with manual installation from anaconda (installs v2.1.0):
  conda install gdal

3. Also pyside is commented out from the install_requires section of setup.py. But sound_speed_manager does require it.

I'm not sure how to fix the inability to find basemap. But I think the swig wrapper for GDAL requires v2.2 or less.

I realize this is probably not the usual method by which you install these things.
Cheers,
Val

url: https://tidesandcurrents.noaa.gov/ofs/gomofs/gomofs_info.html

Basemap is now deprecated: https://matplotlib.org/basemap/users/intro.html#deprecation-notice

The server mode is currently not supported for SIS5.

• SIS url

Issue:
When creating the environment to run Soundspeed Manager and using conda to install gdal:
conda install gdal
Installs gdal v3.2 from the conda-forge feedstock.

Running :

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ogr'

This is related to conda-forge/gdal-feedstock#419 and OSGeo/gdal#3149
Use of :

import gdal
import ogr
import osr

Is depreciated. It is suggested that a refactor be done to use the preferred method:

from osgeo impport gdal, ogr, osr

We are currently surveying in the Antarctic with a Seabird CTD and an OceanScience Underway CTD on board. While importing the Seabird profiles into SSM works like a charm, the OceanScience files either fail or fail partly wherever Temperatures are below zero.

I attached an example *.asc file from one of our casts as well as the error message in the SSM output window.

I'd appreciate any comment on this, especially since we have a rather extensive multibeam/UCTD campaign coming up in June and since it's such an amazing piece of software, I'd love to use SSM for our processing! :)

SSM_debug.zip

DEBUG hyo.soundspeedmanager.widgets.editor.on_input_data:248 > user wants to input data DEBUG hyo.soundspeedmanager.dialogs.import_single_profile_dialog.do_import:173 > user selection: F:/PS118/mbes/data-svp/UCTD/PS118_3-1_down.asc DEBUG hyo.soundspeed.soundspeed.import_data:414 > oceanscience > path: F:/PS118/mbes/data-svp/UCTD/PS118_3-1_down.asc DEBUG hyo.soundspeed.formats.readers.oceanscience.read:28 > *** oceanscience.0.1.0 ***: start WARNING hyo.soundspeed.formats.readers.oceanscience._parse_header:74 > unable to retrieve cast location: newer format? or lat/lon is blank? DEBUG hyo.soundspeedmanager.qt_callbacks.interpret_latitude:108 > lat tokens: ['-64.892'] DEBUG hyo.soundspeedmanager.qt_callbacks.interpret_latitude:126 > DD DEBUG hyo.soundspeedmanager.qt_callbacks.interpret_latitude:193 > latitude: -64.892 DEBUG hyo.soundspeedmanager.qt_callbacks.interpret_longitude:203 > lon tokens: ['-57.236'] DEBUG hyo.soundspeedmanager.qt_callbacks.interpret_longitude:222 > DD DEBUG hyo.soundspeedmanager.qt_callbacks.interpret_longitude:288 > longitude: -57.236 INFO hyo.soundspeed.profile.profile.clone_data_to_proc:1154 > cloning raw data to proc samples DEBUG hyo.soundspeed.formats.readers.oceanscience.read:54 > *** oceanscience.0.1.0 ***: done DEBUG hyo.soundspeed.soundspeed.import_data:422 > data file successfully parsed! DEBUG hyo.soundspeed.atlas.woa09.woa09.query:178 > query: 2019-02-21 @ (-57.236000, -64.892000) DEBUG hyo.soundspeed.atlas.woa09.woa09.load_grids:125 > 0(-89.500, 0.500); step(1.000, 1.000); depths: 33 DEBUG hyo.soundspeed.atlas.woa09.woa09.load_grids:125 > 0(-89.500, 0.500); step(1.000, 1.000); depths: 33 DEBUG hyo.soundspeed.atlas.woa09.woa09.query:228 > at land: 25, 300 DEBUG hyo.soundspeed.atlas.woa09.woa09.query:228 > at land: 25, 301 DEBUG hyo.soundspeed.atlas.woa09.woa09.query:228 > at land: 25, 302 DEBUG hyo.soundspeed.atlas.woa09.woa09.query:228 > at land: 26, 300 DEBUG hyo.soundspeed.atlas.woa09.woa09.query:228 > at land: 26, 301 DEBUG hyo.soundspeed.atlas.woa09.woa09.query:228 > at land: 26, 302 DEBUG hyo.soundspeed.atlas.woa09.woa09.query:228 > at land: 26, 303 DEBUG hyo.soundspeed.atlas.woa09.woa09.query:228 > at land: 26, 304 DEBUG hyo.soundspeed.atlas.woa09.woa09.query:228 > at land: 27, 301 INFO hyo.soundspeed.profile.profile.clone_data_to_proc:1154 > cloning raw data to proc samples INFO hyo.soundspeed.profile.profile.clone_data_to_proc:1154 > cloning raw data to proc samples INFO hyo.soundspeed.profile.profile.clone_data_to_proc:1154 > cloning raw data to proc samples DEBUG hyo.soundspeed.atlas.woa13.woa13.query:185 > query: 2019-02-21 @ (-57.236000, -64.892000) DEBUG hyo.soundspeed.atlas.woa13.woa13.grid_coords:174 > grid coords: 100 491 DEBUG hyo.soundspeed.atlas.woa13.woa13.query:233 > at land: 102, 489 DEBUG hyo.soundspeed.atlas.woa13.woa13.query:233 > at land: 102, 490 DEBUG hyo.soundspeed.atlas.woa13.woa13.query:233 > at land: 102, 491 DEBUG hyo.soundspeed.atlas.woa13.woa13.query:316 > valid: 37 INFO hyo.soundspeed.profile.profile.clone_data_to_proc:1154 > cloning raw data to proc samples Traceback (most recent call last): File "hyo\soundspeedmanager\widgets\editor.py", line 257, in on_input_data File "hyo\soundspeedmanager\mainwin.py", line 395, in data_imported File "hyo\soundspeedmanager\widgets\editor.py", line 738, in data_imported File "hyo\soundspeedmanager\widgets\dataplots.py", line 428, in on_first_draw File "hyo\soundspeedmanager\widgets\dataplots.py", line 206, in _draw_speed TypeError: 'NoneType' object is not subscriptable

I've setup the network transmission as described in the SSM documentation. However, I get an IndexError when trying to transmit the sound speed profile to HYPACK running on the same computer.

Traceback (most recent call last): File "hyo2\soundspeedmanager\widgets\editor.py", line 701, in on_transmit_data File "hyo2\soundspeed\soundspeed.py", line 2114, in transmit_ssp File "hyo2\soundspeed\client\clientlist.py", line 43, in transmit_ssp File "hyo2\soundspeed\client\client.py", line 37, in send_cast File "hyo2\soundspeed\client\client.py", line 84, in send_hyp_format File "hyo2\soundspeed\formats\writers\calc.py", line 47, in convert File "hyo2\soundspeed\formats\writers\calc.py", line 66, in _convert_body IndexError: boolean index did not match indexed array along dimension 0; dimension is 0 but corresponding boolean dimension is 51 2023-03-28 12:56:33 DEBUG hyo2.abc.app.dialogs.exception.exception_dialog._fill:139 > msg: IndexError: boolean index did not match indexed array along dimension 0; dimension is 0 but corresponding boolean dimension is 51

Currently, only the latest version of the html documentation is made available.
It would be better to support multi-version documentation, similar to what is available on Read the Docs (e.g., https://carto-python.readthedocs.io/en/latest/ ).

In Valeport profile reader, the pressure are treated as depth directly. Not sure if other type of profile readers have the similar problem.

Argo drifters provide the required data to compute a vertical sound speed profile (T,S,P) , usually as txt or NetCDF files. (http://www.jcommops.org/board?t=argo ) That information could be used to extend or complement an onboard probe cast when surveying nearby a working Argo drifter.

When using conda to set up env for development, conda installs matplotlib-base=3.3.3. Changes to the backend in this version result in the nav bar crashing.

Solution:

1. Easy - fix conda to use matplotlib=3.2.2. This version does not crash.
2. Harder - Update hyo2.soundspeedmanager.widgets.navtoolbar to be compatible with matplotlib=3.3.3

url: https://www.nodc.noaa.gov/OC5/woa18f/

Possibillity to set a default scale to the plots axis, so every profile is scaled the same in the "Plots per Day" operation?
Do this by manually inputting the values in the setup page so they will be the same for the whole project.

Hi,

If the user export the data with a different date format that is not mm/dd/yyyy, the readers will not worked or convert the date wrongly.

Maybe add a user input option...

The new Datalog X2 dont create a .000 file, so support for .vpd file would be nice.

Hi

We would prefere to use upcast, but sometimes we need to use a downcast. Would it be possible to select upcast/down cast during import?

Best Regards
Bustebartepinn

Trying to get TSS data with add_cur_tss does not give a descriptive error/debug message when it fails or when it returns None.

Starting the program, importing a cast, and clicking TSS resulted in pop up dialog box requesting user to input depth etc data. At this point (with first pop up dialog open) the last message in console was DEBUG hyo2.soundspeedmanager.widgets.editor.on_retrieve_tss:419 > user wants to retrieve transducer sound speed. Only after clicking cancel on both dialog boxes does WARNING hyo2.soundspeed.soundspeed.add_cur_tss:1653 > unable to retrieve tss values appear.

The cause of the problem was that there were two sound speed manager (SSM) instances open, so (I assume) the second instance could not listen to the port and get the data coming in. See attached console output: ssm-tss-error.md for full log (of starting the 2nd SSM, importing cast, attempting to get tss data)

What I expected to see was a debug message saying something to the effect that the process of listening for data was returning Nothing. Or better, checking to see if the port is accessible to it and reporting back failed status. (Because finding a 2nd SSM instance running? Easy. Finding some other program, process, random cool script to give some new functionality that people forgot is running, etc also listening to the same port, maybe in the background? Much harder.)

Let me know if any more info would be useful.

Old generation Kongeberg EM systems can accept 570 data points and newer generation EM systems accept up to 1000 data points in the SVP sent to SIS via SSP datagram.

Currently Sound Speed Manager has a hard coded thinning algorithm that is active when the input profile has 100 or more data points. The thinning occures on both exported .asvp files and network transfered data in SSP datagram format. Currently a profile with 100 data points get over thinned leaving only 16 data points. This gives issues for clients who then receive these .asvp files.

1. The threshold for thinning to occur should be changed from 100 to 570 data points for both types of Kongsberg outputs.

2. When thinning does take place is should stop once the number of points falls below this threshold of 570 data points. The current help files state this already but it does not occur.

Find a way to include the cartopy's offline data with a frozen release.

Continue the work started with this commit: afcb018

The high-level task is to retrieve synthetic profiles from regional OFS similarly to what was done for LEOFS.

Sub-tasks:

• Identify which of the regional OFS models provide regularly-gridded output.
• Add the code to retrieve synthetic profiles from the above models.
• Modify the Import dialog to expose the new models to users.
• Update the manual with the new supported OFS models.

I thought I'd share a couple of issues I had building ssm2 on Windows 10. I'm running conda 23.3.1 and pip 24.0. I'm not sure exactly what is the cause of these problems, and I am not totally confident that I'm doing everything properly. However, applying the following solutions to the issues made everything work properly:

1. I needed to install qt_material from pip since ssm2 was complaining that this package was missing. Currently installed version: 2.14.

2. I needed to downgrade scipy to version 1.6.2 since ssm2 was complaining with the newer release.

3. After building ssm2, I had to manually copy the file "hyo2.xml" in the site-packages\hyo2\abc2\app\app_style\media folder. Otherwise, I was getting a runtime error since this file was missing.

4. After building ssm2, I had to manually copy the folder "media" in the site-packages\hyo2\ssm2\app\gui\soundspeedmanager\media folder. Otherwise, I was getting a runtime error since this folder was missing.

I hope this might help.

There's a problem when using the "import data" function to import an existing project database into a newly created one.

Sound Speed Manager_2019_2_6_bug_1597137834.txt

It has been reported that some invalid values (e.g., 999) for latitude and longitude are parsed as valid, thus making the atlas profile retrieval failing.

I'd like to briefly discuss what would be the most suitable way of linking the "Institutions" and "Vessel" lists in the "Default metadata" section of the "General" tab in the "Setup" section. For me, it would make sense to link these two lists, so that if I choose a specific institution, then all vessels associated with that institution are displayed in the "Vessel" drop-down box.

I think it makes sense to replace the institution and vessel lists in hyo2/ssm2/lib/base/setup_sql.py with a Python dictionary. But I do have one questions pertaining to the existing default institution and vessel lists. Are all currently listed vessel associated with both NOAA and UNH?

Hey, looking at your repo here: https://bitbucket.org/hydroffice/hyo_qctools2/src/master/ it points to here for the source code: https://github.com/hydroffice/hyo_qc2, but this repo doesn't exist.

Where is the source for the main QC Tools project?

Hi there,

We used version 2023.0.10 of sound speed manager on a voyage recently and just noticed a little bug in the Editor page.

On previous versions, if you had used the Zoom in area function to look at a section of a profile more closely, to return to looking at the entire profile you could use Previous view or Reset view. On the new version, those two buttons seem to take you somewhere random in the view rather than returning you to the whole profile. To get back to the whole profile you need to go to the Database and double click on the profile again.

Thanks

Run sanity check for bogus coordinates, particularly for NCEI file output.

Hi,
There is a missing package in hyo2.abc library use by soundspeed in hyo2\soundspeed\atlas\woa09.py and woa13.py

9 from hyo2.abc.lib.onedrive import OneDrive

Version affected:

• hyo2.soundspeed-2022.1.0
• hyo2.abc-0.3.6

I've noticed that the "Profile stats" function returns an error for certain profiles. I wonder if the cause of this might not be in the way the weights are calculated. For example, on line 240 of hyo2/soundspeed/profile/profile.py:

@property
def proc_temp_mean(self):
w = self.calc_weights(self.proc.temp[self.proc_valid])
return np.average(self.proc.temp[self.proc_valid], weights=w)

The argument to the calc_weights function is the temperature profile itself. But isn't the point of the weighted mean of (in this case) the temperature profile to weight the temperature values according to how they are distributed in the water column? In other words, shouldn't the argument to the calc_weights function always be the depth profile?

• https://ci.appveyor.com/project/giumas/hyo-soundspeed

• https://travis-ci.com/hydroffice/hyo2_soundspeed

Kongsberg does not support MR20 datagram with KCtrl.