Comments (5)
Have two PRs that should resolve this:
#117
noaa-ocs-hydrography/drivers#19
I will say that the sample data you have here did not look great in Kluster. I saw missing navigation for most of the file, I believe because there was only a single nav record in the file. Hopefully that is not a result of our file reading.
from kluster.
Have two PRs that should resolve this: #117 noaa-ocs-hydrography/drivers#19
I will say that the sample data you have here did not look great in Kluster. I saw missing navigation for most of the file, I believe because there was only a single nav record in the file. Hopefully that is not a result of our file reading.
Thanks for the updates! I tried out the add_m3_support and all_m3_support branches and the data imports doesn't give the same startbyte error but as you mentioned, there is a lot of missing data. The .all file was exported while the sonar was indoors on a bench facing the ceiling so it makes sense the data isn't the best quality. I will have to retest once field data is available.
from kluster.
I've been trying to extract point cloud data from the sample .all file using Kluster but no luck so far. To see whether the .all file has the necessary data, I put it into Eiva's NaviModel Viewer and was able to get this image:
From this, it looks like the file has the data necessary to generate a pointcloud but I have yet to be able to reproduce the same results using Kluster.
My attempts at extracting data have been to do the following:
from HSTB.kluster.fqpr_intelligence import intel_process
_, fq = intel_process(r'C:\KML\M3_V0254\Recordings\2024_Jan_10_19-56-12.all')
fq = fq[0]
And this results in the following error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[2], line 2
1 from HSTB.kluster.fqpr_intelligence import intel_process
----> 2 _, fq = intel_process(r'C:\KML\M3_V0254\Recordings\2024_Jan_10_19-56-12.all')
3 fq = fq[0]
File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_intelligence.py:1996, in intel_process(filname, outfold, coord_system, epsg, use_epsg, vert_ref, parallel_write, vdatum_directory, force_coordinate_system, cast_selection_method, designated_surface, process_mode, logger, client, skip_dask)
1994 print(e)
1995 while intel.has_actions:
-> 1996 intel.execute_action()
1997 return intel, list(intel.project.fqpr_instances.values())
File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_intelligence.py:1163, in FqprIntel.execute_action(self, idx)
1161 self.parent.kluster_execute_action(self.action_container, 0)
1162 else:
-> 1163 output = self.action_container.execute_action(idx)
1164 if isinstance(output, Fqpr): # if the output is fqpr data
1165 self.project.add_fqpr(output)
File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_actions.py:276, in FqprActionContainer.execute_action(self, idx)
274 self._update_actions()
275 if action:
--> 276 action.execute()
277 output = action.output
278 return output
File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_actions.py:52, in FqprAction.execute(self)
50 self.is_running = True
51 if self.kwargs:
---> 52 self.output = self.function(*self.args, **self.kwargs)
53 else:
54 self.output = self.function(*self.args)
File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_convenience.py:369, in process_multibeam(fqpr_inst, run_orientation, orientation_initial_interpolation, run_beam_vec, run_svcorr, run_georef, run_tpu, add_cast_files, input_datum, use_epsg, use_coord, epsg, coord_system, vert_ref, vdatum_directory, cast_selection_method, only_this_line, only_these_times)
367 fqpr_inst.get_beam_pointing_vectors(subset_time=subset_time)
368 if run_svcorr:
--> 369 fqpr_inst.sv_correct(add_cast_files=add_cast_files, cast_selection_method=cast_selection_method, subset_time=subset_time)
370 if run_georef:
371 fqpr_inst.georef_xyz(vdatum_directory=vdatum_directory, subset_time=subset_time)
File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_generation.py:2884, in Fqpr.sv_correct(self, add_cast_files, cast_selection_method, subset_time, dump_data)
2882 idx_by_chunk = self.return_chunk_indices(applicable_index, pings_per_chunk)
2883 if len(idx_by_chunk[0]): # if there are pings in this system that align with this installation parameter record
-> 2884 self._submit_data_to_cluster(ra, 'sv_corr', idx_by_chunk, max_chunks_at_a_time,
2885 timestmp, prefixes, dump_data=dump_data, skip_dask=skip_dask,
2886 cast_selection_method=cast_selection_method)
2887 else:
2888 if dump_data:
File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_generation.py:3276, in Fqpr._submit_data_to_cluster(self, rawping, mode, idx_by_chunk, max_chunks_at_a_time, timestmp, prefixes, dump_data, skip_dask, prefer_pp_nav, vdatum_directory, cast_selection_method, backscatter_settings)
3274 if self.show_progress and rn != 0: # first run we skip progress as it prints out the run info
3275 print_progress_bar(rn + 1, tot_runs, prefix=f'Loading chunk {rn + 1}/{tot_runs}:')
-> 3276 data_for_workers = chunk_function(*chunkargs, silent=silent)
3277 try:
3278 self.debug_print(f'Running {mode} process...', logging.INFO)
File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_generation.py:1594, in Fqpr._generate_chunks_svcorr(self, ra, cast_chunks, casts, prefixes, timestmp, addtl_offsets, run_index, silent)
1592 if any(c[1] is None for c in cast_chunks):
1593 self.print('Unable to sound velocity correct, one of the data chunks was unable to find a cast, have you imported a cast yet?', logging.ERROR)
-> 1594 raise ValueError('Unable to sound velocity correct, one of the data chunks was unable to find a cast, have you imported a cast yet?')
1596 # this should be the transducer to waterline, positive down
1597 refpt = self.multibeam.return_prefix_for_rp()
ValueError: Unable to sound velocity correct, one of the data chunks was unable to find a cast, have you imported a cast yet?
Seems to have an issue with the sound velocity. In this test setup, the sound velocity was set to a fixed value of 1469 m/s.
from kluster.
Kluster requires a sound velocity profile. This is generally provided in the .all/.kmall file logged through SIS. I am going to guess that the software you used to get the point cloud above skips ray-tracing the corrected beam vectors, or just uses the surface sound velocity value that you mention.
You can import a surface sound velocity profile into Kluster, to get past the correct for sound velocity action. The easiest way to do that is to use Hydroffice (https://www.hydroffice.org/soundspeed/) and generate a synthetic cast, using the WOA. Export the cast to .svp, and drag it into Kluster, which should get you the right action.
https://kluster.readthedocs.io/en/latest/processing/index.html#sound-velocity
from kluster.
There is also a menu item for WOA casts which uses Sound Speed Manager (SSM) for generating the cast. I would suggest just installing SSM and making a cast this time but it is something to keep in mind if this would be a regular part of your workflow.
from kluster.
Related Issues (20)
- expand Accuracy Test to work with GDAL supported formats as well as Bathygrid data
- rework the output progress bars to maybe merge/replace progress information, keep the output text to a minimum
- variable for tracking realtime navigation solution? Does that exist in raw mbes?
- Export shapefile for ship tracklines, grid cells
- add customizable kluster_variables
- UI in linux HOT 17
- Error converting .all files v0.8.8 HOT 20
- Docker file won´t built a working image HOT 10
- Getting an error while processing/georeferenceing .all file HOT 6
- Creating a 'plugin' for Kluster HOT 12
- Unexpected filter behaviour? HOT 7
- Error redrawing 3d points after filtering HOT 23
- set a custom port for the Dask cluster HOT 10
- Array with shape error when processing mulitbeam data HOT 5
- EM124 Data Processing in Kluster HOT 6
- EM2042 support and new KMALL datagram revision HOT 8
- Sonar model not understood 'em2040c' HOT 7
- Error: Multibeam buffer is smaller than requested size HOT 6
- Code examples HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kluster.