Giter Club home page Giter Club logo

Comments (5)

ericgyounkin avatar ericgyounkin commented on July 17, 2024

Have two PRs that should resolve this:
#117
noaa-ocs-hydrography/drivers#19

I will say that the sample data you have here did not look great in Kluster. I saw missing navigation for most of the file, I believe because there was only a single nav record in the file. Hopefully that is not a result of our file reading.

from kluster.

rkeen9 avatar rkeen9 commented on July 17, 2024

Have two PRs that should resolve this: #117 noaa-ocs-hydrography/drivers#19

I will say that the sample data you have here did not look great in Kluster. I saw missing navigation for most of the file, I believe because there was only a single nav record in the file. Hopefully that is not a result of our file reading.

Thanks for the updates! I tried out the add_m3_support and all_m3_support branches and the data imports doesn't give the same startbyte error but as you mentioned, there is a lot of missing data. The .all file was exported while the sonar was indoors on a bench facing the ceiling so it makes sense the data isn't the best quality. I will have to retest once field data is available.

from kluster.

rkeen9 avatar rkeen9 commented on July 17, 2024

I've been trying to extract point cloud data from the sample .all file using Kluster but no luck so far. To see whether the .all file has the necessary data, I put it into Eiva's NaviModel Viewer and was able to get this image:
unnamed
From this, it looks like the file has the data necessary to generate a pointcloud but I have yet to be able to reproduce the same results using Kluster.

My attempts at extracting data have been to do the following:

from HSTB.kluster.fqpr_intelligence import intel_process
_, fq = intel_process(r'C:\KML\M3_V0254\Recordings\2024_Jan_10_19-56-12.all')
fq = fq[0]

And this results in the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 2
      1 from HSTB.kluster.fqpr_intelligence import intel_process
----> 2 _, fq = intel_process(r'C:\KML\M3_V0254\Recordings\2024_Jan_10_19-56-12.all')
      3 fq = fq[0]

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_intelligence.py:1996, in intel_process(filname, outfold, coord_system, epsg, use_epsg, vert_ref, parallel_write, vdatum_directory, force_coordinate_system, cast_selection_method, designated_surface, process_mode, logger, client, skip_dask)
   1994             print(e)
   1995 while intel.has_actions:
-> 1996     intel.execute_action()
   1997 return intel, list(intel.project.fqpr_instances.values())

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_intelligence.py:1163, in FqprIntel.execute_action(self, idx)
   1161     self.parent.kluster_execute_action(self.action_container, 0)
   1162 else:
-> 1163     output = self.action_container.execute_action(idx)
   1164     if isinstance(output, Fqpr):  # if the output is fqpr data
   1165         self.project.add_fqpr(output)

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_actions.py:276, in FqprActionContainer.execute_action(self, idx)
    274 self._update_actions()
    275 if action:
--> 276     action.execute()
    277     output = action.output
    278     return output

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_actions.py:52, in FqprAction.execute(self)
     50 self.is_running = True
     51 if self.kwargs:
---> 52     self.output = self.function(*self.args, **self.kwargs)
     53 else:
     54     self.output = self.function(*self.args)

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_convenience.py:369, in process_multibeam(fqpr_inst, run_orientation, orientation_initial_interpolation, run_beam_vec, run_svcorr, run_georef, run_tpu, add_cast_files, input_datum, use_epsg, use_coord, epsg, coord_system, vert_ref, vdatum_directory, cast_selection_method, only_this_line, only_these_times)
    367     fqpr_inst.get_beam_pointing_vectors(subset_time=subset_time)
    368 if run_svcorr:
--> 369     fqpr_inst.sv_correct(add_cast_files=add_cast_files, cast_selection_method=cast_selection_method, subset_time=subset_time)
    370 if run_georef:
    371     fqpr_inst.georef_xyz(vdatum_directory=vdatum_directory, subset_time=subset_time)

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_generation.py:2884, in Fqpr.sv_correct(self, add_cast_files, cast_selection_method, subset_time, dump_data)
   2882 idx_by_chunk = self.return_chunk_indices(applicable_index, pings_per_chunk)
   2883 if len(idx_by_chunk[0]):  # if there are pings in this system that align with this installation parameter record
-> 2884     self._submit_data_to_cluster(ra, 'sv_corr', idx_by_chunk, max_chunks_at_a_time,
   2885                                  timestmp, prefixes, dump_data=dump_data, skip_dask=skip_dask,
   2886                                  cast_selection_method=cast_selection_method)
   2887 else:
   2888     if dump_data:

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_generation.py:3276, in Fqpr._submit_data_to_cluster(self, rawping, mode, idx_by_chunk, max_chunks_at_a_time, timestmp, prefixes, dump_data, skip_dask, prefer_pp_nav, vdatum_directory, cast_selection_method, backscatter_settings)
   3274 if self.show_progress and rn != 0:  # first run we skip progress as it prints out the run info
   3275     print_progress_bar(rn + 1, tot_runs, prefix=f'Loading chunk    {rn + 1}/{tot_runs}:')
-> 3276 data_for_workers = chunk_function(*chunkargs, silent=silent)
   3277 try:
   3278     self.debug_print(f'Running {mode} process...', logging.INFO)

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_generation.py:1594, in Fqpr._generate_chunks_svcorr(self, ra, cast_chunks, casts, prefixes, timestmp, addtl_offsets, run_index, silent)
   1592 if any(c[1] is None for c in cast_chunks):
   1593     self.print('Unable to sound velocity correct, one of the data chunks was unable to find a cast, have you imported a cast yet?', logging.ERROR)
-> 1594     raise ValueError('Unable to sound velocity correct, one of the data chunks was unable to find a cast, have you imported a cast yet?')
   1596 # this should be the transducer to waterline, positive down
   1597 refpt = self.multibeam.return_prefix_for_rp()

ValueError: Unable to sound velocity correct, one of the data chunks was unable to find a cast, have you imported a cast yet?

Seems to have an issue with the sound velocity. In this test setup, the sound velocity was set to a fixed value of 1469 m/s.

from kluster.

ericgyounkin avatar ericgyounkin commented on July 17, 2024

Kluster requires a sound velocity profile. This is generally provided in the .all/.kmall file logged through SIS. I am going to guess that the software you used to get the point cloud above skips ray-tracing the corrected beam vectors, or just uses the surface sound velocity value that you mention.

You can import a surface sound velocity profile into Kluster, to get past the correct for sound velocity action. The easiest way to do that is to use Hydroffice (https://www.hydroffice.org/soundspeed/) and generate a synthetic cast, using the WOA. Export the cast to .svp, and drag it into Kluster, which should get you the right action.

https://kluster.readthedocs.io/en/latest/processing/index.html#sound-velocity

from kluster.

barry-gallagher avatar barry-gallagher commented on July 17, 2024

There is also a menu item for WOA casts which uses Sound Speed Manager (SSM) for generating the cast. I would suggest just installing SSM and making a cast this time but it is something to keep in mind if this would be a regular part of your workflow.

from kluster.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.