oysstu / pyxtf Goto Github PK
View Code? Open in Web Editor NEWA python library for reading and writing eXtended Triton Format (XTF) files
License: MIT License
A python library for reading and writing eXtended Triton Format (XTF) files
License: MIT License
Hello,
I am trying to use pyxtf to merge port/starboard files from a Kongsberg HISAS. In trying to do this I discovered that when i try and make a file over 255 pings it becomes unreadable.
I went and tried your example script 'write_xtf.py' and tried manually changing the num_pings value to see if i could duplicate the behavior.
When creating files with 255 pings or less, there is no issue and there is no problem using pyxtf to read those files.
When creating files with 256 pings or more, attempting to read that file results in the following error:
2024-06-04 18:05:45,926 - ERROR - raise RuntimeError('Number of bytes to read exceeds the number of bytes remaining in packet.')
Given that its is this 255/256 boundary I thought maybe something like 'ping_count' or similar was 8-bit and couldn't handle a value over 255, however, I have looked through all the code base for pyxtf and so far I can't find a value in any of the file attributes that should be larger than 8-bit and isn't.
Last bit of info is that I can successfully read many other xtf files with pyxtf. It is just the ones written using pyxtf that are over 255 pings that appear to have an issue.
Any help you could provide would be most appreciated. Thanks.
Due to the ordering of data channels, when I ran this code it broke reading my XTF file. I am only trying to access the port and starboard sonar data so changing line 228 in xtf_ctypes from
chan_info = [self.ChanInfo[i] for i in range(0, self.channel_count())]
to
chan_info = [self.ChanInfo[i] for i in range(0, 6) if self.ChanInfo[i].TypeOfChannel in [1,2]]
ended up fixing the problem.
I think this line change the makes the code more generalizable. Instead of assuming the relevant information is in the first two channels (as this code does if self.channel_count=2), it searches over all of the channels until it finds the two with port and starboard imagery.
in xtf_ctypes.py
in the for loop from line 562 to 591, all instances of file_header.sonar_info[i] should instead be file_header.sonar_info[i-1]
the index is off by 1
Following the notebook example for writing XTF data gives the following error when modifying existing XTF packets:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[31], line 14
12 # Write packets
13 for packet in packets:
---> 14 f.write(packet.to_bytes())
TypeError: to_bytes() missing required argument 'length' (pos 1)
It seems the to_bytes method being used in builtins.pyi requires the following: length: SupportsIndex, byteorder: Literal["little", "big"]
this problem seems to come from my XTF files containing the following packets alongside sonar:
XTFHeaderType.navigation
42
XTFHeaderType.attitude
3
XTFHeaderType.sonar
0
I would like to modify the original data file then save a renamed copy, including these additional packets.
First, thanks for the library wich is very useful.
But i had a little problem with the XTF i generated and the same probleme happen with your fake XTF file generated in write.py
the error AttributeError: 'XTFUnknownPacket' object has no attribute 'Year' appear when i try to plot it.
Do you know the origin of it ?
Thanks,
File indexing has been disabled by default now (861a5f5), as it caused problems due to it being applied to xtf files automatically based on the file name alone. The index significantly speeds up reading parts of large files, but should have more protection in place to avoid trying to read a file with an invalid index.
Possible solutions include
The XTF file that write by your own can't be read by plot_sonar_bathy
In the current implementation an index file is produced at the first reading of the XTF and is subsequently loaded in a way that is transparent to the user.
In general, I would not expect a "read" method to "write" something, and I would not expect a method to silently work differently between the first and the second call. (Also I find a bit dangerous to do file I/O inside a generator).
Maybe it could be better to decouple the generation/loading of the index file and leave some control to the user?
Hi @oysstu ,I'm trying to use it to read aC3D side scan sonar.
pyxtf
reads my datafile correctly thanks making pyxtf
available!
I was now following the example notebook but I got an error because of concatenate_channel
is not defined anywhere in the notebook.
Adding from pyxtf import concatenate_channel
fix this issue. I was wondering if you can update the notebook or should I try making a PR
? Thanks!
Hi,
I have a problem when I try to read a XTF file with number 67 XTFHeaderType. In fact, my XTF file contains only R2Sonic TruePix data. Is it something that you are working on ?
Sincerly,
Vincent
I'm the author of another Python XTF implementation: https://github.com/shamrin/pyxtf
Thank you for making the library, @oysstu! If you solution existed in 2013, I would be using it, instead of making my own :)
I've noticed at least one person was confused by our similarly named repos: shamrin/pyxtf#3 (comment)
That's rather unfortunate, I would like more people to know about your library. Your implementation is more up-to-date and actively supported. I'm going to add a note in my repo to let people know about your library: shamrin/pyxtf#5
Please let me know if I could do more :)
When I read a xtf file , it caused a problem (which is shown in the error.txt). The experimental data and error are in the zip file below. How can I solve this ? Thanks a lot.
DataTest.zip
Hi
I recently start working on SSS, and got several SSS data in different formats, such as .875, .jsf. I'm wondering if there's an open-source or free software available for converting different formats to a unified XTF format. I'm considering to develop such a module to remove barriers in the way of sss data processing...
pyxtf is a great module and I appreciate the developer putting it up. I use it to extract track from XTFs. Recently I am getting memory errors with exceptionally large SSS-only XTFs (30 km plus, 50 m range). I think the maximum array size is being exceeded. Computer memory is wide open (55 GB of 64 GB still available). Is there any way, at this initial stage:
input_file = 'yourfile.xtf'
(file_header, packets) = pyxtf.xtf_read(input_file, verbose=True)
to decimate the input so as to only read in, say, every second trace or ping? Failure point is the second line above, so inside the pyxtf module. Thanks for your advice.
Why do I keep getting errors calling the Qt platform when I run plot_sonar_paythy.py in examples, "The following (supported) packets are present (XTFHeaderType:count): ['sonar:10']
"And" qt.qpa.plugin: Could not load the Qt platform plugin "XCB"
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.