domna / tdm_loader Goto Github PK

In several tdm files I've tested, there are more "block" tags than "tdm_channel" tags, which made assert(len(blocks) == len(channel_names)) to fail. This might be i dirty fix to that problem assert(len(blocks) >= len(channel_names)), but I don't have a better idea right now.

What I noted with those tdm file was that the few extra "block" tags are always the last ones, they always have valueType="eInt16Usi", and their id is always linked to a "flags" tag instead of a "double_sequence" or "long_sequence" tag.

Multiple times I have got this error and I don't know why. I guess it happens when OpenFile objects are being destroyed in the background and OpenFile.__del__() is being called.
Error message:

Traceback (most recent call last):
  File "c:\tdm_loader\tdm_loader.py", line 520, in __del__
    self.close()
  File "c:\tdm_loader\tdm_loader.py", line 494, in close
    self._tdx_fobj.close()
AttributeError: 'OpenFile' object has no attribute '_tdx_fobj'

I don't know how to reproduce the error, but it should be solved by replacing the code in OpenFile.close()

self._tdx_fobj.close()

with

if hasattr(self, '_tdx_fobj'):
    self._tdx_fobj.close()

Currently set to utf-8.

The tdm files I'm dealing with are not all well-formed, numbers of the channels maybe different sometimes.
Therefore, I should replace the blank with NaN. However, when I open these tdm/tdx file with tdm_loader.OpenFile() in iPython, I got an ParserError:
ParseError: not well-formed (invalid token): line 1, column 1
Would you please make these baddish tdm/tdx files readable with tdm loader?
Thanks a lot^_^

Hi,

there is an issue with the "get_column_index" function, when using a str instead of an int.
I guess the if statement at the beginning (if channel_group < 0 or channel < 0:) is the root cause for the issue.

Best regards

Florian

currently, only accessing channels of explicit sequence_representation is supported.
that means data stored as float numbers, directly.

If you try to access a time channel, this may fail since time channels typically are even-spaced and therefore can be constructed by very few parameters.
In that case, sequence represetation is implicit_linear.

After some issues with unprocessed measurement data ( AttributeError: 'NoneType' object has no attribute 'get'), I examined the xml structure of the TDM file and also discovered a third sequence represetation type: raw_linear.
That means mapping measurement data on an integer range via two generation parameters (both float).
I think this is reasonnable for typically coarse and even-spaced measurement device resolution.

So I expanded the channel method by a sequence represetation check and handling.

I would like to share this, as #22, as well.

Hello domna

When trying to access TDM data stored by DIAdem 2020 release,
tdm_loader.OpenFile(file.TDM)
will return someting like
ParseError: not well-formed (invalid token): ...

This is because TDM files actually are zip files now.
So you can get around that by renaming file.TDM to file.zip, extracting the contained file and rename that one to file.TDM again.

Anyway, I added some lines to the __init__ that recognize and fix that automatically.
Now I would like to share that solution.

I've been having problems understanding channel indices and column inideces and how to correctly loop through all the channels in a file. What I want to do is to get all the channels in a dict with channel names as keys. The code I'm using now which seems to work fine

df = tdm_loader.OpenFile(tdm_file_name)
tdm_dict = {}
for i_grp in range(df.no_channel_groups()):
    for i_ch in range(df.no_channels(i_grp)):
        name = df.channel_name(i_grp, i_ch)
        ch = df.channel(i_grp, i_ch)
        tdm_dict[name] = np.array(ch)

At first I tried to loop through df.channel_names and use the corresponding index to get the channel with df[i], but that apparently jumble the names vs channels.

It would be nice to have a convenient function for converting to dict of ndarray, e.g. OpenFile.to_dict(). I don't know though how this would work if there are more than one channel group in a file, all my files only have one.

Alternatively give example in README on how to do something like this, or just how to loop through and get correct pairing (channel name, channel data).

I cod as you told:
data_file = tdm_loader.OpenFile('filename.tdm' [, encoding='utf-8'])
and come across an "invalid syntax".
After I tried removing the bracket, there's no "invalid syntax" any more, but a Unicode Encode Error:
''ascii' codec can't encode character u'\xb0' in position 5305: ordinal not in range(128)
Why?

I recently started getting the following error in MemmapColumnFirst.__init__ when loading files:

    self._empty_row[0] = pack('') # initialize all items to zero

ValueError: could not convert string to float: 

The only reason I can find is that numpy was updated from version 1.13.1 to 1.14.2. Loading works if I change the line to self._empty_row[0] = 0, although I have no idea why, but then it instead fails with a TypeError: a bytes-like object is required, not 'int' on the same line if I go back using numpy 1.13.1 again.
I tested the sample file test_sample0001.tdm so you should be able to reproduce the error.

./tdm_loader.py in channel(self, channel_group, channel, occurrence, ch_occurrence)
229 lc = self._root.find(".//localcolumn[@id='{0}']".format(lc_usi))
230 repr = lc.findtext('sequence_representation')
--> 231 data_usi = OpenFile._get_usi_from_txt(lc.findtext('values'))[0]
232 inc = self._root.find(".//{0}[@id='{1}']/values".format(datatype, data_usi))

data_file=tdm_loader.OpenFile('.\data_to_read.TDM','.\data_to_read.tdx')
I can get the information about the file as following:
Number of Channelgroups: 6
Channel Length: 6
but i can't use channel_dict to get the dictionary
if i run data_file.channel_dict(1), i get the error which i display at the beginning.

I can't find the reason.
Besides, i can use the same script to read other files......
I check the file with Diadem but i don't notice any errors.

Can anyone help me?

I have TDM/TDX files I'm trying to read using the following syntax:

import tdm_loader
data_file = tdm_loader.OpenFile('filename.tdm')

And I receive this error:
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 2

I hope there's an easy solution at my end... I uploaded some sample files.

TDM-TDX_Files.zip

First of all, thank you very much for this great package!
We used this to successfully parse large amounts of tdm files.

Recently I noticed that the some of the recently parsed tdm files were corrupt, meaning that the channels haven't been mapped correctly to their corresponding channel names (e.g. for channel A we got data from some other channel C and for channel C we got data from B and so on). The export type of these files is National Instruments USI. For Version 1.1.12 everything works as expected, for version 1.1.14 & 1.1.15 I noticed the problems mentioned above.
Any ideas what changed from 1.1.12 that could cause that behavior?