Data is corrupted when subsampling a large file (8Gb) on windows about mappedtensor HOT 13 CLOSED

Hi Jerome,

That’s troubling. Just to confirm, if you do this:

imagesc(mtData(:, :, 234));
imagesc(mtData(1:2:end, 1:2:end, 234));

there is no problem, but this:

imagesc(mtData(1:5:end, 1:5:end, 234));

doesn’t work?

Is there an error message, or is the data simply corrupted?

Cheers,
Dylan.

from mappedtensor.

Dear Doctor Dylan,

That is exactly what is happening, the data is corrupted after 2gb. Please refer to the attached images.

gap = 5;
x_range = 1:gap:size(datablock,2);
y_range = 1:gap:size(datablock,1);
z_range = 1:1:1400;

val = 234;
imagesc(squeeze(datablock(yrange,xrange,zrange(val))));

This is what it should look like.

Thanks for the fast response.

Best regards,
Jerome

from mappedtensor.

Hi Jerome,

I can't reproduce this with a 3Gb file in double format (1024 x 768 x 512 images x 8 bytes = 3 Gb).
I can subsample as I choose, with no corruption. Could you please make sure you're using the latest commit of MappedTensor, and also let me know the data class of the file you are using?

Thanks,
Dylan.

from mappedtensor.

Dear Dr. Muir,

we started using your framework a while ago to analyze long video files as already mentioned by Jerome.
The easy access to tensors compared to memmapfiles is very much appreciated.

Regarding your questions:
The corruption of data happens at least in the formats uint16, doube and single.
Using the different formats, the corruption occus exactly after 2048MB (for each format) which suggest 16-bit file addressing.

Like you suggested, we tested the latets commit from here as well with the same result. Additionally, the effect was reproduced on two PCs with different compilers.

From the observations, I would conclude that it is not dependent on the file size.

What Jerome did not mention:
The actual data in the tensor is not altered by this:
Trying to display the video content after displaying a subsampled version gives still good results.

We tried Windows 7 and Windows 8.1 64bit with Matlab 2015b 64bit.

Do you need more information?
Thanks again for the excellent framework!

Best regards,

Michael

from mappedtensor.

Hi @mp1988,

Thanks for the extra information. It may be a windows-specific issue.

Could you please recompile the mapped_tensor_shim.c file with DEBUG defined (at the top of the file), and send me the output from running these commands in matlab?

datablock(1:5:20, 1:5:20, end)
datablock(1:20, 1:20, end)

Thanks,
Dylan.

from mappedtensor.

Dear Dr. Muir,

here are the outputs you asked for.

Sincerely,
Jerome

pathname =

D:\MatlabBa\MichaelVideo\40Hz_DC

filename =

PG_Grasshopper3 GS3-U3-23S6M_15355972()_X1920_Y1200_BPP2_LA1_FPS10_TIME2017-01-18-16-57-49_001

mts: Called with command [open]
mts/cof: Opening file [D:\MatlabBa\MichaelVideo\40Hz_DC\PG_Grasshopper3 GS3-U3-23S6M_15355972()_X1920_Y1200_BPP2_LA1_FPS10_TIME2017-01-18-16-57-49_001.dat]
mts/cof: Opening file in read-write mode.
mts/cof: File handle [000007FEEFDF1630], [8] bytes
mts/cof: System format is [ieee-le]
mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEFDF1630]
mts/crc: Chunks:
[-103991295][1][2304000]
mts/crc: Unique indices:
[-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648]...
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22]...
mts/crc: Data size: [1920][1200][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [2304000] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [00000000166D0060] (offset 0)
mts/crc: Seeking to [-207982592]
mts/crc: Single-read chunk, [4608000] bytes
Eavaluation Object created.
mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEFDF1630]
mts/crc: Chunks:
[-103991295][5][4]
[-103981695][5][4]
[-103972095][5][4]
[-103962495][5][4]
mts/crc: Unique indices:
[-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648]
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16]
mts/crc: Data size: [4][4][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [16] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [000000001C033E20] (offset 0)
mts/crc: Seeking to [-207982592]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [000000001C033E28] (offset 8)
mts/crc: Seeking to [-207963392]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [000000001C033E30] (offset 16)
mts/crc: Seeking to [-207944192]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [000000001C033E38] (offset 24)
mts/crc: Seeking to [-207924992]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes

ans =

4×4 uint16 matrix

24385 608 624 656
53250 53250 53250 53250
36866 36866 36866 36866
32770 32770 32770 32770

mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEFDF1630]
mts/crc: Chunks:
[-103991295][1][20]
[-103989375][1][20]
[-103987455][1][20]
[-103985535][1][20]
[-103983615][1][20]
[-103981695][1][20]
[-103979775][1][20]
[-103977855][1][20]
[-103975935][1][20]
[-103974015][1][20]
[-103972095][1][20]
[-103970175][1][20]
[-103968255][1][20]
[-103966335][1][20]
[-103964415][1][20]
[-103962495][1][20]
[-103960575][1][20]
[-103958655][1][20]
[-103956735][1][20]
[-103954815][1][20]
mts/crc: Unique indices:
[-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648]...
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22]...
mts/crc: Data size: [20][20][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [400] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [0000000036F0E920] (offset 0)
mts/crc: Seeking to [-207982592]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0E948] (offset 40)
mts/crc: Seeking to [-207978752]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0E970] (offset 80)
mts/crc: Seeking to [-207974912]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0E998] (offset 120)
mts/crc: Seeking to [-207971072]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0E9C0] (offset 160)
mts/crc: Seeking to [-207967232]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0E9E8] (offset 200)
mts/crc: Seeking to [-207963392]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EA10] (offset 240)
mts/crc: Seeking to [-207959552]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EA38] (offset 280)
mts/crc: Seeking to [-207955712]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EA60] (offset 320)
mts/crc: Seeking to [-207951872]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EA88] (offset 360)
mts/crc: Seeking to [-207948032]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EAB0] (offset 400)
mts/crc: Seeking to [-207944192]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EAD8] (offset 440)
mts/crc: Seeking to [-207940352]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EB00] (offset 480)
mts/crc: Seeking to [-207936512]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EB28] (offset 520)
mts/crc: Seeking to [-207932672]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EB50] (offset 560)
mts/crc: Seeking to [-207928832]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EB78] (offset 600)
mts/crc: Seeking to [-207924992]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EBA0] (offset 640)
mts/crc: Seeking to [-207921152]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EBC8] (offset 680)
mts/crc: Seeking to [-207917312]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EBF0] (offset 720)
mts/crc: Seeking to [-207913472]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EC18] (offset 760)
mts/crc: Seeking to [-207909632]
mts/crc: Single-read chunk, [40] bytes

ans =

20×20 uint16 matrix

Columns 1 through 16

24385 640 688 672 624 608 736 656 608 592 624 640 640 640 688 656
41538 608 640 624 592 592 720 592 704 640 656 624 608 656 608 640
672 656 624 608 688 656 656 656 704 672 592 704 608 640 576 656
640 608 656 656 640 688 608 576 576 704 704 624 624 560 656 656
640 592 704 640 688 608 640 672 640 640 624 640 624 624 592 576
624 672 608 688 672 688 640 656 688 592 704 688 624 576 656 624
592 592 608 576 640 640 656 688 656 624 640 688 608 608 624 592
656 720 704 704 656 640 640 640 656 560 656 656 608 624 576 592
672 608 640 688 640 720 704 592 624 688 592 608 640 672 608 672
608 624 656 624 656 640 672 640 656 640 624 672 688 656 624 672
704 624 672 704 592 640 704 704 656 672 704 688 576 704 640 656
608 592 640 688 656 624 624 640 704 688 672 592 608 640 640 640
640 624 688 640 688 608 592 640 672 624 688 656 688 640 704 592
656 592 672 656 736 672 640 672 624 688 640 704 704 672 752 640
640 592 624 688 672 672 640 624 640 608 576 688 656 656 656 640
608 672 672 608 704 672 640 704 576 672 656 672 688 608 576 640
704 656 656 672 608 640 704 672 736 656 672 656 656 592 704 608
688 624 672 688 640 688 672 704 720 720 656 624 720 704 688 608
704 656 656 560 624 640 656 608 704 656 736 720 656 672 640 688
576 688 560 640 560 592 672 736 688 656 672 672 704 720 688 672

Columns 17 through 20

 640     624     656     656
 624     624     576     608
 624     608     608     640
 592     656     576     656
 640     672     592     656
 560     592     672     672
 656     688     656     576
 640     560     704     560
 592     608     624     672
 608     672     720     656
 624     640     704     608
 624     656     576     560
 672     640     640     656
 560     672     672     608
 688     576     688     624
 656     656     720     656
 624     672     688     688
 624     704     672     656


 640     640     624     656
 656     608     640     656

from mappedtensor.

Hi @JeromeB211 @mp1988,

I have just pushed an update to a new branch iss/15. Please pull that branch, re-build mapped_tensor_shim.c with mex, and see if this commit fixes your problem. Please let me know either way.

There were a couple of lines where non-64 bit versions of fseek may have been used.

Best regards,
Dylan.

from mappedtensor.

Dear Dr. Muir,

Thank you very much for the update. Unfortunately it did not solve the problems we had. The data is even corrupted at the first frame now. Although the values of the corrupted Data changed, they are still not correct. I ran the same code of which i sent you the output last time. Here are the results now.
Thank you very much for your interest in our problem.

Sincerely ,
Jerome

pathname =

D:\MatlabBa\MichaelVideo\40Hz_DC

filename =

PG_Grasshopper3 GS3-U3-23S6M_15355972()_X1920_Y1200_BPP2_LA1_FPS10_TIME2017-01-18-16-57-49_001

mts: Called with command [open]
mts/cof: Opening file [D:\MatlabBa\MichaelVideo\40Hz_DC\PG_Grasshopper3 GS3-U3-23S6M_15355972()_X1920_Y1200_BPP2_LA1_FPS10_TIME2017-01-18-16-57-49_001.dat]
mts/cof: Opening file in read-write mode.
mts/cof: File handle [000007FEEB861630], [8] bytes
mts/cof: System format is [ieee-le]
Eavaluation Object created.
mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEB861630]
mts/crc: Chunks:
[4190976001][5][4]
[4190985601][5][4]
[4190995201][5][4]
[4191004801][5][4]
mts/crc: Unique indices:
[4190976001][4190976006][4190976011][4190976016][4190985601][4190985606][4190985611][4190985616][4190995201][4190995206][4190995211][4190995216][4191004801][4191004806][4191004811][4191004816]
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16]
mts/crc: Data size: [4][4][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [16] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [00000000173937E0] (offset 0)
mts/crc: Seeking to [8381952000]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [00000000173937E8] (offset 8)
mts/crc: Seeking to [8381971200]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [00000000173937F0] (offset 16)
mts/crc: Seeking to [8381990400]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [00000000173937F8] (offset 24)
mts/crc: Seeking to [8382009600]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEB861630]
mts/crc: Chunks:
[4190976001][5][4]
[4190985601][5][4]
[4190995201][5][4]
[4191004801][5][4]
mts/crc: Unique indices:
[4190976001][4190976006][4190976011][4190976016][4190985601][4190985606][4190985611][4190985616][4190995201][4190995206][4190995211][4190995216][4191004801][4191004806][4191004811][4191004816]
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16]
mts/crc: Data size: [4][4][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [16] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [000000001738AF60] (offset 0)
mts/crc: Seeking to [8381952000]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [000000001738AF68] (offset 8)
mts/crc: Seeking to [8381971200]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [000000001738AF70] (offset 16)
mts/crc: Seeking to [8381990400]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [000000001738AF78] (offset 24)
mts/crc: Seeking to [8382009600]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes

ans =

4×4 uint16 matrix

24385 608 624 656
640 608 624 576
640 608 624 576
640 608 624 576

mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEB861630]
mts/crc: Chunks:
[4190976001][1][20]
[4190977921][1][20]
[4190979841][1][20]
[4190981761][1][20]
[4190983681][1][20]
[4190985601][1][20]
[4190987521][1][20]
[4190989441][1][20]
[4190991361][1][20]
[4190993281][1][20]
[4190995201][1][20]
[4190997121][1][20]
[4190999041][1][20]
[4191000961][1][20]
[4191002881][1][20]
[4191004801][1][20]
[4191006721][1][20]
[4191008641][1][20]
[4191010561][1][20]
[4191012481][1][20]
mts/crc: Unique indices:
[4190976001][4190976002][4190976003][4190976004][4190976005][4190976006][4190976007][4190976008][4190976009][4190976010][4190976011][4190976012][4190976013][4190976014][4190976015][4190976016][4190976017][4190976018][4190976019][4190976020][4190977921][4190977922]...
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22]...
mts/crc: Data size: [20][20][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [400] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [000000013D84A140] (offset 0)
mts/crc: Seeking to [8381952000]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A168] (offset 40)
mts/crc: Seeking to [8381955840]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A190] (offset 80)
mts/crc: Seeking to [8381959680]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A1B8] (offset 120)
mts/crc: Seeking to [8381963520]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A1E0] (offset 160)
mts/crc: Seeking to [8381967360]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A208] (offset 200)
mts/crc: Seeking to [8381971200]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A230] (offset 240)
mts/crc: Seeking to [8381975040]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A258] (offset 280)
mts/crc: Seeking to [8381978880]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A280] (offset 320)
mts/crc: Seeking to [8381982720]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A2A8] (offset 360)
mts/crc: Seeking to [8381986560]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A2D0] (offset 400)
mts/crc: Seeking to [8381990400]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A2F8] (offset 440)
mts/crc: Seeking to [8381994240]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A320] (offset 480)
mts/crc: Seeking to [8381998080]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A348] (offset 520)
mts/crc: Seeking to [8382001920]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A370] (offset 560)
mts/crc: Seeking to [8382005760]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A398] (offset 600)
mts/crc: Seeking to [8382009600]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A3C0] (offset 640)
mts/crc: Seeking to [8382013440]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A3E8] (offset 680)
mts/crc: Seeking to [8382017280]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A410] (offset 720)
mts/crc: Seeking to [8382021120]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A438] (offset 760)
mts/crc: Seeking to [8382024960]
mts/crc: Single-read chunk, [40] bytes
mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEB861630]
mts/crc: Chunks:
[4190976001][1][20]
[4190977921][1][20]
[4190979841][1][20]
[4190981761][1][20]
[4190983681][1][20]
[4190985601][1][20]
[4190987521][1][20]
[4190989441][1][20]
[4190991361][1][20]
[4190993281][1][20]
[4190995201][1][20]
[4190997121][1][20]
[4190999041][1][20]
[4191000961][1][20]
[4191002881][1][20]
[4191004801][1][20]
[4191006721][1][20]
[4191008641][1][20]
[4191010561][1][20]
[4191012481][1][20]
mts/crc: Unique indices:
[4190976001][4190976002][4190976003][4190976004][4190976005][4190976006][4190976007][4190976008][4190976009][4190976010][4190976011][4190976012][4190976013][4190976014][4190976015][4190976016][4190976017][4190976018][4190976019][4190976020][4190977921][4190977922]...
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22]...
mts/crc: Data size: [20][20][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [400] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [000000013D8490A0] (offset 0)
mts/crc: Seeking to [8381952000]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8490C8] (offset 40)
mts/crc: Seeking to [8381955840]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8490F0] (offset 80)
mts/crc: Seeking to [8381959680]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849118] (offset 120)
mts/crc: Seeking to [8381963520]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849140] (offset 160)
mts/crc: Seeking to [8381967360]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849168] (offset 200)
mts/crc: Seeking to [8381971200]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849190] (offset 240)
mts/crc: Seeking to [8381975040]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8491B8] (offset 280)
mts/crc: Seeking to [8381978880]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8491E0] (offset 320)
mts/crc: Seeking to [8381982720]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849208] (offset 360)
mts/crc: Seeking to [8381986560]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849230] (offset 400)
mts/crc: Seeking to [8381990400]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849258] (offset 440)
mts/crc: Seeking to [8381994240]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849280] (offset 480)
mts/crc: Seeking to [8381998080]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8492A8] (offset 520)
mts/crc: Seeking to [8382001920]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8492D0] (offset 560)
mts/crc: Seeking to [8382005760]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8492F8] (offset 600)
mts/crc: Seeking to [8382009600]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849320] (offset 640)
mts/crc: Seeking to [8382013440]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849348] (offset 680)
mts/crc: Seeking to [8382017280]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849370] (offset 720)
mts/crc: Seeking to [8382021120]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849398] (offset 760)
mts/crc: Seeking to [8382024960]
mts/crc: Single-read chunk, [40] bytes

ans =

20×20 uint16 matrix

Columns 1 through 16

24385 640 688 672 624 608 736 656 608 592 624 640 640 640 688 656
41538 608 640 624 592 592 720 592 704 640 656 624 608 656 608 640
672 656 624 608 688 656 656 656 704 672 592 704 608 640 576 656
640 608 656 656 640 688 608 576 576 704 704 624 624 560 656 656
640 592 704 640 688 608 640 672 640 640 624 640 624 624 592 576
624 672 608 688 672 688 640 656 688 592 704 688 624 576 656 624
592 592 608 576 640 640 656 688 656 624 640 688 608 608 624 592
656 720 704 704 656 640 640 640 656 560 656 656 608 624 576 592
672 608 640 688 640 720 704 592 624 688 592 608 640 672 608 672
608 624 656 624 656 640 672 640 656 640 624 672 688 656 624 672
704 624 672 704 592 640 704 704 656 672 704 688 576 704 640 656
608 592 640 688 656 624 624 640 704 688 672 592 608 640 640 640
640 624 688 640 688 608 592 640 672 624 688 656 688 640 704 592
656 592 672 656 736 672 640 672 624 688 640 704 704 672 752 640
640 592 624 688 672 672 640 624 640 608 576 688 656 656 656 640
608 672 672 608 704 672 640 704 576 672 656 672 688 608 576 640
704 656 656 672 608 640 704 672 736 656 672 656 656 592 704 608
688 624 672 688 640 688 672 704 720 720 656 624 720 704 688 608
704 656 656 560 624 640 656 608 704 656 736 720 656 672 640 688
576 688 560 640 560 592 672 736 688 656 672 672 704 720 688 672

Columns 17 through 20

 640     624     656     656
 624     624     576     608
 624     608     608     640
 592     656     576     656
 640     672     592     656
 560     592     672     672
 656     688     656     576
 640     560     704     560
 592     608     624     672
 608     672     720     656
 624     640     704     608
 624     656     576     560
 672     640     640     656
 560     672     672     608
 688     576     688     624
 656     656     720     656
 624     672     688     688
 624     704     672     656
 640     640     624     656
 656     608     640     656

from mappedtensor.

Hi,

Sorry it's taken me so long to get back to this. It's difficult to debug because I can't reproduce the problem.

When you say "the data is corrupted at the first frame", do you mean the first frame in the stack? Do you mean when skipping pixels (i.e. 1:5:20) or when reading all pixels (i.e. 1:20), or both?

from mappedtensor.

from mappedtensor.

Hi,

Please pull the latest commit to the iss/15 branch, recompile mapped_tensor_shim.c and try the following:

mt = MappedTensor(<file details>);
a = mt(1:20, 1, end);
b = mt(1:5:20, 1, end);
[a(1:5:end) b]

from mappedtensor.

from mappedtensor.

Hi Michael,

I'm glad this works for you now. I'll pull the changes into the master branch. There were a couple of issues performing file seeks beyond 2^32 bytes, and in mis-computing the file offsets when skipping over data.

from mappedtensor.