[Help] Recovering data based on list of flawed sectors. about exfat HOT 13 CLOSED

Hello,

Let's try! If fuse-exfat manages to mount the image, chances are good.

First of all, I'd ensure that FS metadata is OK. Does exfatfsck detect any errors on the rescued image?

from exfat.

Hello,

First of all, thanks for the answer.

About exfatfsck, it indeed found 2 errors, on the same file (named "something.bin"):

# exfatfsck /mnt/temp/wd_1500G.dd
exfatfsck 1.2.4
Checking file system on /mnt/temp/wd_1500G.dd.
File system version           1.0
Sector size                 512 bytes
Cluster size                512 KB
Volume size                1397 GB
Used space                  988 GB
Available space             409 GB
ERROR: cluster 0x2baa02 of file 'something.bin' is not allocated.
ERROR: cluster 0x2baa03 of file 'something.bin' is not allocated.
Totally 6633 directories and 34131 files.
File system checking finished. ERRORS FOUND: 2.

Note: I see the output only give the file's name, without its path inside the FS; what if I have multiple files with the same name? The output lacks its context, I guess.

As you can see, 409 GiB are free, so the other bad sectors might be inside the empty space.

Thanks for support.

from exfat.

About exfatfsck, it indeed found 2 errors, on the same file

That's strange. This error means cluster bitmap is corrupt, which should not be the case because missing parts start at 0x93B04F1000. Cluster bitmap is located at the beginning of the FS.

I see the output only give the file's name, without its path inside the FS; what if I have multiple files with the same name? The output lacks its context, I guess.

You can enable debug output to make exfatfsck print full paths of all files it checks:

diff --git a/fsck/main.c b/fsck/main.c
index 19628e8..f4bbc94 100644
--- a/fsck/main.c
+++ b/fsck/main.c
@@ -26,7 +26,7 @@
 #include <inttypes.h>
 #include <unistd.h>
 
-#define exfat_debug(format, ...)
+// #define exfat_debug(format, ...)
 
 uint64_t files_count, directories_count;
 

I'd do the following:

1. Mount the image (in read-only mode of course).
2. Run find -type f to list all files paths.
3. For each of those paths do dump/dumpexfat -f FILE IMAGE. It'll print fragments occupied by a FILE (see manpage). Note that you need at least exfat-utils 1.2.5 for this.
4. Compare each printed range with ddrescue log.

This will be slow, but unfortunately exFAT does not have a reverse mapping for clusters, i.e. you cannot easily tell which file uses a particular cluster.

from exfat.

I hope you're fine and safe. :)

from exfat.

Hello!

Sorry, it has been a while without news, but I finally got my scripts working.
By the way, I am fine and safe, thank you, I hope you are as well.

So, I made two Python files, which can be reused efficiently (I hope it will help someone else too, so this message is kind of a tutorial) (note: GitHub doesn't want Python files, so I "gzip-ed" them):

1. The first one, named exfat_dump_script.py.gz, is a Python code/script which requires access (in read-only) to the content of the image (tested with Python 3.4 and Python 3.6, should support Python 3.5 too).
   It only requires the path of the (loop) device on which the image has been attached, to seek its mounted point after being mounted (man findmnt(8)); dumpexfat utility requires the access to the content of the medium, so I used the following mount command: mount -o loop,ro /path/to/image.dd /media/mount_point/
   Use losetup -l for finding out which loop device is used by your image (for me, it was /dev/loop0).
   And then, the Python script will write to stdout (don't forget to save it!) the list of files with the positions and sizes in byte for each files, which is the dumpexfat output. Note 1: if a file is not contigu, then you will have multiple lines, which is not a problem. Note 2: I didn't handle at 100% every filename possibilities (having a new line inside its name for example), but I guess exFAT is limited on this feature (unlike other Linux FS), so it's okay (I guess).
   Now, with this output in our possession, we don't need to use the image (except for the final recovery), and can use the second Python code.

2. This second Python file is named seek_matches.py.gz, and it will now need to content of the ddrescue logfile.
   I won't explain everything, but mostly, it starts reading the ddrescue file content, to get the list of flawed sectors based on their status (3rd column, "-" value) and then it compares the ranges of (a piece of) each files with each flawed sectors entries (using the function "_isCrossing").
   If a crossing is detected, then the path of the file is happened to the JSON output, which is displayed at the end of the execution.

And that's it.

But somehow, I am a bit confused, because the last script didn't find any match on my image with the list of files and the list of flawed sectors. I made two files for some tests, and those work as expected.
So, before I copy some corrupted files, I would like please some reviews on those scripts, and maybe if you want, you can ship them as tools for some recovery.

Sorry for providing some not-so-user-friendly pieces of software, you have to open their content to modify some paths (in the end of the files, before the call for the main function).

I hope to hear from you soon, to be sure I didn't miss some information (I was aware about the value in bytes).

Thanks again, and have a nice day.

from exfat.

Oh, I forgot, here are the test files for the 2nd script:
test_dumpexfat.txt
This one has been renamed with the .txt extension, for github…
test_logfile.txt

from exfat.

@relan would you consider adding the linked scripts to contrib/ in this repository? Or even reimplementing them as part of exfatfsck?

from exfat.

would you consider adding the linked scripts to contrib/ in this repository?

No, the author said those scripts didn't work as expected.

Or even reimplementing them as part of exfatfsck?

Recovery utility would be nice to have, but realistically I'll hardly have any time to implement and (more importantly) test it.

from exfat.

Recovery utility would be nice to have, but realistically I'll hardly have any time to implement and (more importantly) test it.

Understood. Either way if you had at least ideas what and how such "recovery" utility should work, just describe it in your own words in a TODO section in Readme.md and put a "help needed" tag to it. The reason is, that someone understanding exFAT up to such a degree is an extremely scarce species 😉, but programmers being able to write something like that according to a technical advice are thousands.

from exfat.

Hello everyone,

Glad those scripts could help someone else.

would you consider adding the linked scripts to contrib/ in this repository?

No, the author said those scripts didn't work as expected.

Uh, no, I said “I [was] a bit confused, because the last script didn't find any match on my image with the list of files and the list of flawed sectors”; which in facts means “no files were corrupted”, which is a good news.
In fact, it might not be a surprise: on an over than 1.5 TB-sized partition, "only" 900 GB of it were used, and only 4M in not contiguous sectors/blocks were unreadable, not on the 900 GB used part, but on the whole filesystem.

However, I might have to rework some part of those Python scripts, IIRC, I think I made some mistakes, or even to update it to Python 3.7 (subprocess module got another update for its run method).
But it actually was working, I don't why you are saying it didn't. :-)

Best regards,

from exfat.

I said “I [was] a bit confused, because the last script didn't find any match on my image with the list of files and the list of flawed sectors”; which in facts means “no files were corrupted”, which is a good news.

Oh, I misunderstood you then.

from exfat.

Hello,
just a quick note to tell you that @thican's scripts did work for me. They found a few hundred impacted files, and a visual check of jpg files did show some holes in them. Pdf files were missing pages.
I had to slightly adapt the script because I ddrescued a whole disk, so I used kpartx to expose the single partition as a loop device, and parted -l to determine the offset to apply between dumpexfat and ddrescue results, which I hard-coded in the second script.

Beforehand I tried ddru_findbad, which use sleuthkit, that support exfat, so it should work, but not for me. I read somewhere a lucky user, but for me ifind failed, and I don't know enough to dig more.
Best regards

from exfat.

I've also succesfully used slightly modified versions of @thican's scripts - note that seek_matches.py (line 37) does not handle non-scraped ranges (status '\') correctly - they should be considered bad sectors as well, since these ranges are not copied by ddrescue.

from exfat.