maxpat78 / fattools Goto Github PK

So, I like FATtools, It's a very useful package. But, I would like to see a proper documentation instead of some sample code
for example, a question I've been wondering around is, Is it possible to use a VHD .img file as an actual VHD. Instead of an img that just saves bytes. I.e like simulate a FS. I.e have a virtual C: drive that I can write to read to and use as a regular HDD.
Because the sample examples do not explain it well, nor do they show the possibility, I've heard from other sources that it is possible.
But either you have to use 3rd party packages that are not in python or you have to make your own until I stumbled upon FATtools
now to simplify this whole message the questions I have are the following

1. Will there ever be a proper documentation as opposed to some examples?
2. Is it possible to simulate a HDD like in VMWare?

Hello,

I'm trying to create a FAT32 partition on my SD card. It's attached as removeable media with the ID 2 in diskpart:

DISKPART> list disk

  Datenträger ###  Status         Größe    Frei     Dyn  GPT
  ---------------  -------------  -------  -------  ---  ---
  Datenträger 0    Online          100 GB  1024 KB
  Datenträger 1    Kein Medium        0 B      0 B
  Datenträger 2    Online           29 GB      0 B

DISKPART> select disk 2
DISKPART> clean

Now list partition shows me a message that there are no partitions on the disk, so it should be wiped. To create the partition, I call the mkfat.py script in an administrator cmd shell (I have cloned the entire repo in my working directory):

python mkfat.py \\.\PHYSICALDRIVE2 --fstype fat32

This take some time and then shows the following output:

Successfully applied FAT32 to a 29.81 GiB volume.
1952836 clusters of 16.0 KB.
29.80 GiB free in 1952835 clusters.

FAT #1 @0x4000, Data Region @0xEEA400, Root (cluster #2) @0xEEA400

The Windows Explorer shows a empty 32GB FAT32 partition and it seems to work, I could copy files on it. During my tests, I'd like to re-format it again, so my test case was to copy some files on the newly formatted card, run mkfat.py again and I'd expect to have an empty partition again.

But the second call throws an access denied error:

Traceback (most recent call last):
  File "c:\Users\Daniel\Downloads\FATtools-master\mkfat.py", line 102, in <module>
    format(dsk, dsk.size, params=params)
  File "c:\Users\Daniel\Downloads\FATtools-master\FATtools\mkfat.py", line 484, in fat32_mkfs
    stream.write(bytearray(boot.cluster))
  File "c:\Users\Daniel\Downloads\FATtools-master\FATtools\disk.py", line 338, in write
    self._file.write(s[:full_blocks*512])
  File "c:\Users\Daniel\Downloads\FATtools-master\FATtools\disk.py", line 113, in write
    raise BaseException('WriteFile failed with code %d (%s)' % (GetLastError(), FormatError()))
BaseException: WriteFile failed with code 5 (Zugriff verweigert)

I guessed that I may need to remove the existing partition first. The disk management shows an error this request is not supported:

Using diskpart, neither the disk with the id 2, nor the volume (G:) is shown any more

DISKPART> list disk

  Datenträger ###  Status         Größe    Frei     Dyn  GPT
  ---------------  -------------  -------  -------  ---  ---
  Datenträger 0    Online          100 GB  1024 KB
  Datenträger 1    Kein Medium        0 B      0 B
  Datenträger 3    Kein Medium        0 B      0 B
  Datenträger 4    Kein Medium        0 B      0 B

DISKPART> list volume

  Volume ###  Bst  Bezeichnung  DS     Typ         Größe    Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     D   VBox_GAs_6.  CDFS   CD            58 MB  Fehlerfre
  Volume 1         System-rese  NTFS   Partition     50 MB  Fehlerfre  System
  Volume 2     C                NTFS   Partition     99 GB  Fehlerfre  Startpar
  Volume 3                      NTFS   Partition    524 MB  Fehlerfre  Versteck
  Volume 4     E                       Wechselmed      0 B  Kein Medi
  Volume 6     H                       Wechselmed      0 B  Kein Medi
  Volume 7     I                       Wechselmed      0 B  Kein Medi

I had to close the shell window and start diskpart in a new one

DISKPART> list disk

  Datenträger ###  Status         Größe    Frei     Dyn  GPT
  ---------------  -------------  -------  -------  ---  ---
  Datenträger 0    Online          100 GB  1024 KB
  Datenträger 1    Kein Medium        0 B      0 B
  Datenträger 2    Online           29 GB      0 B
  Datenträger 3    Kein Medium        0 B      0 B
  Datenträger 4    Kein Medium        0 B      0 B

DISKPART> list volume

  Volume ###  Bst  Bezeichnung  DS     Typ         Größe    Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     D   VBox_GAs_6.  CDFS   CD            58 MB  Fehlerfre
  Volume 1         System-rese  NTFS   Partition     50 MB  Fehlerfre  System
  Volume 2     C                NTFS   Partition     99 GB  Fehlerfre  Startpar
  Volume 3                      NTFS   Partition    524 MB  Fehlerfre  Versteck
  Volume 4     E                       Wechselmed      0 B  Kein Medi
  Volume 5     G                FAT32  Wechselmed    29 GB  Fehlerfre

I tried to delete the partition using diskpart, but I cannot select it.

DISKPART> select disk 2
DISKPART> list partition

  Partition ###  Typ               Größe    Offset
  -------------  ----------------  -------  -------
* Partition 1    Primär              29 GB      0 B

When I try select partition 1, I get There was no partition selected. Because of the asterix and articles like this, I also called select partition *. It throws an error that the argument is invalid. delete partition also doesn't work and says there was no partition selected.

I try to disconnect and re-connect the USB card reader, restarted the entire VM and also tried it on a Windows 10 host without virtualization. When restarting the VM, Windows did a disk check on the card. But none of them worked. By playing around with the readonly attribute in diskpart

attributes disk clear readonly

it seems to get fixed one time. I tried to reproduce it for this ticket, doesn't work any more.

The event view of Windows shows multiple warning that the devices (2,3 and 4) have assigned a ID like one or more devices who are connected with the system. It seems that all of them are from the cardreader, which has multiple slots and creates a device for each of them. The entry refers to KB2983588, which refers to install MPIO on Windows Server. However, this is a Desktop (Windows 10 21H1 Build 19043) so Dism /online /enable-feature:MultipathIo throws an error that this feature is unknown. I don't know if this warning is problematic.

Do you have any idea why FATtools cause this problems? It seems caused by using this lib, I have flashed multiple images to the card with the same card reader. And I also created 2 times a FAT32 partition with the graphical Windows tools, it worked without any issues.

Edit

I tried it with another card reader. A simple Micro-SD to USB converter. I don't get the warnings about device ids in the logs. But it's still not possible to delete the existing partition. The same error occurs when marking the partition as active. However, all other operations in the computer management GUI seems to work: Changing/deleting the drive letter as well as formatting the partition. Deleting is still not possible. Even after formatting the card with another file system like exFAT or NTFS.

Edit 2: Strange workaround

It was possible to delete the partition with a strange workaround: Formatting it using the GUI without the checkbox for fast formatting, which is checked by default. Then right click on the running formatting in the computer management console and abort it. After that, the partition had changed to type RAW. I could delete it and re-create a FAT32 partition. Deleting seems now possible.

Now I've deleted all partitions (using the second card reader now) and re-run mkfat.py with the same parameters. Just changed PHYSICALDRIVE2 to PHYSICALDRIVE1 since this has changed with the other card reader. The partition gets created, however I'm still unable to delete the partition like before 😢

Edit: Partially working code

I tried to make a custom Python script to break down the problem:

from FATtools import utils, partutils
from FATtools.Volume import vopen
from FATtools.mkfat import *

# Depending on the used cardreader, we have different device ids:
# 1 -> Simple USB-to-Micro-SD-Adapterm  2 -> UGREEN card reader with multiple slots
#dev = "\\\\.\\PHYSICALDRIVE1"
dev = "\\\\.\\PHYSICALDRIVE2"

disk = vopen(dev, 'r+b', 'disk')
print("Disk opened")

gpt = partutils.partition(disk)
# Throws KeyError: '\\\\.\\PHYSICALDRIVE2'
#disk.close()
print("Created partition")

# Also tried partition1 since "list partition" in diskpart shows the first partition as partition 1
# Throws the same error and gave me a 2048 GB GPT protection partition...
# Could not be removed in the computer management UI, but by re-running the script with partition0
part = vopen(dev, 'r+b', 'partition0')
print("Opened partition for formatting")

# Throws BaseException: WriteFile failed with code 5 (permission denied)
exfat_mkfs(part, (gpt.partitions[0].u64EndingLBA-gpt.partitions[0].u64StartingLBA+1)*512)
print("Formatting done")

This seems to work, I can delete the partition without any error. But I wondered that my partition is formatted, even when the exfat_mkfs call throws an access denied error? I thought that my partition would be created but not formatted after this error.

It seems that something is wrong here. I copied some data on the card, deleted the partition in the UI and re-run the script again. Instead of an empty FAT32 card, it contains all the data before deleting the partition!

KeyError on disk.close()

It took out that win32_disk.open_handles in disk.py is empty, since partutils.partition already closes the disk. So it seems for me that this call in mkfat.py is wrong. I just need to re-call vopen to getch the first partition, which should exists after partutils.partition was called successfully. But the formatting still doesn't work and throws code 5 access denied. The program runs as Administrator, so I can't understand this.

Having some issues with sortby, so tried to just use example code, substituting just my drive letter:

from FATtools.Volume import *

# Assuming we have DirA, DirB, DirC in this disk order into X:
root = vopen('G:', 'r+b')

new_order = '''DirB
DirC
DirA'''

root._sortby.fix = new_order.split('\n') # uses built-in directory sort algorithm
root.sort(root._sortby) # user-defined order, in _sortby.fix list
root.sort() # default ordering (alphabetical)

Error returned is:

Traceback (most recent call last):
  File "d:\data\labs\karaoke\testsort.py", line 11, in <module>
    root.sort(root._sortby) # user-defined order, in _sortby.fix list
  File "C:\Program Files (x86)\Python38-32\lib\site-packages\FATtools\FAT.py", line 1850, in sort
    names.sort(key=functools.cmp_to_key(by_func)) # user order
  File "C:\Program Files (x86)\Python38-32\lib\site-packages\FATtools\FAT.py", line 1818, in _sortby
    return cmp(Dirtable._sortby.fix.index(a), Dirtable._sortby.fix.index(b))
NameError: name 'cmp' is not defined

Can you let me know what is wrong here?
Also, I was getting this same error with my own code, which is why I tried your sample.
In my code I am trying to sort files, not directories in a certain order. Is there a difference for this, or can I just feed it the filenames as you have supplied DirA, DirB, DirC, etc...
Also, I have these file names contained in a python list and reformatted them to a string with \n character to more closely fit with your sample. This resulted in the same errors as above, so I think there is more to it than my code.
However, is there a way to feed it a list directly without having to parse it to a string with defined separator?

thanks.

In reference to #2 I wanted to ask if it would be able to provide FATtools as package via PyPI. The standard way to distribute package is as a pip package in either sdist or wheel format. The benefit would be that this package could then be installed easily via pip. Generally speaking, simply providing a proper setup.py, building the packages and then uploading them via twine would be the standard way of distribution.

I'd be happy to support, but due to legal limitations from my employer, the clearance for me doing that might take a while.

I create new virtual machine in VirtualBox with VDI dynamic virtual hard disk.
I mount image of this hard disk with ImDisk in my host windows 8.1 (without creating any file system).
I read this disk in python:

VOLUME_PATH = r'\\.\D:'
with open(VOLUME_PATH, 'r+b') as disk:
    disk.seek(0)
    data=disk.read(512)
data

and I get b'<<< Oracle VM VirtualBox Disk Image >>>\n\x00\x00 ...
Then I unmount this image from host windows.
Then I write my own code (like nodos/nodos in your repository) to image with your library:

from FATtools.Volume import *
disk = vopen('VirtualBox VMs\\ddos\\ddos.vdi', 'r+b', 'disk')
disk.seek(0)
disk.write(b"\xb8\x03\x00\xcd\x10\xb4\x0a\xb0\x71\xbb\x00\x00\xb9\x02\x00\xcd\x10\xeb\xfe")
disk.close()

Then I again mount ddos.vdi with ImDisk to host OS and read it
and I get nothing changes in this vdi, I get again b'<<< Oracle VM VirtualBox Disk Image >>>\n\x00\x00 ...
(VirtualBox also print, that no bootable media, when I try to start it with this image)
Where is my wroted data?

This is more of a question perhaps than an issue (though possibly a feature request??).

1. Can this sort files or only directories? I am looking to sort a list of files alphabetically in the FAT table.
2. If so, do I need to generate the list of files and create an alphabetized list first and feed the list order to this? Or is there a built in function to simply say sort by filename?

More info:
I have been using DriveSort (http://www.anerty.net/software/file/DriveSort/?lang=en) to do these kinds of operations against my USB stick for my car MP3 system. It works great and based on its description I was looking for a python alternative for a project I am looking to do for my daughter's Karaoke player which also needs MP3 sorted.
While I can continue to use DriveSort for this, I have to manipulate additional file data which I am thinking about using python to do (read JSON, modify, and dump back into .ini files for each song).
Since I need to do this in python I was hoping to skip having to use DriveSort and have python also sort the FAT table alphabetically.

thanks

The readme states: Manually install it under site-packages.
However, when I put the repo's root dir in my site-packages, e.g. py -m mkvdisk tells me that C:\Program Files\Python39\python.exe: No module named mkvdisk - unless I am in the folder of the package (but I assume this is not what should be required?).

But even then, I am unable to run samples\Reorder.pyw:

Traceback (most recent call last):
  File "C:\Program Files\Python39\Lib\site-packages\FATtools\samples\Reorder.pyw", line 17, in <module>
    from FATtools.Volume import *
ModuleNotFoundError: No module named 'FATtools.Volume'

Any help/more hints would be highly appreciated. Cheers