Hello @gozora
I was recently informed via a SUSE customer issue report
that ebiso silently corrupts files greater or equal 2GiB.
As far as I understand it the "silently" is the actual issue in ebiso because
the actual root cause is a limit of the ISO 9660 file system according to
https://en.wikipedia.org/wiki/ISO_9660#The_2/4_GiB_file_size_limit
How it can happen that ebiso has to deal with files greater or equal 2GiB:
In ReaR it is possible to include the backup in the ISO
with OUTPUT=ISO
and BACKUP_URL=iso://
, cf.
https://github.com/rear/rear/blob/master/doc/rear.8.adoc
When the backup archive file (e.g. backup.tar.gz) is greater or equal 2GiB
ebiso has to deal with files greater or equal 2GiB.
Because of the 2GiB file size limit in the ISO 9660 file system
the backup archive file gets corrupted in the ISO.
The issue is that ebiso does not detect that problem
so for the user it appears af if everything went OK during "rear mkbackup"
until he finds out later during "rear recover" that his backup got corrupted.
So it would be good if ebiso could detect files greater or equal 2GiB
and show an error message in such cases and finish with non-zero exit code.
If ebiso could not detect (with reasonable effort) files greater or equal 2GiB
we could perhaps do something in ReaR to detect such a problem?
Perhaps in general it would be best to have something in ReaR
when making the ISO (regardless what ISO-making tool is used)
that verifies there is no file greater or equal 2GiB?
Also the ReaR recovery system initrd could become greater or equal 2GiB
(e.g. because of accidentally too much in COPY_AS_IS).
Reproducer script ebiso-test.sh
:
#!/bin/bash
#set -x
mkdir -p ebiso-test
cd ebiso-test
echo "2 GiB = 2 * 1024 * 1024 * 1024 = 2147483648"
mkdir -p below2g exact2g above2g
test -s below2g/testfile || dd if=/dev/urandom of=below2g/testfile bs=1M count=2147483647 iflag=count_bytes status=progress
test -s exact2g/testfile || { cp below2g/testfile exact2g/testfile ; echo -n 'x' >>exact2g/testfile ; }
test -s above2g/testfile || { cp below2g/testfile above2g/testfile ; echo -n 'yy' >>above2g/testfile ; }
for subdir in below2g exact2g above2g ; do
pushd $subdir
echo "in $subdir"
ls -l testfile
md5sum testfile
echo 'dummy' >dummy.efi
ebiso -e dummy.efi -o test.iso .
mkdir -p /tmp/ebiso-test
mount test.iso /tmp/ebiso-test
ls -l /tmp/ebiso-test/testfile
md5sum /tmp/ebiso-test/testfile
umount /tmp/ebiso-test
popd
done
Result of the reproducer script:
# ./ebiso-test.sh
2 GiB = 2 * 1024 * 1024 * 1024 = 2147483648
2096103424 bytes (2.1 GB, 2.0 GiB) copied, 37 s, 56.6 MB/s
2047+1 records in
2047+1 records out
2147483647 bytes (2.1 GB, 2.0 GiB) copied, 37.8585 s, 56.7 MB/s
~/ebiso-test/below2g ~/ebiso-test
in below2g
-rw-r--r-- 1 root root 2147483647 Nov 20 09:53 testfile
e964b39bce5e7a3aafb2656b1e357f69 testfile
mount: /tmp/ebiso-test: WARNING: device write-protected, mounted read-only.
-r-xr-xr-x 1 root root 2147483647 Nov 20 09:53 /tmp/ebiso-test/testfile
e964b39bce5e7a3aafb2656b1e357f69 /tmp/ebiso-test/testfile
~/ebiso-test
~/ebiso-test/exact2g ~/ebiso-test
in exact2g
-rw-r--r-- 1 root root 2147483648 Nov 20 09:54 testfile
9ae5c212a37bd45cf5a6b0a8b0c0d246 testfile
mount: /tmp/ebiso-test: WARNING: device write-protected, mounted read-only.
-r-xr-xr-x 1 root root 2147483647 Nov 20 09:54 /tmp/ebiso-test/testfile
e964b39bce5e7a3aafb2656b1e357f69 /tmp/ebiso-test/testfile
~/ebiso-test
~/ebiso-test/above2g ~/ebiso-test
in above2g
-rw-r--r-- 1 root root 2147483649 Nov 20 09:54 testfile
fde433ea7108fbe31fe68fb39bba13f1 testfile
mount: /tmp/ebiso-test: WARNING: device write-protected, mounted read-only.
-r-xr-xr-x 1 root root 2147483648 Nov 20 09:54 /tmp/ebiso-test/testfile
789f0fb931c30af13be3f68bedf0ebba /tmp/ebiso-test/testfile
~/ebiso-test
So in my case it seems to work up to a file size of 2 GiB - 1 (2147483647 bytes).
But problems start with a file size of 2 GiB (2147483648 bytes) because
then the file in the ISO is listed with only 2147483647 bytes and different md5sum
and same for a a file size of 2 GiB + 1 (2147483649 bytes) where
the file in the ISO is listed with only 2147483648 bytes and different md5sum.