Extend imgmount to support directories in place of the BINARY track in cues about dosbox-staging HOT 23 CLOSED

If space saving is most important you should look into ECM files.
These have their ECC data removed .
https://github.com/alucryd/ecm-tools

*. It an open source tool that runs in GNU/Linux, Windows and Mac OS X.

*. It achieves the size reduction by removing those error correction data from your disc images.

*. Supports a lot of popular disc image formats such as: ISO, NRG, BIN/CUE, CDI, CCD or any format that uses the Raw sector copying.

*. Has two built in tools for compressing and decompressing.

from dosbox-staging.

This is not about that. It's about sharing data between DOS and scummvm and other engines when the DOS game engines can tolerate it. Removing error correction data is almost pointless in compassion to the duplication of data that this requires, because re-implementations of game engines can't use isos.

from dosbox-staging.

@Akuma-Git , awesome suggestion regarding ECM. I've spun this out into its own issue.

I'm a huge fan of ECM because it's reversible (ie: back to original BIN format), saves a considerable amount of disk space without any codec or algorithm needed, and permits efficient 2nd level compression (say, at the filesystem level) that's otherwise squandered due to the ECC portion of each sector being virtual incompressible.

I will run some quick numbers when I'm back home and able, but the one-two punch of ECM offers considerable space saving.

from dosbox-staging.

@kcgen I mean, in reality games are archived as BIN/CUE and thats it. I think until we get further requests for this functionality it's safe to close.

from dosbox-staging.

I opened a bug report with considerably less detail (because i don't expect anything to happen there) on upstream that is basically the same request here https://sourceforge.net/p/dosbox/feature-requests/142/

You can try to upstream this if you end implementing it and say it closes that RFE.

from dosbox-staging.

BTW, there is another case where the host and the guest are opaque to eachother that causes problems in sharing data between scummvm and dosbox; windows 95 and up games, where the user has to put the games into fat16/32 images.

That case has no good solution in the dosbox realm, since the windows filesystem is controlled by the OS, and no one did a windows 95 to host bridge filesystem driver; for good reasons.

I hope the Hangover project eventually solves that while keeping ARM64 portability.

from dosbox-staging.

I don't think I've ever seen a game that definitely needed to be stored in fat16/32 image to work… Do you mean a situation when Windows 95 is running inside DOSBox and a game is started inside it? Personally, I would prefer to use Wine for such situations… but I haven't tried running Window 95 inside dosbox yet (I plan to investigate it when importing relevant patches from ECE).

from dosbox-staging.

Do you mean a situation when Windows 95 is running inside DOSBox and a game is started inside it

Yes

Personally, I would prefer to use Wine for such situations

I do, except for a certain era of games (where the games are logically related to their prequels or era, things like Discworld Noir - which doesn't work on wine too), or when i want the game to be portable to ARM, to play on a raspberry pi 4 for instance.

It's actually quite a lot of trouble to run a dosbox win95 setup, trouble which i'd prefer not to do and just use wine always, especially because of file corruption of FAT or the trouble of making and mounting raw hdd images.

What i do is i have a squash filesystem (compressed, read only) for all my games; then i layer a copy on write filesystem on top for those few platforms that need write permission on those dirs, with a script i did myself on here: https://code.launchpad.net/~i30817/+archive/ubuntu/dosbox-patched

then another script runs that collects all the *.conf files on the mounted 'copy on write' filesystem and starts the games with dosbox -userconf -conf dosbox.conf.

It's complex, as you can probably tell if you read both scripts:
https://bazaar.launchpad.net/~i30817/dosbox/packaging/view/head:/debian/dosboxmount-src/copy-on-write-drive

https://bazaar.launchpad.net/~i30817/dosbox/packaging/view/head:/debian/dosboxmount-src/dosboxmount

Then after all this comes the part that is windows 95 focused. What i do there is to copy the img file containing Windows 95 to the linux ramdisk at /dev/shm and boot from there (instead of using the copy on write disk) in order to try to avoid FAT32 'errors' corrupting the OS and reset it every run. Then i mount the game disk apart and write a shortcut to start the game automatically on the 'ramdisk OS'

For instance this is for discworld noir:

[autoexec]
@echo off
ECHO �[1;33m
ECHO This is a �[1;31mwindows 95�[1;33m game
ECHO Windows 95 runs badly in dosbox because it requires low level disc access. 
ECHO The game and OS are in harddisc images and shutting down the OS improperly 
ECHO (such as by CTRL+F9) can cause hard disc corruption very easily in windows 95 
ECHO (more for the OS than the game since it's in FAT16 for dosbox access and it holds the paging file)
ECHO The game disc image is much less sensitive, since it's FAT32
ECHO Therefore the OS is copied to a �[1;31mLINUX ONLY�[1;33m volatile memory mount 
ECHO in order to 'reset' any errors
ECHO This makes this and any other windows 95 games in this collection linux only
ECHO �[1;31mSavegames go to the game drive otherwise you will lose them!�[1;33m
pause
ECHO �[0m
@echo on
mount v /dev/shm/
IF NOT EXIST v: quit
mount w ../../Windows/
copy w:\win95.img v:\win.img
mount -u w
mount -u v

#fat32 filesystem
imgmount 3 "game/noir.img" -size 512,63,64,640 -t hdd -fs none

#imgmount c win.img -size -size 512,63,8,731
imgmount c "/dev/shm/WIN.IMG" -size 512,63,8,731

#autostart for game
mount g game
copy g:\startup.bat c:\WINDOWS\STARTM~1\PROGRAMS\STARTUP\
mount -u g

boot -l c

Unfortunately as the message says, this ramdisk trick makes it linux only, though dosbox not treating / and \ the same on conf files host paths already does that (something that shouldn't happen imo). Errors can still occur in the 'game hdd' but it's much less of a risk than if both the OS and the game were in the same hdd (and you can reset the game disk by deleting the copy on write cache of it, though you lose the saves).

It also saves some space not to duplicate the OS on every game hdd and keep it separate. However it also means that games that are badly misbehaved and put dlls on the OS dir are not segregated, so adding new games means that i have to mount/modify the 'original' OS hdd and it might break some other game (hasn't yet).

Most of the problems come from Windows managing the filesystem... i can't wait until Hangover works well and i can discard dosbox for this.

from dosbox-staging.

Also needless to say, the copy on write drive works much much better if the game you're playing is on 'transparent' host filesystem (well a squashfs in this case) like normal DOS games. In the case where the (often large, the largest game i have doing this is Black Dahlia which is 6 gb) game is on a FAT hdd image, the copy-on-write is of the whole hdd image...

It basically forces you to wait a few seconds at the start the first time (when the OS tries to mount the filesystem or when the OS open a file as r+w, not sure - the only thing i know is that it always happens at the start).

I've read that dosbox gained a copy-on-write system recently but i don't know how it works myself yet. The copy-on-write-drive is useful for more than dosbox anyway (retroarch computer cores that want to write for instance).

I suspect it might not work for Windows 95 hdd images because those are not controlled by the DOSBOX DOS 'filesystem'.

from dosbox-staging.

@i30817, very cool! I've done something very similar for my rpi3+ DOSBox collection: custom kernel w/ squashfs + zstd and overlay FS. Each game is squashfs'd plus a writable overlay mount. ISOs are part of the squash (usually compress nicely) while the Opus audio tracks ride alongside in a subdirectory.

It makes for a very tidy host layout, having a one directory per game with a config file, squash file, subdir of writes, and optionally a subdir with audio tracks.

One of the challenges I noticed was in dealing with games that sloppily open every file 'rw' even if they never write to it; such as huge .dat files and such. In this case, overlayFS saves a copy of these files simply based on the 'rw' open flag (cluttering the 'writes' directory and wasting space); so I have a patch for DOSBox that tolerates read-only permissions on host files for which it translates the game's 'rw' flags to 'r' to pacify overlayFS, and only throws and error if the game actually attempts to write to the file.

This lets me write-protect (or read-only) almost every game file except for the handful of config/savegame/high score files. It was pretty tedious though working through each game :-)

Great suggestion to allow a "Directory" type in the CUE file for all the reasons you mentioned.

from dosbox-staging.

One of the challenges I noticed was in dealing with games that sloppily open every file 'rw' even if they never write to it; such as huge .dat files and such. In this case, overlayFS

I didn't end up using overlayfs, from what i remember because it needs root (when i want to mount at runtime, not on boot), which unionfs-fuse didn't.
FUSE is the only linux framework that didn't put itself in my way with a 'password required please'. It kind of turned me off from using the kernel stuff, though thankfully squashfs doesn't have that problem if you mount it on fstab.

I don't think unionfs-fuse (which is what i used) has that problem of 'every file opened r+w is copied' or at least i didn't notice that on the few i tried (I spent more time thinking about the config than actually replaying, getting old).

I'm intrigued that you were able to use squash for individual games. I mounted my (large, a absolute bitch to recreate) collection into a single squash, because i can mount without asking for root on the startup of the computer, but if you were able to mount multiple hundreds of games on a folder hierarchy transparently at runtime, i'm interested to know how?

from dosbox-staging.

@i30817 so many good ideas in this thread, thanks for sharing :) Me and @krcroft discussed some ideas about implementing ROM-like files for DOS game distribution (I think at some point we'll need to distill those thoughts into an RFC document and start test implementations to see what works).

Regarding FAT32 filesystem corruption - I would need to investigate more, but I wonder if properly integrating patches with long filenames support would solve large chunk of issues…

though dosbox not treating / and \ the same on conf files host paths already does that (something that shouldn't happen imo).

I agree! I solved this problem once already in Boxtron and intend to port my solution to dosbox-staging; I talked about this with upstream maintainers once already, and they blatantly refused to fix it.

from dosbox-staging.

I agree! I solved this problem once already in Boxtron and intend to port my solution to dosbox-staging; I talked about this with upstream maintainers once already, and they blatantly refused to fix it.

It's just stupid... the only thing that makes conf files habitually non-portable (if you use relative paths) is that. And i'm speaking as someone that deliberately made some conf files not portable to use the shm ramdisk.

It would actually suffice to make '/' work on both unix and windows (windows already accepts '/' as a path separator in some apis IIRC, so it's possible it doesn't even need translation)

GOG does very weird things involving putting every file in the same dir or a 'direct descendant only' (so it doesn't need a separator on the relative path) because of this. It's the reason their conf files are in the main game dir iirc (which i refuse to do, mixing data for dosbox with the game files named for the DOS filesystem is just weird), and the reason they stuff the game isos in the same dir as the game (and rename it to a invented 'hdf' sometimes, iirc).

from dosbox-staging.

For cross-platform .conf files I created #102 - discussion concerning paths let's continue there.

from dosbox-staging.

Seems relevant to our discussion here: https://www.vogons.org/viewtopic.php?f=41&t=34642

from dosbox-staging.

@i30817

It's actually quite a lot of trouble to run a dosbox win95 setup, trouble which i'd prefer not to do and just use wine always, especially because of file corruption of FAT or the trouble of making and mounting raw hdd images.

This is probably obvious, but have you tried a more general purpose pc emulator like pcem for this? Once you get into having to deal with drivers and hdd images, a lot of the advantages of dosbox get thrown out. I don't think Windows 9x ever worked well on dosbox. I seem remember having to do at least some of the installation on dosbox-x because dosbox just didn't emulate enough of the hardware to get through it?

There are definitely reasons to use an emulator over wine for this era of games. Some of them use ddraw in weird ways that aren't supported in wine, for example. Some distros might disable 16-bit support, as well, for whatever security reasons.

from dosbox-staging.

Seems relevant to our discussion here: https://www.vogons.org/viewtopic.php?f=41&t=34642

This patch (and it's 2 alternatives, including the 'very simple one') are currently broken in upstream and are the reason my ppa autobuild is not autobuilding atm.

edit: the simple patch was fixed, and my ppa is using it now.

This is probably obvious, but have you tried a more general purpose pc emulator like pcem for this?

I don't like the complexity of pcem, so no i didn't. I wanted to 'run win3.11 and win95 games automatically and in the same mechanism as the rest' and challenged myself to make it happen in dosbox. I did, though there are definitely problems (like upstream refusing to extend images to > 2 gb for no reason but stubbornness to increase some variables from 16 bits).

Anyway this is not relevant to this RFE (indeed, you can't even see mounted isos in DOSBox in windows 95, so what i usually do is install games on wine then transfer them to a hdd), so maybe any discussion of windows 95 in dosbox maybe should happen on another open issue.

from dosbox-staging.

Quick brain dump after looking through the code.

• directories are handled by the cdromDrive class, which adds some thin file IO functions and dummy MSCDEX audio functions around the localDrive class, which is a simple pass-through to the local filesystem (allowing the DOS-side to read and write files on the host-side).
• images/ISOs/CUEs are handled by the IsoImage and CDROM_Image classes, which operate on sectors, frames, and provide genuine MSCDEX audio functions that tie into raw binary data (if cue+img) or to audio decode rountines (if cue+iso+oggs/flac/opus).

Also, only the iso_image class actually performs the cue parsing.

So to fluidly handle either a directory or an ISO file as the 'data portion' (first track) of a CD mount, followed by zero or more audio tracks, will require these two deeply divided class trees be refactored.

First, their data handling needs to be split off from their MSCDEX audio handling (which right now are entangled).
This way, bonafide audio tracks can be freely added to either the cdromdrive or isoimage classes, both of which will then only handle their data portion.

Finally, the cue handling needs to be lifted out of the isoimage class and moved earlier in the imgmount call tree, so it can determine how to construct the necessary object (dir or iso? fake-mscdex or real audio mscdex?)

from dosbox-staging.

As a question of 'separation of concerns', should this interface be user accessible only from 'imgmount' for cues or with yet another mounting mode for cds/cues in 'mount', or something else?

My bet with this is that you want to use imgmount, but then there is that extra command mount has that imgmount doesn't because it doesn't need it (-label) that dir mounts absolutely need sometimes.

Making mount -t cdrom "a_cue.cue" accept a file is maybe 'simple', but then people might get confused and use a normal cue and it doesn't work.

There is a choice here, allow imgmount to override the iso -label (or 'load' in the case of directory mounts) or allow 'mount' to mount cues in -t cdrom and fail if they're not directory cues.

From what little i've read of dosbox code both those commands are sorta kind of, defined in the same file, so it shouldn't be that important for code, more for user 'surprisingness'.

I believe imgmount is the best option, but then you also need to handle/add '-label' there.

from dosbox-staging.

Yes, they are defined in the same file, but they are written… ekhm… without reusability in mind (to put it mildly) :(

Also, just a heads up: we had a code conflict in this area recently, I suspect upstream might be planning some development in mount/imgmount implementations.

Recently I tried to implement #92 and reduce code complexity in that area, but this code is so tightly coupled, that it's hard to finish this work without introducing code duplication :(

from dosbox-staging.

Absolutely ask if there are plans for anything and delay as long as something is in the works first. I've waited several years with duplicate data for scummvm on the few games affected, i can wait a bit more.

from dosbox-staging.

I think I am persona non grata at Vogons forum, DOSBox maintainers generally stopped talking to me.

from dosbox-staging.

@i30817 , agreed. In cases where the ECC portion is unused in a BIN/CUE pair (such as is the case in some forms of copy protection), then it makes little sense to retain it. Being able to lift out track 1 as a flat directory and subsequent tracks as FLACs would be the most transparent end state for a CDROM and allow for compatibility with Scummvm. (Unfortunately though this will involve a pretty deep refactoring of the filesystem code; patches welcome :-) )

from dosbox-staging.