rtissera / libchdr Goto Github PK
View Code? Open in Web Editor NEWStandalone library for reading MAME's CHDv1-v5 formats.
License: BSD 3-Clause "New" or "Revised" License
Standalone library for reading MAME's CHDv1-v5 formats.
License: BSD 3-Clause "New" or "Revised" License
LRU cache
Threaded read-ahead decompression
ZStandard Codec Support
Follow mamedev/mame#10233
Hi there,
following earlier conversations with @alucryd , I'm fine with resyncing libchdr so that libretro-common, RetroArch and upstream libchdr are all on the same page again.
However, if we may humbly make some tiny requests, there are a couple of things still preventing us from not having to make a single edit to the sources. If these could be addressed then we could get rid of our own fork -
https://github.com/libretro/RetroArch/tree/master/libretro-common/formats/libchdr
For various libretro cores and for RetroArch itself, we set several include dirs for header files. Again, this is a small thing, but the way it is right now in upstream can pose problems on certain compilers/toolchains we care about.
So far, these are the only two things I can think of. They should be relatively small requests hopefully.
If any additions have been made to libchdr in any of the libretro cores that have not been upstreamed yet, hopefully @alucryd can help us out in that endeavor to make sure that all these missing parts are submitted to you as PRs so that we can finally have one central libchdr version with all the features and we no longer have to make these forks.
Thanks for reading.
Paths that aren't representable with the user's current codepage are unable to be accessed. There needs to be a way to open a chd that eventually calls _wfopen
instead of fopen
.
Possible solutions:
chd_open
and chd_read_header
are added that take a FILE*
, allowing users to open their own files with the appropriate call. chd_open_file
is almost this for chd_open
, but it's impossible to set (*chd)->owns_file
from outside of the library, complicating closing of files.chd_open
and chd_read_header
are added that take a wchar_t*
and call _wfopen
on windowschd_open
and chd_read_header
always treat input as utf-8, and internally convert to utf-16 before calling _wfopen
on WindowsI just played around with Delphi and libchdr and had a problem with data alignment as chd_header was 200 with Visual Studio and 196 in Delphi.
Disabling alignment for the data structures in chd.h fixed the problem.
#pragma pack(push, 1)
...
#pragma pack(pop)
libchdr currently makes an assumption that any compressed hunk can fit within an uncompressed hunk. This assumption can be wrong however for completely valid CHDs, if they were created with low compression levels (as implicitly some internal compression header/footer could be present along with the possibly not very compressed data). It can also be trivially wrong for any maliciously created CHD, since the compressed hunk length is just something read from the hunk map.
I'm getting a few crash reports such as:
EXCEPTION_ACCESS_VIOLATION_WRITE
Currently using 2781322
I assume these are due to corrupted CHDs but I haven't been able to reproduce the crash, even when hacking the code.
it looks like an overflow of the decoder->huffnode
array.
May be add this on line 215?:
if (repcount + curnode > decoder->numcodes)
return HUFFERR_INVALID_DATA;
I'd like to consider using it on a python project and it'd be swell if it was on pypi already. My usecase is simple identification of redump dumps of cue/bin and gdi/bin dumps, which necessarily needs to divide the tracks streams, so a way to get the track extents and the stream without file I/O would be nice. The 'data sha1sum' is not sufficient because redump does not have it.
If this ever gets write support, particularly write of delta chds, i'd like to make that xdelta->delta chd child feature in my other RFE if it's not done by this project, because it's a major usability barrier for romhack users, especially if they have to divide the tracks every time. FSM knows MAME upstream probably never will, with the way they think romhacks are for other projects and they 'must' control all dumps themselves.
Would it be possible that it could be made so that a libchdr shared library could be built so it can be installed system wide? My main interest is that I would be able to reuse it in downstream projects instead of requiring libchdr to be built statically. This would greatly help in comparing upstream libchdr with downstream implementations.
Additionally it would also be nice to be able to build libchdr with system version of libraries like flac. At least with flac, a system version works fine with libchdr in RetroArch.
Hello,
PR #93 introduced a compilation error:
pico/cd/libchdr/src/libchdr_flac.c:16:10: fatal error: dr_flac.h: No such file or directory
16 | #include <dr_flac.h>
| ^~~~~~~~~~~
compilation terminated.
make: *** [Makefile:405: pico/cd/libchdr/src/libchdr_flac.o] Error 1
Indeed, dr_flac.h
is in deps/dr_libs/
but the include in src/libchdr_flac.c
is:
#include <dr_flac.h>
Of course I can fix the include path myself, but just wanted to let you know.
The macros core_fseek
and core_ftell
are defined in a way that libretro VFS support is broken. Here is a patch that would solve this:
index c8ec622..338a0f0 100644
--- a/include/libchdr/coretypes.h
+++ b/include/libchdr/coretypes.h
@@ -31,7 +31,11 @@ typedef int8_t INT8;
#define core_file FILE
#define core_fopen(file) fopen(file, "rb")
-#if defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(__WIN64__)
+
+#if defined USE_LIBRETRO_VFS
+ #define core_fseek fseek
+ #define core_ftell ftell
+#elif defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(__WIN64__)
#define core_fseek _fseeki64
#define core_ftell _ftelli64
#elif defined(_LARGEFILE_SOURCE) && defined(_FILE_OFFSET_BITS) && _FILE_OFFSET_BITS == 64```
cmd line used : chdman cratecd -c none -i <src.cue> -o <dest.chd>
As of current libretro libchdr, it does not support uncompressed chd files. updating it will fix this bu not sure if this is known already, but reading audio tracks off from an uncompressed CHD will cause [libretro ERROR] chd_read_sector failed lba=16410 error=14 to be displayed despite game running.
Original post:
https://github.com/libretro/beetle-saturn-libretro/issues/163#issuecomment-696682686
This is the most useful compression type of chd, since it allows multiple cd games to minimize duplication.
cd1 is parent of cd2 and cd3 and cd4 in a game with 4 cds is a option and all the textures and models that are duplicated from cd1 are deduplicated without problems of the compression or the fact the files are 'contiguous' to the OS file system interfering.
or game hacks.
It would be cool if the relationship could be a tree. for instance in a game with 2 cds and hack:
cd1
|\
| \
| \
| \
cd2 cd1hack
|
cd2hack
but that is currently unsupported by chdman (it would be really could if it was contributed upstream though)-
I get this from the lr-picodrive buildbot run for the libretro-build-libnx-aarch64 job:
aarch64-none-elf-gcc -c -opico/cd/libchdr/deps/zstd-1.5.6/lib/decompress/huf_decompress.o pico/cd/libchdr/deps/zstd-1.5.6/lib/decompress/huf_decompress.c -Ipico/cd/libchdr/deps/lzma-24.05/include -Ipico/cd/libchdr/include -I/builds/libretro/picodrive -I platform/libretro/libretro-common/include -I platform/libretro/libretro-common/include/compat -I platform/libretro/libretro-common/include/encodings -I platform/libretro/libretro-common/include/formats -I platform/libretro/libretro-common/include/streams -I platform/libretro/libretro-common/include/string -I platform/libretro/libretro-common/include/vfs -DREVISION=\"-dab9c6e\" -O3 -fomit-frame-pointer -ffast-math -I/opt/devkitpro/libnx/include/ -fPIE -Wl,--allow-multiple-definition -specs=/opt/devkitpro/libnx/switch.specs -D__SWITCH__ -DHAVE_LIBNX -DARM -D__aarch64__=1 -march=armv8-a -mtune=cortex-a57 -mtp=soft -ffast-math -mcpu=cortex-a57+crc+fp+simd -ffunction-sections -Ifrontend/switch -ftree-vectorize -D__LIBRETRO__ -DUSE_LIBRETRO_VFS -O3 -DNDEBUG -I/builds/libretro/picodrive -falign-functions=2 -DREVISION=\"-dab9c6e\" -DUSE_LIBCHDR -Izlib -DEMU_F68K -D_USE_CZ80 -Ipico/cd/libchdr/deps/zstd-1.5.6/lib -Wno-unused
huf_decompress_amd64.S
pico/cd/libchdr/deps/zstd-1.5.6/lib/decompress/huf_decompress_amd64.S:12: fatal error: opening dependency file /pico/cd/libchdr/deps/zstd-1.5.6/lib/decompress/huf_decompress_amd64.d: No such file or directory
12 |
|
compilation terminated.
Other aarch64 targets seem to compile fine though.
Hi,
I'm searching to build a (fake) minimal image that I could use in CI test for Debian.
It can't be a real game, it needs to be free data.
Greetings
As was discussed in libretro/beetle-psx-libretro#587, support for this would be beneficial for various retroarch cores, especially for PS1. Presumably the frontend/core would handle parent locating, so that part at least shouldn't be necessary to implement in this library.
Hi,
This is some fallout from the time64 rebuild in Debian with -Werror=implicit-function-declaration
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1069527
I can certainly fix it in some ugly way, I 'll have a look later.
I guess the same bug can be reproduced on a i386 chroot with the right compilation flags.
https://www.gnu.org/software/libc/manual/html_node/File-Positioning.html#index-ftello
I can't seem to load CHD files larger than 2GiB, all I get is CHDERR_DECOMPRESSION_ERROR
, I used cmake and VS2019 x64 build.
Everything compiles fine and loading smaller CHD files seems to work.
Software: PCSX fork using libchdr: https://github.com/DrUm78/pcsx_rearmed
Hardware: FunKey S handheld device with armv7 NEON
Not sure what commit broke it but PS1 ROMs with .bin or .img format do not load anymore, the user is just sent back to the desktop. Nothing relevant in the logs I got from GMenu2X. That works with 045f2a7 but not with last 9882eea.
.PBP and .CHD still work however.
Interesting thing: last commit works well with .bin/.img on PicoDrive emulator on the same platform but breaks .cso file format too.
Steps:
Hi,
Could you publish a release ?
I'd like to package this for Debian and strip the vendored copies from ares
, mrboom
and retroarch
.
Having versionned releases makes it much easier.
Greetings,
Add VS project files so a static/dynamic library can easily be created.
This would make adding this to other projects easier.
The ssize_t type is undefined when using Visual Studio 2019.
This can be fixed by defining it in coretypes.h or similar as follows:
#ifdef _WIN32
#ifdef _MSC_VER
typedef size_t ssize_t;
#endif
#endif
Using: cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DWITH_SYSTEM_ZLIB=1 -DCMAKE_C_COMPILER=clang -D CMAKE_C_STANDARD=11 -D CMAKE_C_EXTENSIONS=0
-- The C compiler identification is Clang 16.0.2
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found ZLIB: /lib/libz.so (found version "1.2.13")
-- Configuring done (0.2s)
-- Generating done (0.0s)
-- Build files have been written to: /var/tmp/spkg/libchdr-master/build
[ 4%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/Alloc.c.o
[ 8%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/Bra86.c.o
[ 12%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/BraIA64.c.o
[ 16%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/CpuArch.c.o
[ 20%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/Delta.c.o
[ 24%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/LzFind.c.o
[ 28%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/Lzma86Dec.c.o
[ 32%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/LzmaDec.c.o
[ 36%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/LzmaEnc.c.o
[ 40%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/Sort.c.o
[ 44%] Linking C static library liblzma.a
[ 44%] Built target lzma
[ 48%] Building C object CMakeFiles/chdr-static.dir/src/libchdr_bitstream.c.o
[ 52%] Building C object CMakeFiles/chdr-static.dir/src/libchdr_cdrom.c.o
[ 56%] Building C object CMakeFiles/chdr-static.dir/src/libchdr_chd.c.o
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:814:10: warning: implicit conversion from enumeration type 'enum huffman_error' to different enumeration type 'chd_error' (aka 'enum _chd_error') [-Wenum-conversion]
return err;
~~~~~~ ^~~
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2993:6: error: call to undeclared function 'ftello'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
p = core_stdio_ftell_impl(fp);
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2987:32: note: expanded from macro 'core_stdio_ftell_impl'
#define core_stdio_ftell_impl ftello
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2993:6: note: did you mean 'ftell'?
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2987:32: note: expanded from macro 'core_stdio_ftell_impl'
#define core_stdio_ftell_impl ftello
^
/usr/include/stdio.h:718:17: note: 'ftell' declared here
extern long int ftell (FILE *__stream) __wur;
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2994:2: error: call to undeclared function 'fseeko'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
core_stdio_fseek_impl(fp, 0, SEEK_END);
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2986:32: note: expanded from macro 'core_stdio_fseek_impl'
#define core_stdio_fseek_impl fseeko
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2994:2: note: did you mean 'fseek'?
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2986:32: note: expanded from macro 'core_stdio_fseek_impl'
#define core_stdio_fseek_impl fseeko
^
/usr/include/stdio.h:713:12: note: 'fseek' declared here
extern int fseek (FILE __stream, long int __off, int __whence);
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:3031:9: error: call to undeclared function 'fseeko'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
return core_stdio_fseek_impl((FILE)file->argp, offset, whence);
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2986:32: note: expanded from macro 'core_stdio_fseek_impl'
#define core_stdio_fseek_impl fseeko
^
1 warning and 3 errors generated.
make[2]: *** [CMakeFiles/chdr-static.dir/build.make:104: CMakeFiles/chdr-static.dir/src/libchdr_chd.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:118: CMakeFiles/chdr-static.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
People are starting to use the new CHD support in PPSSPP, and are complaining about bad performance.
It turns out that CHDs created with createcd
or createraw
are very slow to access with PSP-sized sector accesses.
How can I detect such CHDs and warn the user or reject them through the API? I guess chd_getheader, but what fields should I check?
Just check hunkbytes == 2048?
I fixed this locally by adding set(CMAKE_POSITION_INDEPENDENT_CODE ON)
to CMakeLists.txt, but I'm not familiar enough with cmake to be certain that would have no drawbacks.
[ 98%] Linking C static library libchdr-static.a
[100%] Linking C shared library libchdr-shared.so
[100%] Built target chdr-static
/usr/bin/ld: libflac-static.a(stream_decoder.c.o): relocation R_X86_64_PC32 against symbol `FLAC__STREAM_METADATA_APPLICATION_ID_LEN' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
CMakeFiles/chdr-shared.dir/build.make:202: recipe for target 'libchdr-shared.so' failed
make[2]: *** [libchdr-shared.so] Error 1
CMakeFiles/Makefile2:221: recipe for target 'CMakeFiles/chdr-shared.dir/all' failed
make[1]: *** [CMakeFiles/chdr-shared.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2
AVHuff is the format used for laserdisks.. its used since v3 afaik
How would one go about doing it to add CHD support to them?
On error, zlib_codec_init
frees the first argument, but that's in various cases not something meant to be freed:
In function 'zlib_codec_init',
inlined from 'cdfl_codec_init' at libchdr_chd.c:794:8:
libchdr_chd.c:2604:17: warning: 'free' called on pointer 'codec' with nonzero offset 184 [-Wfree-nonheap-object]
2604 | free(data);
| ^~~~~~~~~~
libchdr_chd.c:2604:17: warning: 'free' called on pointer 'codec' with nonzero offset 184 [-Wfree-nonheap-object]
In function 'zlib_codec_init',
inlined from 'cdzl_codec_init' at libchdr_chd.c:689:8,
inlined from 'cdzl_codec_init' at libchdr_chd.c:671:18:
libchdr_chd.c:2604:17: warning: 'free' called on pointer 'codec' with nonzero offset 1136 [-Wfree-nonheap-object]
2604 | free(data);
| ^~~~~~~~~~
In function 'zlib_codec_init',
inlined from 'cdlz_codec_init' at libchdr_chd.c:603:8:
libchdr_chd.c:2604:17: warning: 'free' called on pointer 'codec' with nonzero offset 1176 [-Wfree-nonheap-object]
2604 | free(data);
| ^~~~~~~~~~
libchdr_chd.c:2604:17: warning: 'free' called on pointer 'codec' with nonzero offset 1176 [-Wfree-nonheap-object]
This is a copy of a request at the rust port of this library. I know this is 'overreach' of the purpose of the library of trying to reimplement the format, not chdman, but i fully believe this feature would make chd more popular because the vast vast vast majority of existing hacks are in xdelta, not to mention that i doubt that hacks would ever be distributed as chd child 'roms' in spite of being equivalent in not being the actual game, just for appearances sake, because .chd is a actual rom format as far as the internet is concerned. And if it's implemented downstream maybe MAME will actually get moving on this.
Xdelta (with some outliers in certain platforms of ppf) is the standard way to apply romhacks to cds currently in romhacking.net. It would simplify the life of users a lot, and in the case this library is ever used to create a command line application, if chd had the ability to take a .xdelta patch and apply it to a chd, producing not changes to the chd but a 'child' chd.
If you'd like to go beyond, ppf would be nice too. I believe both ppf and xdelta have checksum verification of the result and maybe the source too.
There is a very unfortunate binary format being used for saturn and dreamcast hacking called sega saturn patcher which is a zip file with several kind of hacks, options and transformations¹ wrapped around a executable which doesn't actually work outside of windows or in wine, which could also be a target, except i fucking hate it (see: doesn't work in linux or wine) and think it would be a mistake to support, since it does many transformations of the original dumps to adapt the source to the result (ie: 'tries not to rely on specific rips').
Anyway and since i don't mind a bit of hypocrisy in a good cause, the usecase is, obviously, not having to decompress and reformat back to cue/bin or toc/bin a file before applying a patch. I'm actually unsure if this is actually possible in all cases since afaik, xdelta patches usually apply to a single track, the binary track. Would be kind of a bummer if they couldn't apply because most patches target the redump format with separate tracks and chd outputs the format with 1 single track. Ideally the target track could be autodetected or as fallback, specified.
Regardless because of that doubt and because of the need to uncompress, you can see why this would make users life easier no? 1 command is easier to explain than 3, and it's not the average user that will feel comfortable extracting a chd, then separating a .bin into tracks, then applying the xdelta, then creating a cue file for them before creating the child chd and deleting all the intermediate files.
¹ A occasional needed conversion in romhacks source discs is turning a 'MODE 1 / 2352' binary track into a iso without checksum data 'MODE 1 / 2048' iirc. This sometimes happens in hacks for sega consoles, for 'some' reason (probably the completely terrible dump situation on those consoles and them using cdi in the older ages, so the hackers first turned the cd image to iso before starting to replace files). This is probably the main 'problem' sega saturn patcher attempts to solve - the alternative to adapt the patch to other dumps is to create the patched rom the usual way, extract the files and attempt to glue them to redump/MAME dumps manually, a complicated process that may require rebuilding the cd image in subtle ways. I wouldn't complain about a switch that did this before applying a patch.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.