askeksa / shrinkler Goto Github PK
View Code? Open in Web Editor NEWCruncher for Amiga executables
License: Other
shrinkler's People
Contributors
Stargazers
Watchers
Forkers
b0nefish jeffv03 bartmanabyss unforeseenocean pfusik remy-luisant kieranhj michaeljtroughton icodein happy-ferret-entertainment rasky brushesmshrinkler's Issues
Header for compressed data files.
When compressing files in --data mode the compressed file has no header.
It would be useful to have the possibility to add a small header, e.g. 4-byte ID + uncompressed size + compressed size.
Windows compilation error
There's a compilation error:
C:\0\src\Shrinkler>make PLATFORM=windows-64
x86_64-w64-mingw32-g++ -Wall -Wno-sign-compare -O3 -I decrunchers_bin cruncher/Shrinkler.cpp -c -o build/windows-64/Shrinkler.o
In file included from cruncher/HunkFile.h:24,
from cruncher/Shrinkler.cpp:22:
cruncher/HunkFile.h: In member function 'bool HunkFile::analyze()':
cruncher/doshunks.h:65:27: error: narrowing conversion of '-2147483648' from 'long int' to 'unsigned int' [-Wnarrowing]
65 | #define HUNKF_FAST (1L<<31)
| ^
cruncher/HunkFile.h:380:9: note: in expansion of macro 'HUNKF_FAST'
380 | case HUNKF_FAST:
| ^~~~~~~~~~
C:\0\src\Shrinkler>x86_64-w64-mingw32-g++ --version
x86_64-w64-mingw32-g++ (Rev6, Built by MSYS2 project) 10.2.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
An easy fix works for me:
C:\0\src\Shrinkler>git diff
diff --git a/cruncher/doshunks.h b/cruncher/doshunks.h
index 190239e..ea4b440 100644
--- a/cruncher/doshunks.h
+++ b/cruncher/doshunks.h
@@ -62,7 +62,7 @@
#define HUNKB_FAST 31
#define HUNKF_ADVISORY (1L<<29)
#define HUNKF_CHIP (1L<<30)
-#define HUNKF_FAST (1L<<31)
+#define HUNKF_FAST (1u<<31)
Out of bounds memory use in make_suffix_array
Found using Valgrind, while trying to debug an unrelated issue:
Original After 1st pass After 2nd pass
==1533662== Invalid read of size 1
==1533662== at 0x1158B0: MatchFinder::make_suffix_array() (MatchFinder.h:85)
==1533662== by 0x10F11A: MatchFinder::MatchFinder(unsigned char*, int, int, int, int) (MatchFinder.h:128)
==1533662== by 0x10C90C: packData(unsigned char*, int, int, PackParams*, Coder*, RefEdgeFactory*, bool) (Pack.h:79)
==1533662== by 0x126B68: DataFile::compress(PackParams*, RefEdgeFactory*, bool) (DataFile.h:44)
==1533662== by 0x1111E0: DataFile::crunch(PackParams*, RefEdgeFactory*, bool) (DataFile.h:118)
==1533662== by 0x10DE90: main2(int, char const**) (Shrinkler.cpp:308)
==1533662== by 0x10EAC1: main (Shrinkler.cpp:411)
==1533662== Address 0x4edbbc0 is 0 bytes after a block of size 1,001,296 alloc'd
==1533662== at 0x483EF3F: operator new(unsigned long) (vg_replace_malloc.c:417)
==1533662== by 0x126796: __gnu_cxx::new_allocator<unsigned char>::allocate(unsigned long, void const*) (new_allocator.h:121)
==1533662== by 0x126730: std::allocator_traits<std::allocator<unsigned char> >::allocate(std::allocator<unsigned char>&, unsigned long) (alloc_traits.h:460)
==1533662== by 0x12633F: std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_allocate(unsigned long) (stl_vector.h:346)
==1533662== by 0x125FC3: std::vector<unsigned char, std::allocator<unsigned char> >::_M_default_append(unsigned long) (vector.tcc:635)
==1533662== by 0x125E45: std::vector<unsigned char, std::allocator<unsigned char> >::resize(unsigned long) (stl_vector.h:940)
==1533662== by 0x1110CF: DataFile::load(char const*) (DataFile.h:89)
==1533662== by 0x10DE39: main2(int, char const**) (Shrinkler.cpp:304)
==1533662== by 0x10EAC1: main (Shrinkler.cpp:411)
==1533662==
==1533662== Invalid read of size 1
==1533662== at 0x1158C0: MatchFinder::make_suffix_array() (MatchFinder.h:85)
==1533662== by 0x10F11A: MatchFinder::MatchFinder(unsigned char*, int, int, int, int) (MatchFinder.h:128)
==1533662== by 0x10C90C: packData(unsigned char*, int, int, PackParams*, Coder*, RefEdgeFactory*, bool) (Pack.h:79)
==1533662== by 0x126B68: DataFile::compress(PackParams*, RefEdgeFactory*, bool) (DataFile.h:44)
==1533662== by 0x1111E0: DataFile::crunch(PackParams*, RefEdgeFactory*, bool) (DataFile.h:118)
==1533662== by 0x10DE90: main2(int, char const**) (Shrinkler.cpp:308)
==1533662== by 0x10EAC1: main (Shrinkler.cpp:411)
==1533662== Address 0x4edbbd7 is 23 bytes after a block of size 1,001,296 alloc'd
==1533662== at 0x483EF3F: operator new(unsigned long) (vg_replace_malloc.c:417)
==1533662== by 0x126796: __gnu_cxx::new_allocator<unsigned char>::allocate(unsigned long, void const*) (new_allocator.h:121)
==1533662== by 0x126730: std::allocator_traits<std::allocator<unsigned char> >::allocate(std::allocator<unsigned char>&, unsigned long) (alloc_traits.h:460)
==1533662== by 0x12633F: std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_allocate(unsigned long) (stl_vector.h:346)
==1533662== by 0x125FC3: std::vector<unsigned char, std::allocator<unsigned char> >::_M_default_append(unsigned long) (vector.tcc:635)
==1533662== by 0x125E45: std::vector<unsigned char, std::allocator<unsigned char> >::resize(unsigned long) (stl_vector.h:940)
==1533662== by 0x1110CF: DataFile::load(char const*) (DataFile.h:89)
==1533662== by 0x10DE39: main2(int, char const**) (Shrinkler.cpp:304)
==1533662== by 0x10EAC1: main (Shrinkler.cpp:411)
==1533662==
This happens pretty much with any input. The input in question was the Shrinkler executable file itself, using -d.
Note that this is not a simple off-by-one, there is a read several bytes further than intended.
Simple Encoder and Decoder in C
Hi,
Shrinkler is an excellent compression algorithm with potential applications outside of the demoscene. Would you consider publishing a plain C version of the encoder and decoder for those unfamiliar with C++?
Since it would require a lot of work to translate the encoder, even just a decoder in C would be great for writing depackers targeting architectures other than the Z80 and 68K.
g++ 9.3 warnings
g++ -Wall -Wno-sign-compare -O3 -I decrunchers_bin cruncher/Shrinkler.cpp -c -o build/native/Shrinkler.o
In file included from cruncher/Shrinkler.cpp:22:
cruncher/HunkFile.h: In member function 'HunkFile* HunkFile::crunch(PackParams*, bool, bool, std::string*, unsigned int, RefEdgeFactory*, bool)':
cruncher/HunkFile.h:874:44: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type '__gnu_cxx::__alloc_traits<std::allocator<Longword>, Longword>::value_type' {aka 'class Longword'}; use assignment or value-initialization instead [-Wclass-memaccess]
874 | memset(&ef->data[dpos], 0, header1_size);
| ^
In file included from cruncher/HunkFile.h:25,
from cruncher/Shrinkler.cpp:22:
cruncher/AmigaWords.h:44:7: note: '__gnu_cxx::__alloc_traits<std::allocator<Longword>, Longword>::value_type' {aka 'class Longword'} declared here
44 | class Longword {
| ^~~~~~~~
In file included from cruncher/Shrinkler.cpp:22:
cruncher/HunkFile.h:923:49: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type '__gnu_cxx::__alloc_traits<std::allocator<Longword>, Longword>::value_type' {aka 'class Longword'}; use assignment or value-initialization instead [-Wclass-memaccess]
923 | memset(&ef->data[dpos], 0, rounded_text_size);
| ^
In file included from cruncher/HunkFile.h:25,
from cruncher/Shrinkler.cpp:22:
cruncher/AmigaWords.h:44:7: note: '__gnu_cxx::__alloc_traits<std::allocator<Longword>, Longword>::value_type' {aka 'class Longword'} declared here
44 | class Longword {
| ^~~~~~~~
g++ -s build/native/Shrinkler.o -o build/native/Shrinkler
Bytestream decoder?
Would it be possible to add an option to Shrinkler to avoid having to fetch one bit at a time during decoding? I'm working on a port to a MIPS platform where bit management in general isn't very fast. In particular this hot loop:
while (ctx->intervalsize < 0x8000) {
ctx->intervalsize <<= 1;
ctx->intervalvalue = (ctx->intervalvalue << 1) | shr_get_bit(ctx);
}in the range decoder seems to take lot of time. If it was possible to instead fetch one byte at a time, that would be great.
upkr has a similar functionality:
https://github.com/exoticorn/upkr/blob/f647449497967dc682b9cd3d47e8857f8fa886f0/c_unpacker/unpack.c#L15-L30
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.