Hi flanglet, wanted to share with you my results - in order to see where Kanzi is positioned among some top performers.
Oh, and before I forgot, is the name of that famous monkey used for your project?
Okay, crunching DNA...
D:\TEXTORAMIC_benchmarking_2019-Apr-29>dir *.kanzi
06/30/2018 02:43 PM 382,715,759 SILVA_123_SSURef_Nr99_tax_silva.fasta.MODE1.BLOCK128MB.Kanzi
06/30/2018 12:41 PM 383,311,150 SILVA_123_SSURef_Nr99_tax_silva.fasta.MODE1.Kanzi
06/30/2018 02:44 PM 255,457,674 SILVA_123_SSURef_Nr99_tax_silva.fasta.MODE2.BLOCK128MB.Kanzi
06/30/2018 12:41 PM 261,736,754 SILVA_123_SSURef_Nr99_tax_silva.fasta.MODE2.Kanzi
06/30/2018 02:45 PM 251,208,169 SILVA_123_SSURef_Nr99_tax_silva.fasta.MODE3.BLOCK128MB.Kanzi
06/30/2018 12:47 PM 257,686,441 SILVA_123_SSURef_Nr99_tax_silva.fasta.MODE3.Kanzi
06/30/2018 02:49 PM 58,966,467 SILVA_123_SSURef_Nr99_tax_silva.fasta.MODE4.BLOCK128MB.Kanzi
06/30/2018 12:50 PM 125,295,300 SILVA_123_SSURef_Nr99_tax_silva.fasta.MODE4.Kanzi
06/30/2018 02:52 PM 56,291,224 SILVA_123_SSURef_Nr99_tax_silva.fasta.MODE5.BLOCK128MB.Kanzi
06/30/2018 12:52 PM 119,932,618 SILVA_123_SSURef_Nr99_tax_silva.fasta.MODE5.Kanzi
06/30/2018 02:58 PM 55,168,123 SILVA_123_SSURef_Nr99_tax_silva.fasta.MODE6.BLOCK128MB.Kanzi
06/30/2018 12:55 PM 118,568,658 SILVA_123_SSURef_Nr99_tax_silva.fasta.MODE6.Kanzi
In above box, I tested the default 1MB block and 128GB, for modes 1..3 the gain is poor, why so? Maybe you decided to go after symmetricalness!?
D:\TEXTORAMIC_benchmarking_2019-Apr-29>DUMP_HEX_header.exe SILVA_123_SSURef_Nr99_tax_silva.fasta
First 512 bytes of: 'SILVA_123_SSURef_Nr99_tax_silva.fasta' 947,966,398 bytes:
0000 3e 48 50 34 35 31 37 34 39 2e 36 2e 31 37 39 34 20 45 75 6b 61 72 79 6f 74 61 3b 4f 70 69 73 74 >HP451749.6.1794 Eukaryota;Opist
0020 68 6f 6b 6f 6e 74 61 3b 4e 75 63 6c 65 74 6d 79 63 65 61 3b 46 75 6e 67 69 3b 44 69 6b 61 72 79 hokonta;Nucletmycea;Fungi;Dikary
0040 61 3b 42 61 73 69 64 69 6f 6d 79 63 6f 74 61 3b 50 75 63 63 69 6e 69 6f 6d 79 63 6f 74 69 6e 61 a;Basidiomycota;Pucciniomycotina
0060 3b 50 75 63 63 69 6e 69 6f 6d 79 63 65 74 65 73 3b 50 75 63 63 69 6e 69 61 6c 65 73 3b 50 75 63 ;Pucciniomycetes;Pucciniales;Puc
0080 63 69 6e 69 61 63 65 61 65 3b 50 75 63 63 69 6e 69 61 3b 50 75 63 63 69 6e 69 61 20 74 72 69 74 ciniaceae;Puccinia;Puccinia trit
00a0 69 63 69 6e 61 0a 43 43 55 47 47 55 55 47 41 55 43 43 55 47 43 43 41 47 55 41 47 55 43 41 55 41 icina.CCUGGUUGAUCCUGCCAGUAGUCAUA
00c0 55 47 43 55 55 47 55 43 55 43 41 41 41 47 41 55 55 41 41 47 43 43 41 55 47 43 41 55 47 55 43 55 UGCUUGUCUCAAAGAUUAAGCCAUGCAUGUCU
00e0 41 41 47 55 41 55 41 41 41 43 41 41 43 55 41 55 41 43 41 47 55 47 0a 41 41 41 43 55 47 43 47 41 AAGUAUAAACAACUAUACAGUG.AAACUGCGA
0100 41 55 47 47 43 55 43 41 55 55 41 41 41 55 43 41 47 55 55 41 55 41 47 55 55 55 41 55 55 55 47 41 AUGGCUCAUUAAAUCAGUUAUAGUUUAUUUGA
0120 55 47 41 55 41 43 43 55 55 41 43 55 41 43 41 55 47 47 41 55 41 41 43 55 47 55 47 47 55 41 41 55 UGAUACCUUACUACAUGGAUAACUGUGGUAAU
0140 55 43 55 41 47 41 47 0a 43 55 41 41 55 41 43 41 55 47 43 55 47 41 41 41 41 47 43 43 43 43 41 41 UCUAGAG.CUAAUACAUGCUGAAAAGCCCCAA
0160 43 43 55 55 55 47 47 41 41 47 47 47 47 55 47 55 41 55 55 55 41 55 55 41 47 41 55 41 41 41 41 41 CCUUUGGAAGGGGUGUAUUUAUUAGAUAAAAA
0180 41 43 43 41 41 55 47 47 43 55 55 55 43 47 47 47 55 43 55 43 55 55 55 47 0a 47 55 47 41 55 55 43 ACCAAUGGCUUUCGGGUCUCUUUG.GUGAUUC
01a0 41 55 41 41 55 41 41 43 55 55 43 55 43 47 41 41 55 43 47 43 41 55 47 47 43 43 55 55 47 55 47 43 AUAAUAACUUCUCGAAUCGCAUGGCCUUGUGC
01c0 43 47 47 55 47 41 55 47 43 55 55 43 41 55 55 43 41 41 41 55 41 55 43 55 47 43 43 43 55 41 55 43 CGGUGAUGCUUCAUUCAAAUAUCUGCCCUAUC
01e0 41 41 43 55 55 55 43 47 41 0a 55 47 47 55 41 47 47 41 55 41 47 41 47 47 43 43 55 41 43 43 41 55 AACUUUCGA.UGGUAGGAUAGAGGCCUACCAU
This file is taken from Kirill's SCB:
http://kirill-kryukov.com/study/naf/
https://github.com/KirillKryukov/naf
http://kirr.dyndns.org/sequence-compression-benchmark/
The used 'SILVA' corpus is here:
https://www.arb-silva.de/no_cache/download/archive/release_132/Exports/
The testmachine is my slowest laptop i5-2430M @2.40GHz 16GB DDR3 1333MHz, Windows 7:
D:\TEXTORAMIC_benchmarking_2019-Apr-29>lzbench173 -c4 -i1,15 -o3 -etornado,16/blosclz,9/brieflz/crush,2/csc,5/density,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lz4hc,1,12/lizard,19,29,39,49/lzf,1/lzfse/lzg,9/lzjb/lzlib,9/lzma,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,1,9/xz,9/yalz77,12/yappy,99/zlib,1,5,9/zling,4/shrinker/wflz/lzmat SILVA_123_SSURef_Nr99_tax_silva.fasta
lzbench 1.7.3 (64-bit Windows) Assembled by P.Skibinski
The results sorted by column number 4:
Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename
lzlib 1.8 -9 0.57 MB/s 115 MB/s 947966398 49927392 5.27 SILVA_123_SSURef_Nr99_tax_silva.fasta
tornado 0.6a -16 0.67 MB/s 317 MB/s 947966398 51876911 5.47 SILVA_123_SSURef_Nr99_tax_silva.fasta
lzma 16.04 -9 0.92 MB/s 219 MB/s 947966398 53482886 5.64 SILVA_123_SSURef_Nr99_tax_silva.fasta
xz 5.2.3 -9 1.01 MB/s 169 MB/s 947966398 53483925 5.64 SILVA_123_SSURef_Nr99_tax_silva.fasta
csc 2016-10-13 -5 1.62 MB/s 158 MB/s 947966398 63122428 6.66 SILVA_123_SSURef_Nr99_tax_silva.fasta
lizard 1.0 -29 0.51 MB/s 1666 MB/s 947966398 92138920 9.72 SILVA_123_SSURef_Nr99_tax_silva.fasta
Nakamichi 'Ryuugan-ditto-1TB' 1197 MB/s 92184094 ! outside lzbench !
lizard 1.0 -49 0.49 MB/s 1672 MB/s 947966398 96151827 10.14 SILVA_123_SSURef_Nr99_tax_silva.fasta
crush 1.0 -2 0.16 MB/s 521 MB/s 947966398 120354575 12.70 SILVA_123_SSURef_Nr99_tax_silva.fasta
xpack 2016-06-02 -9 7.32 MB/s 689 MB/s 947966398 144557535 15.25 SILVA_123_SSURef_Nr99_tax_silva.fasta
zling 2016-01-10 -4 29 MB/s 256 MB/s 947966398 163033199 17.20 SILVA_123_SSURef_Nr99_tax_silva.fasta
libdeflate 0.7 -12 2.07 MB/s 737 MB/s 947966398 168227973 17.75 SILVA_123_SSURef_Nr99_tax_silva.fasta
ucl_nrv2e 1.03 -9 0.28 MB/s 341 MB/s 947966398 176282228 18.60 SILVA_123_SSURef_Nr99_tax_silva.fasta
zlib 1.2.11 -9 1.59 MB/s 335 MB/s 947966398 177147955 18.69 SILVA_123_SSURef_Nr99_tax_silva.fasta
lzsse2 2016-05-14 -17 0.53 MB/s 3080 MB/s 947966398 177807250 18.76 SILVA_123_SSURef_Nr99_tax_silva.fasta
ucl_nrv2d 1.03 -9 0.28 MB/s 356 MB/s 947966398 178368952 18.82 SILVA_123_SSURef_Nr99_tax_silva.fasta
lzsse4 2016-05-14 -17 0.15 MB/s 3095 MB/s 947966398 178506100 18.83 SILVA_123_SSURef_Nr99_tax_silva.fasta
ucl_nrv2b 1.03 -9 0.28 MB/s 357 MB/s 947966398 181389542 19.13 SILVA_123_SSURef_Nr99_tax_silva.fasta
lzsse8 2016-05-14 -17 0.21 MB/s 2870 MB/s 947966398 181980090 19.20 SILVA_123_SSURef_Nr99_tax_silva.fasta
lzg 1.0.8 -9 0.25 MB/s 752 MB/s 947966398 186464811 19.67 SILVA_123_SSURef_Nr99_tax_silva.fasta
lz4hc 1.8.0 -12 1.80 MB/s 2056 MB/s 947966398 194162877 20.48 SILVA_123_SSURef_Nr99_tax_silva.fasta
lizard 1.0 -19 1.51 MB/s 2267 MB/s 947966398 194639562 20.53 SILVA_123_SSURef_Nr99_tax_silva.fasta
lzo1b 2.09 -999 1.79 MB/s 801 MB/s 947966398 194821390 20.55 SILVA_123_SSURef_Nr99_tax_silva.fasta
lizard 1.0 -39 1.49 MB/s 2179 MB/s 947966398 200121731 21.11 SILVA_123_SSURef_Nr99_tax_silva.fasta
lzmat 1.01 1.83 MB/s 314 MB/s 947966398 218007177 23.00 SILVA_123_SSURef_Nr99_tax_silva.fasta
zlib 1.2.11 -5 11 MB/s 193 MB/s 947966398 244043474 25.74 SILVA_123_SSURef_Nr99_tax_silva.fasta
yalz77 2015-09-19 -12 26 MB/s 344 MB/s 947966398 245078611 25.85 SILVA_123_SSURef_Nr99_tax_silva.fasta
lzfse 2017-03-08 39 MB/s 385 MB/s 947966398 262755056 27.72 SILVA_123_SSURef_Nr99_tax_silva.fasta
yappy 2014-03-22 -99 27 MB/s 2411 MB/s 947966398 284086695 29.97 SILVA_123_SSURef_Nr99_tax_silva.fasta
libdeflate 0.7 -1 102 MB/s 372 MB/s 947966398 295835903 31.21 SILVA_123_SSURef_Nr99_tax_silva.fasta
xpack 2016-06-02 -1 25 MB/s 296 MB/s 947966398 297319392 31.36 SILVA_123_SSURef_Nr99_tax_silva.fasta
zlib 1.2.11 -1 33 MB/s 177 MB/s 947966398 329027419 34.71 SILVA_123_SSURef_Nr99_tax_silva.fasta
brieflz 1.1.0 90 MB/s 152 MB/s 947966398 341108765 35.98 SILVA_123_SSURef_Nr99_tax_silva.fasta
quicklz 1.5.0 -3 39 MB/s 809 MB/s 947966398 378438165 39.92 SILVA_123_SSURef_Nr99_tax_silva.fasta
gipfeli 2016-07-13 143 MB/s 332 MB/s 947966398 393326740 41.49 SILVA_123_SSURef_Nr99_tax_silva.fasta
lzrw 15-Jul-1991 -5 23 MB/s 420 MB/s 947966398 397501507 41.93 SILVA_123_SSURef_Nr99_tax_silva.fasta
pithy 2011-12-24 -9 201 MB/s 809 MB/s 947966398 408863225 43.13 SILVA_123_SSURef_Nr99_tax_silva.fasta
lzvn 2017-03-08 27 MB/s 818 MB/s 947966398 410636374 43.32 SILVA_123_SSURef_Nr99_tax_silva.fasta
snappy 1.1.4 156 MB/s 688 MB/s 947966398 431965139 45.57 SILVA_123_SSURef_Nr99_tax_silva.fasta
density 0.12.5 beta -3 103 MB/s 326 MB/s 947966398 446112238 47.06 SILVA_123_SSURef_Nr99_tax_silva.fasta
slz_zlib 1.0.0 -3 149 MB/s 198 MB/s 947966398 491482457 51.85 SILVA_123_SSURef_Nr99_tax_silva.fasta
lz4hc 1.8.0 -1 65 MB/s 1256 MB/s 947966398 493265115 52.03 SILVA_123_SSURef_Nr99_tax_silva.fasta
fastlz 0.1 -2 215 MB/s 370 MB/s 947966398 543326925 57.31 SILVA_123_SSURef_Nr99_tax_silva.fasta
blosclz 2015-11-10 -9 92 MB/s 283 MB/s 947966398 548624289 57.87 SILVA_123_SSURef_Nr99_tax_silva.fasta
lzf 3.6 -1 215 MB/s 426 MB/s 947966398 552336487 58.27 SILVA_123_SSURef_Nr99_tax_silva.fasta
lzjb 2010 214 MB/s 387 MB/s 947966398 626861264 66.13 SILVA_123_SSURef_Nr99_tax_silva.fasta
wflz 2015-09-16 162 MB/s 536 MB/s 947966398 687187700 72.49 SILVA_123_SSURef_Nr99_tax_silva.fasta
shrinker 0.1 48 MB/s 5658 MB/s 947966398 943719706 99.55 SILVA_123_SSURef_Nr99_tax_silva.fasta
memcpy 6034 MB/s 6041 MB/s 947966398 947966398 100.00 SILVA_123_SSURef_Nr99_tax_silva.fasta
Oodle 'Leviathan' impresses most, this time around, second best seems to be the supermonster lzturbo 39, decompression-speed-wise.
D:\TEXTORAMIC_benchmarking_2019-Apr-29>"turbobench_v18.05_-_build_04_May_2018" SILVA_123_SSURef_Nr99_tax_silva.fasta -ebzip2/lzlib,9d30fb273/lzham,4fb258:x4:d30/lzma,9d30:fb273:mf=bt4/oodle,89,91,95,99,111,115,119,129,139/lzturbo,19,12,10,29,22,20,39,32,30,59/brotli,11d30/trle -I3 -J31 -k1 -B2G
41170037 4.3 0.20 423.47 brotli 11d30 SILVA_123_SSURef_Nr99_tax_silva.fasta
41175596 4.3 0.50 270.33 lzma 9d30:fb273:mf=bt4 SILVA_123_SSURef_Nr99_tax_silva.fasta
43226170 4.6 0.28 1289.30 lzturbo 39 SILVA_123_SSURef_Nr99_tax_silva.fasta
44350957 4.7 4.72 24.33 lzturbo 59 SILVA_123_SSURef_Nr99_tax_silva.fasta
46338866 4.9 0.07 1887.44 oodle 139 'Leviathan' SILVA_123_SSURef_Nr99_tax_silva.fasta
47401385 5.0 0.05 1124.12 oodle 129 'Hydra' SILVA_123_SSURef_Nr99_tax_silva.fasta
47655604 5.0 0.09 1134.04 oodle 89 'Kraken' SILVA_123_SSURef_Nr99_tax_silva.fasta
62538908 6.6 0.29 1567.57 lzturbo 29 SILVA_123_SSURef_Nr99_tax_silva.fasta
64352852 6.8 0.15 1556.81 oodle 99 'Mermaid' SILVA_123_SSURef_Nr99_tax_silva.fasta
72130095 7.6 0.41 1501.21 oodle 95 'Mermaid' SILVA_123_SSURef_Nr99_tax_silva.fasta
85823872 9.1 0.16 1735.64 oodle 119 'Selkie' SILVA_123_SSURef_Nr99_tax_silva.fasta
92184094 1197 Nakamichi 'Ryuugan-ditto-1TB' ! outside turbobench !
98454437 10.4 0.46 1692.66 oodle 115 'Selkie' SILVA_123_SSURef_Nr99_tax_silva.fasta
129524831 13.7 6.79 21.35 bzip2 SILVA_123_SSURef_Nr99_tax_silva.fasta
134312319 14.2 44.34 1097.28 lzturbo 32 SILVA_123_SSURef_Nr99_tax_silva.fasta
187031693 19.7 46.16 1814.22 lzturbo 22 SILVA_123_SSURef_Nr99_tax_silva.fasta
194672481 20.5 0.24 2840.00 lzturbo 19 SILVA_123_SSURef_Nr99_tax_silva.fasta
235566020 24.8 70.18 2681.77 lzturbo 12 SILVA_123_SSURef_Nr99_tax_silva.fasta
285590842 30.1 111.84 476.37 lzturbo 30 SILVA_123_SSURef_Nr99_tax_silva.fasta
292872859 30.9 94.61 1815.50 oodle 91 'Mermaid' SILVA_123_SSURef_Nr99_tax_silva.fasta
420135691 44.3 353.19 832.80 lzturbo 20 SILVA_123_SSURef_Nr99_tax_silva.fasta
541288106 57.1 131.91 1749.63 oodle 111 'Selkie' SILVA_123_SSURef_Nr99_tax_silva.fasta
594113486 62.7 362.27 1454.73 lzturbo 10 SILVA_123_SSURef_Nr99_tax_silva.fasta
891632906 94.1 115.44 985.34 trle SILVA_123_SSURef_Nr99_tax_silva.fasta
Notes:
- Latest available ‘oo2core_6_win64.dll’ was used;
- The tweak value spacespeedtradeoff=[64..1024] “tweak size vs decode time” has not been played with!
- Oodle Levels: HyperFast = -4..-1; None = 0; SuperFast = 1; VeryFast = 2; Fast = 3; Normal = 4; Optimal1,2,3,4 = 5,6,7,8;
- Oodle Compressors: Kraken = 8 (Default); Leviathan = 13 (Best); Mermaid = 9 (Crazy fast); Selkie = 11 (Fastest); Hydra = 12 (Tuneable composite of above).
And some strong crunchers outside RAM-2-RAM benches:
48,764,228 SILVA_123_SSURef_Nr99_tax_silva.fasta.L9Dict1024.xz ! "xz_v5.2.3_x64.exe" -z -k -f -9 -e -v -v --lzma2=dict=1024MiB --threads=1 SILVA_123_SSURef_Nr99_tax_silva.fasta !
49,105,765 SILVA_123_SSURef_Nr99_tax_silva.fasta.2GB.L22.zst ! zstd-v1.4.0-win64.exe --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 SILVA_123_SSURef_Nr99_tax_silva.fasta !
51,465,690 SILVA_123_SSURef_Nr99_tax_silva.fasta.method511.zpaq ! "zpaq_v7.05_x64.exe" add SILVA_123_SSURef_Nr99_tax_silva.fasta.method511.zpaq SILVA_123_SSURef_Nr99_tax_silva.fasta -method 511 -threads 1 !
63,601,401 SILVA_123_SSURef_Nr99_tax_silva.fasta.O16.PPMd_varI ! PPMd_varI_rev2_Intel15_32bit.exe e -o16 -m256 -fSILVA_123_SSURef_Nr99_tax_silva.fasta.O16.PPMd_varI SILVA_123_SSURef_Nr99_tax_silva.fasta !
69,514,622 SILVA_123_SSURef_Nr99_tax_silva.fasta.method211.zpaq ! "zpaq_v7.05_x64.exe" add SILVA_123_SSURef_Nr99_tax_silva.fasta.method211.zpaq SILVA_123_SSURef_Nr99_tax_silva.fasta -method 211 -threads 1 !
92,184,094 SILVA_123_SSURef_Nr99_tax_silva.fasta.Nakamichi
96,851,916 SILVA_123_SSURef_Nr99_tax_silva.fasta.rar560_m5_m1g ! rar-x64-560.exe a -m5 -ma5 -md1g SILVA_123_SSURef_Nr99_tax_silva.fasta.rar560_m5_m1g SILVA_123_SSURef_Nr99_tax_silva.fasta !
168,268,686 SILVA_123_SSURef_Nr99_tax_silva.fasta.ST6Block1024.bsc ! "bsc_v3.1.0_x64.exe" e SILVA_123_SSURef_Nr99_tax_silva.fasta SILVA_123_SSURef_Nr99_tax_silva.fasta.ST6Block1024.bsc -b1024 -m6 -cp -Tt !
174,231,115 SILVA_123_SSURef_Nr99_tax_silva.fasta.O6.PPMd_varI ! PPMd_varI_rev2_Intel15_32bit.exe e -o6 -m256 -fSILVA_123_SSURef_Nr99_tax_silva.fasta.O6.PPMd_varI SILVA_123_SSURef_Nr99_tax_silva.fasta !
194,978,369 SILVA_123_SSURef_Nr99_tax_silva.fasta.12.lz4 ! lz4_v1_9_0_win64.exe -12 SILVA_123_SSURef_Nr99_tax_silva.fasta SILVA_123_SSURef_Nr99_tax_silva.fasta.12.lz4 !
947,966,398 SILVA_123_SSURef_Nr99_tax_silva.fasta
And for good measure latest Zstd and LZ4:
D:\TEXTORAMIC_benchmarking_2019-Apr-29>zstd-v1.4.0-win64.exe -b1e22 -i9 --priority=rt "SILVA_123_SSURef_Nr99_tax_silva.fasta"
Note : switching to real-time priority .fasta...
1#9_tax_silva.fasta : 947966398 -> 178911916 (5.299), 124.4 MB/s , 573.7 MB/s
Note : switching to real-time priority .fasta...
2#9_tax_silva.fasta : 947966398 -> 197639514 (4.796), 150.4 MB/s , 461.8 MB/s
Note : switching to real-time priority .fasta...
3#9_tax_silva.fasta : 947966398 -> 149661634 (6.334), 214.5 MB/s , 618.8 MB/s
Note : switching to real-time priority .fasta...
4#9_tax_silva.fasta : 947966398 -> 147498216 (6.427), 196.0 MB/s , 644.2 MB/s
Note : switching to real-time priority .fasta...
5#9_tax_silva.fasta : 947966398 -> 188131516 (5.039), 69.8 MB/s , 497.8 MB/s
Note : switching to real-time priority .fasta...
6#9_tax_silva.fasta : 947966398 -> 170660167 (5.555), 73.9 MB/s , 549.6 MB/s
Note : switching to real-time priority .fasta...
7#9_tax_silva.fasta : 947966398 -> 158332784 (5.987), 49.0 MB/s , 602.0 MB/s
Note : switching to real-time priority .fasta...
8#9_tax_silva.fasta : 947966398 -> 152178033 (6.229), 36.9 MB/s , 621.6 MB/s
Note : switching to real-time priority .fasta...
9#9_tax_silva.fasta : 947966398 -> 140101920 (6.766), 24.4 MB/s , 663.2 MB/s
Note : switching to real-time priority .fasta...
10#9_tax_silva.fasta : 947966398 -> 136449466 (6.947), 23.4 MB/s , 669.4 MB/s
Note : switching to real-time priority .fasta...
11#9_tax_silva.fasta : 947966398 -> 135240559 (7.009), 21.5 MB/s , 696.6 MB/s
Note : switching to real-time priority .fasta...
12#9_tax_silva.fasta : 947966398 -> 124191769 (7.633), 4.10 MB/s , 778.8 MB/s
Note : switching to real-time priority .fasta...
13#9_tax_silva.fasta : 947966398 -> 101866858 (9.306), 4.33 MB/s , 893.3 MB/s
Note : switching to real-time priority .fasta...
14#9_tax_silva.fasta : 947966398 -> 97726488 (9.700), 3.93 MB/s , 841.1 MB/s
Note : switching to real-time priority .fasta...
15#9_tax_silva.fasta : 947966398 -> 86286159 (10.99), 2.06 MB/s , 825.5 MB/s
Note : switching to real-time priority .fasta...
16#9_tax_silva.fasta : 947966398 -> 74265519 (12.76), 2.08 MB/s , 774.7 MB/s
Note : switching to real-time priority .fasta...
17#9_tax_silva.fasta : 947966398 -> 67723270 (14.00), 1.73 MB/s , 771.4 MB/s
Note : switching to real-time priority .fasta...
18#9_tax_silva.fasta : 947966398 -> 66641989 (14.22), 1.51 MB/s , 735.9 MB/s
Note : switching to real-time priority .fasta...
19#9_tax_silva.fasta : 947966398 -> 62603077 (15.14), 1.04 MB/s , 769.3 MB/s
Note : switching to real-time priority .fasta...
20#9_tax_silva.fasta : 947966398 -> 55306862 (17.14), 0.99 MB/s , 790.0 MB/s
Note : switching to real-time priority .fasta...
21#9_tax_silva.fasta : 947966398 -> 52169242 (18.17), 0.84 MB/s , 824.5 MB/s
Note : switching to real-time priority .fasta...
22#9_tax_silva.fasta : 947966398 -> 50823066 (18.65), 0.70 MB/s , 844.3 MB/s
D:\TEXTORAMIC_benchmarking_2019-Apr-29>lz4_v1_9_0_win64.exe -b1e12 -i9 --no-frame-crc SILVA_123_SSURef_Nr99_tax_silva.fasta
Benchmarking levels from 1 to 12
1#9_tax_silva.fasta : 947966398 -> 429351653 (2.208), 318.9 MB/s ,1459.3 MB/s
2#9_tax_silva.fasta : 947966398 -> 429351653 (2.208), 92.8 MB/s ,1595.4 MB/s
3#9_tax_silva.fasta : 947966398 -> 412845449 (2.296), 13.8 MB/s ,1740.3 MB/s
4#9_tax_silva.fasta : 947966398 -> 341175162 (2.779), 37.1 MB/s ,1963.4 MB/s
5#9_tax_silva.fasta : 947966398 -> 292454847 (3.241), 25.2 MB/s ,2140.2 MB/s
6#9_tax_silva.fasta : 947966398 -> 257848296 (3.676), 16.7 MB/s ,2184.8 MB/s
7#9_tax_silva.fasta : 947966398 -> 230959312 (4.104), 3.3 MB/s ,2405.7 MB/s
8#9_tax_silva.fasta : 947966398 -> 210632835 (4.501), 7.2 MB/s ,2488.3 MB/s
9#9_tax_silva.fasta : 947966398 -> 200556101 (4.727), 4.4 MB/s ,2513.3 MB/s
10#9_tax_silva.fasta : 947966398 -> 213436118 (4.441), 5.7 MB/s ,2494.9 MB/s
11#9_tax_silva.fasta : 947966398 -> 194190162 (4.882), 2.6 MB/s ,2611.8 MB/s
12#9_tax_silva.fasta : 947966398 -> 194162877 (4.882), 2.3 MB/s ,2556.4 MB/s
And the console log of Nakamichi during compression:
D:\_TEXTUAL_MADNESS_bare-minimum_2019-Feb-17\TESTDATAFILES>timer64 "Nakamichi_Ryuugan-ditto-1TB_RAM_(5GB)_Intel150.exe" SILVA_123_SSURef_Nr99_tax_silva.fasta SILVA_123_SSURef_Nr99_tax_silva.fasta.Nakamichi 27 208000 E
...
Nakamichi 'Ryuugan-ditto-1TB', written by Kaze, inspired by Haruhiko Okumura sharing, based on Nobuo Ito's LZSS source, babealicious suggestion by m^2 enforced, muffinesque suggestion by Jim Dempsey enforced.
Note0: Nakamichi 'Dragoneye' is 100% FREE, licenseless that is.
Note1: Hamid Buzidi's LzTurbo ([a] FASTEST [Textual] Decompressor, Levels 19/29/39) retains kingship, his TurboBench (2017-Apr-07) proves the supremacy of LzTurbo, Turbo-Amazing!
Note2: Conor Stokes' LZSSE2 ([a] FASTEST Textual Decompressor, Level 17) is embedded, all credits along with many thanks go to him.
Note3: The matchfinder is either 'Railgun_Trolldom' (matches longer than 18, except 36 and 64) or Leprechaun's B-tree order 3.
Note4: Instead of '_mm_loadu_si128' '_mm_lddqu_si128' is used.
Note5: Maximum compression ratio is 44:1, for 704 bytes long matches within 1TB Sliding Window.
Note6: Please send me (at [email protected]) decompression results obtained on machines with fast CPU-RAM subsystems.
Note7: In this compile, clock() was replaced with time() - to counter bigtime stats misreporting.
Note8: Multi-way hashing allows each KeySize to occupy its own HASH pool, thus less RAM is in use - the LEAF is smaller.
Note9: In this revision, B-tree heuristics are in use, allowing skipping many unnecessary memmem() invocations.
NoteA: The file being compressed should be 64 bytes or longer due to Building-Blocks being in range 4..18, 36, 64.
NoteB: In this compile, the keysizes in the LEAF are not HEXed i.e. not doubled.
NoteC: In this latest (2019-Apr-24) compile, keysizes 36/64 are no longer hashed with SHA3-224, it is slow for this case.
Current priority class is REALTIME_PRIORITY_CLASS.
Allocating Source-Buffer 904 MB ...
Allocating Source-Buffer 904 MB (REVERSED) ...
Allocating Target-Buffer 936 MB ...
Allocating Verification-Buffer 904 MB ...
Leprechaun: Memory pool for B-tress is 208,000 MB.
Leprechaun: In this revision 1,024MB 10-way hash is used which results in 10 x 134,217,728 external B-Trees of order 3.
Leprechaun: In this revision, 128 passes are to be executed.
Leprechaun: Allocating HASH memory 10,737,418,305 bytes ... OK
Leprechaun: Allocating/ZEROing 218,103,808,014 bytes swap file ... OK
Leprechaun: Size of input file: 947,966,398
Leprechaun: Inserting keys/BBs of order 004 into B-trees, free RAM in B-tree pool is 00,207,988 MB; Pass #128 of 128 ... DONE; 00,000,246,906 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 006 into B-trees, free RAM in B-tree pool is 00,207,891 MB; Pass #128 of 128 ... DONE; 00,002,136,886 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 008 into B-trees, free RAM in B-tree pool is 00,207,647 MB; Pass #128 of 128 ... DONE; 00,006,517,040 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 010 into B-trees, free RAM in B-tree pool is 00,206,980 MB; Pass #128 of 128 ... DONE; 00,017,724,196 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 012 into B-trees, free RAM in B-tree pool is 00,205,190 MB; Pass #128 of 128 ... DONE; 00,042,176,707 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 014 into B-trees, free RAM in B-tree pool is 00,202,079 MB; Pass #128 of 128 ... DONE; 00,081,573,960 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 016 into B-trees, free RAM in B-tree pool is 00,197,932 MB; Pass #128 of 128 ... DONE; 00,125,366,437 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 018 into B-trees, free RAM in B-tree pool is 00,192,687 MB; Pass #128 of 128 ... DONE; 00,179,346,212 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 036 into B-trees, free RAM in B-tree pool is 00,174,474 MB; Pass #128 of 128 ... DONE; 00,273,214,316 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 064 into B-trees, free RAM in B-tree pool is 00,125,820 MB; Pass #128 of 128 ... DONE; 00,391,919,211 B-trees have been rooted so far.
Leprechaun: Total Searches-n-Inserts Per Second: 10,456,803 SNIPS
Leprechaun: RAM needed to house B-trees (relative to the file being ripped): 89N = 80,513MB
Leprechaun: Total IOPS for 10,698,482,483 'freads' and 10,081,382,386 'fwrites' (of packets 170 bytes long) during loading traversing all orders: 179,076 IOPS
Compressing 947,966,398 bytes ...
|; Each rotation means 64KB are encoded; Speed: 0,000,285 B/s; Done 100%; Compression Ratio: 10.28:1; Matches(16/24/48): 1,445,744/2,822,352/1,782,082; 128[+] long matches: 1,668,715; ETA: 0.00 days
NumberOfFullLiterals (lower-the-better): 10148
Tsuyo_HEURISTIC_APPLIED_thrice_back-to-back: 0
NumberOf(Tiny)Matches[Micro]Window (4)[16B]: 265094
NumberOfMatches[Bheema]Window [128GB window]: 2474051
RAM-to-RAM performance: 285 B/s.
Compressed to 92,184,094 bytes.
Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xf401,08aa
Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x18e9,9edd
Decompressing 92,184,094 (being the compressed stream) bytes ...
RAM-to-RAM performance: 1151 MB/s.
Verification (input and output sizes match) OK.
Verification (input and output blocks match) OK.
Kernel Time =433452.226 = 12%
User Time =2349298.385 = 68%
Process Time =2782750.611 = 80% Virtual Memory = 34755 MB
Global Time =3437074.672 = 100% Physical Memory = 12161 MB
Or, 947,966,398/3,437,074 = 275 B/s compression rate on SSD Crucial MX200 250GB (DRAM DDR3-1600 512MB), and i5-2430M @2.40GHz 16GB DDR3 1333MHz, Windows 7.