Giter Club home page Giter Club logo

compactor's Introduction

Compactor

A friendly user interface to Windows 10 filesystem compression

With modern lightweight compression algorithms running at gigabytes per second per core, it's practically a no-brainer to apply them to filesystems to make better use of storage and IO.

Half-recognising this, Windows 10 ships with a reworked compression system that, while fast and effective, is only exposed to users via a command-line tool — compact.exe.

Compactor is here to plug that gap, with a simple GUI utility anyone can use.

Installation v0.10.1 Downloads

Downloads are available from the Github Releases page under Assets, or you can use these direct links:

The 64-bit version is recommended for most users.

If you get "Windows protected your PC" trying to run it, it's just SmartScreen upset the binaries aren't (yet) signed. Click "More info" and "Run anyway" if you judge things to be above-board.

Note this is beta software and comes with no warranty.

Features

Real-time Progress Updates

Compactor's directory analysis updates as it goes. You too can experience the satisfaction of watching the disk-space used counter tick down with each file compressed.

Pause, Resume, Stop

All operations can be paused and interrupted safely at any time. Compactor will finish off what it's doing and stop, or restart where it left off.

Compresstimation

Compactor performs a statistical compressibility check on larger files before passing them off to Windows for compaction. A large incompressible file can be skipped in less than a second instead of tying up your disk for minutes for zero benefit.

Machine Learning

Using advanced condition-based AI logic, Compactor can skip over files that have been previously found to be incompressible, making re-running Compactor on a previously compressed folder much quicker.

(Yes, it's an if statement and a trivial hash database, hush)

Scalable and Fast

Written in Rust, a modern compiled systems programming language from Mozilla, Compactor can cope easily with large folders containing millions of files.

Caveats

Beta Software

While it has been used successfully by thousands of people, Compactor should be used with care. It is intended for compressing replacable software, not precious files.

Make backups. Report bugs. Be nice. You are reminded:

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

It's in shouty legal text so you know I mean it.

Data Corruption

There has been one report of data corruption with open SQLite database files. The author has been unable to reproduce this, but file locking was added to version 0.10 which should prevent them from being modified.

Permissions

Compactor currently has no mechanism to elevate its privileges using UAC for protected files. If you're using a limited account you may need to run the program with elevated permissions.

Be careful what you compress. System files should be skipped automatically, and the Windows folder should be in the list of default exclusions (if you want to compact Windows, check out its CompactOS feature), but you almost certainly don't want to blindly run this across your entire C:\ drive.

Modifiable Files

Compaction is designed for files that rarely change — any modifications result in the file being uncompressed in its entirety. In fact, simply opening a file in write mode will hang until the file is uncompressed, even if no changes are made.

This generally doesn't matter much for application folders, but it's not great for databases, logs, virtual machine images, and various other things that hopefully mostly live elsewhere.

If a game uses large files and in-place binary patching for updates, it might be worth adding to the exclusions list.

Compatibility with Other Operating Systems

Compaction is only supported on Windows 10 - earlier versions of Windows will be unable to access compressed files, though the rest of the filesystem should remain fully accessible.

Linux and other operating systems will experience similar issues if the NTFS driver they're using lacks support. As of time of writing, NTFS-3G has a third-party plugin for the feature.

This is available on FreeBSD under sysutils/fusefs-ntfs-compression, while users of lesser platforms may need to manually install it like savages.

Compression Results

A totally-not-cherry-picked sample of compression results with the default settings:

Program Size Compacted Ratio
AI War 2 2.43 GiB 1.42 GiB 0.59x
Big Pharma 1.1 GiB 711 MiB 0.37x
Crusader Kings 2 2.19 GiB 1.29 GiB 0.59x
Deus Ex MD 41.31 GiB 28.06 GiB 0.68x
Infinifactory 1.71 GiB 742 MiB 0.58x
Satisfactory 15.82 GiB 10.45 GiB 0.66x
Space Engineers 16.28 GiB 9.4 GiB 0.58x
Stellaris 7.76 GiB 5.21 GiB 0.67x
Subnautica BZ 10.62 GiB 6.40 GiB 0.60x
The Long Dark 7.42 GiB 5.64 GiB 0.76x
Microsoft SDKs 5.91 GiB 2.45 GiB 0.41x
Visual Studio 2017 9.63 GiB 4.77 GiB 0.50x
Windows Kits 5.38 GiB 2.03 GiB 0.38x

A more comprehensive database of results is maintained by the CompactGUI project.

Future

There are many things I want to do with Compactor in future. These include, but are certainly not limited to:

  • Make analysis optional. It isn't fundamentally needed.
  • Multithreaded analysis/compaction for SSDs.
  • GUI rework of some description. The longer I leave this the better Rust should get at it :P
  • Installer. Why does this involve so much XML oh god.
  • Sign the binaries/installer. This appears to involve money.
  • Scheduled task or a background service for set-it-and-forget-it operation.

Feature requests can be discussed in the forum, or you may open an issue.

Alternatives

  • compact.exe is a command-line tool that ships with Windows 10. If you're familiar with the command line and batch files, maybe you'd prefer that. Weirdo.
  • CompactGUI is a popular Visual Basic program that shells out to compact.exe to do its work, instead of using the Windows API directly as Compactor does. It has some... performance issues, particularly with larger folders.
  • NTFS has supported LZNT1 compression since 1995, hidden behind a checkbox under PropertiesAdvanced Attributes. It's less flexible and has a reputation for poor performance and issues with fragmentation, but is more set-it-and-forget-it.

Are you aware of any others? Do let me know.

Nerdy Technical Stuff

Compactor is primarily written in Rust. The front-end is basically an embedded website driven by the web-view crate. It does not depend on any remote resources or open any ports.

Under the hood it uses DeviceIoControl with FSCTL_SET_EXTERNAL_BACKING and FSCTL_DELETE_EXTERNAL_BACKING, and a few functions from WofApi (Windows Overlay Filesystem). This is, of course, in part thanks to the winapi crate. Eventually I hope to get around to finishing off some of my bindings and contributing them back.

Compresstimation uses a simple linear sampling algorithm, passing blocks through LZ4 level 1 as a compressibility check and averaging across the entire file. The code is available on Github.

The incompressible-files database is simply an append-only list of SipHash128 path hashes. It should be safe to share between multiple instances if you want to compress different drives at the same time. It lives in %APPDATA%\Local\Freaky\Compactor.

Author

Compactor is written by Thomas Hurst, a nerdy, aloof weirdo from the north-east of England, and a programmer for about 25 years.

He mostly works with FreeBSD and focuses on Unix platforms, but uses Windows because he plays games instead of having a social life.

You can find him on Mastodon at @[email protected], or bug him on IRC as Freaky on libera.chat.

compactor's People

Contributors

dr-emann avatar freaky avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

compactor's Issues

Prune excluded directories

I have the following folders excluded for all drives:

*:\Windows*
*:$Recycle.Bin*
*:\Documents and Settings*
*:\System Volume Information*

Yet Compactor stills goes through each file in said folders whilst adding them to the total 'excluded' value, would it not be more efficient to not scan any excluded directories and subdirectories?

Winget repo

Could be possible to make this app so you could get it from winget?

Compactor ignoring excluded folder

I have the folder path %APPDATA%\Hamsket added to the exclusion list, but as you can see in the pictures Compactor is still compressing the files in said folder.

Untitled

Untitle2d

Improve app close behaviour

As the title says ...
I instantly opened task manager to see if it was still running ... luckily it was ...
Opening the exe another time would open a new window
how do i reopen the same window
also if possible please add a tray icon

Skip frequently updated files

Files that get updated a lot, particularly if they're large, are poor candidates for compaction.

An obvious simple heuristic that seems likely to be relatively effective is to check file modification time, and simply have a configurable cutoff - if it's changed in the past n days, skip it. Tiering this by file size might also make sense - frequently decompressing and recompressing a small file is much less costly than a large one.

This sort of thing will be more important should Compactor migrate to a background system service with patrol compactions.

Program Immediately Closes

When I open Compactor.exe, it opens a blank white window for about half a second and then just closes. I've tried running it as an Administrator too with no change in behaviour. Windows 10, up to date.

image

Dry run compaction

Offer a read-only dry-run mode using the compression API, simulating the effects of compacting a folder without actually doing it.

This allows for a safe way to present a user with potential savings, and offers an opportunity to validate excludes and compresstimation both more efficiently than actually performing compaction, and more accurately than relying on lz4 as a stand-in.

This links with the desire to benchmark algorithms, which also requires compression API access.

Unable to compact Microsoft (Xbox) games

Whenever I try to compact an Xbox game on PC (Forza Horizon 4 for example) it says Access Denied: OS error 5. Any way to solve this? I have full admin control on my user and I've tried launching the .exe as admin.

PC Crash

My PC crashes with no BSOD when in the middle of compressing Metro Exodus.

Compactor Seems to copy files to AppData\Local\VirtualStore\ when compressing files on C drive.

Sorry if this is an inappropriate usage of Github, I normally only use this site as a user and I don't really use my account for much. This is my first time reporting an issue of any kind for anything on Github.

So it's pretty much as the title suggests. I noticed it after I did some digging in Appdata as I tried compressing a large game only to see I had lost 50gb of space. It really had me worried.

When compressing folders from OTHER drives, Compactor works normally and with CompactGUI, ALL my drives compress normally.

I can get into more specific details if need be, but I thought it might be worth including that my C drive is an SSD and my other 2 drives are HDDs.

Automaticly Close App / Restart / Shutdown PC

Since some Files can take relativly long to Compress I think a toggable option where the User decides wherever the Program should close itself after it's done compressing or shutting down the computer could come in really handy for some users!

Clarification on the compression methods?

On the settings, there's an option for choosing compression method. One of them, LZX, says it's slow but it has high compression rate. Does it being slow just mark the act of compressing it? Or would it also make the accessing of the files slower?

Negative space saved on initial analysis

Noticed a few folder scans on two different machines that reported small (few hundred KiB) negative amounts of space saved. Seems to only happen on an initial scan - a re-scan immediately afterwards usually doesn't see the same thing.

Skipping too many files that can be compressed

I used Compactor on genshin impact because why not and It seems like it skips alot of files that could be compressed, I tried Compactgui instead and it Compressed it much better then compactor. Can you add a setting to allow us to change this so it compresses every file?

Multi-threaded analysis/compaction

On my PC, the scanning part only uses ~11% of CPU (Ryzen 2600, 11% is usually single threaded stuff) and ~6% of my SSD (Crucial P1 NVME 1TB). Would it be possible to somehow speed this up?

Also, sometimes the compression itself doesn't use the PC fully either, usually disk usage is below 50% and CPU usage below 30%, I think there might be a way to optimize that too.

Corruption of SQLite Databases in Use

Compacting SQLite databases while they are in use causes them to become corrupted. I've not checked if this is limited to SQLite but after compacting my system drive, all the programs I faced problems with had SQLite databases. These include Thunderbird, and Sticky Notes (From MS), among others.
Simply excluding .sqlite or .db files and their corresponding -shm and -wal files isn't enough, as programs such as Chrome use SQLite DB files without any extensions.

Configurable compresstimation

The compresstimator currently cuts-off at 0.95 - it might be nice to make this value (and indeed compresstimation in general) configurable so users can choose between quickly getting the low-hanging fruit or getting every last byte of potential saving.

This would be best done alongside changes to the incompressible database to store the estimate.

*.bin/*.cas isn't in the exclusion lists, but Compactor still excludes it

I'd like the option to tell Compactor to tell it to compress bin files. bin files can be anything. They aren't always binaries that are uncompressible, and it could save a lot of space for certain games.

It seems that Compactor forces itself to not compress these. Can this be adjusted?

EDIT: It also seems to ignore CAS files. Is there something it detects to prevent compression?

I find that CompactGUI compresses some folders way more than Compactor, sometimes saving a couple extra GB. (I've tested the same data decompressed)

Windows Defender history is broken after compression

After a restart, Windows Defender history will appear blank and non functional. I did recover this in the past but I don't remember the exact steps and what files are involved.

I'll update the issue if I will find a solution

Path filter incompatibility across 32-bit and 64-bit

When testing an updated i386 build, it's noted that the path database does not filter out known-incompressibles found by the 64-bit build. This doesn't really make sense since the databases should be bit-identical across architectures.

Compressing Discord app folder would reset the session

I've noticed that if Compactor compresses Discord in %localappdata%, on next launch Discord will force expire the session and reset the settings, require the user to login and configure it again.

I propose to add Discord to the blacklist

No signs of compression

Compactor claims that some files were compressed and tells me how many GiB were saved, but my disk is still as full as before. Free space didn't change at all, numbers are still the same. Why is that?

Update mechanism

We've just had one release fixing a data corruption issue, and another one fixing a CPU use bug, and I have limited reach to inform people of these updates.

At its simplest, I could simply put a link to the releases page on the initial window and encourage people to manually check it from time to time. Easy to do, but equally easy to ignore and makes work for the user.

A custom check shouldn't be that difficult, but is bit of a can of worms with regard to how it's triggered, how the user can opt-out, and even just the basics of how it works - I guess checking a bit of JSON hosted on Github, triggered from a button the user hits.

Automated updates... can probably wait for the Steam release. Which would also be a convenient if somewhat inefficient place for people to donate...

Exclude more standard files/paths by default

As an addition to the great if statement trivial hash machine learning advanced condition-based AI logic database (sorry.. couldn't resist) i would suggest to implement a list of files/folders in categories (like archives, database, etc.) that are not meant to be compressed instead of a huge list with little to no explanation.

For files this would be etl, etw and other database or log files (like from Windows Search):

*.etl
*.etw
*.evtx
*.jrs
*.jtx
*.jcp
*.jfm
*.edb
*.crwl
*.gthr
%ALLUSERSPROFILE%\Microsoft\Windows Defender
%ALLUSERSPROFILE%\Microsoft\Search
%ALLUSERSPROFILE%\NVIDIA
%TEMP%

This would speed up the tool even more and SSD wear would be less.

Also a really cool option would be a way to scan for magic numbers: Read the first four bytes of a file (if the file extension is unknown) and compare that with a list of known magic numbers and decide on how to compress these.

A possible solution to programmatically get some of those files/paths would be to read the registry and see where some of those files are stored...

Thanks for this great tool! Works better than anything else i used before as it's using the API instead of just being a wrapper for compact.exe. Very nice!

Recompress option, and dry run without analyze. Detect and block Windows 7 or fallback to a compatible algorithm.

I already used the Properties → Advanced → "compress contents to save disk space" attribute for years, and it saved me 55GB on my tiny 250GB SSD. (I compress almost everything, even things you exclude.) But now I discovered this amazing tool and over the past two days I uncompressed and recompressed my entire drive, having to do it folder by folder. (As there was no way to free up 50GB for doing it all in one go.) I now have 10 GB additional free space! (65GB saved.)

Please add a Recompress option that achieves this automatically; Uncompress files one by one, and immediately recompress them. Or a single folder at a time.

Please also add a Compress without analyzing button, as I trust this tool and don't care about waiting for the predicted details, only the actual results. I had to wait a long time per each folder. I wish I could have just "dry run" it.

I also tried on Windows 7: My entire C/: drive compressed offline. It was a bad idea:

  • Now boots much slower.
  • Biometric login: Fingerprint reader not working anymore.
  • On-screen keyboard opening exceedingly slowly.
  • I'm unable to access any of my installed games or programs. Windows apps are okay, just way slower.
  • Compactor itself unfortunately opens without warning and runs fine and even appears to work as well, but is unable to actually decompress anything.
  • In fact after clicking decompress all options disappear and I can only analyze.

Please warn Win7 users upon opening the program. And Win10 users dual booting Win7...

Luckily an offline decompress saved my data and everything is back to normal.

I would like to have a "fallback to a compatible algorithm" option for Win7 users.

Question/Suggestion: Use WimBootCompress.ini as default exclusion list. Allows safe compression of C:\

WimBootCompress.ini
or
WimBootReCompress.ini
found in the Windows\System32 folder
are exclusion lists created during WinNTSetup installs on Wimboot mode and WinNTSetup installs on Compress mode, respectivly.

So as I understand things, using it/them as the default exclusion list should make it safe to use Compactor in the whole system disk; C:
??

More info is available in the below link of an app that enables Windows Compact all the way down to Win 7:
http://reboot.pro/topic/22007-wofcompress-tool-for-win7-win10/?p=211055

ALSO:
I noticed that the wimlib algorithms are said to be more performant than the Win 10 algorithms.
Faster and smaller files:
https://wimlib.net/compression.html#Benchmarks

I'm no programmer but thought I'd mention this in the hopes that they might be used to improve this excellent app?

Thank you!

Hello Thomas!

I just wanted to pop in and say thank you for creating this handy piece of software. It's helped myself and many I know easily free up more bits and bytes. Great job!

All the best,
Blake

Wont open anymore - software exception

Every time I'm trying to launch the program I get a blank window and then it closes itself, sometimes I get this error (so far after 10 tries or so I got it twice):

Se produjo una excepción unknown software exception (0x80000003) en la aplicación en la ubicación 0x00007FFD5E799173.

Haga clic en Aceptar para finalizar este programa

I've tried deleting both AppData folders to no avail. This error has happened just now, last time I launched the program which was just a week more or less ago, it launched normally.

Option to avoid LZX on large files

Taken from LZX — new Windows 10 NTFS compression algorithm:

I strongly recommend avoiding LZX and using xpress16k instead.

The xpress algorithms do not require a full read of the file before performing operations, but can be done on the fly.

Thus, if you have a 1 GB file and you move it to another drive (say), with LZX your OS will sit at 0 until the whole thing is decompressed, and then it will eventually begin to start copying after sitting around for 30 seconds.

If you use xpress4k, xpress8k or xpress16k, the file will begin copying immediately.

And the compression improvement from xpress16k to lzx is always like 1.8 to 1.9, for example, it is not worth the overhead especially for large files.

I would suggest an option to use xpress16k on larger files even if LZX is choosen with a key to override it if really needed.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.