Comments (2)
I tried this and it mostly made the strings grow in size and the few that were smaller were only smaller by 1%.
from chihaya.
I'm not familiar with how much of a benefit you'd get from doing so, looking at a variety of PeerID values from this test there doesn't seem like much compressible patterns, beyond some cases of repeated values.
If these are stored as strings, you could probably binary pack the charset directly as a lookup, if the number of valid characters in such a string is less than 128 you could compress these to roughly half the size I think? Or a quarter with a limit of 64 characters. Assuming each character is using 1 byte. Maybe there's a Go package that provides similar functionality.
I don't know how the IDs are used though, might not be worth it if they're frequently accessed and "decompressed" depending on CPU overhead.
Many of those seem to have a 6 character prefix surrounded by -
on each end (client ID?), that's a little more compressible, 1 bit or bool flag to mark clients with that pattern can exclude the two -
. There seems to be some structure in the values for those too which can additionally compress better (either via static lookup or dynamic runtime generated LUT), or some data structures (I forget the name, prefix trie?).
There's a Go implementation of zstd too, with a large enough sample set, you might be able to train the compressor on that and it may recognize the prefix part I just mentioned building a dict for you to compress with, but I think the remainder might be wasted on training (not sure, perhaps it'll recognize the limited charset and optimize for that?).
Just figured I'd chime in with that if it's helpful at all π
from chihaya.
Related Issues (20)
- Explore serving UDP without standard lib HOT 3
- Bencoded dictionaries' keys are not always sorted HOT 5
- Explore using freecache for memory storage HOT 1
- Chihaya has reached a GoLang Limitation: concurrent operations on a single file or socket HOT 8
- Load config from dotfiles & create default config HOT 4
- Migrate chat off freenode HOT 2
- Explore replacing net.IP with inet.af/netaddr.IP HOT 2
- Generational Garbage Collection
- State of Chihaya? No release since 2016? Will a new release be published? HOT 3
- Adopt goreleaser
- Change storage interface to take a slice of peers as input
- Does chihaya not support redis cluster modeοΌ HOT 2
- How to read metrics number? HOT 2
- Explore garbage collection using golang.org/x/exp/maps
- How to connect to redis with unix domain socket? HOT 2
- Ideas regarding limiting concurrency and reducing allocations
- Data Race in UDP Frontend
- Use cases for peer-to-peer cloud software deployments HOT 3
- Investigate zeropool
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chihaya.