Comments (4)
Since
decoder
is mostly documented already, would it be feasible to expose those primitives as public without much work, possibly behind a feature flag?
It surely would be feasible! Could you send a pull request for review?
I don't think a feature flag is needed here - just make sure that the newly exposed API is documented and that the GitHub workflows pass.
from lzma-rs.
I did some internal refactoring to expose the raw decoder as a primitive where the LZMA compression parameters are encoded aas const generics. I also did a similar wrapper for LZMA2 although I'm not too sure about the usefulness of that in my fork.
I believe this is a better approach than just exposing LzmaParams
and I was able to expose this API without changing the existing public API, but at the cost of having to make some relatively sizeable internal changes, particularly with DecoderState
to also allow reusing the existing allocation.
If you could take a quick look to see if you're comfortable with those changes and the shape of the LzmaDecoder
and Lzma2Decoder
APIs I'd be happy to clean it up slightly and PR it upstream.
from lzma-rs.
Some comments:
- In the current state of things, I'm not sure about the value of using const generics to encode the compression parameters. The
LzmaDecoder
struct just converts these const generics parameters into simply callingLzmaParams
. It seems to me that something more useful could be achieved withfeature(generic_const_exprs)
(i.e.literal_probs
could be a const generic array rather than aVec
), but this is still an unstable feature. I'm open to exploring that if it's possible to gate the use of const generics by a feature flag (i.e. on stable Rustlzma-rs
compiles without using any const generics, and on nightly it can leverage const generics if the feature is enabled). - Having a
reset
function that fills the default values in place rather than re-allocating and copying seems like a valuable addition :) It's unclear how much improvement this brings after compiler optimizations (the compiler may replace the array copy by a simple in-place writing of the default data) - but the intent is clearer and it also compiles better in debug mode. - What was the reason to moving the
output
outside of the decoder state (07c2d7e)? This adds quite a bit of verbosity in passing it along on each function call.
Feel free to break down your changes into multiple PRs (exposing LzmaParams
, improving the reset
implementation, adding some const generics, etc.) to make it easier to review.
from lzma-rs.
- Fair criticism of using const generics. In my fork the motivation was mostly for expressiveness (i.e. an
LzmaDecoder<3,0,2>
would be type-incompatible with anLzmaDecoder<0,0,0>
) but I'm open to feature-gating this for consumers that need it or just scrapping it all together in favour of usingLzmaParams
directly or anew_with_params
constructor. - After some unscientific benchmarking,
reset
was slightly faster in Release as well, but this is heavily dependent on compiler optimizations and specific usage. I think intentwise,reset
is a good to have. - Taking
output
outside of the decoder state was a necessary prerequisite for a simplerreset
.- Conceptually,
reset
prepares the decoder state for a new buffer of input that may be written into a new output. Keepingoutput
in the decoder state tied the lifetime of the output buffer to the state of the decoder which wouldn't make much sense for a mutable reset function. This wasn't an issue before since the public APIs were one-shot. - Keeping the lifetime attached meant that a new
DecoderState
must be created for each possible newoutput
where'decoder : 'output
. There is some added verbosity at the callsite but I think separating the lifetime of the decoder state and the output more than makes up for the complexity budget there. - Detaching the lifetime of
output
fromDecoderState
also allowedLzmaDecoder
to not be lifetime-parameterized and reuse the sameDecoderState
for differentoutput
buffers.
- Conceptually,
With that said I'd be happy to split decoupling output
from DecoderState
into a separate PR since that doesn't affect any public APIs and build on top of that.
from lzma-rs.
Related Issues (20)
- Publish some benchmarks on `README.md`
- add `xz_decompress_with_options` API
- Panic on decoding xz file with sha256 checksum
- Test against the XZ test suite HOT 4
- Behavior mismatches between liblzma and lzma-rs HOT 1
- Provide features to import parts of the crate HOT 3
- Enable no_std support HOT 4
- Add configurable memory limit
- Support multi-stream files
- .lz (lzip) HOT 2
- Add unit tests
- Report code coverage
- Reach 100% coverage.
- Problems when porting lzma code Java -> Rust HOT 1
- New release
- XZ: Unknown filter id 7 HOT 2
- API Suggestion: Add encoder and decoder structs
- Extraction fails with `Unknown filter id 8` HOT 2
- lzma2_compress and zx_compress do not compress input HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lzma-rs.