Comments (8)
It'd be useful if a Read
interface were also provided (compare flate2
which has both read::DeflateDecoder
and write::DeflateDecoder
).
from lzma-rs.
This is a very interesting point!
I was wondering whether there was a generic way of transforming an io::Write
into an io::Read
. The opposite would be quite simple (read bytes from an io::Read
and write them into an io::Write
), but this looks trickier. Maybe that could be possible with async functions/generators? Or with a separate process - or simply a thread - that "writes" data to the main thread, which reads it (like with Unix pipes).
In the meantime, I think the easiest way to support streaming would be to extract the loop body of the process
function (https://github.com/gendx/lzma-rs/blob/master/src/decode/lzma.rs#L215) into a step
function. Then, in the streaming case, use a temporary buffer as the io::Write
for the current decoder ; the read
method of your io::Read
would repeatedly call step
and copy the bytes from the tmp buffer into the read buffer.
I probably won't have time to look at it more closely this week, but feel free to send a PR if you want to give it a try!
from lzma-rs.
Thanks for the explanation!
Regarding the process
function and the temporary buffer -- indeed this is how I thought it can be done as well.
I'm not sure I'll have time in the coming days as well though. Maybe I'll come to that later if/when I get rid of other libraries that bind to OS libraries, and will be otherwise on pure Rust.
from lzma-rs.
Hi,
I am maintaining swf-parser, a library to parse SWF files. These files can be encoded with LZMA and I am using this library to decode them. To support streaming parsing of SWF files, support in LZMA is required first. A low level API similar to the one used by the inflate
crate would be nice.
Using this API, you create a stream inflater maintaining the internal state of parser (for LZMA it would correspond to dictionaries and temporary buffers). You can manually feed data to the decoder it and read the result.
from lzma-rs.
I've been working on an implementation for this ticket based off of the LzmaDec_TryDummy function in libhtp's port of the LZMA SDK. The main issue in incrementally executing the loop is that you may end up in a partially corrupted state if you are in the middle of a function and you fail to read the next byte because it isn't available yet.
Also, I used the std::io::Write
trait instead of std::io::Read
to create an interface like flate2::write::DeflateDecoder
.
I'll publish this soon. It will most likely be dependant on #50 .
from lzma-rs.
I'm now wondering whether integrating with async/await would be the way to go to implement this. Something like taking futures::io::AsyncRead as input and writing to a futures::io::AsyncWrite or a futures::stream::Stream of bytes as output.
I don't know what the performance overhead of that would be, but from a programming perspective the code should be similar to the current one (with some extra async
keywords). The streaming mode would be gated by a feature flag.
from lzma-rs.
@gendx I published a PR for this if you want to have a look. I haven't really thought of implementing it using futures but that's an interesting idea. It would add a couple extra dependencies for those who want to use a streaming API and possibly require a runtime. I was looking for a solution that uses an std::io::Write
interface to have an API consistent with flate2::write::DeflateDecoder
to implement a generic decoder.
from lzma-rs.
Reading line by line is very important, for example, flate2
can read .gz
files line by line:
let f_in = std::fs::File::open("sample.txt.xz").unwrap();
let d = flate2::read::GzDecoder::new(f_in);
let mut buf_reader = std::io::BufReader::new(d);
for line in buf_reader.lines() {
println!("{}", line)
}
from lzma-rs.
Related Issues (20)
- Publish some benchmarks on `README.md`
- Travis/Bors integration broken HOT 2
- Panic on decoding xz file with sha256 checksum
- Test against the XZ test suite HOT 4
- Behavior mismatches between liblzma and lzma-rs HOT 1
- Provide features to import parts of the crate HOT 3
- Enable no_std support HOT 4
- Add configurable memory limit
- Support multi-stream files
- .lz (lzip) HOT 2
- Expose decoder/encoder primitives as public HOT 4
- Add unit tests
- Report code coverage
- Reach 100% coverage.
- Problems when porting lzma code Java -> Rust HOT 1
- New release
- XZ: Unknown filter id 7 HOT 2
- API Suggestion: Add encoder and decoder structs
- Extraction fails with `Unknown filter id 8` HOT 2
- lzma2_compress and zx_compress do not compress input HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lzma-rs.