backblaze / javareedsolomon Goto Github PK
View Code? Open in Web Editor NEWBackblaze Reed-Solomon Implementation in Java
Home Page: https://www.backblaze.com/open-source-reed-solomon.html
License: MIT License
Backblaze Reed-Solomon Implementation in Java
Home Page: https://www.backblaze.com/open-source-reed-solomon.html
License: MIT License
Is the idea to just copy and paste the source files directly instead of using a build system like gradle?
Hi there,
I want to know the license of image files on your blog post:
https://www.backblaze.com/blog/reed-solomon/
Such as:
https://www.backblaze.com/blog/wp-content/uploads/2015/06/blog-rs-7.png
The reason is one of the image is used in a github project, which I'm trying to package to Debian.
It's required to confirm license of every file in the project before going into Debian.
Your blog post already claimed the source code is in MIT license. So I guess the image files are also in MIT license. Could you kindly help to confirm it? Thank you!
Cheers,
Roger
(This is not an issue with the java implementation, but something you might find interesting, if you don't feel free to close the issue)
Since I may eventually do an assembler version of the main loop, I was researching if there was anything out there, and I found this: Screaming Fast Galois Field Arithmetic, which can do constant multiplications using two 16 bytes tables, which is perfect for SSE3. [their implementation]
This requires the main loop to be slightly different (in Go)
// output should be cleared, or first iRow loop should overwrite
for c := 0; c < r.DataShards; c++ {
in := inputs[c]
for iRow := 0; iRow < outputCount; iRow++ {
m := matrixRows[iRow][c]
o := outputs[iRow]
for iByte := 0; iByte < byteCount; iByte++ {
o[iByte] ^= galMultiply(m, in[iByte])
}
}
}
Now the one value (m) is a constant in the entire inner loop, so I should be able to do the multiplication with the byte[256][16] table, which only requires two look-ups for the entire inner loop.
I have generated the tables needed, and I will begin writing the assembler when I get a bit more time.
- This table was generated by java_tables.py, and the
- unit tests check it against the Java implementation.
https://github.com/Backblaze/JavaReedSolomon/blob/master/com/backblaze/erasure/Galois.java#L56
It looks like there are no tests?
You can pre-calculate results of Galois.multiply into a 64KB 2D table. This also gets rid of the branching.
I am porting the library to Go, and for me that gives a nice speed-up (~30%), and I suspect it will be even more in Java.
I'm trying to benchmark throughput using this Java Reed Solomon code. Correct me if I'm wrong in assuming that using the gradle build file to run the README-mentioned tests is the best way to do that. When I try to run gradle build
as indicated in the README, but I'm running into the following issue:
FAILURE: Build failed with an exception.
* Where:
Build file '.../JavaReedSolomon/build.gradle' line: 8
* What went wrong:
A problem occurred evaluating root project 'JavaReedSolomon'.
> Could not find method testCompile() for arguments [{group=junit, name=junit, version=4.+}] on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 910ms
I'm wondering if this is an issue related to the JavaReedSolomon project itself. Any help would be appreciated to help me run the tests.
Extra Information:
I was quite surprised when I found this.
After researching a bit, I found this in the blog post:
The math—and the code—works with any numbers as long as you have at least one data shard and don’t have more than 256 shards total.
If there is no way for this to work on bigger shard numbers, it should be mentioned in the library, and you should check it in the constructor.
Also - is the above correct? Shouldn't it be "and don’t have more than 256 DATA shards total". I don't have any problem creating an encoder with 256 data and 256 parity shards. But as soon as there is more than 256 data shards it fails.
In the generateLogTable
function, why does the expression b = ((b - FIELD_SIZE) ^ polynomial);
ensure that b
is set to FIELD_SIZE
different and non-repeating values?
Looking forward to your explanation, thank you.
If there are no plans for publishing this to the Central Maven Repo (see #15), then there should be a section for the built JAR files to download in order to make it easier to use this library and to promote its use, which would hopefully lead to developers actually taking care of this repository.
Do you have any plans to publish this to the central Maven repo?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.