pyeclib's Introduction
This is v1.0 of PyECLib. This library provides a simple Python interface for implementing erasure codes and is known to work with Python v2.6, 2.7 and 3.x. To obtain the best possible performance, the library utilizes liberasurecode, which is a C based erasure code library. Please let us know if you have any other issues building or installing (email: [email protected] or [email protected]). This library makes use of Jesasure for Reed-Solomon as implemented by the liberasurecode library and provides its' own flat XOR-based erasure code encoder and decoder. Currently, it implements a specific class of HD Combination Codes (see "Flat XOR-based erasure codes in storage systems: Constructions, efficient recovery, and tradeoffs" in IEEE MSST 2010). These codes are well-suited to archival use-cases, have a simple construction and require a minimum number of participating disks during single-disk reconstruction (think XOR-based LRC code). Examples of using this library are provided in "tools" directory: Command-line encoder:: tools/pyeclib_encode.py Command-line decoder:: tools/pyeclib_decode.py Utility to determine what is needed to reconstruct missing fragments:: tools/pyeclib_fragments_needed.py PyEClib initialization:: ec_driver = ECDriver(k=<num_encoded_data_fragments>, m=<num_encoded_parity_fragments>, ec_type=<ec_scheme>)) Supported ``ec_type`` values: * ``jerasure_rs_vand`` => Vandermonde Reed-Solomon encoding, based on Jerasure [1] * ``jerasure_rs_cauchy`` => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [2] * ``flat_xor_hd_3``, ``flat_xor_hd_4`` => Flat-XOR based HD combination codes, liberasurecode [3] * ``isa_l_rs_vand`` => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4] * ``shss`` => NTT Lab Japan's Erasure Coding Library A configuration utility is provided to help compare available EC schemes in terms of performance and redundancy:: tools/pyeclib_conf_tool.py The Python API supports the following functions: - EC Encode Encode N bytes of a data object into k (data) + m (parity) fragments:: def encode(self, data_bytes) input: data_bytes - input data object (bytes) returns: list of fragments (bytes) - EC Decode Decode between k and k+m fragments into original object:: def decode(self, fragment_payloads) input: list of fragment_payloads (bytes) returns: decoded object (bytes) *Note*: ``bytes`` is a synonym to ``str`` in Python 2.6, 2.7. In Python 3.x, ``bytes`` and ``str`` types are non-interchangeable and care needs to be taken when handling input to and output from the ``encode()`` and ``decode()`` routines. - EC Reconstruct Reconstruct "missing_fragment_indexes" using "available_fragment_payloads":: def reconstruct(self, available_fragment_payloads, missing_fragment_indexes) - Minimum parity fragments needed for durability gurantees:: def min_parity_fragments_needed(self) - Fragments needed for EC Reconstruct Return the indexes of fragments needed to reconstruct "missing_fragment_indexes":: def fragments_needed(self, missing_fragment_indexes) - Get EC Metadata Return an opaque buffer known by the underlying library:: def get_metadata(self, fragment) - Verify EC Stripe Consistency Use opaque buffers from get_metadata() to verify a the consistency of a stripe:: def verify_stripe_metadata(self, fragment_metadata_list) - Get EC Segment Info Return a dict with the keys - segment_size, last_segment_size, fragment_size, last_fragment_size and num_segments:: def get_segment_info(self, data_len, segment_size) - Get EC Segment Info given a data length and segment size:: def get_segment_info_byterange(self, ranges, data_len, segment_size) Assume a range request is given for an object with segment size 3K and a 1 MB file: Ranges = (0, 1), (1, 12), (10, 1000), (0, segment_size-1), (1, segment_size+1), (segment_size-1, 2*segment_size) This will return a map keyed on the ranges, where there is a recipe given for each range: { (0, 1): {0: (0, 1)}, (10, 1000): {0: (10, 1000)}, (1, 12): {0: (1, 12)}, (0, 3071): {0: (0, 3071)}, (3071, 6144): {0: (3071, 3071), 1: (0, 3071), 2: (0, 0)}, (1, 3073): {0: (1, 3071), 1: (0,0)} } Quick Start Install pre-requisites: * Python 2.6, 2.7 or 3.x (including development packages), argparse, setuptools * liberasurecode v1.0.4 or greater [3] * Erasure code backend libraries, gf-complete and Jerasure [1],[2], ISA-L [4] etc Install PyECLib:: $ sudo python setup.py install Run test suite included:: $ ./.unittests If all of this works, then you should be good to go. If not, send us an email! If the test suite fails because it cannot find any of the shared libraries, then you probably need to add /usr/local/lib to the path searched when loading libraries. The best way to do this (on Linux) is to add '/usr/local/lib' to:: /etc/ld.so.conf and then make sure to run:: $ sudo ldconfig References [1] Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org [2] Greenan, Kevin M et al, "Flat XOR-based erasure codes in storage systems", http://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf [3] liberasurecode, C API abstraction layer for erasure coding backends, https://bitbucket.org/tsg-/liberasurecode [4] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version [5] Kota Tsuyuzaki <[email protected]>, Ryuta Kon <[email protected]>, "NTT SHSS Erasure Coding backend" -- 1.0
pyeclib's People
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.