The anelace from anjor

anelace's People

Contributors

Watchers

Forkers

10d9e

anelace's Issues

Add Deterministic output tests

The current output has inconsistent output likely due to concurrency synchronization problems. It's important for us to build out a set of test cases to ensure data integrity/determinism.

Steps to reproduce:

❯ cat ../maint/testdata/deterministic/random-1K-payload.dat | ./anelace --emit-stdout car-v1-stream --emit-stderr none | stream-commp

Ran on 10-core/10-thread Apple M1 Pro
Processing took 0.00 seconds using 0.94 vCPU and 5.97 MiB peak memory
Performing 2 system reads using 1.00 vCPU at about 1.43 MiB/s
Ingesting payload of:            1,024 bytes

Forming DAG covering:            1,079 bytes of 2 logical nodes

Dataset deduped into:            1,024 bytes over 1 unique leaf nodes
Linked as streams by:               55 bytes over 1 unique DAG-PB nodes
Taking a grand-total:            1,079 bytes, 105.37% of original, 0.9x smaller


CommPCid: baga6ea4seaqa2bkanxejmzehurpsnltfj62o4ti6py6frmiray4nrztaicleqdy
Payload:                1337 bytes
Unpadded piece:         2032 bytes
Padded piece:           2048 bytes

CARv1 detected in stream

and again:

❯ cat ../maint/testdata/deterministic/random-1K-payload.dat | ./anelace --emit-stdout car-v1-stream --emit-stderr none | stream-commp

Ran on 10-core/10-thread Apple M1 Pro
Processing took 0.00 seconds using 0.81 vCPU and 5.88 MiB peak memory
Performing 2 system reads using 0.99 vCPU at about 1.36 MiB/s
Ingesting payload of:            1,024 bytes

Forming DAG covering:            1,079 bytes of 2 logical nodes

Dataset deduped into:            1,024 bytes over 1 unique leaf nodes
Linked as streams by:               55 bytes over 1 unique DAG-PB nodes
Taking a grand-total:            1,079 bytes, 105.37% of original, 0.9x smaller


CommPCid: baga6ea4seaqpfe7llywlyuv7lthty7m3taugxv45dwji6vznylr33i5xkzfzijy
Payload:                1337 bytes
Unpadded piece:         2032 bytes
Padded piece:           2048 bytes

CARv1 detected in stream

Note commp is baga6ea4seaqa2bkanxejmzehurpsnltfj62o4ti6py6frmiray4nrztaicleqdy in the first case and baga6ea4seaqpfe7llywlyuv7lthty7m3taugxv45dwji6vznylr33i5xkzfzijy in the second case. These values should be identical.