ptravers / spq Goto Github PK

We currently force big endianess during write in the Storage package. We want to validate that this is appropriate on little endian machines.

Given A little endian system
When An entry is written in to a durable storage instance
Then The entry should be read back with the same key

FIx all clippy warnings in queue and server

We would like to:

1. Make memory usage in storage layer configurable
2. Make SPQ itself detect memory limits and configure storage layer appropriately

Given Multiple nodes are in a cluster able to contact each other
When A request to take an item is sent
Then Every node in the cluster should process the request
Then Every node if peeked should have the same next item

Currently we are using an ugly hack to test durability. This is mostly due to my lack of knowledge and understanding of docker networking.

We do not want to require the user to grant super user whilst running the integration tests. We do want to be able to test if the database is durable between instances.

Given A SPQ instance has collected some data
When The SPQ instance exits and is restarted
Then The SPQ instance should have the same data as before it exited

WTH is a node or node value (long story short it's a mess)

Each node in SPQ has a set of associated values. A node represents a feature in a path of features. For a feature space with 2 features there are at minimum two nodes.

Currently the following are tracked per node per value in the feature hierarchy:
value - the value in the feature space this node value represents
items at index - the number of items in paths through the tree below this node
child index - the index of the node that contains the children of this value
hash - the index in the storage layer at which the epoch of the value is being stored

Each node contains
A list of node values
The name of the feature the node represents
If the node has leaf nodes beneath it

We want to flatten this structure ideally into one rocksdb table

Currently the queue if killed part way through dequeuing or enqueuing an item will create invalid partial state:

1. During enqueue the number of items available on the path will have been decremented without an item having been removed. If the enqueue fails after any node value in the path has had the number if items available decremented.
2. The epoch can have been incremented without having been used during dequeue. Which does not have a pathological effect on the system but is definitely not clean.
3. We would in the case of partial failure on enqueue. Add an item to the items table and never add it's features to the feature space. Thereby accruing junk data in the items table. The item would eventually be dequeued, if another item with the same features was added successfully later. The possibility of failure during enqueue resulting in a successful dequeue is pathological.

At minimum we need the process of removing an item to be atomic. As we can otherwise arrive in a state where we are not returning the fairest item.

Currently the queue is not safe for concurrency. We will want to remove the read write lock on the queue in the server

Given A running instance of SPQ
When A request is sent to create a queue
Then A new addressable queue should be created
Then All other operations must reference the queue they wish to work with by name

We may wish to refactor as part of this to have a single RPC for all queue operations.

Given Multiple SPQ nodes
When The nodes attempt to replicate
Then They can discover all the other nodes

We should wait for performance evaluations before completing this ticket to make sure we have some data around what bottlenecks this refactor introduces or removes

we added a hack comparing if the prefix of the key equals exactly the prefix provided to counter issues with the prefix iterator returning partial matches to the prefix.

We would like to instead find a way to do this in rocksdb not in the prefix storage struct.