ptravers / spq Goto Github PK
View Code? Open in Web Editor NEWSorting Priority Queue
Sorting Priority Queue
We currently force big endianess during write in the Storage package. We want to validate that this is appropriate on little endian machines.
Given A little endian system
When An entry is written in to a durable storage instance
Then The entry should be read back with the same key
FIx all clippy warnings in queue and server
We would like to:
Given Multiple nodes are in a cluster able to contact each other
When A request to take an item is sent
Then Every node in the cluster should process the request
Then Every node if peeked should have the same next item
Currently we are using an ugly hack to test durability. This is mostly due to my lack of knowledge and understanding of docker networking.
We do not want to require the user to grant super user whilst running the integration tests. We do want to be able to test if the database is durable between instances.
Given A SPQ instance has collected some data
When The SPQ instance exits and is restarted
Then The SPQ instance should have the same data as before it exited
Each node in SPQ has a set of associated values. A node represents a feature in a path of features. For a feature space with 2 features there are at minimum two nodes.
Currently the following are tracked per node per value in the feature hierarchy:
value - the value in the feature space this node value represents
items at index - the number of items in paths through the tree below this node
child index - the index of the node that contains the children of this value
hash - the index in the storage layer at which the epoch of the value is being stored
Each node contains
A list of node values
The name of the feature the node represents
If the node has leaf nodes beneath it
We want to flatten this structure ideally into one rocksdb table
Currently the queue if killed part way through dequeuing or enqueuing an item will create invalid partial state:
At minimum we need the process of removing an item to be atomic. As we can otherwise arrive in a state where we are not returning the fairest item.
Currently the queue is not safe for concurrency. We will want to remove the read write lock on the queue in the server
Given A running instance of SPQ
When A request is sent to create a queue
Then A new addressable queue should be created
Then All other operations must reference the queue they wish to work with by name
We may wish to refactor as part of this to have a single RPC for all queue operations.
Given Multiple SPQ nodes
When The nodes attempt to replicate
Then They can discover all the other nodes
We should wait for performance evaluations before completing this ticket to make sure we have some data around what bottlenecks this refactor introduces or removes
we added a hack comparing if the prefix of the key equals exactly the prefix provided to counter issues with the prefix iterator returning partial matches to the prefix.
We would like to instead find a way to do this in rocksdb not in the prefix storage struct.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.