I feel like I parse too early however any work on this should wait until I can really profile this.

This implies the following functionality:

• Index support through the stack down to the page level.
• The concept of unique indexes.
• Transactional support for indexes.
• Failure of a statement on constraint violation. Unsure if I'll end up with a general constraint system from this.

Based on reading this really means implementing Btree indexes. They don't seem to be that bad to understand/implement.

First and most important question, how should the index layers work?
Are they transactional? (I don't think so until I implement a visability map)
How should the low level layer function?
Should I have an Index config struct I pass around or just a table + columns + unique or not + type
Index Config it is

Index Manager -> for a given table
IO Manager -> Handle Page Load / Store / Update

Implemented the formats but I think I need to add locking to the I/O manager.
At a minimum I need to support a get for update, update and release lock.
I'm not sure I understand how this should work :(. I think need to commit to another layer.

Back to indexes for now. I need to make a decision on how to handle them hitting the file system.
Postgres uses a series of OIDs to map onto disk.

I've been using uuids, I think I'm going to continue that. That would also solve the postgres fork approach.

Next up implementing the index manager to add entries to the index.

I'm having a hard time figuring this out, I might work to do the operations on the tree before I keep messing with the serialization protocols. I'm just worries they are directly linked.

Got further into the index manager. Unfortunately I need a lock manager to let it even pass the smell test. Time to go on a wild goose chase again! (This project is great for someone with ADHD to have fun on!)

The lock manager design/code is done but I'm not happy with using a rwlock to protect a tag. I really want to have the lock protect the content but that needs a way for me to support writeback. I think I need to build out two more things, a WAL mechanism and a buffer manager.

I guess I need to commit to doing this for reals. However I am worried about reaching a point of partially working for a while like when I did the type fixing. We'll see how this goes.

For now, the index implementation is now on hold until I get an integrated I/O subsystem and a stubbed out WAL.

Right now I have an index manager but its not part of the actual implementation.

Time to integrate it.

So I have a fully ready free space map but I can't avoid the locking issue anymore despite it also being the next item on the todo list.

So I started trying to bake in locks and I've hit my normal do I go for a read or write lock approach? I think I may have screwed up the lock manager design, oops too much in silos!

So what I really need in a lock manager is this:

• multi-threaded access
• ability to read page
• ability to write a page without causing collisions

I'm debating do I need to put a check-in / check-out layer on top of file manager.
So I could expose an api for the following:
Read - Easy
ReadForUpdate - This is the hard part
Ideally
Append - Needs to be serialized
//Ignoring Delete/shrink for now

I should treat this as a lesson for not implementing something outside the greater system and realizing I can't integrate it.

Right now I have licensing issues as identified by Fossa. These are not due to the code but the various support tools around the code.

To fix this I will need to restructure the project to separate testing tools from the core code. They are already separated but its hard for Fossa to tell. However since I'm not selling presenting this code as production ready, meh, I'll fix it later.

Implement support for running a fuzzer against the code base to ensure we are keeping the code at a high quality.

I've been using a series of layered providers I've been calling managers and its gotten overly complex.

I need a better naming convention and package organization structure.

See where I can pass read only data by reference instead of uisng Arc everywhere

Need to make a decision on the future of the project.

Currently page offsets are generated as part of the insertion into the file system. However this results in a bad assumption in a couple layers of the stack.

My issue is that the code ends up assuming when a new page is inserted that it will result in the logical next page.

If I can avoid the extraction I would prefer it, we'll see.

Postgres has arbitrary precision types that are critical for financial applications.

Implementing them will require a picking up a rust arbitrary precision library such as https://www.postgresql.org/docs/current/datatype-numeric.html

Right now the most reasonable library seems to have major limitations https://github.com/alkis/decimal and is basically a wrapper of the C library http://speleotrove.com/decimal/.

• https://github.com/reorg/pg_repack
• Investigate if the zheap table format would be better to implement.
• Until I get past a WAL implementation and planner costs I don't think its worth it.
• • Since I extended the size of transaction IDs, I probably have a larger issue on my hands than normal postgres.
• • • Reading into the zheap approach I'm thinking that I might have some space saving options availible for me. In particular if a tuple is frozen so its always availible I could remove the xmin/xmax and pack more into the page. Need more thinking however my approach of questioning the storage efficency of each part of data seems to be worth it.

Right now pgbench fails on an initial attempt to drop the tables. We should add drop support.

drop table if exists pgbench_accounts, pgbench_branches, pgbench_history, pgbench_tellers

• The codec that parses the network traffic is pretty naive. You could make the server allocate 2GB of data for a DDOS easily.
• • We should either add state to the codec or change how it parses to produce chunked requests. That means that when the 2GB offer is reached the server can react and terminate before we accept too much data. Its a little more nuanced than that, 2GB input might be okay but we should make decisions based on users and roles.

Currently new pages are selected based on appending to the end instead of based on pages being in use or not.

Switch this to using just the free space map.

Row manager will make use of the free/full markings.
Index manager will make use of them as inuse/not used.

At the moment I have the license set to AGPL3.0 but that IS NOT conducive to getting people interested in this as a library. Once I get to a MVP I need to consider strongly relicensing to BSD/Apache/MIT something.

As I write this I might make the decision sooner than later.

Did more thinking, I should implement postgres's streams concept so that I don't need to do lookups to find associated metadata on an object.
I thought I was going to get to use Uuid+page offset. I think its now going to be uuid+page offset+ type.

struct PageId + enum PageType should do it (done).

Currently all pages in the system are being handled via Tokio's BytesMut struct and related traits. This struct is awesome for giving me easy read/write methods on an underlying set of pointers.

However I have discovered a critical issue that I am uncertain how to solve effectively.

So right now I have a combination of features in my lock/cache system:

• Reader/Writer locks for a given page
• A least recently used cache of pages
• A way to check out pages and if discarded a way to purge the cache.

The issue I am facing is as follows:

1. A page is is checked out for read or write.
2. Read/writes are done to it but are discarded for any number of reasons.
3. The next process checks out the same page.
4. The process will not see the entire page since it has been consumed partially. This leads to interesting and painful bugs as I discovered in my index work.

Right now the main function runs the server from primitives. The Tokio Tower layer will probably do it better.

A lot of my low level code is very heavy and sensitive to locks. The Tokio project has a library called loom that seems to offer a reasonable way to test this type of code for correctness.

I suspect I'll have to move away from Github based runners once I do so.

https://github.com/tokio-rs/loom

The BTreeLeafs have a doubly linked list to scan back and forth once you reach the bottom.

They are stored in multiple pages so if the database crashes they could be corrupted.

To work around this until I figure out a better way I can scan each leaf on database startup.

Will need to determine if the database shuts down unclean or not.

Acceptance Criteria:

• Should be able to update a row either inside a page or not without loosing commits.
• This is independent of transaction control so I think this sits below/in row manager.

Will likely need to have to start tracing columns from analyizing through to later stages.