If we send an APPEND to an lbs server with an incorrect starting block number, the server does an exit(1). However, INTERFACES says:

APPEND: Request type = 0x00000002
...
        Response if the starting block # is incorrect:
        [4 byte status code = 1]

so that sounds like the server should be able to recover.

However, storage_write() contains:

        /* Sanity-check the write position. */
        if (blkno != S->nextblk) {
                warn0("Attempt to append data with wrong blkno");
                warn0("(%016" PRIx64 ", should be %016" PRIx64 ")",
                    blkno, S->nextblk);
                goto err2;
        }

which goes back to worker.c's workthread:

                case 1: /* Write */
                        if (storage_write(ctl->sstate,
                            ctl->blkno, ctl->nblks, ctl->buf)) {
                                warnp("Failure writing blocks");
                                exit(1);
                        }
                        break;

I'm happy to either:

• modify INTERFACES to eliminate this response, or
• modify the lbs server to recover from an APPEND with a bad starting block number

According to INTERFACE,

ADD:    Request type = 0x00000112
...
        Response (value set):
        [4 byte status code = 0]

        Response (value not set):
        [4 byte status code = 1]

However, this seems to be the opposite of what actually happens. kvlds/dispatch_mr.c, on line 419, sets:

                /* Record if we did something. */
                if (op != OP_NONE)
                        req->opdone = 1;

Then, on line 513,

                case PROTO_KVLDS_ADD:
                        if (proto_kvlds_response_add(B->WQ, R->ID,
                            req->opdone))
                                goto err0;
                        break;

where proto_kvlds_response_add() has been defined in lib/proto_kvlds_proto_kvlds.h as:

#define proto_kvlds_response_add(Q, ID, status) \
        proto_kvlds_response_status(Q, ID, status)

As a result, if we successfully add a key (say, starting from an rm -rf'd LBS server), we receive status 1.
(At first glance, this applies to MODIFY as well.)

I'm content change the code and update the tests to not fail [*], but I'm wondering if you actually want another status code. Namely (with extra verbosity),

• 0: value was set as a result of this ADD
• 1: value was not set because an error occurred
• 2: value was not set because there was already a key of this name

[*] at the moment, the only two tests for kvlds ADD are no-ops. One of those is explicitly marked as a no-op, but the other lacks a (should be a no-op) tag.

I don't think OpenSSL has any mechanism for preventing SIGPIPE. In libcperciva/network we use the POSIX-standard MSG_NOSIGNAL but we can't do that in network_ssl since OpenSSL is handling the socket itself.

@gperciva Can you test to confirm that we have a problem here? I'd suggest "open a TLS socket and write endless garbage into it, then kill the (TLS) listening daemon" -- the only catch being that you'll need to do something about certificates in order to get the TLS handshake to pass.

Assuming we indeed have an issue here, we probably want to set SO_NOSIGPIPE. Which doesn't work on Linux... but better to say "TLS sockets aren't available on Linux" than to have utilities which use them randomly crash.

Say vs. redis and memcache?

It's cool this server is FOSS. Keep doing awesome stuff.

The lbs-dynamodb daemon is currently crash-safe; that is, if lbs-dynamodb is killed and a new process is launched to replace it, data will be in a consistent state and every operation which has been acknowledged will be visible.

However, it is not concurrency-safe; if you launch a second lbs-dynamodb process operating against the same table, the two processes will walk over each other and generally screw up everything.

I don't have time to address this right now so I'm leaving some notes to myself about how I plan on fixing this. The key is to use DynamoDB conditional operations.

1. When storing blocks of data -- blocks which are only ever stored once and then deleted later, without ever being overwritten -- we can perform operations using the attribute_not_exists function. This of course would cause problems if a connection breaks and we need to reissue the DynamoDB request... but maybe in the rare case that we get a ConditionalCheckFailedException error we could check whether the value present is the one we were trying to store? (Or maybe we can test for attribute_not_exists OR the current value is the one we were storing?) This would however not prevent them from overwriting each other's metadata items.

2. Right now, an append operation consists of

• Storing new metadata indicating how far we plan on writing,
• Storing all of the data blocks except the last one,
• Storing the last block.
  The append operation is "committed" by the final block being stored. It might be better to change this to:
• Storing new metadata indicating how far we plan on writing,
• Storing all of the data blocks,
• Storing new metadata indicating that we have indeed written,
  i.e., adding a "last known valid block" pointer to the metadata in addition to the "first known unused block" pointer (and the "everything prior to this has been deleted" pointer). This would make it possible to rely solely on the metadata items to determine the state of the store.

3. If we make the above change, it should be possible to use conditional writes of the metadata items to avoid having concurrent lbs-dynamodb processes writing -- probably making use of the "generation number" values (which are used to find the most recent metadata item in the wear-leveling buffer) to avoid having two processes write at the same time, and a random "process ID" to ensure that if a write needs to be retried then lbs-dynamodb doesn't think it's conflicting with itself.

This would all require changes to the dynamodb-kv daemon as well, of course, since it currently doesn't have any knowledge of conditional operations.

1. if one uses an invalid TABLE name and receives an AccessDeniedException, it would be helpful if the error was something other than

dynamodb-kv: parsenum failed on : Invalid argument

arising from dynamodb-kv/capacity.c, line 137. I can work on printing a better message for that.

2. when creating a DybamoDB table, what should the primary key / partition key be?

The function SSL_set1_host is defined in LibreSSL 2.9 and later. I'm not 100% certain but I think we need to turn off some of the compatibility code to avoid symbol conflicts there?

Hi.

An error occurs

COMMAND: sudo ../../kvlds/kvlds -s stor/sock_kvlds -l stor/sock_lbs

ERROR: kvlds: Address must contain port number: stor/sock_kvlds
kvlds: Error resolving socket address: stor/sock_kvlds

http://www.tarsnap.com/kivaloo.html
https://github.com/Tarsnap/kivaloo/releases

tl;dr: There's an API mismatch between lbs/dispatch.h and kvlds/dispatch.h (which has carried over to all the other programs).

lbs/ and mux/ have a dispatch_init(), which obviously allocates things; and dispatch_done(), which unconditionally cleans up and frees the dispatcher. That seems nice and clean to me.

By contrast, kvlds/ (and likely other programs) doesn't have a dispatch_init(); instead, dispatch_accept() allocates a new dispatcher. More problematically, dispatch_done() can only be called if dispatch_alive(D) has previously returned zero. In particular, it checks that (D->dying == 1), which only happens if it has called dropconnection().

1. are you sure that you want to allocate & free the dispatch object all the time in kvlds? Of course we need to ensure that each connection starts from a known state, but surely that could be restricted to dispatch_accept(), while a new dispatch_init() handles parameters that don't change, such as kmax and vmax.

2. I'm sure that kvlds needs a function to handle cleaning up after a dying connection (including sanity checks), but it's a bit confusing for that to have the same name as the "unconditional clean up" dispatch_done() from lbs. Could we rename one or both types of functions? My first thought would be rename dispatch_done() -> dispatch_zombie_cleanup() in dynamodb-kv, kvlds, lbs-dynamodb, lbs-s3, s3. (I'm not a huge fan of zombies; I'm just trying to be clear about the conditional nature of that clean-up.)

And then I'd like to add a dispatch_done() which acts the same as lbs dispatch_done(), namely unconditional clean-up.

On the other hand, given that there's only two programs which have unconditional clean-up, maybe it's better to leave the other programs alone, and rename the function in lbs and mux to dispatch_unconditional_done() or something like that.

Do you have a particular intuition behind taking a particular time range (such as 50 to 60 seconds for bulk_update)?

In the attached pngs, it looks like the number of operations per second in bulk_update are randomly distributed. Here's 3 tests (I cancelled the last one a little bit early).

I'd be tempted to use the median, or the 25% & 75% quadrants, rather than the mean of a specific time range.
(As it happens, I spent the past 2 days working on perftests for spiped, so I have this in my mind.)

After building everything:

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        dynamodb-kv/dynamodb-kv
        lbs-dynamodb/lbs-dynamodb
        perftests/dynamodb_kv/test_dynamodb_kv
        tests/kvlds-ddbkv/test_kvlds