Snapshot transfer performance about rqlite HOT 8 OPEN

My test host is an AWS EC2 instance m5.2xlarge with 8 vCPUs, 32.0 GiB RAM, and EBS SSD storage.

from rqlite.

@aderouineau -- when a Snapshot is transferred to a node, in your mental model how many copies of the SQLite database are made on the receiving node?

from rqlite.

On the receiving node two copies are made of the received snapshot -- one to that copies the snapshot to its internal store, the other which then takes that copy, and makes the working SQLite copy that rqlite uses to serve queries.

But I see you're saying the network transfer speed itself is slower than you expect. One other thing to be aware is that rqlite gzip-compresses the snapshot on the sending side, and decompresses it on the receiving side. It does this to reduce network bandwidth.

In your testing are you streaming the data over the local network, in addition to running your dd test? It doesn't look like it. If you want to simulate closely with two rqlite nodes are doing, it's like this.

• node1 is reading from disk
• node1 is compressing
• node1 is sending over the network
• node 2 is receiving on the network
• node 2 is decompressing
• node 2 is writing to disk

from rqlite.

Both nodes are running on the same machine and using localhost

from rqlite.

Both nodes are running on the same machine and using localhost

I understand, I'm just pointing that the dd test you are running is not fully simulating what is actually happening when one node transfers a snapshot to another node (assuming I'm not missing anything).

from rqlite.

I'll look into a better benchmark and see if this is still a problem.

from rqlite.

Maybe run a test using netcat. I think if you put gzip and netcat between the dd invocations that could be a good test.

dd -> gzip -> nc -> nc -> gunzip -> dd

from rqlite.

Correct comment above.

from rqlite.