Multithread scalability issue -- BinlogQueue lock contention about ssdb-rocks HOT 2 OPEN

Hi,

1. The purpose of Binlog is for replication, the server will log every write to it and send it to a slave, the slave will replay the write operation. If you don't need master-slave replication, then you can set binlog: no in ssdb.conf.
2. The lock is necessary. For set operation, actually it is two writes: the key-value, and the binlog item, the two must be performed in one WriteBatch.

from ssdb-rocks.

Thanks, since I do not need replication for my use-case, I have eliminated the use of WriteBatch and made each Put and Get operation go directly to the rocksdb database, ie: db->Put() and db->Get(), instead of adding operations to a lock-protected WriteBatch and then calling db->Write().

However, I still see a 15 ms performance gap between average and 95th percentile latency. Have you observed this as well? I see a 15 ms performance gap between avg and p95 latency even when I only use 1 reader and 1 writer and I thread-pin them to separate cores to avoid interference from the linux scheduler. So I wonder if there is some rare event (that occurs ~5% of the time) that causes ssdb to sleep or delay execution of a request for ~15ms. I am currently going through the code in detail, but if you have any ideas for explaining the latency spikes that occur 5% of the time, that would be very helpful. Thank you!

from ssdb-rocks.