Consider setting CPU affinity for fdbserver processes about foundationdb HOT 5 OPEN

This turns out to be rather significant for us since we use the memory storage engine. Kernel-level automatic NUMA balancing "kernel.numa_balancing" doesn't help us (enough), basically we discovered that hot spots (certain processes would be much hotter CPU-wise than others) are largely the result of the kernel scheduling the process to a node and having memory allocation local to that node and then subsequently moving the process to another node (the latency impacts are significant in that case and difficult to debug).

[fdbserver.4500]
command = /usr/bin/numactl -m 0 -N 0 /usr/sbin/fdbserver

We are directly managing core and memory affinity with numactl hence.

from foundationdb.

I'd guess the difference will be small on an otherwise-unloaded machine unless you manage to make an fdbserver allocate some memory on the wrong NUMA domain. That should definitely cause noticeable overhead.

from foundationdb.

The performance effects of this sort of tuning tend to be fairly hardware and OS version specific, so I would recommend diverse testing before making this the default behavior out of the box. But an fdbserver parameter (and thus, config file option) to set affinity might be low risk and convenient.

Ideally operating systems should be able to do a reasonable job placing essentially single-threaded programs, even on NUMA machines. But I think setting affinity is a legitimate solution to a particular situation with network card interrupt delivery. It rings a bell for me. There might have been something written on this subject at some point in FDB history, if the Apple team wants to try to dig it up.

from foundationdb.

We messed with pinning the network queues to cores, etc. It only made minor differences with highly unusual test workloads. Never a practical improvement. I agree with @davidscherer 's suggestion of adding an option for CPU affinity.

from foundationdb.

The numactl command can also handle the set-cpu-affinity work.
Unless we want to start a (fdbserver) process on a particular core, it seems using the linux tools to set the affinity is enough.

from foundationdb.