Comments (5)
This turns out to be rather significant for us since we use the memory storage engine. Kernel-level automatic NUMA balancing "kernel.numa_balancing" doesn't help us (enough), basically we discovered that hot spots (certain processes would be much hotter CPU-wise than others) are largely the result of the kernel scheduling the process to a node and having memory allocation local to that node and then subsequently moving the process to another node (the latency impacts are significant in that case and difficult to debug).
[fdbserver.4500]
command = /usr/bin/numactl -m 0 -N 0 /usr/sbin/fdbserver
We are directly managing core and memory affinity with numactl hence.
from foundationdb.
I'd guess the difference will be small on an otherwise-unloaded machine unless you manage to make an fdbserver allocate some memory on the wrong NUMA domain. That should definitely cause noticeable overhead.
from foundationdb.
The performance effects of this sort of tuning tend to be fairly hardware and OS version specific, so I would recommend diverse testing before making this the default behavior out of the box. But an fdbserver parameter (and thus, config file option) to set affinity might be low risk and convenient.
Ideally operating systems should be able to do a reasonable job placing essentially single-threaded programs, even on NUMA machines. But I think setting affinity is a legitimate solution to a particular situation with network card interrupt delivery. It rings a bell for me. There might have been something written on this subject at some point in FDB history, if the Apple team wants to try to dig it up.
from foundationdb.
We messed with pinning the network queues to cores, etc. It only made minor differences with highly unusual test workloads. Never a practical improvement. I agree with @davidscherer 's suggestion of adding an option for CPU affinity.
from foundationdb.
The numactl command can also handle the set-cpu-affinity work.
Unless we want to start a (fdbserver) process on a particular core, it seems using the linux tools to set the affinity is enough.
from foundationdb.
Related Issues (20)
- Separate critical system metadata from other non-critical ones
- DR constantly growing disk space HOT 5
- 7.1 with tenanting upgrade to 7.3 crash HOT 3
- ClientWorkload bindings unstable ABI
- add Windows client dll tar package HOT 3
- Open FoundationDB in a Dev Container HOT 2
- Segmentation fault when emitting trace during a snapshot operation (7.3.27) HOT 10
- Expose PSI metrics in host metrics
- Deprecate support for Python2
- Remove the use of six.py in FoundationDB Python package
- Remove dependency of trollies in FDB Python package
- Add typing for FDB python binding package
- FoundationDB is unresponsive on MacOS ARM64 when disk usage is high HOT 1
- TestOpenNotExistTenant will always fail on release-7.3/release-7.1
- GetEstimatedRangeSizeBytes can cause PacketLimitExceeded error on commit proxies
- Benchmark `Trace.cpp:SuppressionMap`
- ASSERTion in flow may be improved by bringing in less branches HOT 3
- FDBLibTLS being used? HOT 1
- Use pathlib.Path in Directory layer in Python bindings if possible
- DR fails atomicity in 7.3.43 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from foundationdb.