Swift Actor/Tasks concurrency on Linux - Lock contention in rand() about swift-corelibs-libdispatch HOT 18 CLOSED

#804

from swift-corelibs-libdispatch.

Is there any plan to solve this issue? It's reaching a third year since reported :(

from swift-corelibs-libdispatch.

CC @ktoso / @rokhinip / @rjmccall do you know what the plans w.r.t. Concurrency runtime and dispatch are? Clearly there are a bunch of things that need addressing and corelibs-dispatch isn't getting updated...

from swift-corelibs-libdispatch.

Hi everyone,
we've been doing a lot of thinking about the executor pool provided here by (corelibs) Dispatch for Swift Concurrency.
We're leaning towards a fresh thread-pool implementation, that would be tailored towards Swift Concurrency's needs specifically, rather than thinking about the DispatchQueue APIs explicitly. Instead of explicitly having 1:1 dispatch API's back Swift Concurrency on various platforms, I think we're more interested in specialized executor implementations per platform, which then Swift Concurrency uses.

Sadly this is no small feat and a large project in itself. We don't currently have more details to share on the long term.

We'd certainly welcome any PRs that could help remove the short-term pain, but medium-to-long term we think replacing the executors backing Swift Concurrency may be a preferable direction here.

from swift-corelibs-libdispatch.

The #804 helps the CPU usage of our stress workload, reduction from 100% utilisation of 30 CPU cores to ~18 cores on a 32 core system. Sadly, the throughput numbers of the workload do not change.

The lock contention in rand() is now replaced with spending 20% in _dispatch_root_queue_mediator_is_gone() :

When larger amount of Actors are present, apple/swift#68299 reproduces.

from swift-corelibs-libdispatch.

Thanks for verifying on your end @freef4ll

Merged and should be part of 5.9.3 https://forums.swift.org/t/development-open-for-swift-5-9-3-for-linux-and-windows/69020

from swift-corelibs-libdispatch.

@ilya-fedin thanks for the PR, we'll try to rerun the original workload test (@freef4ll could you please try to give it a spin with a benchmark perhaps so we can get before/after numbers and add that to #804 as further input).

We'd certainly welcome any PRs that could help remove the short-term pain, but medium-to-long term we think replacing the executors backing Swift Concurrency may be a preferable direction here.

@ktoso and @rjmccall - thanks for clarifying your thoughts on the possible future direction of the shared concurrency pool for Swift, I think the above quoted sentence would be pragmatic.

I think what you suggest is the right call (I remember when we spent some time getting libdispatch to work on Solaris back in the day - the mach/kqueue bits makes it a bit challenging to keep the codebase in sync as have been seen - Windows doesn't make it easier).

I also know that the needs of various users on different platforms can differ quite a bit too (e.g. back in the day we wanted to prioritise latency over energy efficiency and had a spin thread for picking up new work as a configurable default, for data center usage with many cores available, it was a great improvement for the stuff we did). One could envision a few different variations there even on the same OS platform. To structure the code base of such an API such that not only multiple platforms are easy to support, but also such that one could have a couple of variants per platform would be nice.

Also, just to mention a need to put it into your thought process - it's desirable to be able to pin executors to a given thread and it'd be nice if a future API made that fairly straightforward (not sure how the interaction with the concurrent pool and additional such threads would look like, but it'd be nice if it could be managed by the same code base...) - this is especially interesting for I/O threads where one might pin the thread to a specific core, which one designates to handle the interrupts from the network card (with the use of a user-land networking stack, this allows for low latency processing of inbound packets with good cache behaviour...).

from swift-corelibs-libdispatch.

A semi-related issue was present in Windows, that was resolved with: #453 / #455

from swift-corelibs-libdispatch.

Would it be possible to use the Windows optimization for Linux also possibly to work around this?

from swift-corelibs-libdispatch.

CC @rokhinip

from swift-corelibs-libdispatch.

Silence :(

from swift-corelibs-libdispatch.

What does this mean for me if I use libdispatch directly in a C++ application as a lock-free thread pool implementation? Is this variant of libdispatch officially discontinued?

from swift-corelibs-libdispatch.

We'd certainly welcome any PRs that could help remove the short-term pain

Would such a PR be ok?

diff --git a/src/shims/yield.h b/src/shims/yield.h
index 53eb800..c8e5eed 100644
--- a/src/shims/yield.h
+++ b/src/shims/yield.h
@@ -98,7 +98,7 @@ void *_dispatch_wait_for_enqueuer(void **ptr);
 #define _dispatch_contention_spins() \
                ((DISPATCH_CONTENTION_SPINS_MIN) + ((DISPATCH_CONTENTION_SPINS_MAX) - \
                (DISPATCH_CONTENTION_SPINS_MIN)) / 2)
-#elif defined(_WIN32)
+#else
 // Use randomness to prevent threads from resonating at the same frequency and
 // permanently contending. Windows doesn't provide rand_r(), so use a simple
 // LCG. (msvcrt has rand_s(), but its security guarantees aren't optimal here.)
@@ -108,12 +108,6 @@ void *_dispatch_wait_for_enqueuer(void **ptr);
                os_atomic_store(&_seed, _next * 1103515245 + 12345, relaxed); \
                ((_next >> 24) & (DISPATCH_CONTENTION_SPINS_MAX)) | \
                                (DISPATCH_CONTENTION_SPINS_MIN); })
-#else
-// Use randomness to prevent threads from resonating at the same
-// frequency and permanently contending.
-#define _dispatch_contention_spins() ({ \
-               ((unsigned int)rand() & (DISPATCH_CONTENTION_SPINS_MAX)) | \
-                               (DISPATCH_CONTENTION_SPINS_MIN); })
 #endif
 #define _dispatch_contention_wait_until(c) ({ \
                bool _out = false; \

from swift-corelibs-libdispatch.

Is this variant of libdispatch officially discontinued?

No. The Swift project is just considering whether it should continue to use libdispatch as the standard implementation of our thread pool on non-Darwin platforms.

What does this mean for me if I use libdispatch directly in a C++ application as a lock-free thread pool implementation?

Without any changes to make the thread pools coordinate, they'd both create their own threads. Out of the box, that means you could end up with 2 * ncpus threads — or more, given that normal uses of libdispatch don't hard-cap the thread count. IIRC, you can manually cap libdispatch's thread usage with environment variables, and I assume we'd want to offer the same capability with a Swift thread pool. We might also choose to offer direct APIs to schedule work on Swift's thread pool.

from swift-corelibs-libdispatch.

Without any changes to make the thread pools coordinate, they'd both create their own threads.

Both? Perhaps there's a misunderstanding: I don't use Swift, I just use libdispatch.

from swift-corelibs-libdispatch.

We'd certainly welcome any PRs that could help remove the short-term pain

Would such a PR be ok?

That seems abstractly right; could you adjust the comment and make a real PR?

from swift-corelibs-libdispatch.

Without any changes to make the thread pools coordinate, they'd both create their own threads.

Both? Perhaps there's a misunderstanding: I don't use Swift, I just use libdispatch.

Then you would not be affected by Swift's use of a different thread pool implementation.

from swift-corelibs-libdispatch.

Well, my question was about what it means for libdispatch bugs priority & maintenance in general :)

from swift-corelibs-libdispatch.