Calling CompletionStage.asDeferred() on the same future many times causes stack overflows about kotlinx.coroutines HOT 3 OPEN

Root cause

The root cause seems to be a bad interaction between CompletionStage.asDeferred and CompletableFuture. Specifically:

1. When future completes, it calls the CompletionStage.handle callback at

   kotlinx.coroutines/kotlinx-coroutines-core/jdk8/src/future/Future.kt

   Line 126 in 1bffe67

   handle { value, exception ->

   .
2. The callback calls result.completeExceptionally.
3. Completion of result triggers cancelFutureOnCompletion at

   kotlinx.coroutines/kotlinx-coroutines-core/jdk8/src/future/Future.kt

   Line 141 in 1bffe67

   result.cancelFutureOnCompletion(future)

   .
4. CompletableFuture.cancel triggers postComplete at https://github.com/openjdk/jdk/blob/jdk-21%2B26/src/java.base/share/classes/java/util/concurrent/CompletableFuture.java#L2512, which in turn runs (1) for the next Deferred, recursively.

Normally CompletableFuture runs all the callbacks iteratively, but due to triggering the cancel() function in a recursive context, it starts running callbacks recursively, and causes a stack overflow.

from kotlinx.coroutines.

Workaround

The above issue can be worked around by breaking the recursive CompletableFuture.cancel call. For example:

fun <T> CompletableFuture<T>.safeAsDeferred(): Deferred<T> {
    val parent = this
    val safeCompletableFuture = object : CompletableFuture<T>() {
        override fun cancel(mayInterruptIfRunning: Boolean): Boolean {
            return if (!parent.isDone) {
                parent.cancel(mayInterruptIfRunning)
            } else {
                parent.isCancelled
            }
        }
    }
    parent.handle {
        result, exception ->
        if (exception == null) {
            safeCompletableFuture.complete(result)
        } else {
            safeCompletableFuture.completeExceptionally(exception)
        }
    }
    return safeCompletableFuture.asDeferred()
}

When used in place of asDeferred in the reproducer above, this fixes the issue:

$ JAVA_HOME=/opt/homebrew/Cellar/openjdk@21/21.0.3/libexec/openjdk.jdk/Contents/Home kotlin test.main.kts 10000
Expected additional transformation to run. Actual: true
Expected 10000 Deferred<Unit> to complete exceptionally. Actual: 10000

And if you don't care about cancelling the Deferred, you can simply create another chained CompletableFuture, since apparently CompletableFuture cancellations doesn't propagate upstream:

future.thenApply { it }.asDeferred()

from kotlinx.coroutines.

Great job figuring out the problem!

I think this is worth reporting to the JDK maintainers as well. As far as I can see, nothing in the docs indicates that this may happen. Though nothing says that it won't, I think:

• https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/CompletableFuture.html#cancel(boolean) says that a successful cancellation will complete the dependents as well.
• https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/CompletableFuture.html#handle(java.util.function.BiFunction) says that it creates a dependent future that will synchronously invoke the handler in the current thread.
• The top-level docs say: "When two or more threads attempt to complete, completeExceptionally, or cancel a CompletableFuture, only one of them succeeds." It doesn't say what happens when one thread calls cancel and then calls cancel again before the first one finishes.

I think this should be fixed on the side of the JDK as well by detecting the same thread entering the cancellation procedure more than once. That said, we can work around this explicitly on our side: #4173 The downside of this workaround is that when several threads cancel coroutines related to one future, only one of them will deal with the work related to the cancellation now (as opposed to many threads sharing the load), but I don't think that's a significant issue.

from kotlinx.coroutines.