@fjetter offline you expressed concern for dask-expr optimization performance. I'm observing a 50~150ms slowdown for the full TPCH queries.
IMHO it's negligible.

runtime for graph definition + optimization.
Note that it incorporates fetching the metadata of the input dataframe from s3, which I suspect takes the lion share's of both the mean time and the variance (this is just an intuition; I didn't collect numerical evidence about it).

end-to-end runtime on the Coiled cluster:

The other TPCH queries show similar behaviour.

from dask.

All PRs are now only waiting for review

from dask.

Summary of changes

• tokenize() is now deterministic, within the same interpreter, in most cases
• in the rare edge cases where it is not, you can trust tokenize(..., ensure_determinstic=True) to raise robustly
• there should be no expectation of determinism across interpreter restarts, hosts, OSs, or dependency versions

The issue of key collision on the scheduler (same key, but different run_spec and possibly different dependencies), which is chiefly caused by #9888, has been mitigated:

Legend

✔️ produces correct output
📛 cluster crashes or hangs on AssertionError
😕 task completes successfully, but output is wrong
⚠️ emits warning in the scheduler log

[1] this is not great and could deserve a follow-up

from dask.