tagging system interaction with concurrent load/stores about cassette.jl HOT 2 OPEN

So I think this is running into known "undefined" behavior of Cassette's tagging system: "context-unobservable" mutation of "context-observed" state in the target program can corrupt the state's contextual metadata (and if the mutation is performed concurrently from different tasks, it can similarly induce race conditions in the corresponding metadata mutation).

Here, IIUC, a task is being popped from a global task queue before that task's metadata has even been enqueued in the task queue's metadata storage.

I've basically been ignoring this problem - the plan was to just document it as undefined behavior and call it a day - but now that people are trying to use the tagging system for nonnumerical work it seems now is the time to figure this out...

from cassette.jl.

Okay, I have a very vague start of an idea of how to proceed here.

Basically, my intuition is that Cassette will have to consider all Julia tasks + Julia's built-in scheduler/work queue to always be "context-observable", regardless of what the Julia program is actually doing. This should be similar to the way Cassette currently treats "tagged modules" - just part of the state of "Julia itself" that must be tracked as it could result in non-local interactions with contextual metadata.

I think this addition to Cassette's semantics might at least make it possible to discuss a definition for this behavior, though I'm still not sure yet what the definition will even be. For example, I'm not sure this helps that much with being able to actually identify when two tasks concurrently interfere with the same contextual metadata, or helps us decide what to do if primal read/writes are torn from metadata read/writes across tasks...hmmm...

from cassette.jl.