Deprecate compute keyword for set_index about dask HOT 2 CLOSED

Yeah that might make more sense, the current keyword is just weird

from dask.

Yeah, that code path does seem like an old optimization to me. When I first saw this issue, I thought the compute arg may have something to do with the new divisions calculation. However, as far as I can tell, this block is indeed persisting the shuffled collection to disk.

Personal Thoughts
I can actually imagine many cases where a disk-based persist mechanism like this can be useful. However, (1) I don't think many people are using "disk" for the cases that would benefit, and (2) I don't think this kind of mechanism makes much sense within the shuffle API like this. Instead, this kind of mechanism should be exposed to the user in such a way that it is clear that the query is being segmented. For example, the user should need to call something like df.persist(storage=..., shuffle_method=...) (just an example, not necessarily a suggested API).

from dask.