combine_first: conditional type-cast to rhs's dtype about dask HOT 6 OPEN

@jrbourbeau @phofl how should we deal with this? It's impossible to replicate pandas' current (and deprecated) behaviour, as it would require us to eagerly compute everything until that moment for lhs in order to determine its dtype:

out.dtype = dd.isna(lhs).all().compute() ? rhs.dtype : lhs.dtype

The current behaviour (which I'm not too excited about) is to have out.dtype = in.dtype for the dask collection and to match pandas for the computed output; in other words out.dtype != out.compute().dtype.
There may however be FutureWarnings that don't happen in pandas as the isna(lhs).all() is tested individually for every chunk (the PR linked below makes a trivial example).
For datetime64, the current behaviour is to always return lhs.dtype, which differs from pandas.

One option is to quietly perform rhs.astype(lhs.dtype); this means that dask will change its output overnight to what's going to be in pandas 3.0(?).
Another option is do nothing, which means dask will keep occasionally producing spurious FutureWarnings and/or have mismatched dtype at times.

from dask.

A third option is to print a RuntimeWarning at graph definition time every time lhs.dtype != rhs.dtype, and then implement pandas 3's behaviour of sticking with lhs.dtype.

from dask.

we can just filter the warning and be done with it?

from dask.

we can just filter the warning and be done with it?

that would mean that out.compute().dtype != out.dtype, and that the output dtype will change overnight when the user upgrades to pandas 3.

from dask.

that would mean that out.compute().dtype != out.dtype

This happens in multiple places, I wouldn't worry too much about it.

that the output dtype will change overnight when the user upgrades to pandas 3.

Not great but could be worse imo.

I wouldn't spend too much time on this, having all nan partitions in this combination is not something that is very likely and combine_first is not used much in the pandas space

from dask.

#10932 fixes the flaky test

from dask.