Area under the receiving operating characteristic curve (AUROC) calculation. about dask-ml HOT 2 OPEN

from dask-ml.

I don't know enough about this specific ML metric, but maybe if we can translate the problem to something more general then I can be of use.

For example, If someone had a distribution of values spread in an unsorted way across many machines and they wanted to plot an approximate sorted distribution of those points then there are a couple things they could do:

1. The simplest would be to just sample. I suspect that it would be easy to get very good accuracy by sampling.

2. But if for some reason they wanted to have all points influence the result then my recommendation would be to calculate a lot of approximate quantiles, for which there are good distributed algorithms. I suspect once one had, say, 100 percentiles (0%, 1%, 2%, ..., 99%, 100%) that one could take those values and reconstruct a distribution curve that was pretty close to full accuracy.

   Depending on the exact curve necessary one might need to invert the delta between neighboring percentiles or something similar. This might get mildly complex, but is the sort of thing one can probably do with ten minutes of thought and, importantly, without distributed systems experience.

I may be entirely on the wrong track through. @stephenpardy can you help by reducing your problem a little bit to general numbers / computational terms? I suspect that it'll be easier for folks to help in that case.

from dask-ml.