Modify feature importance ranking to use method 'min' about triage HOT 8 CLOSED

I would actually vote for mean over min—this is what most rank correlation procedures use—so [2,2,4,5,2] in @jtwalsh0's example.

from triage.

If features share the same feature importance score, pandas.DataFrame.rank(method='dense') gives them the same rank but does not increment the following ranks how we need them, e.g. if a list of feature importances looks like [0.1, 0.1, 0.2, 0.3, 0.1], rank would assign the following values: [1,1,2,3,1]. To calculate the feature percentile, we would need [1,1,4,5,1].

from triage.

@thcrock Use the min method rather than dense

from triage.

Thinking on this a bit more, percentiles are a bit trickier. I tend to default to the %-below definition because this is what's used in educational testing applications, but depending on how these are used internally, a different definition might be preferable. And depending on the definition, they have different relations to ranks and scores. This stack overflow discussion gets into some of the ickier parts of percentiles. http://math.stackexchange.com/questions/33488/how-to-calculate-percentile-is-it-possible-to-get-100-percentile

from triage.

Relevant to ranking discussions, seems like scipy.stats.percentileofscore is able to implement a few different methods of percentile rankings
https://docs.scipy.org/doc/scipy-0.19.0/reference/generated/scipy.stats.percentileofscore.html

from triage.

Is there one of those percentile methods that makes the most sense for the use case we would have for percentile rankings? Keeping in mind that we can always compute percentile rankings for edge cases using the raw scores if need be; so I think we can get away with just picking the most commonly useful method.

from triage.

I haven't been in on the most recent discussions on ranking, but this is what i went with for the custom baseline PercentileRankOneFeature (actually the title of the issue, lol):

>>> from scipy import stats
>>> ranks = stats.rankdata([1, 1, 1, 2, 2, 3, 4, 5], 'min')
>>> percentiles = [(rank - 1) / len(ranks) for rank in ranks]
>>> ranks
array([1, 1, 1, 4, 4, 6, 7, 8])
>>> percentiles
[0.0, 0.0, 0.0, 0.375, 0.375, 0.625, 0.75, 0.875]

This creates percentiles that follow the definition I gave above (percent of scores below this one), so the lowest score(s) has a percentile of 0. Have to subtract 1 when making percentiles because the ranks are 1-indexed.

But for the ranks themselves, I think I would want:

>>> from scipy import stats
>>> ranks = stats.rankdata([1, 1, 1, 2, 2, 3, 4, 5], 'average')
>>> ranks
array([ 2. ,  2. ,  2. ,  4.5,  4.5,  6. ,  7. ,  8. ])

So average for ranks but min for percentile rank.

from triage.

Moved to dssg/catwalk#4

from triage.