Binning method for continuous values about lime HOT 4 CLOSED

Hello,
I did quartiles just because it was quick and easy to implement. I've also had it happen to me that most point lie in one quartile, so that the discretization is not so good.

Whether or not we want to discretize is a question that I've been pondering for a while. Anyway, to answer your question: if we are discretizing, I think smarter discretization methods would probably be helpful.

from lime.

Hi Marco,

I've implemented a entropy based discretizer, using scikit entropy decision trees, here: https://github.com/marcbllv/lime if ever you have time and would like to try it.

The discretizers are in a separated file, with an abstract class & children classes where specific binnings are implemented. I mostly used your code for the base class. The labels now needs to be passed to the explainer to use supervized binning, but not passing them is ok if keeping the QuartileDiscretizer (default None), so it should keep compatibility.

I've tested it on classification (not regression yet) and it seems to give nice results as it explores more the extreme values than with the quartiles.

I don't know if this needs to be integrated in your own repo ? I mostly did it for testing purposes on my side but if you're interested let me know :-)

from lime.

I like how you've used the Decision Tree to do the discretization. I think it would perhaps make more sense if the discretization did not depend on the labels, but only on the data itself.
Anyway, I'm a bit slammed for the next few weeks, but as soon as I have some time I will try out your discretization. If it seems to make more sense for a few datasets, we should merge it in LIME. If you have particular examples where the quartile discretizer is bad and the entropy one is good, send them to me : ).

Thanks,

from lime.

@mrkaiser included this in a recent pull request, so closing this issue.

from lime.