How big is the calibration data, mapie uses to compute intervals about mapie HOT 3 CLOSED

Also wouldnt it make sense, in case of a regression problem, to sort the training data by the target feature before fitting it on mapie? I mean if you have samples which are sorted from lets say 100.000k to 600.000k Saleprice, in case of a housing problem, the leave-one-out-cv would basically calculate intervals in a space of values, which are simliar to each other and thus make more sense. For example fold1 = 100.000k -120.000k, fold2= 120.000k-140.000k and so on...

from mapie.

@vtaquet , I think the picture was specific to classification, at a time where we had not implemented cross-validation yet, only the split-conformal with cv="prefit" option. This picture is thus obsolete, and the size of the calibration set is defined by the number of calibration folds cv.

from mapie.

@gmartinonQM , the picture is indeed obsolete and should be updated in a future PR.

@nilslacroix , sorting the training data before splitting it into folds is up to the user and needs to be done before calling MAPIE. Your cross-validation strategy can be defined using the desired sklearn BaseCrossValidator object like KFold but keep in mind that the training and calibration sets need to have similar distributions.

from mapie.