Is your feature request related to a problem? Please describe.
At the moment, crossval aggregation proposed in Algorithm 2 of Romano et al. (2020) is only available for the "score" method.

Describe the solution you'd like
Implement the crossval aggregation also for the "cumulated_score" method.

Is your documentation request related to a problem? Please describe.
In your documentation you show time complexity but not space complexity for your different methods.
I was quite confused when the CV+ ( keyword cv=10 and method="plus" ) for example needed to construct
a matrice of n_training_samples*n_test_samples. This gets big quite fast, even with around 60 features and 200k samples i needed around 50 GB.

Describe the solution you'd like
Adding space complexity to the table.

Describe the bug
Using this regressor with the normal fit function produced an key error.
I tried converting between numpy and pandas but it did not solve the issue.

Code:

mapie = MapieRegressor(regressor, method="plus", cv=4)
mapie.fit(pd.DataFrame(X_train), pd.DataFrame(y_train))

``

ClfSwitcher(estimator=LGBMRegressor(bagging_fraction=0.4, bagging_freq=4,
                                    bagging_seed=15871193,
                                    feature_fraction=0.11,
                                    feature_fraction_seed=15871193,
                                    learning_rate=0.007, max_bin=63,
                                    min_data_in_leaf=10, n_estimators=4000,
                                    num_leaves=6, objective='regression',
                                    random_state=15871193))

This was the error I received, for some reason indexes 648-2591 seem to be missing (samplesize =2592).
The scikit vanilla gscv.fit function works flawless.

`---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_16976/240006600.py in
4 az = akc[0][2].fit(pipe_nyst.fit_transform(X_train), y_train)
5 mapie = MapieRegressor(az, method="plus", cv=4)
----> 6 mapie.fit(pd.DataFrame(pipe_nyst.fit_transform(X_train)), pd.DataFrame(y_train))
7 #y_pred[name], y_pis[name] = mapie.predict(X_test, alpha=0.05)

~\miniconda3\envs\Master_ML\lib\site-packages\mapie\regression.py in fit(self, X, y, sample_weight)
495 self.n_samples_val_ = [X.shape[0]]
496 else:
--> 497 outputs = Parallel(n_jobs=self.n_jobs, verbose=self.verbose)(
498 delayed(self._fit_and_predict_oof_model)(
499 clone(estimator),

~\miniconda3\envs\Master_ML\lib\site-packages\joblib\parallel.py in call(self, iterable)
1039 # remaining jobs.
1040 self._iterating = False
-> 1041 if self.dispatch_one_batch(iterator):
1042 self._iterating = self._original_iterator is not None
1043

~\miniconda3\envs\Master_ML\lib\site-packages\joblib\parallel.py in dispatch_one_batch(self, iterator)
857 return False
858 else:
--> 859 self._dispatch(tasks)
860 return True
861

~\miniconda3\envs\Master_ML\lib\site-packages\joblib\parallel.py in _dispatch(self, batch)
775 with self._lock:
776 job_idx = len(self._jobs)
--> 777 job = self._backend.apply_async(batch, callback=cb)
778 # A job can complete so quickly than its callback is
779 # called before we get here, causing self._jobs to

~\miniconda3\envs\Master_ML\lib\site-packages\joblib_parallel_backends.py in apply_async(self, func, callback)
206 def apply_async(self, func, callback=None):
207 """Schedule a func to be run"""
--> 208 result = ImmediateResult(func)
209 if callback:
210 callback(result)

~\miniconda3\envs\Master_ML\lib\site-packages\joblib_parallel_backends.py in init(self, batch)
570 # Don't delay the application, to avoid keeping the input
571 # arguments in memory
--> 572 self.results = batch()
573
574 def get(self):

~\miniconda3\envs\Master_ML\lib\site-packages\joblib\parallel.py in call(self)
260 # change the default number of processes to -1
261 with parallel_backend(self._backend, n_jobs=self._n_jobs):
--> 262 return [func(*args, **kwargs)
263 for func, args, kwargs in self.items]
264

~\miniconda3\envs\Master_ML\lib\site-packages\joblib\parallel.py in (.0)
260 # change the default number of processes to -1
261 with parallel_backend(self._backend, n_jobs=self._n_jobs):
--> 262 return [func(*args, **kwargs)
263 for func, args, kwargs in self.items]
264

~\miniconda3\envs\Master_ML\lib\site-packages\mapie\regression.py in _fit_and_predict_oof_model(self, estimator, X, y, train_index, val_index, k, sample_weight)
366
367 """
--> 368 X_train, y_train, X_val = X[train_index], y[train_index], X[val_index]
369 if sample_weight is None:
370 estimator = fit_estimator(estimator, X_train, y_train)

~\miniconda3\envs\Master_ML\lib\site-packages\pandas\core\frame.py in getitem(self, key)
3459 if is_iterator(key):
3460 key = list(key)
-> 3461 indexer = self.loc._get_listlike_indexer(key, axis=1)[1]
3462
3463 # take() does not accept boolean indexers

~\miniconda3\envs\Master_ML\lib\site-packages\pandas\core\indexing.py in _get_listlike_indexer(self, key, axis)
1312 keyarr, indexer, new_indexer = ax._reindex_non_unique(keyarr)
1313
-> 1314 self._validate_read_indexer(keyarr, indexer, axis)
1315
1316 if needs_i8_conversion(ax.dtype) or isinstance(

~\miniconda3\envs\Master_ML\lib\site-packages\pandas\core\indexing.py in _validate_read_indexer(self, key, indexer, axis)
1372 if use_interval_msg:
1373 key = list(key)
-> 1374 raise KeyError(f"None of [{key}] are in the [{axis_name}]")
1375
1376 not_found = list(ensure_index(key)[missing_mask.nonzero()[0]].unique())

KeyError: "None of [Int64Index([ 648, 649, 650, 651, 652, 653, 654, 655, 656, 657,\n ...\n 2582, 2583, 2584, 2585, 2586, 2587, 2588, 2589, 2590, 2591],\n dtype='int64', length=1944)] are in the [columns]"`

Expected behavior
I would expect a fitted estimator

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS Windows 10
• Python 3.9, Jupyter Notebook 6.4, conda 4.12

The J+aB and APS methods should be described in the documentation (in tutorials and/or examples). Tutorials in .rst format should be built directly from notebooks with jupyter nbconvert for instance.

As described in section 2.2 of this tutorial:
http://people.eecs.berkeley.edu/~angelopoulos/publications/downloads/gentle_intro_conformal_dfuq.pdf

Is your documentation request related to a problem? Please describe.
I found this library interesting and was going through the library code examples where I found a code error in the plotting of interval widths for the section "Estimating the uncertainty with different sklearn-compatible regressors". It uses a variable y_preds which is not defined earlier anywhere in the code. Besides, a few import statements are missing from sklearn methods, I have made the changes. Should I send a PR for the same?

Describe the solution you'd like
The incorrect line of code can be fixed by using y_pis instead of undefined y_preds and using the correct dimensions for the data. The import statements for the missing sklearn methods can be added as well.

Describe the bug
Dear colleagues, I am creating a system to classify customers in 2 binary classes and then apply a regression model to one of the classes.

Some of my features are string that I obviously need to encode. In this case with one hot encoding.

To Reproduce

My code is as follows:

from sklearnex import patch_sklearn
patch_sklearn()

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer, TransformedTargetRegressor, make_column_selector as selector 
from sklearn.preprocessing import StandardScaler, OneHotEncoder, LabelEncoder 
from sklearn.impute import SimpleImputer
#from sklearn.model_selection import cross_val_score, cross_validate
from sklearn.multioutput import RegressorChain, MultiOutputRegressor
from sklearn.feature_selection import VarianceThreshold
from sklearn.metrics import mean_absolute_error, make_scorer, mean_tweedie_deviance, auc
from sklearn.model_selection import RandomizedSearchCV, train_test_split, LeaveOneGroupOut, LeavePGroupsOut, cross_validate
from sklearn.metrics import roc_auc_score, plot_roc_curve, roc_curve, confusion_matrix, classification_report, ConfusionMatrixDisplay
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingRegressor, HistGradientBoostingClassifier

from sklearn import set_config
from mapie.regression import MapieRegressor

data_train, data_test, target_train, target_test = train_test_split(
    df.drop(columns=target_reg + target_class + METADATA_COLUMNS), 
    df[target_reg + target_class], 
    random_state=42)

categorical_columns_of_interest = ['col1', 'col2', 'col3', 'col4', 'col5', 'col6', 'col7']
numerical_columns = ml_data.drop(columns=target_reg + target_class + METADATA_COLUMNS).select_dtypes(include=np.number).columns
numerical_columns = [x for x in MY_FEATURES if x not in FEATURES_NOT_TO_IMPUTE]
numerical_columns = [x for x in numerical_columns if x not in categorical_columns_of_interest]

categorical_transformer = OneHotEncoder(handle_unknown="ignore")
numeric_transformer = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="mean")), 
        ("scaler", StandardScaler()),
        ("variance_selector", VarianceThreshold(threshold=0.03))
        ]
)
preprocessor = ColumnTransformer(
    transformers=[
        ("numeric_only", numeric_transformer, numerical_columns),
        ("get_dummies", categorical_transformer, categorical_columns_of_interest)])

pipeline_hist_boost_reg= Pipeline([('preprocessor', preprocessor),
                             ('estimator', HistGradientBoostingRegressor())])

regressor = TransformedTargetRegressor(pipeline_hist_boost_reg, func=np.log1p, inverse_func=np.expm1)

mapie_estimator = MapieRegressor(pipeline_hist_boost_reg)
mapie_estimator.fit(data_train, target_train)

Expected behavior
After this I will expect that I can run:

y_pred, y_pis = mapie_estimator.predict(data_test)

Screenshots

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-43-61f70ee71787> in <module>
      1 mapie_estimator = MapieRegressor(pipeline_hist_boost_reg)
----> 2 mapie_estimator.fit(X_train_reg, y_train_reg)

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/mapie/regression.py in fit(self, X, y, sample_weight)
    457         cv = self._check_cv(self.cv)
    458         estimator = self._check_estimator(self.estimator)
--> 459         X, y = check_X_y(
    460             X, y, force_all_finite=False, dtype=["float64", "int", "object"]
    461         )

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/sklearn/utils/validation.py in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
    962         raise ValueError("y cannot be None")
    963 
--> 964     X = check_array(
    965         X,
    966         accept_sparse=accept_sparse,

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/sklearn/utils/validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
    683     if has_pd_integer_array:
    684         # If there are any pandas integer extension arrays,
--> 685         array = array.astype(dtype)
    686 
    687     if force_all_finite not in (True, False, "allow-nan"):

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/pandas/core/generic.py in astype(self, dtype, copy, errors)
   5804         else:
   5805             # else, only a single dtype is given
-> 5806             new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
   5807             return self._constructor(new_data).__finalize__(self, method="astype")
   5808 

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/pandas/core/internals/managers.py in astype(self, dtype, copy, errors)
    412 
    413     def astype(self: T, dtype, copy: bool = False, errors: str = "raise") -> T:
--> 414         return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
    415 
    416     def convert(

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/pandas/core/internals/managers.py in apply(self, f, align_keys, ignore_failures, **kwargs)
    325                     applied = b.apply(f, **kwargs)
    326                 else:
--> 327                     applied = getattr(b, f)(**kwargs)
    328             except (TypeError, NotImplementedError):
    329                 if not ignore_failures:

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/pandas/core/internals/blocks.py in astype(self, dtype, copy, errors)
    590         values = self.values
    591 
--> 592         new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
    593 
    594         new_values = maybe_coerce_values(new_values)

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/pandas/core/dtypes/cast.py in astype_array_safe(values, dtype, copy, errors)
   1298 
   1299     try:
-> 1300         new_values = astype_array(values, dtype, copy=copy)
   1301     except (ValueError, TypeError):
   1302         # e.g. astype_nansafe can fail on object-dtype of strings

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/pandas/core/dtypes/cast.py in astype_array(values, dtype, copy)
   1246 
   1247     else:
-> 1248         values = astype_nansafe(values, dtype, copy=copy)
   1249 
   1250     # in pandas we don't store numpy str dtypes, so convert to object

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/pandas/core/dtypes/cast.py in astype_nansafe(arr, dtype, copy, skipna)
   1083         flags = arr.flags
   1084         flat = arr.ravel("K")
-> 1085         result = astype_nansafe(flat, dtype, copy=copy, skipna=skipna)
   1086         order: Literal["C", "F"] = "F" if flags.f_contiguous else "C"
   1087         # error: Item "ExtensionArray" of "Union[ExtensionArray, ndarray]" has no

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/pandas/core/dtypes/cast.py in astype_nansafe(arr, dtype, copy, skipna)
   1190     if copy or is_object_dtype(arr.dtype) or is_object_dtype(dtype):
   1191         # Explicit copy, or required since NumPy can't view from / to object.
-> 1192         return arr.astype(dtype, copy=True)
   1193 
   1194     return arr.astype(dtype, copy=copy)

ValueError: could not convert string to float: 'group C'

Being this value part of one of the categorical columns which its being encoded by the preprocessor.

When training the model without mapie everything works correctly:

Desktop (please complete the following information):

import platform
print(platform.machine())
print(platform.version())
print(platform.platform())
print(platform.system())
print(platform.processor())

x86_64
#58~18.04.1-Ubuntu SMP Wed Jul 28 23:14:18 UTC 2021
Linux-5.4.0-1056-azure-x86_64-with-glibc2.10
Linux
x86_64

Scikit learn dependencies:

scikit-learn==1.0.2
scikit-learn-intelex==2021.5.1
imbalance-learn==0.9
mapie==0.3.1

In your documentation we can see that for classification, MAPIE implements only multiclass classification.
In the industry we know that many classification problems are binary classification.

One possibility is to add the link to the Probability calibration API from scikit-learn as into MAPIE or showing some example in the documentation to show that there are possibilities for binary classification.

As described in section 2.1 of this tutorial :
https://sites.google.com/berkeley.edu/dfuq21/tutorial-resources

Is your feature request related to a problem? Please describe.
MAPIE has not yet a dedicated method for time series regression.

Describe the solution you'd like
A valid method for time series.

As described in this paper :
https://arxiv.org/abs/1802.06300

Or the EnbPI method proposed by Xu & Xie (2021) :
http://proceedings.mlr.press/v139/xu21h.html
https://arxiv.org/pdf/2010.09107.pdf

MapieClassifier should accept the method argument as an Iterable so conformity scores just need to be calibrated once to estimate prediction sets with different conformal methods.

Publish MAPIE to conda-forge for installation with conda

As described in section 1 of this tutorial :
https://sites.google.com/berkeley.edu/dfuq21/tutorial-resources

As described in this paper :
https://arxiv.org/abs/1904.06019

Is your feature request related to a problem? Please describe.
Certain unit tests are copy-pasted from test_regression.py to test_classification.py.

Describe the solution you'd like
These tests could be refactorized in a test_common.py file, with pytest.mark.parametrize to repeat a piece of code across different instances of MapieRegressor and MapieClassifer

Additional context
These tests are test_initialized, test_fit, test_fit_predict, test_no_fit_predict, test_default_sample_weight, test_default_alpha, test_invalid_estimator, test_sklearn_compatible_estimator.

Others tests can probably be factorized easily this way.

Is your documentation request related to a problem? Please describe.
The reproduction of the online notebook of ChenXu examples, testing and evaluating is method to different data set was missing

Describe the solution you'd like
Add plot_kim2020_simulations.py to examples/regression/ to reproduce, with a little more modularity, the Chen Xu example

Is your feature request related to a problem? Please describe.
The function regression_mean_width_score and classification_mean_width_score are circumvented in some examples, with code almost copy-pasted.

Describe the solution you'd like
The functions could be imported and called instead.

Is your feature request related to a problem? Please describe.
There are if_else switches according to "cv" value in the regression.py code that make the code unclear and could become more and more cumbersome with new methods.

Describe the solution you'd like
Use the matrix k_ for all methods

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Is your documentation request related to a problem? Please describe.

On the main site, on the example you have

from maple.estimators import MapieRegressor

but I had an error message saying that estimators was not found. Then I went inside your codes and you have

from mapie.regression import MapieRegressor

This is working.

Describe the solution you'd like
You may want to update the main site.

Is your documentation request related to a problem? Please describe.
I recently read this Medium post: https://towardsdatascience.com/mapie-explained-exactly-how-you-wished-someone-explained-to-you-78fb8ce81ff3

The author fits mapie on an extra dataset which was splitted from the training data, I guess in order to have independent confidence intervals. The tutorials in the documentation on the other hand fit Mapie directly on the training set.

Which is the right thing to do? Is Mapie doing this split for calibrated data in the background? How much data is used for the calibrated dataset? Would be nice to know since I have less samples to train on if thats the case

Best regards

Is your feature request related to a problem? Please describe.
Currently, estimators are required to have scikit-learn API. This can be problematic when using some of the deep learning frameworks. While it is easy to create a wrapper, sometimes generating predictions itself can take some time or be run on different hardware (e.g. get predictions on GPU instance, save to disk, and then create conformal prediction sets on CPU instance). Additionally, given compute costs for deep learning models and the diminishing returns of a larger calibration set, a Split Conformal framework is often ideal for deep learning problems.

As an aside, it would also potentially be faster to prototype and release new methods for the Split Conformal framework.

Describe the solution you'd like
Here's an example template:

class SplitConformal():
  def __init__(self):
    pass
  
  def fit(self, Y_calib, Y_calib_pred, conformity_score_fn):
    pass

 def predict(self, Y_test, Y_test_pred, inv_conformity_score_fn):
    pass

The above setup generalizes to both classification and regression through appropriate specification of conformity_score_fn and inv_conformity_score_fn. Furthermore, it is also easily to extendible to whatever score function a user comes up with (quantile based, uses both mean and std error estimate etc.).

Describe alternatives you've considered
We can use cv="prefit" as part of MapieRegressor to do Split Conformal calibration. However, the problem remains that the predictions have to be generated within predict (no way to calibrate existing predictions).

Is your feature request related to a problem? Please describe.
Running the tutorial 3. Estimating the uncertainty with different sklearn-compatible regressors produces the error No valid loss function found. You must provide a loss function to train. [...] when fitting the Keras mlp.
It also produces the future warning UserWarning: ``build_fn`` will be renamed to ``model`` in a future release, at which point use of ``build_fn`` will raise an Error instead. from scikeras.wrappers import KerasRegressor.

Describe the solution you'd like
The issue can be solved by changing

mlp_model = KerasRegressor(
    build_fn=mlp,
    epochs=500,
    verbose=0
)

to

mlp_model = KerasRegressor(
    model=mlp,
    epochs=500,
    verbose=0,
    loss='mse'
)

As described in section 3 of this tutorial :
http://people.eecs.berkeley.edu/~angelopoulos/publications/downloads/gentle_intro_conformal_dfuq.pdf

and in this paper :
https://arxiv.org/abs/2101.02703

Is your feature request related to a problem? Please describe.
At the moment, the tutorials were created with .rst files "by hand" from notebooks with manual copy/paste of blocks, triggering some outdated blocks and no script testing.

Describe the solution you'd like
We shall create a notebooks folder containing the tutorials as ipynb files, synchronized in .md for code review, which can automatically be converted to .rst fot the documentation.

As described in this paper :
https://link.springer.com/article/10.1007%2Fs10472-013-9368-4

Or the recent method by Romano et al. (2020) inspired by the Jackknife+/CV+ algorithms: https://arxiv.org/abs/2006.02544

Describe the bug
Assertion error: For binary classification problems with y in [0, 1]
To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context
https://github.com/scikit-learn-contrib/MAPIE/blob/master/mapie/classification.py#L513

assert type_of_target(y) in ["binary", "multiclass"]

Following this paper : http://proceedings.mlr.press/v128/messoudi20a.html

Is your documentation request related to a problem? Please describe.
I found this picture of Mapie in your docs, which shows the workflow of Mapie. My question is: How big is this
calibration dataset in case for example of the "cv+" method?

I find this question important, because the calibrated data is subtracted from the training data and can't be changed in size. This could lead to performance issues if the number of samples is small.

When all possible labels are not present in the calibration set, the code result in an error because of a shape mismatch if a label in the test set was not in the calibration set

I think their is 2 possible solutions :

• Make it mandatory to have all possible classes in the calibration set
• Fit the LabelBinarizer on a fake label which have all possible classes (typically a list [0, 1, 2, 3, ..., 999] if we have 1000 classes)

Is your documentation request related to a problem? Please describe.
The description of J-a-B method in the theoretical description was missing

Describe the solution you'd like
Add the description of J-a-B in theoretical_description_regression.rst

The naive method principle is to take all the classes with the higher softmax output such that their sum is almost equal to 1-alpha (the confidence level). It would able the user to compare conformal predictions to a simpler method.

For the top-k method, the goal is to watch the distribution of k (k being the rank of the label in the sorted softmax output). Then take the 1-alpha quantile of this distribution, giving k*, which will be the size of the prediction set (always takings values from the higher to the lower one)

It would be great to add sub-sections in the example galleries both for regression and classification. I suggest the following subsections :

1. Basic examples
2. Advanced analyses
3. Reproduction of scientific papers

Problem description
Currently, the regression confidence intervals are based on the absolute value of the residuals self.residuals_ = np.abs(np.ravel(y) - y_pred). From my experience, in some cases, this may not be appropriate, for two main reasons:

• We expect the confidence interval not to be symmetrical around the predicted value.
• We expect the confidence interval to be conditioned (scaled for instance) on the predicted value.

I ended up with these conclusions by applying MAPIE to a house price dataset from kaggle. My initial hunch was that the confidence intervals should scale with the predicted value. For instance, I would prefer to get the two following house prices 100 k€ +/- 10 k€ and 1000 k€ +/- 100 k€ instead of 100 k€ +/- 10 k€ and 1000 k€ +/- 10 k€.

Describe the solution you'd like
I suggest to add an argument in the fit method of MapieRegressor class. The argument would be an instance of a class (that I chose to name ResidualScore). The confidence intervals would then be conditioned on the residual scores (I prefer to call them "scores" instead of "residuals" since they may not simply be np.ravel(y) - y_pred which is the definition of residuals if I am correct). Different inherited classes may be created from ResidualScore. For instance I wrote the class AbsoluteResidualScore which gives the same results as currently and the class GammaResidualScore which computes the residual scores by (y - y_pred) / y_pred.

Illustrations
I have tested the proposed approach on the house price dataset. The two figures depict the predicted values with their confidence intervals based on the absolute value of the residuals (on the left) and on the proposed gamma residual score (on the right).

NB
I know that the quantile regression may be a solution to meet my expectations regarding this particular problem. I do not know if my proposal could be another tool to address such problems or if quantile regression should/must be prioritized.

Hi, great work on the library! I'm reading the documentation, it's very interesting. I spotted a few typos in the documentation, would it be okay if I proof-read the documentation, corrected the typos and opened a PR for that?

As we don't use chec_X_y_array anymore, we can remove check_image_is_input and its related argument in fit function.

Describe the bug
I want to use mapie on a model, which I obtained from gscv.best_estimator_ . The model uses a pipeline, which looks like this:

Pipeline(steps=[('preprocessor',
                 ColumnTransformer(n_jobs=-1,
                                   transformers=[('enc_plz',
                                                  BinaryEncoder(drop_invariant=True),
                                                  ['Postcode']),
                                                 ('enc_obj',
                                                  OneHotEncoder(drop_invariant=True),
                                                  ['PropertyType']),
                                                 ('features', 'passthrough',
                                                  Index(['YearSurvey', 'GeoY', 'SecondBathroom', 'Income'], dtype='object')),
                                                 ('log',
                                                  FunctionTransformer(func=<ufunc 'log1p'>,
                                                                      validate...
                                                  Index(['Pensioner', 'Balcony', 'YearModernization', 'YearBuilt'], dtype='object'))])),
                ('scaler', RobustScaler()),
                ('clf',
                 ClfSwitcher(estimator=LGBMRegressor(bagging_fraction=0.4,
                                                     bagging_freq=4,
                                                     bagging_seed=15871193,
                                                     feature_fraction=0.11,
                                                     feature_fraction_seed=15871193,
                                                     learning_rate=0.007,
                                                     max_bin=63,
                                                     min_data_in_leaf=10,
                                                     n_estimators=4000,
                                                     num_leaves=6,
                                                     objective='regression',
                                                     random_state=15871193)))])

When I use the lines

mapie = MapieRegressor(best_estimator_, method="plus", cv=4)
mapie.fit(X_train, y_train)

Mapie throws the exception:

ValueError: could not convert string to float: 'EFH'

in

~\miniconda3\envs\Master_ML\lib\site-packages\mapie\regression.py in fit(self, X, y, sample_weight) 457 cv = self._check_cv(self.cv) 458 estimator = self._check_estimator(self.estimator) --> 459 X, y = check_X_y( 460 X, y, force_all_finite=False, dtype=["float64", "int", "object"] 461 )

X_train and y_train are still in raw format (strings, not scaled, ....) the pipeline was designed to adress this.

My guess is that when mapie.fit is called on X_train the categorical variable "EFH" produces the error because it
is not float64, int or object type. However the pipeline would adress this by using an encoder.

Expected behavior
I would expect for the pipeline to preprocess my data before throwing a exception because of a wrong datatype.

Hi there,

scikit-learn metrics names are usually suffixed by _score or _error
From what I understand the coverage is a scoring function (?)

Also, could the name of the function be a bit more specific. I am a newcomer and just had a quick look at the paper,
but coverage can mean many things to me.
Could it be renamed something like target_coverage_score ??

Finally, an example could be added in the doc.
The shortest example I could get is the following

>>> import numpy as np; from sklearn.linear_model import LinearRegression; from sklearn.datasets import make_regression
>>> regressor = LinearRegression()
>>> X, y = make_regression(n_samples=500, n_features=1, noise=20, random_state=59)
>>> from mapie.estimators import MapieRegressor
>>> mapie = MapieRegressor(regressor).fit(X, y)
>>> X_pi = np.linspace(X.min(), X.max(), 100).reshape(-1, 1)
>>> y_preds = mapie.predict(X_pi)
>>> from mapie.metrics import coverage
>>> coverage(*y_preds.T)
1.0

Describe the bug
The test test_toy_dataset_predictions does not pass on Windows jobs in Github CI at random.

To Reproduce
Just check the jobs in open PRs.

Expected behavior
Fix the test probably due to some machine precision problems.

Following https://arxiv.org/abs/2107.08353

Using python 3.8 and latest version of sklearn
This is the error I get:
from sklearn.utils.validation import ( ImportError: cannot import name '_check_y' from 'sklearn.utils.validation' (/Users/nimashahbazi/PycharmProjects/distrust/venv/lib/python3.9/site-packages/sklearn/utils/validation.py)
Any ideas how to fix it?

Implement the Jackknife+-after-Bootstrap method : https://arxiv.org/abs/2002.09025

Is your feature request related to a problem? Please describe.
In the context of genetic programming, I need to evaluate a given "model" (i.e a mathematical expression) on my data, many times. One advantage of GAs being the ability to use a custom fitness function, I was thinking of integrating regression_coverage_score inside my fitness function, so that my GA would be "looking for a performant and confident model" (still not sure of this sentence though)

From a purely practical point of view, how whould I get the y_pred_low and y_pred_high arguments to be passed to regression_coverage_score ? I that even possible outside of MapieRegressor ?

Describe the solution you'd like

from mapie.metrics import regression_coverage_score
import numpy as np
from sklearn.metrics import r2_score

def fitness(model, X, y_true):
    y_pred_low, y_pred_up = function_i_am_looking_for(model, X)
    y_pred = make_prediction(model, X)  # just make a classic prediction on X
    cov = regression_coverage_score(y_true, y_pred_low, y_pred_up)
    r2 = r2_score(y_pred, y_true)
    return (r2 + cov) / 2  # take the mean but could take another weighted combination of r2 and cov

NB.

1. same would hold for classification_coverage_score
2. I have very limited understanding of MAPIE, please forgive my ignorance is that question ends up being silly

Describe the bug

• The environment in environment.notebooks.yml is corrupted and cannot be created.
• Furthermore, scikit-learn and pandas versions are not aligned with de dev environment
• Finally, the name of the environment should contain "-" and not "_"

To Reproduce

conda env create -f environment.notebooks.yml

Note however that this alternative environment works :

dependencies:
    - ipykernel=6.9.0
    - jupyter=1.0.0
    - pandas=1.3.5
    - python=3.10.1
    - scikit-learn=1.0.1

Expected behavior
The environment should be created without conflicts.

Screenshots
Here is the output of the command line :

Collecting package metadata (repodata.json): done
Solving environment: \
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                                                                 \
Solving environment: -
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                      \                                                                                                                          \

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package python_abi conflicts for:
pandas=1.2.4 -> python_abi[version='3.7|3.7.*|3.9.*|3.8.*',build='*_cp38|*_cp37m|*_pypy37_pp73|*_cp39']
matplotlib=3.5.1 -> tornado -> python_abi[version='3.6|3.6.*',build='*_cp36m|*_pypy36_pp73']
scikit-learn=1.0.0 -> numpy[version='>=1.19.5,<2.0a0'] -> python_abi[version='3.10.*|3.6.*|3.6',build='*_cp36m|*_cp310|*_pypy36_pp73']
xgboost=1.3.0 -> python_abi[version='3.6.*|3.6|3.7.*|3.7|3.9.*|3.8.*',build='*_cp38|*_pypy37_pp73|*_cp39|*_cp37m|*_pypy36_pp73|*_cp36m']
nbconvert=6.4.0 -> python_abi[version='3.10.*|3.7.*|3.9.*|3.7|3.8.*',build='*_cp38|*_cp37m|*_cp39|*_cp310|*_pypy37_pp73']
jupyterlab=3.0.16 -> ipython -> python_abi[version='2.7.*|3.10.*|3.8.*|3.9.*|3.7.*|3.7|3.6|3.6.*',build='*_cp27m|*_pypy37_pp73|*_cp37m|*_cp38|*_cp310|*_cp39|*_pypy36_pp73|*_cp36m']
ipykernel=6.4.1 -> python_abi[version='3.8.*|3.9.*',build='*_cp38|*_cp39']
pip=21.0.1 -> python[version='>=3.6,<3.7.0a0'] -> python_abi[version='3.10.*|3.6|3.7|3.7.*|3.8.*|3.9.*|3.6.*',build='*_cp37m|*_pypy37_pp73|*_pypy36_pp73|*_cp38|*_cp39|*_cp310|*_cp36m']
jupyter=1.0.0 -> python_abi[version='3.10.*|3.7|3.8.*|3.9.*|3.7.*|3.6.*|3.6',build='*_cp36m|*_cp38|*_pypy37_pp73|*_cp39|*_cp310|*_cp37m|*_pypy36_pp73']
scikit-learn=1.0.0 -> python_abi[version='3.7.*|3.7|3.9.*|3.8.*',build='*_pypy37_pp73|*_cp39|*_cp37m|*_cp38']
matplotlib=3.5.1 -> python_abi[version='3.10.*|3.7|3.7.*|3.8.*|3.9.*',build='*_cp38|*_cp37m|*_pypy37_pp73|*_cp310|*_cp39']
nbconvert=6.4.0 -> entrypoints[version='>=0.2.2'] -> python_abi[version='2.7.*|3.6|3.6.*',build='*_cp27m|*_cp36m|*_pypy36_pp73']
jupytext=1.11.2 -> python[version='>=3.6'] -> python_abi[version='2.7.*|3.10.*|3.6|3.7|3.7.*|3.8.*|3.9.*|3.6.*',build='*_cp27m|*_cp38|*_cp37m|*_pypy37_pp73|*_pypy36_pp73|*_cp310|*_cp39|*_cp36m']
tensorflow=2.6.2 -> python_abi[version='3.7.*|3.8.*|3.9.*',build='*_cp38|*_cp37m|*_cp39']
pandas=1.2.4 -> numpy[version='>=1.19.2,<2.0a0'] -> python_abi[version='3.10.*|3.6.*|3.6',build='*_cp36m|*_cp310|*_pypy36_pp73']
tensorflow=2.6.2 -> python[version='>=3.7,<3.8.0a0'] -> python_abi==3.7[build=*_pypy37_pp73]
ipykernel=6.4.1 -> appnope -> python_abi[version='2.7.*|3.10.*|3.7.*|3.7|3.6|3.6.*',build='*_cp27m|*_cp37m|*_cp310|*_pypy37_pp73|*_pypy36_pp73|*_cp36m']

Package pip conflicts for:
jupyterlab=3.0.16 -> python[version='>=3.6'] -> pip
ipykernel=6.4.1 -> python[version='>=3.10,<3.11.0a0'] -> pip
jupyter=1.0.0 -> python[version='>=3.10,<3.11.0a0'] -> pip
python=3.10.1 -> pip
jupytext=1.11.2 -> python[version='>=3.6'] -> pip
pip=21.0.1
tensorflow=2.6.2 -> python[version='>=3.7,<3.8.0a0'] -> pip
matplotlib=3.5.1 -> python[version='>=3.7,<3.8.0a0'] -> pip
pandas=1.2.4 -> python[version='>=3.8,<3.9.0a0'] -> pip
xgboost=1.3.0 -> python[version='>=3.7,<3.8.0a0'] -> pip
scikit-learn=1.0.0 -> python[version='>=3.7,<3.8.0a0'] -> pip

Package send2trash conflicts for:
jupyter=1.0.0 -> notebook -> send2trash[version='>=1.5.0|>=1.8.0']
jupyterlab=3.0.16 -> jupyter_server[version='>=1.4,<2'] -> send2trash

Package gdbm conflicts for:
jupyter=1.0.0 -> pypy3.7[version='>=7.3.7'] -> gdbm[version='>=1.18,<1.19.0a0']
nbconvert=6.4.0 -> pypy3.7[version='>=7.3.7'] -> gdbm[version='>=1.18,<1.19.0a0']
matplotlib=3.5.1 -> pypy3.7[version='>=7.3.7'] -> gdbm[version='>=1.18,<1.19.0a0']
scikit-learn=1.0.0 -> pypy3.7[version='>=7.3.5'] -> gdbm[version='>=1.18,<1.19.0a0']
xgboost=1.3.0 -> pypy3.7[version='>=7.3.3'] -> gdbm[version='>=1.18,<1.19.0a0']
pandas=1.2.4 -> pypy3.7[version='>=7.3.3'] -> gdbm[version='>=1.18,<1.19.0a0']

Package python-dateutil conflicts for:
pandas=1.2.4 -> python-dateutil[version='>=2.7.3']
matplotlib=3.5.1 -> matplotlib-base[version='>=3.5.1,<3.5.2.0a0'] -> python-dateutil[version='>=2.7']
ipykernel=6.4.1 -> jupyter_client[version='<8.0'] -> python-dateutil[version='>=2.1']

Package typing_extensions conflicts for:
tensorflow=2.6.2 -> tensorflow-base==2.6.2=cpu_py37h5759434_2 -> typing_extensions[version='>=3.7.4,<3.8']
jupytext=1.11.2 -> markdown-it-py[version='>=1.0,<2.0'] -> typing_extensions[version='>=3.7.4']
ipykernel=6.4.1 -> importlib-metadata[version='<5'] -> typing_extensions[version='>=3.6.4']

Package nest-asyncio conflicts for:
nbconvert=6.4.0 -> nbclient[version='>=0.5.0,<0.6.0'] -> nest-asyncio
jupyter=1.0.0 -> ipykernel -> nest-asyncio[version='>=1.5']
ipykernel=6.4.1 -> jupyter_client[version='<8.0'] -> nest-asyncio[version='>=1.5']

Package six conflicts for:
jupyterlab=3.0.16 -> packaging -> six
pandas=1.2.4 -> python-dateutil[version='>=2.7.3'] -> six[version='>=1.5']
ipykernel=6.4.1 -> traitlets[version='>=4.1.0,<6.0'] -> six
tensorflow=2.6.2 -> tensorflow-base==2.6.2=cpu_py37h5759434_2 -> six[version='>=1.15,<1.16']
nbconvert=6.4.0 -> bleach -> six[version='>=1.9.0']

Package numpy conflicts for:
pandas=1.2.4 -> numpy[version='>=1.16.6,<2.0a0|>=1.19.2,<2.0a0|>=1.19.5,<2.0a0|>=1.17.5,<2.0a0']
xgboost=1.3.0 -> py-xgboost==1.3.0=py37h5186d4c_3 -> numpy
scikit-learn=1.0.0 -> numpy[version='>=1.18.5,<2.0a0|>=1.19.5,<2.0a0']
tensorflow=2.6.2 -> tensorflow-base==2.6.2=cpu_py37h5759434_2 -> numpy[version='>=1.19.2,<1.20']
scikit-learn=1.0.0 -> scipy -> numpy[version='1.10.*|1.11.*|1.12.*|1.13.*|>=1.11|>=1.11.3,<2.0a0|>=1.14.6,<2.0a0|>=1.16.6,<2.0a0|>=1.21.2,<2.0a0|>=1.15.1,<2.0a0|>=1.9.3,<2.0a0|>=1.21.5,<2.0a0|>=1.21.4,<2.0a0|>=1.17.5,<2.0a0|>=1.19.4,<2.0a0|>=1.16.5,<2.0a0|>=1.19.2,<2.0a0|>=1.18.1,<2.0a0|>=1.9']
matplotlib=3.5.1 -> matplotlib-base[version='>=3.5.1,<3.5.2.0a0'] -> numpy[version='>=1.17|>=1.19.5,<2.0a0|>=1.18.5,<2.0a0|>=1.21.4,<2.0a0']

Package nbformat conflicts for:
jupyter=1.0.0 -> ipywidgets -> nbformat[version='>=4.2.0|>=4.4']
nbconvert=6.4.0 -> nbformat[version='>=4.4']
jupyterlab=3.0.16 -> jupyter_server[version='>=1.4,<2'] -> nbformat
jupytext=1.11.2 -> nbformat
nbconvert=6.4.0 -> nbclient[version='>=0.5.0,<0.6.0'] -> nbformat[version='>=5.0']

Package pyparsing conflicts for:
matplotlib=3.5.1 -> matplotlib-base[version='>=3.5.1,<3.5.2.0a0'] -> pyparsing[version='>=2.2.1']
jupyterlab=3.0.16 -> packaging -> pyparsing[version='<3,>=2.0.2|>=2.0.2|>=2.0.2,!=3.0.5|>=2.0.2,<3']

Package pyzmq conflicts for:
jupyterlab=3.0.16 -> jupyter_server[version='>=1.4,<2'] -> pyzmq[version='>=17']
jupyter=1.0.0 -> ipykernel -> pyzmq[version='!=21|<21|>=17|>=17.1']
ipykernel=6.4.1 -> jupyter_client[version='<8.0'] -> pyzmq[version='>=13']

Package certifi conflicts for:
ipykernel=6.4.1 -> tornado[version='>=4.2,<7.0'] -> certifi
matplotlib=3.5.1 -> matplotlib-base[version='>=3.5.1,<3.5.2.0a0'] -> certifi[version='>=2020.06.20']
pandas=1.2.4 -> setuptools[version='<60.0.0'] -> certifi[version='>=2016.09|>=2016.9.26']
pip=21.0.1 -> setuptools -> certifi[version='>=2016.09|>=2016.9.26']

Package pickleshare conflicts for:
jupyterlab=3.0.16 -> ipython -> pickleshare
ipykernel=6.4.1 -> ipython[version='>=7.23.1,<8.0'] -> pickleshare

Package jupyter_core conflicts for:
jupyter=1.0.0 -> nbconvert -> jupyter_core[version='>=4.4.0|>=4.6.0|>=4.6.1']
jupyterlab=3.0.16 -> jupyter_server[version='>=1.4,<2'] -> jupyter_core[version='>=4.4.0|>=4.6.0']
nbconvert=6.4.0 -> jupyter_core
jupytext=1.11.2 -> nbformat -> jupyter_core
ipykernel=6.4.1 -> jupyter_client[version='<8.0'] -> jupyter_core[version='>=4.6.0']
jupyterlab=3.0.16 -> jupyter_core

Package entrypoints conflicts for:
nbconvert=6.4.0 -> entrypoints[version='>=0.2.2']
jupyter=1.0.0 -> nbconvert -> entrypoints[version='>=0.2.2']
ipykernel=6.4.1 -> jupyter_client[version='<8.0'] -> entrypoints
jupyterlab=3.0.16 -> jupyterlab_server[version='>=2.3,<3'] -> entrypoints[version='>=0.2.2']

Package ipython conflicts for:
ipykernel=6.4.1 -> matplotlib-inline[version='>=0.1.0,<0.2.0'] -> ipython
ipykernel=6.4.1 -> ipython[version='>=7.23.1,<8.0']

Package tornado conflicts for:
jupyterlab=3.0.16 -> tornado[version='>=6.1']
matplotlib=3.5.1 -> tornado
ipykernel=6.4.1 -> jupyter_client[version='<8.0'] -> tornado[version='>=4.1|>=4.1,<6']
jupyter=1.0.0 -> ipykernel -> tornado[version='>=4|>=4,<6|>=4.0|>=4.2|>=4.2,<7.0|>=6.1|>=5.0|>=5.0,<7|>=4.1,<7']
ipykernel=6.4.1 -> tornado[version='>=4.2,<7.0']
jupyterlab=3.0.16 -> jupyter_server[version='>=1.4,<2'] -> tornado[version='>=6.1.0']

Package testpath conflicts for:
jupyter=1.0.0 -> nbconvert -> testpath
nbconvert=6.4.0 -> testpath

Package pypy3.6 conflicts for:
jupyter=1.0.0 -> ipykernel -> pypy3.6[version='7.3.*|7.3.0.*|7.3.1.*|7.3.2.*|7.3.3.*|>=7.3.1|>=7.3.2']
xgboost=1.3.0 -> pypy3.6[version='>=7.3.3']
jupytext=1.11.2 -> python[version='>=3.6'] -> pypy3.6[version='7.3.0.*|7.3.1.*|7.3.2.*|7.3.3.*|>=7.3.3|>=7.3.2|>=7.3.1']
jupyterlab=3.0.16 -> ipython -> pypy3.6[version='7.3.0.*|7.3.1.*|7.3.2.*|7.3.3.*|>=7.3.1|>=7.3.3|>=7.3.2']
jupyter=1.0.0 -> pypy3.6[version='>=7.3.3']
xgboost=1.3.0 -> python[version='>=3.6,<3.7.0a0'] -> pypy3.6[version='7.3.*|7.3.0.*|7.3.1.*|7.3.2.*|7.3.3.*']
matplotlib=3.5.1 -> tornado -> pypy3.6[version='>=7.3.1|>=7.3.2|>=7.3.3']
pandas=1.2.4 -> numpy[version='>=1.19.2,<2.0a0'] -> pypy3.6[version='>=7.3.1|>=7.3.2|>=7.3.3']
ipykernel=6.4.1 -> appnope -> pypy3.6[version='>=7.3.1|>=7.3.2|>=7.3.3']
scikit-learn=1.0.0 -> numpy[version='>=1.19.5,<2.0a0'] -> pypy3.6[version='>=7.3.1|>=7.3.2|>=7.3.3']
nbconvert=6.4.0 -> entrypoints[version='>=0.2.2'] -> pypy3.6[version='>=7.3.1|>=7.3.2|>=7.3.3']
pip=21.0.1 -> python[version='>=3.6,<3.7.0a0'] -> pypy3.6[version='7.3.0.*|7.3.1.*|7.3.2.*|7.3.3.*|>=7.3.3|>=7.3.2|>=7.3.1']

Package matplotlib-inline conflicts for:
ipykernel=6.4.1 -> matplotlib-inline[version='>=0.1.0,<0.2.0']
ipykernel=6.4.1 -> ipython[version='>=7.23.1,<8.0'] -> matplotlib-inline[version='>=0.1.2']

Package pygments conflicts for:
nbconvert=6.4.0 -> jupyterlab_pygments -> pygments[version='>=2.4.1,<3']
nbconvert=6.4.0 -> pygments[version='>=2.4.1']

Package nbconvert conflicts for:
jupyterlab=3.0.16 -> jupyter_server[version='>=1.4,<2'] -> nbconvert
nbconvert=6.4.0
jupyter=1.0.0 -> notebook -> nbconvert[version='<6.0']
jupyter=1.0.0 -> nbconvert

Package setuptools conflicts for:
python=3.10.1 -> pip -> setuptools
nbconvert=6.4.0 -> bleach -> setuptools
ipykernel=6.4.1 -> ipython[version='>=7.23.1,<8.0'] -> setuptools[version='>=18.5']
scikit-learn=1.0.0 -> joblib[version='>=0.11'] -> setuptools
jupyterlab=3.0.16 -> ipython -> setuptools[version='>=18.5']
pip=21.0.1 -> setuptools
pandas=1.2.4 -> setuptools[version='<60.0.0']

Package jsonschema conflicts for:
jupyterlab=3.0.16 -> jupyterlab_server[version='>=2.3,<3'] -> jsonschema[version='>=3.0.1']
nbconvert=6.4.0 -> nbformat[version='>=4.4'] -> jsonschema[version='>=2.4,!=2.5.0']
jupytext=1.11.2 -> nbformat -> jsonschema[version='>=2.0,!=2.5.0|>=2.4,!=2.5.0']

Package scikit-learn conflicts for:
scikit-learn=1.0.0
xgboost=1.3.0 -> py-xgboost==1.3.0=py37h5186d4c_3 -> scikit-learn

Package ca-certificates conflicts for:
jupyter=1.0.0 -> python[version='>=2.7,<2.8.0a0'] -> ca-certificates
tensorflow=2.6.2 -> openssl[version='>=1.1.1l,<1.1.2a'] -> ca-certificates
python=3.10.1 -> openssl[version='>=1.1.1l,<1.1.2a'] -> ca-certificates

Package backports_abc conflicts for:
ipykernel=6.4.1 -> tornado[version='>=4.2,<7.0'] -> backports_abc[version='>=0.4']
matplotlib=3.5.1 -> tornado -> backports_abc[version='>=0.4']

Package pathlib2 conflicts for:
ipykernel=6.4.1 -> importlib-metadata[version='<5'] -> pathlib2
nbconvert=6.4.0 -> testpath -> pathlib2
jupyterlab=3.0.16 -> ipython -> pathlib2

Package markupsafe conflicts for:
nbconvert=6.4.0 -> jinja2[version='>=2.4'] -> markupsafe[version='>=0.23|>=2.0|>=2.0.0rc2']
jupyterlab=3.0.16 -> jinja2[version='>=2.10'] -> markupsafe[version='>=0.23|>=2.0|>=2.0.0rc2']

Package mkl_fft conflicts for:
pandas=1.2.4 -> numpy[version='>=1.19.2,<2.0a0'] -> mkl_fft[version='>=1.0.14,<2.0a0|>=1.0.6,<2.0a0|>=1.2.1,<2.0a0']
scikit-learn=1.0.0 -> numpy[version='>=1.19.5,<2.0a0'] -> mkl_fft

Package ipykernel conflicts for:
jupyter=1.0.0 -> ipywidgets -> ipykernel[version='>=4.1|>=4.2.2|>=4.5.1']
ipykernel=6.4.1
jupyter=1.0.0 -> ipykernel

Package pypy3.7 conflicts for:
xgboost=1.3.0 -> pypy3.7[version='>=7.3.3']
xgboost=1.3.0 -> python[version='>=3.7,<3.8.0a0'] -> pypy3.7[version='7.3.*|7.3.3.*|7.3.4.*|7.3.5.*|7.3.7.*']

Package wheel conflicts for:
pip=21.0.1 -> wheel
python=3.10.1 -> pip -> wheel
tensorflow=2.6.2 -> tensorflow-base==2.6.2=cpu_py37h5759434_2 -> wheel[version='>=0.35,<1']

Package configparser conflicts for:
nbconvert=6.4.0 -> entrypoints[version='>=0.2.2'] -> configparser[version='>=3.5']
ipykernel=6.4.1 -> importlib-metadata[version='<5'] -> configparser[version='>=3.5']

Package packaging conflicts for:
matplotlib=3.5.1 -> matplotlib-base[version='>=3.5.1,<3.5.2.0a0'] -> packaging[version='>=20.0']
jupyterlab=3.0.16 -> packaging
pip=21.0.1 -> wheel -> packaging[version='>=20.2']
nbconvert=6.4.0 -> bleach -> packaging

As described in this paper :
https://arxiv.org/abs/2103.03323

Describe the bug

The Cifar10 notebook is facing a problem with the uncertainty estimation using the "naive" and "cumulated_score" methods. It produces abnormal calibration plots with null prediction sets. Other examples built in the documentation are unchanged though.

The problem appears after the validation of PR #137 named "Include crossval for cumulatedscore" on March 11 : f622549

A dedicated unit test is needed to address and solve this issue.

To Reproduce
Steps to reproduce the behavior:
Simply run the Cifar10.ipynb notebook using the MAPIE versions before/after this PR.

Expected behavior
Calibration plots should be monotonic with non-empty prediction sets for the naive and cumulative score methods.

Describe the bug
For MapieClassifier, when cv is set to "prefit", the calibration set does not necessarily contain all the classes of the original dataset. Therefore, the base classifier could have been trained on a training set containing a larger number of classes. This discrepancy triggers a bug for the cumulated_score in the fit method.

To Reproduce
Simply run a simulation with a toy dataset in which the calibration set does not contain all the classes of the original dataset, with cv="prefit" and method="cumulated_score".

Expected behavior
This discrepancy can be fixed by using sklearn.preprocessing.label_binarize instead of sklearn.preprocessing.LabelBinarizer in order to explicitely list the number of classes when "binarizing" the labels.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Describe the bug
The command make type-check raise a very high number of errors.

Expected behavior
The command should run as-is after git clone

Additional context
It seems that there is a bug in mapie/_typing.py because the import statement from np.typing import ArrayLike raises an error (catched, but not sure this is the normal behaviour).

Allow image as input of MapieClassifier: if we create model scikit-learn compatible model for image classification for instance, we should be able to have X as a matrix of images