Aim: Testing how the performance of different clustering algorithms for different datasets change on adding noise with different dimensions:

To be done: A jupyter notebook documentation describing the effect of the addition of different dimensions of noise on a dataset. Here different types of synthetic datasets are generated on which the experiment is performed. To these datasets gaussian noise of different dimensions are added, and the performance of each clustering algorithm is measured after noise addition. This is repeated for noise with different variances.

Expected output: The plots that compare the effect of varying noise dimensions on different clustering algorithms for each of the datasets. In this set of subplots, the variance of the added noise changes along the column and the dataset changes along the row.

Link to the code: https://nbviewer.jupyter.org/github/sree0917/team-forbidden-forest/blob/master/Sree/Clustering%20comparison%20%281%29.ipynb

problem

Rn, we have to override and copy a lot of custom code to make fit and partial_fit work in subclasses of BaseDecisionTree inside sktree.

possible soln

We should track the kwarg parameters needed to instantiate the:

• criterion
• splitter
• tree

These should be then easily accessible in subclasses. E.g.

class BaseDecisionTree
      _criterion_kwargs = ['n_samples', ...']
      _splitter_kwargs = ['criterion', 'max_features', ...]
...

Then ideally a subclass just has to add these additional kwargs to the __init__ structure and then override the corresponding _criterion/splitter/tree_kwargs.

cc: @PSSF23 from our discussion

Describe the workflow you want to enable

First and foremost, we appreciate your NeuroData work focused on Scikit-Learn trees. This is brilliant. Anyway, we did some tree modifications, but it surely compiles on Linux and MacOS without much trouble, so we built the wheel on MacOS (local) and re-used it in our dependencies instead of recompiling the whole Scikit-Learn fork, and it works, thus speeding up the installation of our project, which uses an update of Scikit-Learn-Tree. However, in our Docker container, doing the same appears to produce a problem with the wheel generated after installation.

Is there a method to create the wheels we want (MacOS and Linux) so that our end users can use both (open source) instead of having to re compile the whole thing?

Cheers

Describe your proposed solution

None

Describe alternatives you've considered, if relevant

Did not find any yet

Additional context

No response

scikit-learn#26391 now adds missing-value support for random forests.

It will be good to integrate this into the submodulev2 PR branch #44

Description

The sklearn's example on comparing the different classifier accuracies does not have multiple settings for testing various scenarios.
There is no concrete example showing when some of these algorithms win and when they lose.
One scenario to consider is - given a dataset of a relatively low order dimension, how does the accuracy of classifiers change with respect to the addition of noise dimensions.

Noise dimensions are any features added across the dimensions of the dataset which bears no relevance to the original signal dimensions.

Goal
To check the performance of Random Forest, Support Vector Machine and K Nearest Neighbours as three different classifiers for the additions of gaussian noise across three different variance values.

Proposed changes in the form of PR
I am proposing a new tutorial in the form of a jupyter notebook containing all the code from data generation to the computation of accuracies across noise dimensions.
The final figure will contain a plot of the original datasets adopted from https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html
and 9 different plots of "Accuracy Vs Number of Noise Dimensions" for the 3 different datasets and 3 different variances of gaussian noise. The plot will containing the testing accuracies across 50 trials of the experiment.

Here is a link to the code:
https://github.com/NeuroDataDesign/team-forbidden-forest/blob/master/Sahana/FINAL_PR_classifiers.ipynb

Describe the workflow you want to enable

I am part of the @neurodata team. Binning features have resulted in highly efficient and little loss in performance in gradient-boosted trees. This feature should not only be used in gradient-boosted trees, but should be available within all decision trees [1].

By including binning as a feature for decision trees, we would enable massive speedups for decision trees that operate on high-dimensional data (both in features and sample sizes). This would be an additional tradeoff that users can take. The intuition behind binning for decision trees would be exactly that of Gradient Boosted Trees.

Describe your proposed solution

We propose introducing binning to the decision tree classifier and regressor.

An initial PR is proposed here: #24 (review)
However, it seems that many of the files were copied and it is not 100% clear if needed. Perhaps we can explore how to consolidate the _binning.py/pyx files using the current versions under ensemble/_hist_gradient_boosting/*.

Changes to the Cython codebase

TBD

Changes to the Python API

The following two parameters would be added to the DecisionTreeClassifier and Regressor:

hist_binning=False,
max_bins=255

where the default number of bins follows that of histgradient boosting.

Additional context

These changes can also trivially be applied to Oblique Trees.

References:
[1] https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree

Describe the bug

During the unit-test of RandomForestClassifier and ExtraTreesClassifier, it seems that there is some segmentation fault arising from partial_fit tree builder build function.

I suspect, this has something to do with either:

i) pickling and then accessing something that isn't restored properly
ii) accessing memory that isn't allocated

It would be great if it could be reproduced, but I fail to do so locally. This suggests to me it could be an edge case, but then that's even more important to fix.

Steps/Code to Reproduce

TBD.

Expected Results

There should be no segmentation fault.

Actual Results

.....................................................Fatal Python error: Segmentation fault

Thread 0x00007000107d4000 (most recent call first):
  File "/usr/local/miniconda/envs/testvenv/lib/python3.11/threading.py", line 320 in wait
  File "/usr/local/miniconda/envs/testvenv/lib/python3.11/site-packages/joblib/externals/loky/backend/queues.py", line 113 in _feed
  File "/usr/local/miniconda/envs/testvenv/lib/python3.11/threading.py", line 975 in run
  File "/usr/local/miniconda/envs/testvenv/lib/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/local/miniconda/envs/testvenv/lib/python3.11/threading.py", line 995 in _bootstrap

Thread 0x000070000f7d1000 (most recent call first):
  File "/usr/local/miniconda/envs/testvenv/lib/python3.11/selectors.py", line 415 in select
  File "/usr/local/miniconda/envs/testvenv/lib/python3.11/multiprocessing/connection.py", line 930 in wait
  File "/usr/local/miniconda/envs/testvenv/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py", line 611 in wait_result_broken_or_wakeup
  File "/usr/local/miniconda/envs/testvenv/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py", line 557 in run
  File "/usr/local/miniconda/envs/testvenv/lib/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/local/miniconda/envs/testvenv/lib/python3.11/threading.py", line 995 in _bootstrap

Versions

sklearn submodulev3 branch

We want a CI pipeline (and/or local pipeline) that can build wheels for:

• windows
• Linux
• Mac Intel (x64)
• Mac M1 (Arm)

and we can attach those wheels to a specific release we make (these are called "nightly wheels" in scipy/pytorch/tensorflow/etc.). Then we can pip install directly from those wheels. Use https://cibuildwheel.readthedocs.io/en/stable/.

Then create a stable release that has all these wheels pip installable

Then have sktree rely on these wheels for v0.1

I added more notes on the call, feel free to add them here for documentation @jshinm.

Describe the workflow you want to enable

Currently, the sklearn.manifold.SpectralEmbedding is restricted to symmetric affinity matrices, and if an asymmetric matrix is passed it is converted through sklearn.utils.validation.check_symmetric into a symmetric matrix. However, in doing so, one loses the underlying asymmetries and potential directional clusters present in the adjacency matrix of the directed graph input.

Describe your proposed solution

The algorithms I propose adding use singular value decomposition, as opposed to eignendecomposition, and a modified Laplacian to perform spectral embedding on directed graphs/asymmetric matrices. Specifically I would like to propose adding Adjacency / Laplacian spectral embedding, ASE and LSE respectively. My thoughts would be to add a new class,sklearn.manifold.DirectedSpectralEmbedding, which users may call directly, or is dispatched to when an asymmetric matrix is passed to sklearn.manifold.SpectralEmbedding. Similarly to SpectralEmbedding, users would specify between ASE and LSE through an affinity parameter.

Additional context

These algorithms have been implemented in GraSPy, (available at ASE and LSE), inputing a graph represented as a dense or sparse matrix, and returning the appropriate embedding.

Description

This demo provides an example of how to tune a Random Forest classifier using GridSearchCV on the MNIST dataset.

Steps/Code to Reproduce

link to code: https://github.com/NeuroDataDesign/team-forbidden-forest

It will be a document showing the Effect of Dimensionality Reduction on the accuracy of different classifiers.
The document will have simulations On High dimensional dataset of different shapes:
Each dataset is synthesized from sklearn established Datasets. Each dataset has 1000 dimensions with only 2 dimensions of data and rest are noise dimensions.

Questions to answer:

1. Analyzing how dimensionality reduction helps in classification for different classifiers.
2. Analyzing how classifiers perform with a different number of reduced datasets from the main high dimensional dataset.

Pipeline to be followed:

1. Defining a dataset with sklearn established synthetic datasets with High dimensions.
2. Performing classification on data and measuring accuracy for quantification of the process.
3. performing the Dimensionality reduction technique keeping varying numbers of reduced dimensions.
4. Checking the performance of classification again after reducing dimension after each iteration.

The output of the PR would be a figure showing different datasets, comparing accuracies of different classifiers with and without dimensionality reduction and a plot showing varying accuracies over reduced dimensions.

Experiments to follow:
https://github.com/NeuroDataDesign/team-forbidden-forest/blob/master/Parimal%20Joshi/Final_pr_2.ipynb

Some segfaults arose in scikit-tree during the implementation of this PR: neurodata/treeple#249.
Some actionable items came to mind to improve what we have:

Documentation:

Next, we prolly want to include a clear description for developers on what the differences are here:

Features, or documentation
Part of sklearn handles monotonic constraints and n_constant_features tracking. From first glance it is also not clear that these are actually tracked. I.e. is the monotonic constraint and n_constant_features if we do fit and then partial_fit for two subsets of the data different from if we just did fit for the entire dataset? If they are different, what does this imply?

In an ideal world the state is the same.