Hello,
lets say i have a set of genes from a non-model organism, but i do have gene annotations/descriptions. Can I use those? (without resorting to write javascript). I read the documentation and didn't find anything about it, so it seems i can't. I was thinking something like passing this annotation table/df to js and use that instead of retrieving data from the web could be a very useful option.

https://github.com/MaayanLab/clustergrammer-docs

• need to update python API
• document signature methods and other API updates

Describe the bug
Sometimes the widget causes Jupyter Lab to become unresponsive. I've attached a screenshot of the console logs below.

The problem appears to be that Clustergrammer2 is trying to hide a DOM element that does not exist and this is causing an error whenever the mouse is moved. The solution is probably to just put an if statement to check if the element exists before trying to hide it. This bug will be resolved by fixing Clustergrammer-GL and releasing a new version.

To Reproduce
Steps to reproduce the behavior:

1. Generate a Clustergrammer2 heatmap in Jupyter Lab (see https://github.com/ismms-himc/clustergrammer2-notebooks for examples)
2. Sometimes the page becomes unresponsive
3. See error in the console log

Expected behavior
The page should not be unresponsive.

Screenshots

Desktop (please complete the following information):

• MacOS
• Browser Chrome

Additional context

Allow users to choose between using the original SVG clustergrammer-js and the new WebGL clustergrammer-gl when generating a widget. The default will be WebGL. This will allow me to make updates to the original SVG visualization that will not effect the original clustergrammer-widget (e.g. no further releases). Clustergrammer2 will take over as the default widget and clustergrammer-widget will effectively be deprecated.

This will also enable the SVG widget to be used in cloud based jupyter notebooks.

The only downside is the requirement of downloading additional JavaScript code. But I think the benefits outweigh the costs.

It works with the Clustergrammer2 widget, but in order to have functionality we need to add widget communication to the back end. See Blog post. Look at other dashboards for inspiration.

Use cases

• have one widget talk to another (e.g. clicking a cluster of cells will display the corresponding single cell data in another heatmap)
• clicking a gene row can bring up a set of violin plots for the gene's expression across cell types
• on the fly enrichment analysis of the visible rows in a heatmap

Describe the bug
Clustergrammer2 0.5.2 is not working in JLab (1.0 and I think 0.3.5)

To Reproduce
Steps to reproduce the behavior:

1. Pip install 0.5.2
2. Launch JLab
3. Make Widget
4. See error about 0.5.1 JavaScript being loaded

Expected behavior
We should see a console log of 0.5.2 in the console. Clustergrammer2 0.5.2 is working in Jupyter Notebook so I suspect that I did not update the JLab widget extension when publishing 0.5.2.

Plan
Release 0.5.3 and make sure to build new JLab extension.

Add downloads per month etc badges to README.md - follow this example

https://github.com/ismms-himc/ipython-file-upload/blob/master/README.rst

Possibly switch from markdown to rst

Make a method that simplifies the addition of row/column categories based on some external category dataset (e.g. a table of clinical parameters, or a dictionary of categories).

Add SPACE button for row/column reordering. Space will reorder components by their spatial positions if available. MIBI (MICSS or spatial genomics) images for instance will have spatial positioning. Rows/genes might also have positioning if getting data from FISH (https://www.nature.com/articles/s41467-017-02714-7), but position data will be for the most part concerning cells.

Spatial Transcriptomics
http://www.spatialtranscriptomicsresearch.org/datasets/

Is your feature request related to a problem? Please describe.
Clustergrammer2 should be able to read/write from Zarr. This format can be used to store single cell gene expression matrices.

Describe the solution you'd like
IO to Zarr

Describe alternatives you've considered
IO to Parquet

Additional context
https://www.youtube.com/watch?v=qyJXBlrdzBs

Add to list and viz list. Add to tutorial list when kaggle examples are done

https://github.com/seandavi/awesome-single-cell#interactive-visualization-and-analysis

See if we can build a tableau extension for clustergrammer2

See

• https://github.com/tableau/extensions-api/

• https://www.tableau.com/developer/extensions#reveal

Describe the bug
We should be seeing selected rows/columns in the dendrogram tooltip. But they are not showing up. The console log from Clustergrammer-GL suggests that the correct number of rows/cols is being found. It may be a problem with selecting the right element within the notebook. This functionality works in Clustergrammer-GL alone - not in the widget.

To Reproduce
Steps to reproduce the behavior:

1. Mouse over a dendrogram in a heatmap
2. Note that there are no row col names being shown in the input section

Expected behavior
We expect to see these names, see Clustergrammer-GL

Screenshots

Desktop (please complete the following information):

• MacOS Mojave
• Browser Chrome
• Version 75.0.3770.142 (Official Build) (64-bit)

When zooming in and out the display seem to be stuck.

Used Windows 10 and Chrome.

Check out this tutorial: http://savvastjortjoglou.com/nba-play-by-play-movements.html

https://github.com/neilmj/BasketballData/tree/master/2016.NBA.Raw.SportVU.Game.Logs

https://github.com/rajshah4/NBA_SportVu

https://www.analyticsvidhya.com/blog/2019/05/scraping-nba-data-analyze-1000-basketball-games-python/

https://github.com/swar/nba_api

https://towardsdatascience.com/which-nba-players-are-most-similar-machine-learning-provides-the-answers-r-project-b903f9b2fe1f

shot chart

http://savvastjortjoglou.com/nba-shot-sharts.html

https://medium.com/analytics-vidhya/nba-shot-data-analytics-visualization-with-python-pandas-and-matplotlib-part-1-the-basics-33643b10f0a6

Repo to check out

https://gitlab.com/jphwang/online_articles

deprecate up/down values for matrix

This feature allows each matrix cell to be composed of two values (positive and negative) which is relevant for enrichment analysis heatmaps (e.g. a term is enriched based on up- and down-regulated genes). However, implementing this is pretty complicated and this feature is very rarely used. So we'll simplify things and drop this feature.

Google Colab does not currently support custom widgets - see googlecolab/colabtools#498 (comment). However, @blois made an example interactive figure using IPython.display (this will be useful for cases where widgets are not supported).

The following enhancements should be made:

• clean up IPython.display implementation (e.g. maybe make a net.colab method in clustergrammer2)
• move reorder buttons and control panel above heatmap
• get tooltip working
• figure out how to pre-load data into shareable notebook (@manugarciaquismondo has worked on this). Maybe we can use Kaggle data API, or public data on Google Drive, or Github.

This link shows an example of Clustergrammer visualizing a random matrix:
https://colab.research.google.com/gist/cornhundred/b7154ee964c71cec6a2378f9cbba8eeb/clustergrammer2_example.ipynb?authuser=1

• /Users/nickfernandez/anaconda3/envs/voila/lib/python3.6/site-packages/clustergrammer2/clustergrammer_fun/load_data.py:68: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  header=col_arr)
• replace inst_rc with axis for all methods

Have the width of the widget be controllable as an input parameter and/or from the front-end. Also, add a full screen option.

Is your feature request related to a problem? Please describe.
We need more slices in the dendrogram.

Describe the solution you'd like
Save the entire linkage matrix in the visualization JSON, then slice the dendrogram on the front-end

Describe alternatives you've considered
More slices are an alternative, but not as good.

Additional context
This issue is a dependency of this issue in clustergrammer-gl ismms-himc/clustergrammer-gl#30. Passing the linkage matrix needs to be first done on the python back end and then addressed on the front-end.

We can use value based categories to denote how similar a cell is to its closest cell type signature. This way we can identify cells that may have been incorrectly assigned a cell type - based on surface marker or gene expression.

fix mybinder links

Need to check behavior on Kaggle since they are using a older version.

Describe the bug
Fix these warnings

/srv/conda/envs/notebook/lib/python3.7/site-packages/clustergrammer2/clustergrammer_fun/run_filter.py:104: FutureWarning:
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#ix-indexer-is-deprecated
df['mat'] = df['mat'].ix[keep_rows]
/srv/conda/envs/notebook/lib/python3.7/site-packages/clustergrammer2/clustergrammer_fun/run_filter.py:74: FutureWarning:
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#ix-indexer-is-deprecated
df = df.ix[keep_rows]
/srv/conda/envs/notebook/lib/python3.7/site-packages/clustergrammer2/clustergrammer_fun/make_views.py:28: FutureWarning:
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#ix-indexer-is-deprecated
tmp_df['mat'] = tmp_df['mat'].ix[keep_rows]

To Reproduce

1. Go to https://github.com/ismms-himc/clustergrammer2-notebooks
2. Run Jupyter Lab by clicking launch jupyter lab badge
3. Run notebook 1.0 Running Clustergrammer2
4. See warnings

Expected behavior
No warnings

Desktop (please complete the following information):

• MacOS (warnings are on cloud deployment, MyBinder)
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Is your feature request related to a problem? Please describe.
I would like to try to use UMAP to project data down to one dimension, then rank the data points in this one dimensional space to use as an ordering for the points in a heatmap. I plan on doing this for rows and columns separately. My hope is that this will enable a faster way of arranging/clustering data points and dimensions in a heatmap (it will lack the hierarchical clustering information obtained via hierarchical clustering). But it may be useful as a quick and dirty way of producing a heatmap from a large dataset.

@lmcinnes, would you think this seems like a reasonable idea?

Describe the solution you'd like
I'll use UMAP with n_componenets=1 (see https://github.com/lmcinnes/umap/blob/3f6e92f0ece650423d52eb560fac31db7e3f7dc7/doc/parameters.rst#n_components) then proceed as described above.

Describe the solution you'd like
The interactive dendrogram should be able to assemble a pandas dataframe with cell metadata based on clicking clusters and assigning categories. We should be able to pass in an optional metadata DataFrame that gets appended to by the dendrogram. We already know how to have front end to back end communication.

This will simplify filtering for a labeled subset of cells and the addition of additional metadata. We probably need to think about keeping track of the global metadata for all cells while still working on sunsets.

We should also keep track of what level and cluster of the dendrogram a category was assigned so that it is transparent and reproducible.

Initialize a blank canvas category (option in widget) that will be used to convey the state of the new metadata over time as the user populates this metadata (by interacting with different levels of the dendrogram). This will be easier than having to resize all elements in the visualization (which we can avoid for now) and will only require recoloring categories and changing tooltip contents.

Describe alternatives you've considered
This type of solution that seems best for this purpose of manually labeling cells based on the interactive dendrogram.

Additional context
We can have the state of the new categories shown in the dendrogram (stored in the front end) and/or as new categories (which will be hard to do).

Allow users to order the heatmap based on their initial custom ordering as well as set the default ordering for the heatmap from the widget. Thanks for the issue @jeremymsimon

deprecate additional pre-filtered views

Add contributing guidelines similar to this one:

https://github.com/voila-gallery/gallery/blob/master/CONTRIBUTING.md

Is your feature request related to a problem? Please describe.
I would like to install Clustergrammer2 via conda.

Describe the solution you'd like
We need to upload a conda installable version of Clustergrammer2

I have a few examples where we generate gene expression signatures based on some single cell categories (e.g. cell type) and then add gene-level (row) categories based on which cell category the gene is most highly expressed in. This usually only adds categories for a subset of genes since not all genes are considered "signature" genes. We should simplify this process and have the following features:

• Assign each gene a category based on which cell type (or generally category) it is most highly expressed
• Calculate this in the front end after setting manual categories
• Show the P-value (-log10) as a value-based category of the one-vs-all T-test. This allows users to easily identify significantly differentially up-regulated genes in a category

Save this information so that:

• the front-end can on the fly re-calculate the original values
• we can show the mean and variance as value based categories

Hi, as the clustergrammer_widget appears very slow with a large dataset, I would like to try the very promising clustergrammer2 but I run into difficulties when trying to get the programme recognised by jupyter. The install was through the normal python install setup.py approach.

At the activation stage of the widgets (which depended on the joblib module) this happens:

bash-5.0# jupyter nbextension enable --py widgetsnbextension
Enabling notebook extension jupyter-js-widgets/extension...
- Validating: OK
bash-5.0# jupyter nbextension enable --py --sys-prefix clustergrammer2
clustergrammer2 backend version 0.4.0
Enabling notebook extension clustergrammer2/extension...
- Validating: problems found:
- require? X clustergrammer2/extension

What can have gone wrong??

Use existing infrastructure from widget-ts-cookiecutter to set up back-end and front-end testing.

Start with Python I/O - see https://clustergrammer.readthedocs.io/matrix_format_io.html and https://github.com/ismms-himc/clustergrammer2/blob/master/examples/introduction_nb.ipynb

Clear WebGL contexts when making a new widget. This will prevent crashing when many widgets are made in a single notebook.

Is your feature request related to a problem? Please describe.
The Clustergrammer2 python API needs to be cleaned up.

• move UMI normalization into normalization
• update sparse matrix loading methods
• replace inst_rc with `axis

Describe the solution you'd like
Clean up API and document in documentation

We need to generalize row/column categories for rows/columns that are themselves clusters. The way we have tried to do this previously is to show the majority category followed by a value based category that indicates the proportion of the majority category. It would be better to show a stacked bar plot of the category breakdown (that would also be generalizable to a rectangle and not just a square). See https://github.com/effiken/scDissector cluster categories for an example (probably using plotly).

Use case. Find up/down genes based on clustering. Calculate the vector of gene expression of the up/down genes. Run enrichment analysis for this gene list across many libraries (locally stored GMTs) and aggregate the enriched terms in a heatmap. The columns of the heatmap are enriched terms with library category (e.g. Gene Ontology) and P-value value-based category. The rows are genes (with value based category for average expression). The matrix cells are binary associations of genes with an enriched term.

Is your feature request related to a problem? Please describe.
We want to load a pre-clustered view of our data - possibly load the visualization JSON

Describe the solution you'd like
We want to be able to save a clustered view and load this view into a heatmap. This will be useful for dashboarding ismms-himc/codex_dashboard#5

Describe alternatives you've considered
We could save the clustered view as a visualization JSON or some new TSV format.

Additional context
This will be useful for dashboarding ismms-himc/codex_dashboard#5

Describe the bug
I tried the last version of clustergrammer2 in a fresh conda environment with the last version of jupyterlab. I installed clustergrammer and activated the lab extension using the instructions from the README.md,

$ pip install clustergrammer2
$ jupyter labextension install @jupyter-widgets/jupyterlab-manager

and this is what i have:

$ conda list | egrep  'python |ipywidgets|jupyterlab|clustergrammer'
clustergrammer2           0.5.18                   pypi_0    pypi
ipython                   7.13.0           py37hc8dfbb8_2    conda-forge
ipywidgets                7.5.1                      py_0    conda-forge
jupyterlab                2.0.1                      py_0    conda-forge
jupyterlab_server         1.1.0                      py_0    conda-forge
python                    3.7.6           h8356626_5_cpython    conda-forge

when i tried to use it in jupyterlab notebook I got: (This is just an excerpt)

react_devtools_backend.js:6 Plugin 'jupyter.extensions.clustergrammer2' failed to activate.
r	@	react_devtools_backend.js:6
(anonymous)	@	index.js:276
Promise.catch (async)		
(anonymous)	@	index.js:275
e.start	@	index.js:274
o	@	index.out.js:1011
load (async)	
...
...
(anonymous)	@	vendors~main.535a1ca….js:2
react_devtools_backend.js:6 Error: No provider for: jupyter.extensions.jupyterWidgetRegistry.
    at f.e.resolveRequiredService (index.js:191)
    at index.js:149

When i checked the extensions installed, I got:

$ jupyter labextension list
JupyterLab v2.0.1
Known labextensions:
   app dir: /home/mibu/miniconda3/envs/py_almost-update/share/jupyter/lab
        @jupyter-widgets/jupyterlab-manager v2.0.0  enabled  OK
        clustergrammer2 v0.5.17  enabled  OK

An odd thing is that says clustergrammer2 v0.5.17 even though i have 0.5.18 installed. I uninstalled the extension and reinstalled using other method:

$ jupyter labextension install clustergrammer2
Building jupyterlab assets (build:prod:minimize)
$ jupyter labextension list
JupyterLab v2.0.1
Known labextensions:
   app dir: /home/mibu/miniconda3/envs/py37/share/jupyter/lab
        @jupyter-widgets/jupyterlab-manager v2.0.0  enabled  OK
        clustergrammer2 v0.5.18  enabled  OK

But still I'm getting the same js error. I tried multiple times with fresh environments, installing directly the extension using jupyter labextension install clustergrammer2 to avoid getting the wrong version of it but that doesn't help. The widget does work in notebook 6.0.3 and in jupyter-lab 1.2.7 following the instructions from the README

Desktop (please complete the following information):

• OS: Linux 5.5.13-1-MANJARO
• Chrome 80.0.3987.162

Full javascript error:

react_devtools_backend.js:6 Plugin 'jupyter.extensions.clustergrammer2' failed to activate.
r	@	react_devtools_backend.js:6
(anonymous)	@	index.js:276
Promise.catch (async)		
(anonymous)	@	index.js:275
e.start	@	index.js:274
o	@	index.out.js:1011
load (async)		
ANye	@	index.out.js:1057
a	@	bootstrap:84
0	@	main.ca893ad….js:1
a	@	bootstrap:84
r	@	bootstrap:45
i	@	bootstrap:32
(anonymous)	@	vendors~main.535a1ca….js:2
react_devtools_backend.js:6 Error: No provider for: jupyter.extensions.jupyterWidgetRegistry.
    at f.e.resolveRequiredService (index.js:191)
    at index.js:149
    at Array.map (<anonymous>)
    at f.e.activatePlugin (index.js:149)
    at index.js:275
    at Array.map (<anonymous>)
    at f.e.start (index.js:274)
    at o (index.out.js:1011)
r	@	react_devtools_backend.js:6
(anonymous)	@	index.js:277
Promise.catch (async)		
(anonymous)	@	index.js:275
e.start	@	index.js:274
o	@	index.out.js:1011
load (async)		
ANye	@	index.out.js:1057
a	@	bootstrap:84
0	@	main.ca893ad….js:1
a	@	bootstrap:84
r	@	bootstrap:45
i	@	bootstrap:32
(anonymous)	@	vendors~main.535a1ca….js:2

Describe the bug
It has been reported that the widget does not work with Jupyter Lab 1.0

To Reproduce
Steps to reproduce the behavior:

White space is showing in the place where you expect to see the controls.

Link Voronoi diagram from codex. Calculate nearest neighbor cell type breakdown. Make heatmap of nearest neighbor breakdown and combine with marker levels.

Look into Kepler.gl widget for Voronoi plots.

Is your feature request related to a problem? Please describe.
I want to be able to change the color scheme - currently it is only red/blue for positive/negative values, respectively.

Describe the solution you'd like
I would like to be able to pass in two colors into the net.widget method (e.g. named colors or hexcode) and have those override the positive/negative colors.

Additional context
This issue depends on this issue in Clustergrammer-GL ismms-himc/clustergrammer-gl#33

Make kaggle examples using public 10X genomics data. The notebooks should be proof of concepts that demonstrate how we can use Clustergrammer2 to perform exploratory data analysis. They should also include static UMAP and t-SNE plots of cells and genes to highlight the relationship of a heatmap and dimensionality reduction views of data points and dimensions.

Make Seurat, Scanpy, and scVI kernels, save their outputs, and visualize them using Clustergrammer2. We can compare the clustering and cell type labels from the different methods in a summary notebook (that loads the saved data from each kernel).

scVI link: https://github.com/YosefLab/scVI/blob/master/tests/notebooks/Linearly_decoded_VAE_for_gene_interpretation.ipynb

Pre-calculate UMAP coordinates for the rows and columns of a heatmap. Save these in the Visualization JSON. If UMAP coordinates are available to Clustergrammer-GL, show UMAP reordering buttons for rows and columns. Pressing the UMAP buttons for rows/columns will cause the matrix to disappear, rows/columns rectangles (viz-aid-triangles) will rearrange (animated ismms-himc/clustergrammer-gl#4) themselves into a UMAP (not zoomable for now). If making a UMAP of columns, then row categories and row names will remain visible. Clicking row categories or rows from the invisible matrix will change the colors of the UMAP squares/points.

https://github.com/iaconogi/bigSCale2

Hello and thank you for the great tool.

I am using the Python version of clustergrammer2, specifically in jupyter notebook, and am unable to generate a sim_mat from my dataframe. The widget visualization of the same dataframe otherwise works when running

net.load_df(filtered_df)
net.cluster()
net.widget()

The code that fails is:
net.load_df(filtered_df)
net.cluster(sim_mat=True)

Stack trace:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-30-504a038db812> in <module>
      1 net.load_df(filtered_df)
----> 2 net.cluster(sim_mat=True)

~/anaconda2/envs/sd2py3/lib/python3.6/site-packages/clustergrammer_widget/clustergrammer/__init__.py in cluster(self, dist_type, run_clustering, dendro, views, linkage_type, sim_mat, filter_sim, calc_cat_pval, run_enrichr, enrichrgram)
     97                                    calc_cat_pval=calc_cat_pval,
     98                                    run_enrichr=run_enrichr,
---> 99                                    enrichrgram=enrichrgram)
    100 
    101   def make_clust(self, dist_type='cosine', run_clustering=True,

~/anaconda2/envs/sd2py3/lib/python3.6/site-packages/clustergrammer_widget/clustergrammer/make_clust_fun.py in make_clust(net, dist_type, run_clustering, dendro, requested_views, linkage_type, sim_mat, filter_sim, calc_cat_pval, sim_mat_views, run_enrichr, enrichrgram)
     67 
     68   if sim_mat is not False:
---> 69     sim_net = make_sim_mat.main(net, inst_dm, which_sim, filter_sim, sim_mat_views)
     70 
     71     net.sim = {}

~/anaconda2/envs/sd2py3/lib/python3.6/site-packages/clustergrammer_widget/clustergrammer/make_sim_mat.py in main(net, inst_dm, which_sim, filter_sim, sim_mat_views)
     11 
     12     sim_dict[inst_rc] = dm_to_sim(inst_dm[inst_rc], make_squareform=True,
---> 13                              filter_sim=filter_sim)
     14 
     15   sim_net = {}

~/anaconda2/envs/sd2py3/lib/python3.6/site-packages/clustergrammer_widget/clustergrammer/make_sim_mat.py in dm_to_sim(inst_dm, make_squareform, filter_sim)
     51 
     52   if filter_sim > 0:
---> 53     filter_sim = adjust_filter_sim(inst_sim_mat, filter_sim)
     54     inst_sim_mat[ np.abs(inst_sim_mat) < filter_sim] = 0
     55 

~/anaconda2/envs/sd2py3/lib/python3.6/site-packages/clustergrammer_widget/clustergrammer/make_sim_mat.py in adjust_filter_sim(inst_dm, filter_sim, keep_top)
     69 
     70     inst_series = pd.Series(val_vect)
---> 71     inst_series.sort(ascending=False)
     72 
     73     sort_values = inst_series.values

~/.local/lib/python3.6/site-packages/pandas/core/generic.py in __getattr__(self, name)
   5065             if self._info_axis._can_hold_identifiers_and_holds_name(name):
   5066                 return self[name]
-> 5067             return object.__getattribute__(self, name)
   5068 
   5069     def __setattr__(self, name, value):

AttributeError: 'Series' object has no attribute 'sort'

Add YouTube video tutorials to the beginning of the example notebooks, e.g. https://github.com/ismms-himc/clustergrammer2-notebooks. This way users know what to expect (especially since Kaggle does not support embedding of interactive widgets).

Also, add YouTube links to READMEs.

To see the visualization, you need to execute each cell first. This is not convenient.