allonkleinlab / spring_dev Goto Github PK

Python 11.53% Shell 0.01% HTML 0.86% JavaScript 39.09% CSS 16.21% OpenEdge ABL 0.01% Jupyter Notebook 32.28%

spring_dev's Introduction

Installing Python libraries

To run SPRING Viewer locally, make sure Python 2.7 is installed (and that it's your active version). You will also need the following Python libraries:

scikit-learn
numpy
scipy
matplotlib
h5py
networkx
fa2
python-louvain

We recommend Anaconda to manage your Python libraries. You can download it here (be sure to get the Python 2.7 version): https://conda.io/miniconda.html. Libraries can then be installed using the command conda. To do so, open Terminal (Mac) or Anaconda Prompt (Windows) and enter:

conda install scikit-learn numpy scipy matplotlib h5py

The remaining libraries can be installed using pip. Note that if you're a Windows user, you'll first need to install Microsoft Visual C++ compiler for Python (available from http://aka.ms/vcpython27). Enter the following into Terminal or Anaconda Prompt:

pip install networkx fa2 python-louvain

Setting up a SPRING data directory

See the example notebooks:
Hematopoietic progenitor FACS subpopulations
Mature blood cells (10X Genomics 4k PBMCs)
CITE-seq data from 10X Genomics
PBMCs from 10X Genomics

A SPRING data set consist of a main directory and any number of subdirectories, with each subdirectory corresponding to one SPRING plot (i.e. subplot) that draws on a data matrix stored in the main directory. The main directory should have the following files, as well as one subdirectory for each SPRING plot.

matrix.mtx
counts_norm_sparse_cells.hdf5
counts_norm_sparse_genes.hdf5
genes.txt

Each subdirectory should contain:

categorical_coloring_data.json
cell_filter.npy
cell_filter.txt
color_data_gene_sets.csv
color_stats.json
coordinates.txt
edges.csv
graph_data.json
run_info.json

Running SPRING Viewer

Open Terminal (Mac) or Anaconda Prompt (Windows) and change directories (cd) to the directory containing this README file (SPRING_dev/).
Start a local server by entering the following: python -m CGIHTTPServer 8000
Open web browser (preferably Chrome; best to use incognito mode to ensure no cached data is used).
View data set by navigating to corresponding URL: http://localhost:8000/springViewer_1_6_dev.html?path_to/main/subplot. In the example above, if you wanted to view a SPRING plot called FullDataset_v1 in the main directory 10X_PBMCs_Signac_GitHub, then you would navigate to http://localhost:8000/springViewer_1_6_dev.html?datasets/10X_PBMCs_Signac_GitHub/FullDataset_v1

Signac

To classify cellular phenotypes in single cell data, SignacX was integrated with the files output by SPRING (specifically, the matrix.mtx, genes.txt, edges.csv and categorical_coloring_data.json files), such that SPRING data can be classified by Signac in R with only a few lines of code. First, install SignacX in R:

# load the Signac library
install.packages('SignacX')
library(SignacX)

Now classify the cellular phenotypes in R, which allows them to be visualized in SPRING Viewer:

# dir is the subdirectory generated by the Jupyter notebook; it is the directory that contains the 'categorical_coloring_data.json' file.
dir = "./FullDataset_v1" 

# load the expression data
E = CID.LoadData(dir)

# generate cellular phenotype labels
labels = Signac(E, spring.dir = dir, num.cores = 4)
# alternatively, if you're in a hurry, use:
# labels = SignacFast(E, spring.dir = dir, num.cores = 4)
celltypes = GenerateLabels(labels, E = E, spring.dir = dir)

# write cell types and Louvain clusters to SPRING
dat <- CID.writeJSON(celltypes, spring.dir = dir)

Subsequently, the single cell data together with the cellular phenotypes (and Louvain clusters) can be visualized in SPRING Viewer (Signac writes the cell type and clustering information to the categorical_coloring_data.json file).

spring_dev's People

Contributors

Stargazers

Watchers

spring_dev's Issues

The python2.7 was so older that i can not install the module that your workflow need

Hi, developer
I want to return the spring_dev, but i get the module miss error, like below:

this module can not be install by pip.i already had solve the the preprocessing_python and the helper_functions module.
but the tables can not be solved by myself. it was complicated than i though.
Hope you help to fix this issue.
Best
Hanhuihong

unable to find genes when export to SPRING object

Hi SPRING team,
I convert Seurat obj to anndata with:
mda_ad <- Convert(from = mda, to = "anndata",filename = 'mda_day27_ad.h5ad')

mda_ad
AnnData object with n_obs × n_vars = 6039 × 19331
obs: 'n_genes', 'n_counts', 'orig_ident', 'percent_mito', 'S_Score', 'G2M_Score', 'Phase', 'old_ident', 'res_1_2'
var: 'gene.mean', 'gene.dispersion', 'gene.dispersion.scaled'
obsm: 'X_pca', 'X_tsne'

Then I run as scanpy's tutorial https://github.com/theislab/scanpy_usage/blob/master/171111_SPRING_export/SPRING_export.ipynb
to export SPRING, however, I was unable to find no response when I enter a gene in the web browser.
I have all required files in the folder.

Any suggestion how to solve it?

Thanks!

Changing expression colors

Hi SPRING_dev team!

Thanks for the awesome tools!

I may have missed this, can you please guide me to where can I change the expression colors?

Currently it's green in color. For better visualization, I want more tones to it maybe a viridis.

Thanks!
Justine

Problem loading counts matrices - HPCs tutorial

Hi
I am a python novice and I am interested in using SPRING
With some adaptations I successfully managed to run the pbmc4k tutorial on python 3.7.3
Instead when trying the HPCs tutorial, upon loading the counts matrices at this stage...

for s in sample_name:
print '_________________', s

if os.path.isfile(input_path + s + '.raw_counts.unfiltered.npz'):
    print('Loading from npz file')
    D[s]['E'] = scipy.sparse.load_npz(input_path + s + '.raw_counts.unfiltered.npz')
else:
    print('Loading from text file')
    E,cell_bcs = load_text(file_opener(input_path + s + '.counts.tsv.gz'), delim = '\t', load_cell_bcs=True)
    D[s]['E'] = E
    D[s]['cell_bcs'] = cell_bcs
    scipy.sparse.save_npz(input_path + s + '.raw_counts.unfiltered.npz', D[s]['E'], compressed = True)
print(D[s]['E'].shape)

....I get the following error:

UnicodeDecodeError Traceback (most recent call last)
in
10 else:
11 print('Loading from text file')I cannot
---> 12 E,cell_bcs = load_text(file_opener(input_path + s + '.counts.tsv.gz'), delim = '\t', load_cell_bcs=True)
13 D[s]['E'] = E
14 D[s]['cell_bcs'] = cell_bcs

~/SPRING_dev-spring-of-rebirth/data_prep/spring_helper.py in load_text(file_data, delim, load_cell_bcs)
149 start_column = -1
150 start_row = -1
--> 151 for row_ix, dat in enumerate(file_data):
152 dat = dat.strip('\n').split(delim)
153 if start_row == -1:

~/anaconda3/lib/python3.7/gzip.py in readline(self, size)
372 def readline(self, size=-1):
373 self._check_not_closed()
--> 374 return self._buffer.readline(size)
375
376

~/anaconda3/lib/python3.7/_compression.py in readinto(self, b)
66 def readinto(self, b):
67 with memoryview(b) as view, view.cast("B") as byte_view:
---> 68 data = self.read(len(byte_view))
69 byte_view[:len(data)] = data
70 return len(data)

~/anaconda3/lib/python3.7/gzip.py in read(self, size)
461 # jump to the next member, if there is one.
462 self._init_read()
--> 463 if not self._read_gzip_header():
464 self._size = self._pos
465 return b""

~/anaconda3/lib/python3.7/gzip.py in _read_gzip_header(self)
404
405 def _read_gzip_header(self):
--> 406 magic = self._fp.read(2)
407 if magic == b'':
408 return False

~/anaconda3/lib/python3.7/gzip.py in read(self, size)
89 self._read = None
90 return self._buffer[read:] +
---> 91 self.file.read(size-self._length+read)
92
93 def prepend(self, prepend=b''):

~/anaconda3/lib/python3.7/codecs.py in decode(self, input, final)
320 # decode input (taking the buffer into account)
321 data = self.buffer + input
--> 322 (result, consumed) = self._buffer_decode(data, self.errors, final)
323 # keep undecoded input until the next call
324 self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

I tried with no luck several workarounds on the file_opener module of spring_helper.py (I got the latest version with - Added cell_BC export - ) as my hypothesis is that this is somehow related to the GzipFile missing the utf-8 encoding.

I might be completely wrong and/or missing something totally obvious...Could you help me?

Issue with loading the localhost:8000/springViewer_1_6_dev.html page in all web browsers

Hi
I am having problems loading the localhost page in static mode both on Firefox and Chrome (I am working on a Linux machine and starting local server both in python 2.7 and 3.6 enviroments)

I can load the map on dynamic mode
http://localhost:8000/springViewer_1_5_dev.html?datasets/192567415/full
But when I try to switch to the static mode
http://localhost:8000/springViewer_1_6_dev.html?datasets/192567415/full
it's like the page does not fully load
in firefox I get a grey page with no map and only the top tabs
in chrome I get a black background with no map but I can type and browse the gene list in the top menu

Last time I tried it was Decembed 2019 and I was able to load these very same maps in both modes and play with the static mode (e.g. creating subplots and louvain clustering) with no problem
Both Firefox and Chrome browsers are up to date to the latest version

Can you help me?
thank you

BrokenPipeError from /cgi-bin/load_counts.py

Hi,

I'm trying to get the SPRING web interface to work offline. But when I try to load the gene counts I come across a BrokenPipeError. What can be the cause of this? Using python2.7 and MacOS.

Regards,

Anton

python2.7 -m CGIHTTPServer 8000
Serving HTTP on 0.0.0.0 port 8000 ...
127.0.0.1 - - [24/Feb/2020 12:23:41] “GET /springViewer_1_6_dev.html?datasets/pbmc4k/full HTTP/1.1” 200 -
127.0.0.1 - - [24/Feb/2020 12:23:41] code 404, message File not found
127.0.0.1 - - [24/Feb/2020 12:23:41] “GET /scripts_1_6_dev/text_annotations.js HTTP/1.1" 404 -
127.0.0.1 - - [24/Feb/2020 12:23:41] code 404, message File not found
127.0.0.1 - - [24/Feb/2020 12:23:41] “GET /datasets/pbmc4k/full/mutability.txt HTTP/1.1” 404 -
127.0.0.1 - - [24/Feb/2020 12:23:41] “POST /cgi-bin/load_counts.py HTTP/1.1" 200 -
Exception ignored in: <_io.TextIOWrapper name=‘’ mode=‘w’ encoding=‘UTF-8’>
BrokenPipeError: [Errno 32] Broken pipe

Exception happened during processing of request from (‘127.0.0.1’, 60592)
Traceback (most recent call last):
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py”, line 295, in _handle_request_noblock
self.process_request(request, client_address)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py”, line 321, in process_request
self.finish_request(request, client_address)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py”, line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py”, line 655, in init
self.handle()
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py”, line 340, in handle
self.handle_one_request()
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py”, line 328, in handle_one_request
method()
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/CGIHTTPServer.py”, line 62, in do_POST
self.run_cgi()
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/CGIHTTPServer.py”, line 239, in run_cgi
if not self.rfile.read(1):
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py”, line 384, in read
data = self._sock.recv(left)
error: [Errno 54] Connection reset by peer

127.0.0.1 - - [24/Feb/2020 12:23:41] code 404, message File not found
127.0.0.1 - - [24/Feb/2020 12:23:41] “GET /stuff/colorpicker_images/custom_hsb_b.png HTTP/1.1" 404 -
127.0.0.1 - - [24/Feb/2020 12:23:41] code 404, message File not found
127.0.0.1 - - [24/Feb/2020 12:23:41] “GET /stuff/colorpicker_images/custom_hsb_s.png HTTP/1.1” 404 -
127.0.0.1 - - [24/Feb/2020 12:23:41] code 404, message File not found
127.0.0.1 - - [24/Feb/2020 12:23:41] “GET /stuff/colorpicker_images/custom_hsb_h.png HTTP/1.1" 404 -
127.0.0.1 - - [24/Feb/2020 12:23:41] code 404, message File not found
127.0.0.1 - - [24/Feb/2020 12:23:41] “GET /datasets/pbmc4k/full/animation.txt HTTP/1.1” 404 -
127.0.0.1 - - [24/Feb/2020 12:23:41] code 404, message File not found
127.0.0.1 - - [24/Feb/2020 12:23:41] “GET /datasets/pbmc4k/full/clone_map.json?=1582543421216 HTTP/1.1" 404 -
127.0.0.1 - - [24/Feb/2020 12:23:41] code 404, message File not found
127.0.0.1 - - [24/Feb/2020 12:23:41] “GET /datasets/pbmc4k/full/sticky_notes_data.json HTTP/1.1” 404 -
127.0.0.1 - - [24/Feb/2020 12:23:41] code 404, message File not found
127.0.0.1 - - [24/Feb/2020 12:23:41] “GET /datasets/pbmc4k/full/PAGA_data.json?=1582543421219 HTTP/1.1" 404 -

Problem about SPRING_py3 ./start_server.sh

Hi!

I followed the SPRING_py3 pipeline and created the environment, however when I started ./start_server.sh, the sever showed

Not Found
The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

Do you have any suggestions as to what to do?

Updates

First to say, this is a fantastic project. I am working to updating the code-base of a portal application wrapping the spring tool from python 2.7 to 3.x. The code-bases are independent but this had me thinking about the spring viewer code base itself.

Are there any plans to do this for the public source? Or are there any significant private updates made to the project not posted on github?

a question about cell grouping file when running spring browser version

Hope your week is going well and thanks for your work on the spring force directed graph online viewer! I have a question regarding the input file "cell groupings". Based on the instruction:

I constructed my cell grouping csv files as:

My goal is to visualize the cell identity clusters on the spring plot. So the first column as the cell identity label followed by cell barcode in the second column. But apparently this is not correct because it returns the error message:
Error: Cell grouping "HSC" has a different number of entries (1) than the number of cells (6613) in the expression matrix. I have no luck in googling...
would you mind clarifying what exactly should the format be? Thank you very much for your time!

Question about save SPRING file

I am learning the SPRING Viewer by repeating the code of tutorial pbmc 4K.
I have some trouble with the last step(Save SPRING files).
I got the error : TypeError: 1386 is not JSON serializable(https://github.com/njumay/SPRING/blob/master/spring_example_pbmc4k.ipynb).
I do not kown how to solve this problem.
I hope someone can help me. Thanks a lot.

Setting colors for different conditions of cells

I'm trying to figure out how to display cells with two conditions in two different colors. I don't think there are specific cell labels for each cell input into SPRING because only the X matrix and the gene list are used in the function make_spring_subplot. However, I suppose the indexes from 0 to the total number of cells can be used automatically as cell labels,

I guessed that the build_categ_colors function would be used to make the input for parameter cell_groupings. Then, cell_groupings was set to d_condition_labels as shown in the code below in make_spring_subplot. But the graph generated in SPRING is not what I want. So I don't think I set it up right. Can you help me to solve this problem? Thank you very much!

# test #0-5 indicate the cell label 
# Dict d_condition_labels contains cell labels belonging to the corrsponding condition
d_condition_labels = {"Infected":['3','4','5'],"Control":['0','1','2']}

# Then I tested the function to check the structure of cell_groupings
build_categ_colors({},d_condition_labels)
{'Control': {'label_colors': {'0': '#00d4ff', '1': '#00007f', '2': '#ffe500'},
  'label_list': ['0', '1', '2']},
 'Infected': {'label_colors': {'3': '#00007f', '4': '#ffe500', '5': '#00d4ff'},
  'label_list': ['3', '4', '5']}}

Error - file not found

Hi there,

I tried to run the SPRING Viewer but I get the following message and nothing is displayed:

127.0.0.1 - - [08/Nov/2018 15:12:22] "GET /scripts_1_6_dev/jquery-3.1.0.min.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/newSpringPlot.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/colorBar.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/d3-queue.v3.min.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/d3.min.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/forceLayout_script.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/rotation_script.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/selection_script.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/settings_script.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/diffex_script.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/jspdf.min.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/coexpression_script.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/simg.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/canvg.min.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/FileSaver.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/pixi.min.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/spin.min.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/make_new_SPRINGplot_script.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/LineSprite.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/clone_viewer.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/stickyNote.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/smoothing_imputation.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/downloadSelectedExpr_script.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_dev_shared/jquery.autocomplete.min.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/sweetalert.min.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/doublet_detector.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/PAGA_viewer.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/cluster2_script.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/colorpicker.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] code 404, message File not found
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/text_annotations.js HTTP/1.1" 404 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/selection_logic.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/colorpicker_eye.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/colorpicker_layout.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/colorpicker_utils.js HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /scripts_1_6_dev/sound_effects/icon_mute.svg HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] code 404, message File not found
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /Users/luca_work/Desktop/spring//coordinates.txt HTTP/1.1" 404 -
127.0.0.1 - - [08/Nov/2018 15:12:23] code 404, message File not found
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /Users/luca_work/Desktop/spring//mutability.txt HTTP/1.1" 404 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /stuff/colorpicker_images/custom_background.png HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /stuff/colorpicker_images/custom_rgb_r.png HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] code 404, message File not found
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /stuff/colorpicker_images/custom_hsb_b.png HTTP/1.1" 404 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /stuff/colorpicker_images/custom_submit.png HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /stuff/colorpicker_images/colorpicker_overlay.png HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /stuff/colorpicker_images/custom_rgb_b.png HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /stuff/colorpicker_images/custom_rgb_g.png HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /stuff/colorpicker_images/custom_hex.png HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] code 404, message File not found
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /stuff/colorpicker_images/custom_hsb_h.png HTTP/1.1" 404 -
127.0.0.1 - - [08/Nov/2018 15:12:23] code 404, message File not found
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /stuff/colorpicker_images/custom_hsb_s.png HTTP/1.1" 404 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /stuff/colorpicker_images/colorpicker_select.gif HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:23] "GET /stuff/colorpicker_images/custom_indic.gif HTTP/1.1" 304 -
127.0.0.1 - - [08/Nov/2018 15:12:31] "GET /stuff/ex-mark.svg HTTP/1.1" 304 -

Any idea of what is causing this?

Thank you!

please create tags or release

I am responsible for maintaining open source scientific software. I would like to request that you create git tags or git releases in order that we can better document what versions are installed. This also helps to make science reproducible. thank you,

cell barcodes IDs for download files

Hi!

Great tool! For the various downloads (doublet scores, coordinates, etc), can I assume that the values are in the same cell order as the input matrix? Or is there a way to add the cell barcode IDs to the download?

Thanks!

Is more documentation planned?

Are you planning on extending the documentation of this version? It's a great tool, but the technical side of the documentation is a bit sparse, both in the README, the source code, and the web front-end. Some details have clearly changed from the publication, and it required some effort for the user to understand what's going on.

For example, is there a reference for the v-score filtering of genes? From the code it's clear that it's filtering based on the distribution of variability, but what is the quantity being calculated from the fano factors? It's also not so clear to the less technically versed front-end user whether to in- or decrease that parameter based on its name, so more documentation could be useful here.

Export to spring issue (AttributeError: 'numpy.ndarray' object has no attribute 'tocsc')

Hi,

I am having an issue exporting my h5ad file to spring for visualization. I have been following the spring export tutorial but every time I try to run sc.export_to.spring_project I get the following error:

sc.export_to.spring_project(adata1, '/home/seth/anaconda3/write', 'draw_graph')
Writing subplot to /home/seth/anaconda3
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-9-656bbf38658a> in <module>
----> 1 sc.export_to.spring_project(adata1, '/home/seth/anaconda3/write', 'draw_graph', cell_groupings='louvain_r0.8_sub')

~/anaconda3/envs/ddocent_env/lib/python3.7/site-packages/scanpy/_exporting.py in spring_project(adata, project_dir, embedding_method, subplot_name, cell_groupings, custom_color_tracks, total_counts_key, overwrite)
    101     # Ideally, all genes will be written from adata.raw
    102     if adata.raw is not None:
--> 103         E = adata.raw.X.tocsc()
    104         gene_list = list(adata.raw.var_names)
    105     else:

AttributeError: 'numpy.ndarray' object has no attribute 'tocsc'

I can't seem to get around this error. I tried posting in the scanpy github page but received no responses. Any help would be greatly appreciated!

Thanks,
Seth

Downloads - firefox

It appears Spring is generally compatible with Firefox, but we noticed download options are not functioning in Firefox. I was able to resolve by a small update to the downloadFile function. It is a one line change suggested from this link:
https://support.mozilla.org/en-US/questions/968992

Apparently the hidden link being created needs to be attached to the DOM but not in Chrome.

Before this line -

SPRING_dev/scripts_1_6_dev/forceLayout_script.js

Line 577 in efe6ab4

hiddenElement.click();

I added this to attach to DOM:
document.body.appendChild(hiddenElement)

socket.error: [Errno 98] Address already in use

input "python -m CGIHTTPServer 8000" and then got noting but socket.error: [Errno 98] Address already in use

Default Hide Edges

I am interested to have edges hidden by default. I thought it would be straight-forward but hit my 1 hour time-box :) Do you have any quick guidance of how I might accomplish? Not as a feature request, but which code I might update on my fork.

how to pre-process annotations (custom color tracks and groupings)

Dear colleagues,
I'm trying to pre-process the required input for a local SPRING server by following the notebook in data_prep/spring_example_pbmc4k.ipynb
However, I am unable to run the function that generates the required files in a way that it would include my custom annotations with continuous and categorical data.

Here is how I load the annotation files (that I previuously saved in the same format that I would normally use for the SPRING server that is hosted in Allon Klein Lab)

import csv
with open(main_spring_dir + '../../spring.groupings.csv') as csvfile:
    reader = csv.reader(csvfile)
    cell_groupings = {}
    for row in reader:
        key = row[0]
        cell_groupings[key] = row[1:]
with open(main_spring_dir + '../../spring.custom.color.tracks.csv') as csvfile:
    reader = csv.reader(csvfile)
    custom_colors = {}
    for row in reader:
        key = row[0]
        custom_colors[key] = row[1:]

But when later I call the function below to generate the subplots and processed files, it breaks:

out = make_spring_subplot(E, gene_list, save_path, 
                    normalize = False, tot_counts_final = total_counts,
                    min_counts = 3, min_cells = 3, min_vscore_pctl = 60,show_vscore_plot = True, 
                    num_pc = 60, 
                    k_neigh = 5, 
                    num_force_iter = 500,
                    cell_groupings = cell_groupings,
                    custom_colors = custom_colors)

Displaying the following:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-13-e5e49c56d331> in <module>()
     11                     num_force_iter = 500,
     12                     cell_groupings = cell_groupings,
---> 13                     custom_colors = custom_colors)
     14 
     15 np.save(save_path + '/cell_filter.npy', np.arange(E.shape[0]))

/restricted/projectnb/crem-bioinfo/project_code/00_pan_project/SPRING_dev/data_prep/spring_helper.pyc in make_spring_subplot(E, gene_list, save_path, base_ix, normalize, exclude_dominant_frac, min_counts, min_cells, min_vscore_pctl, show_vscore_plot, exclude_gene_names, num_pc, sparse_pca, pca_norm, k_neigh, cell_groupings, num_force_iter, output_spring, precomputed_pca, gene_filter, custom_colors, exclude_corr_genes_list, exclude_corr_genes_minCorr, dist_metric, use_approxnn, run_doub_detector, dd_k, dd_frac, dd_approx, tot_counts_final)
    778             save_spring_dir_sparse_hdf5(E, gene_list, save_path, list(links),
    779                             custom_colors = custom_colors,
--> 780                             cell_groupings = cell_groupings)
    781         else:
    782             save_spring_dir_sparse_hdf5(E, gene_list, save_path, list(links),

/restricted/projectnb/crem-bioinfo/project_code/00_pan_project/SPRING_dev/data_prep/spring_helper.pyc in save_spring_dir_sparse_hdf5(E, gene_list, project_directory, edges, custom_colors, cell_groupings)
    654     # save custom colors
    655     custom_colors['Uniform'] = np.zeros(E.shape[0])
--> 656     write_color_tracks(custom_colors, project_directory+'color_data_gene_sets.csv')
    657 
    658     # create and save a dictionary of color profiles to be used by the visualizer

/restricted/projectnb/crem-bioinfo/project_code/00_pan_project/SPRING_dev/data_prep/spring_helper.pyc in write_color_tracks(ctracks, fname)
    598     out = []
    599     for name,score in ctracks.items():
--> 600         line = name + ',' + ','.join(['%.3f' %x for x in score])
    601         out += [line]
    602     out = sorted(out,key=lambda x: x.split(',')[0])

TypeError: float argument required, not str

Here's a view of the input of those annotations:

head -n 5 spring.custom.color.tracks.csv| cut -f 1-4 -d ","
nCount_RNA,15703,18128,41380
nFeature_RNA,4231,3411,6802
percent.mt,5.48302872062663,4.08208296557811,3.33977767037216
nCount_SCT,8331,7728,7891
nFeature_SCT,3526,2333,3155

head -n 5 spring.groupings.csv| cut -f 1-4 -d ","
orig.ident,F00431,F01380,F01391
Diagnosis,IPF,IPF,IPF
Sample_Name,TILD001,TILD028,VUILD64
Sample_Source,NTI,NTI,Vanderbilt
Status,ILD,ILD,ILD

Am I reading the annotations in the wrong format?
I was wondering if you had any version of the example notebook that would include annotations (continuous and categorical).
Thanks a lot for your help!

How to download vector graphs ?

Hello,
SPRING is so convenient for visualization. I want to know how to download the vector graph while I only found the "screenshot " button on SPRING interface and the png file is not clear enough.
Thanks for help !

Unable to see gene expression in SPRING viewer

Hi!

I followed the spring_example_pmbc_4k example as well as the cell labeling with SignacX using my own data, and was able to generate a SPRING plot that looks like what I would have expected based on other clustering packages. However, when I try to look at expression by gene, nothing shows up despite the expression scalebar changing depending on what gene is entered. Similarly, I do not get a list of enriched genes when I select a group of cells from the cell labels. For instance, this is the output for two genes in my dataset showing the different scale bars:

and this is the empty gene list for a group of selected cells:

Do you have any suggestions as to what to do from here?

How to visualize distinct 10x datasets in the same SPRING plot

Hi
I would like to generate a single SPRING plot from two or more 10X datasets and be able to

color them independently
select and highlight them separately from the side menu

Could you please share an example that I can run on my jupyter console?
(Apologies in advance if I have missed this information in the tutorials)

Thank you!

RuntimeError: Invalid DISPLAY variable

Hello,
In the last section "Save SPRING files" of pbmc4k, I have the error:
RuntimeError: Invalid DISPLAY variable

Then it works after I add following code before make_spring_subplot function:
import matplotlib.pyplot as plt
plt.switch_backend('agg')

Docker image

Hello there, SPRING Team!

I love your web-tool and I particurlaly have been using it a lot to show fellow non-bioinformaticians colleagues my results in an interactive and visual attractive way.

I would like, however, to also be able to run a SPRING docker image of your functioning web-tool at our local computing cluster. This because the current dataset I'm working with is about ~150k cells big, and I think it's too demanding for the web-tool, overloading both your servers and the web connection.

Is there anyway someone on the team might give me some support on building a docker image? It then could be made freely avaiable, bypassing many of the technical issues currently open (requests for more documentation,etc).

I'll be working on this and would very much appreciate some support.

Best regards,

Davi

Tutorial pbmc4k

Hello,
I am trying the SPRING new version.
I have some trouble with the tutorial pbmc 4K.

The preprocessing is OK but when I try to ,visualize I have a lot of issues...

So I run this in the SPRING_dev-master folder:
python -m CGIHTTPServer 8080

then:

(idem with springViewer_1_5) and I have this on the terminal:

127.0.0.1 - - [15/May/2018 15:19:20] code 404, message File not found
127.0.0.1 - - [15/May/2018 15:19:20] "GET /datasets/pbmc4k/graph_data.json HTTP/1.1" 404 -
127.0.0.1 - - [15/May/2018 15:19:20] code 404, message File not found
127.0.0.1 - - [15/May/2018 15:19:20] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [15/May/2018 15:19:21] code 404, message File not found
127.0.0.1 - - [15/May/2018 15:19:21] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [15/May/2018 15:20:40] "GET /springViewer_1_5_dev.html?datasets/pbmc4k/full HTTP/1.1" 200 -
127.0.0.1 - - [15/May/2018 15:20:40] code 404, message File not found
127.0.0.1 - - [15/May/2018 15:20:40] "GET /datasets/pbmc4k/full/mutability.txt HTTP/1.1" 404 -
127.0.0.1 - - [15/May/2018 15:20:40] code 404, message File not found
127.0.0.1 - - [15/May/2018 15:20:40] "GET /datasets/pbmc4k/selected_cells.txt HTTP/1.1" 404 -

FYI in pbmc4k folder there is :
/counts_norm.npz
/counts_norm_sparse_cells.hdf5
/counts_norm_sparse_genes.hdf5
/genes.txt
/total_counts.txt

and folder /full with
/categorical_coloring_data.json
/cell_filter.npy
/cell_filter.txt
/color_data_gene_sets.csv
/color_stats.json
/coordinates.txt
/edges.csv
/graph_data.json
/intermediates.npz
/run_info.json

What did I miss?

Thanks you for your help!

Camille

Missing Files?

Hi,

I recently clone the repo and try to run Spring locally, but I got bunch of 404 errors which showed me some files were missing, like text_annotation.js so on.

Is there any reason behind? or I just made any mistakes.