euxhenh / cellar Goto Github PK

Hello.

Great Tool!
I could run Cellar in one of my samples, 10x data, worked great.
I still don't know the better way to upload the whole batch as one file to be analyzed.
Can you give a help with it?

At the moment, uploading a new dataset to the DATA_PATH through command line (not through cellar UI) doesn't change the drop-down content in the dataset selector. So, every time I upload a dataset to the server, cellar needs to be restarted. One way to resolve it would be to update the drop-down content on every click to Load Data. Do you think it's a reasonable solution? I could make a PR to fix it.

Hi,
It's an amazing job,and I want to create a mirror on my website with my example data.

May I ask how can I create a site mirror with default data?

It's like when you go to a website and show the content after click on "LOAD" to display .

Hello,
whether it is running locally with docker or running on the web, it is found that the website is not loaded correctly, and some styles are not displayed. What do I need to do to use it normally?

Thank you for developing this amazing tool!
I'm trying to deploy it for our datasets, and simply using the docker throws an error on dataset upload:

docker run --rm -p 8050:8050 -v /home/azureuser/data:/home/nonroot/cellar/data euxhen/cellar

ERROR: Exception on /_dash-update-component [POST]
Traceback (most recent call last):
File "/home/nonroot/.conda/envs/cellar/lib/python3.8/site-packages/flask/app.py", line 2077, in wsgi_app
response = self.full_dispatch_request()
File "/home/nonroot/.conda/envs/cellar/lib/python3.8/site-packages/flask/app.py", line 1525, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/nonroot/.conda/envs/cellar/lib/python3.8/site-packages/flask/app.py", line 1523, in full_dispatch_request
rv = self.dispatch_request()
File "/home/nonroot/.conda/envs/cellar/lib/python3.8/site-packages/flask/app.py", line 1509, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/home/nonroot/.conda/envs/cellar/lib/python3.8/site-packages/dash/dash.py", line 1383, in dispatch
response.set_data(func(*args, outputs_list=outputs_list))
File "/home/nonroot/.conda/envs/cellar/lib/python3.8/site-packages/dash/_callback.py", line 151, in add_context
output_value = func(*func_args, **func_kwargs) # %% callback invoked %%
File "/home/nonroot/cellar/controller/data_uploader.py", line 46, in upload_data
with open(os.path.join(DATA_PATH, 'uploaded', filename), "wb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'data/uploaded/processed.h5ad'

If I log in to the container (docker run --rm -p 8050:8050 -v /home/azureuser/data:/home/nonroot/cellar/data -it --entrypoint /bin/bash euxhen/cellar) and create a subfolder manually (mkdir data/uploaded), it starts working fine.

UPDATE: I just realized that the problem was in the -v /home/azureuser/data:/home/nonroot/cellar/data part. It overrides the whole data folder, which is incorrect. Instead, the data folder should be mounted to /home/nonroot/cellar/data/server. So the solution would be to update the documentation, describing that either the mounted directory should have subfolders server and uploaded, or that it should be mounted to /home/nonroot/cellar/data/server.

UPDATE 2: Looking at other content of the data folder, it looks like normal functioning of Cellar requires the following files:

data/ASCT-B_tables
data/geneIdsNamesHuman.csv
data/geneIdsNamesMouse.csv

So, full mounting of the data folder should not be recommended, I guess?

Hi, and thanks for sharing your work!

We're running this at DUG Technology on behalf of a client and experiencing a crash when it runs on a machine that doesn't support AVX2. The failure mode looks something like this:

$ python -v main.py 
...
# code object from '/usr/local/conda/cellar/lib/python3.8/site-packages/faiss/__pycache__/swigfaiss.cpython-38.pyc'
# extension module 'faiss._swigfaiss' loaded from '/usr/local/conda/cellar/lib/python3.8/site-packages/faiss/_swigfaiss.so'
# extension module 'faiss._swigfaiss' executed from '/usr/local/conda/cellar/lib/python3.8/site-packages/faiss/_swigfaiss.so'
import 'faiss._swigfaiss' # <_frozen_importlib_external.ExtensionFileLoader object at 0x2b5e21c82b50>
import 'faiss.swigfaiss' # <_frozen_importlib_external.SourceFileLoader object at 0x2b5e21b2f970>
import 'faiss.loader' # <_frozen_importlib_external.SourceFileLoader object at 0x2b5e2151dd00>
import 'faiss' # <_frozen_importlib_external.SourceFileLoader object at 0x2b5e2150e8b0>
# /usr/local/conda/cellar/lib/python3.8/site-packages/faiss/__pycache__/swigfaiss_avx2.cpython-38.pyc matches /usr/local/conda/cellar/lib/python3.8/site-packages/faiss/swigfaiss_avx2.py
# code object from '/usr/local/conda/cellar/lib/python3.8/site-packages/faiss/__pycache__/swigfaiss_avx2.cpython-38.pyc'
Illegal instruction

Upon closer investigation of the faiss package it does include non-AVX2 python bindings, and we tracked the problem down to this line (introduced in 8226fe1):

controller/cellar/core/_cluster.py:5:from faiss.swigfaiss_avx2 import compute_PQ_dis_tables_dsub2

Which pulls in the AVX2 implementation regardless of what CPU type cellar is actually running on. faiss actually imports all the symbols from swigfaiss into the top-level namespace (see faiss/loader.py), so changing cluster.py to the following is enough to fix the problem:

from faiss import compute_PQ_dis_tables_dsub2

and this will still use AVX2 when available.

While using your public web-server, it looks like the uploaded datasets are not persistent. However, if I run an instance myself, they're simply stored in the uploaded sub-folder. Consequently, every user sees everything anyone uploads to the server. And in case of a name conflict, an older dataset gets overwritten without any warning. Is there any way to make these uploaded datasets temporary and user-specific?

To my understanding, currently there is no way to color the dataset by anything but cluster id. Having a color column selector would add a lot to usability. For example, adding a drop-down to the left of "Change Color Palette". Seems like a very easy fix. Do you plan adding such functionality? If it would help, I can submit a PR for that.