Comments (6)
karenfeng
commented on June 9, 2024
HTSlib is a C library; I'm not sure if it was the root of the problem. How do you have HTSlib installed?
from glow.
Hoeze
commented on June 9, 2024
conda install -c bioconda htslib
from glow.
karenfeng
commented on June 9, 2024
I'm having trouble replicating this. Can you share your environment via pip list/conda list, as well as your Spark settings?
from glow.
Hoeze
commented on June 9, 2024
Yes, sure. Unfortunately, it's a quite long package list...
Thanks for looking into it :)
# packages in environment at /opt/modules/i12g/anaconda/envs/florian3:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 0_gnu conda-forge
_py-xgboost-mutex 2.0 cpu_0 conda-forge
_tflow_select 2.1.0 gpu
absl-py 0.9.0 py37_0 conda-forge
adjusttext 0.7.3.1 py_0 conda-forge
aioeasywebdav 2.4.0 py37_1000 conda-forge
aiohttp 3.6.2 py37h516909a_0 conda-forge
alabaster 0.7.12 py_0 conda-forge
alembic 1.4.1 py_0 conda-forge
alsa-lib 1.1.5 h516909a_1002 conda-forge
anndata 0.7.1 py37_0 conda-forge
ansiwrap 0.8.4 py_0 conda-forge
appdirs 1.4.3 py_1 conda-forge
argcomplete 1.11.1 pypi_0 pypi
argh 0.26.2 pypi_0 pypi
arrow 0.15.5 py37_0 conda-forge
arrow-cpp 0.15.1 py37h38a5dc4_3 conda-forge
asciitree 0.3.3 py_2 conda-forge
asn1crypto 1.3.0 py37_0 conda-forge
astor 0.7.1 py_0 conda-forge
async-timeout 3.0.1 py_1000 conda-forge
attrs 19.3.0 py_0 conda-forge
autoconf 2.69 pl526h14c3975_9 conda-forge
automake 1.16.1 pl526_1003 conda-forge
babel 2.8.0 py_0 conda-forge
backcall 0.1.0 py_0 conda-forge
backports 1.0 py_2 conda-forge
backports.lzma 0.0.14 py37hf26bc8e_1 conda-forge
backports.tempfile 1.0 py_0 conda-forge
backports.weakref 1.0.post1 py37_1000 conda-forge
bcolz 1.2.1 py37hb3f55d8_1001 conda-forge
bcrypt 3.1.7 py37h516909a_0 conda-forge
beautifulsoup4 4.8.2 pypi_0 pypi
binaryornot 0.4.4 py_1 conda-forge
binutils_impl_linux-64 2.33.1 h53a641e_8 conda-forge
binutils_linux-64 2.33.1 h9595d00_16 conda-forge
black 19.10b0 py37_0 conda-forge
bleach 3.1.0 py_0 conda-forge
blosc 1.17.1 he1b5a44_0 conda-forge
bokeh 1.4.0 py37_0 conda-forge
boost-cpp 1.70.0 h8e57a91_2 conda-forge
boto3 1.11.10 py_0 conda-forge
botocore 1.14.10 py_0 conda-forge
brotli 1.0.7 he1b5a44_1000 conda-forge
bzip2 1.0.8 h516909a_2 conda-forge
c-ares 1.15.0 h516909a_1001 conda-forge
ca-certificates 2019.11.28 hecc5488_0 conda-forge
cachetools 3.1.1 py_0 conda-forge
cachey 0.1.1 py_0 conda-forge
cairo 1.16.0 hfb77d84_1002 conda-forge
certifi 2019.11.28 py37_0 conda-forge
cffi 1.13.2 py37h8022711_0 conda-forge
cftime 1.0.4.2 py37hc1659b7_0 conda-forge
chardet 3.0.4 py37_1003 conda-forge
click 7.0 py_0 conda-forge
cloudpickle 1.2.2 py_1 conda-forge
cmake 3.16.3 h28c56e5_0 conda-forge
colorama 0.4.3 pypi_0 pypi
colorcet 2.0.1 py_0 conda-forge
coloredlogs 10.0 py37_1000 conda-forge
colorlog 4.1.0 pypi_0 pypi
configargparse 0.13.0 py_1 conda-forge
configparser 3.7.3 py37_1 conda-forge
contextlib2 0.6.0.post1 py_0 conda-forge
cookiecutter 1.7.0 py_0 conda-forge
coreutils 8.31 h516909a_0 conda-forge
coverage 5.0.4 py37h8f50634_0 conda-forge
cryptography 2.8 py37h72c5cf5_1 conda-forge
cudatoolkit 10.0.130 0
cudnn 7.6.5 cuda10.0_0
cupti 10.0.130 0
curl 7.65.3 hf8cf82a_0 conda-forge
cycler 0.10.0 py_2 conda-forge
cython 0.29.14 py37he1b5a44_0 conda-forge
cytoolz 0.10.1 py37h516909a_0 conda-forge
cyvcf2 0.11.6 pypi_0 pypi
dask 2.10.1 py_0 conda-forge
dask-core 2.10.1 py_0 conda-forge
dask-glm 0.2.0 py_1 conda-forge
dask-ml 1.2.0 py_0 conda-forge
dask-xgboost 0.1.10 py_0 conda-forge
databricks-cli 0.9.1 py_0 conda-forge
datashader 0.10.0 py_0 conda-forge
datashape 0.5.4 py_1 conda-forge
datrie 0.8 py37h516909a_0 conda-forge
dbus 1.13.6 he372182_0 conda-forge
decorator 4.4.1 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
deprecation 2.0.7 pypi_0 pypi
descartes 1.1.0 py_4 conda-forge
distributed 2.10.0 py_0 conda-forge
docker-py 4.2.0 py37_0 conda-forge
docker-pycreds 0.4.0 py_0 conda-forge
docutils 0.15.2 py37_0 conda-forge
double-conversion 3.1.5 he1b5a44_2 conda-forge
dropbox 9.4.0 py_0 conda-forge
entrypoints 0.3 py37_1000 conda-forge
et_xmlfile 1.0.1 py_1001 conda-forge
expat 2.2.9 he1b5a44_2 conda-forge
fasteners 0.14.1 py_3 conda-forge
fftw 3.3.8 nompi_h7f3a6c3_1110 conda-forge
filechunkio 1.8 py_2 conda-forge
filelock 3.0.12 pypi_0 pypi
flake8 3.7.9 py37hc8dfbb8_1 conda-forge
flask 1.1.1 py_1 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 2.001 hab24e00_0 conda-forge
font-ttf-source-code-pro 2.030 hab24e00_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.13.1 h86ecdb6_1001 conda-forge
fonts-conda-forge 1 0 conda-forge
freetype 2.10.0 he983fc9_1 conda-forge
fribidi 1.0.5 h516909a_1002 conda-forge
fsspec 0.6.2 py_0 conda-forge
ftputil 3.4 py_0 conda-forge
funcsigs 1.0.2 pypi_0 pypi
future 0.18.2 py37_0 conda-forge
gast 0.2.2 py_0 conda-forge
gcc_impl_linux-64 7.3.0 hd420e75_5 conda-forge
gcc_linux-64 7.3.0 h553295d_16 conda-forge
gdk-pixbuf 2.38.2 h3f25603_0 conda-forge
gettext 0.19.8.1 hc5be6a0_1002 conda-forge
gffutils 0.10.1 pypi_0 pypi
gflags 2.2.2 he1b5a44_1002 conda-forge
ghostscript 9.22 hf484d3e_1001 conda-forge
giflib 5.2.1 h516909a_1 conda-forge
gitdb2 2.0.6 py_0 conda-forge
gitpython 3.0.5 py_0 conda-forge
glib 2.58.3 py37h6f030ca_1002 conda-forge
glog 0.4.0 he1b5a44_1 conda-forge
glow-py 0.2.0 pypi_0 pypi
gmp 6.2.0 he1b5a44_1 conda-forge
gobject-introspection 1.58.2 py37h5503ade_1002 conda-forge
google 2.0.3 pypi_0 pypi
google-api-core 1.16.0 py37_1 conda-forge
google-auth 1.11.0 py_0 conda-forge
google-cloud-core 1.3.0 py_0 conda-forge
google-cloud-storage 1.24.1 py37_1 conda-forge
google-pasta 0.1.8 py_0 conda-forge
google-resumable-media 0.5.0 py_1 conda-forge
googleapis-common-protos 1.51.0 py37_1 conda-forge
gorilla 0.3.0 py_0 conda-forge
gradle 6.1.1 h4d321a9_0 conda-forge
graphite2 1.3.13 hf484d3e_1000 conda-forge
graphviz 2.42.3 h0511662_0 conda-forge
grpc-cpp 1.25.0 h18db393_0 conda-forge
grpcio 1.23.0 py37he9ae1f9_0 conda-forge
gst-plugins-base 1.14.5 h0935bb2_1 conda-forge
gstreamer 1.14.5 h36ae1b5_1 conda-forge
gtfparse 1.2.0 py_0 bioconda
gunicorn 20.0.4 py37hc8dfbb8_1 conda-forge
gxx_impl_linux-64 7.3.0 hdf63c60_5 conda-forge
gxx_linux-64 7.3.0 h553295d_16 conda-forge
h5netcdf 0.7.4 py_0 conda-forge
h5py 2.10.0 nompi_py37h513d04c_101 conda-forge
harfbuzz 2.4.0 h9f30f68_3 conda-forge
hdf4 4.2.13 hf30be14_1003 conda-forge
hdf5 1.10.5 nompi_h3c11f04_1104 conda-forge
heapdict 1.0.1 py_0 conda-forge
holoviews 1.12.7 py_0 conda-forge
htslib 1.10.2 h78d89cc_0 bioconda
humanfriendly 4.18 py37_1 conda-forge
hvplot 0.5.2 py_0 conda-forge
hyperopt 0.2.2 py_0 conda-forge
icu 64.2 he1b5a44_1 conda-forge
idna 2.8 py37_1000 conda-forge
igraph 0.7.1 h9e3b1fc_1007 conda-forge
imageio 2.6.1 py37_0 conda-forge
imagemagick 7.0.9_20 pl526ha9fe49d_0 conda-forge
imagesize 1.2.0 py_0 conda-forge
importlib-metadata 0.19 py37_0 conda-forge
importlib_metadata 1.5.0 py37_0 conda-forge
inflect 4.1.0 py37_0 conda-forge
ipykernel 5.1.4 py37h5ca1d4c_0 conda-forge
ipython 7.12.0 py37h5ca1d4c_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 7.5.1 py_0 conda-forge
itsdangerous 1.1.0 py_0 conda-forge
jaraco.itertools 5.0.0 py_0 conda-forge
jbig 2.1 h14c3975_2001 conda-forge
jdcal 1.4.1 py_0 conda-forge
jedi 0.16.0 py37_0 conda-forge
jinja2 2.11.1 py_0 conda-forge
jinja2-time 0.2.0 py_2 conda-forge
jmespath 0.9.4 py_0 conda-forge
joblib 0.14.1 py_0 conda-forge
jpeg 9c h14c3975_1001 conda-forge
jsonschema 3.2.0 py37_0 conda-forge
jupyter_client 5.3.4 py37_1 conda-forge
jupyter_core 4.6.1 py37_0 conda-forge
keras 2.3.1 py37_0 conda-forge
keras-applications 1.0.8 py_1 conda-forge
keras-preprocessing 1.1.0 py_0 conda-forge
kipoi 0.6.25 pypi_0 pypi
kipoi-conda 0.2.2 pypi_0 pypi
kipoi-utils 0.3.8 pypi_0 pypi
kipoiseq 0.3.2 pypi_0 pypi
kiwisolver 1.1.0 py37hc9558a2_0 conda-forge
koalas 0.30.0 dev_0 <develop>
krb5 1.16.4 h2fd8d38_0 conda-forge
lcms2 2.9 hbd6801e_1 conda-forge
ld_impl_linux-64 2.33.1 h53a641e_8 conda-forge
libblas 3.8.0 14_openblas conda-forge
libcblas 3.8.0 14_openblas conda-forge
libclang 9.0.1 default_hde54327_0 conda-forge
libcroco 0.6.13 h8d621e5_0 conda-forge
libcurl 7.65.3 hda55be3_0 conda-forge
libdeflate 1.3 h516909a_0 conda-forge
libedit 3.1.20170329 hf8c457e_1001 conda-forge
libevent 2.1.10 h72c5cf5_0 conda-forge
libffi 3.2.1 he1b5a44_1006 conda-forge
libgcc-ng 9.2.0 h24d8f2e_2 conda-forge
libgfortran-ng 7.3.0 hdf63c60_5 conda-forge
libgomp 9.2.0 h24d8f2e_2 conda-forge
libgpuarray 0.7.6 h14c3975_1003 conda-forge
libiconv 1.15 h516909a_1005 conda-forge
liblapack 3.8.0 14_openblas conda-forge
libllvm8 8.0.1 hc9558a2_0 conda-forge
libllvm9 9.0.1 hc9558a2_0 conda-forge
libnetcdf 4.7.3 nompi_h9f9fd6a_101 conda-forge
libopenblas 0.3.7 h5ec1e0e_6 conda-forge
libpng 1.6.37 hed695b0_0 conda-forge
libprotobuf 3.8.0 h8b12597_0 conda-forge
librsvg 2.46.4 h33a7fed_0 conda-forge
libsodium 1.0.17 h516909a_0 conda-forge
libssh2 1.8.2 h22169c7_2 conda-forge
libstdcxx-ng 9.2.0 hdf63c60_2 conda-forge
libtiff 4.1.0 hc3755c2_3 conda-forge
libtool 2.4.6 h14c3975_1002 conda-forge
libuuid 2.32.1 h14c3975_1000 conda-forge
libuv 1.34.0 h516909a_0 conda-forge
libwebp 1.0.2 h56121f0_5 conda-forge
libxcb 1.13 h14c3975_1002 conda-forge
libxgboost 0.90 he1b5a44_4 conda-forge
libxkbcommon 0.9.1 hebb1f50_0 conda-forge
libxml2 2.9.10 hee79883_0 conda-forge
llvmlite 0.31.0 py37h8b12597_0 conda-forge
locket 0.2.0 py_2 conda-forge
louvain 0.6.1 py37he1b5a44_2 conda-forge
lz4-c 1.8.3 he1b5a44_1001 conda-forge
lzo 2.10 h14c3975_1000 conda-forge
m4 1.4.18 h14c3975_1001 conda-forge
mako 1.1.0 py_0 conda-forge
markdown 3.1.1 py_0 conda-forge
markupsafe 1.1.1 py37h516909a_0 conda-forge
matplotlib 3.1.3 py37_0 conda-forge
matplotlib-base 3.1.3 py37h250f245_0 conda-forge
maven 3.6.0 0 conda-forge
mccabe 0.6.1 py_1 conda-forge
mistune 0.8.4 py37h516909a_1000 conda-forge
mizani 0.6.0 py_0 conda-forge
mlflow 1.7.1 py37hc8dfbb8_0 conda-forge
mock 3.0.5 py37_0 conda-forge
monotonic 1.5 py_0 conda-forge
more-itertools 8.2.0 py_0 conda-forge
msgpack-python 0.6.2 py37hc9558a2_0 conda-forge
multidict 4.7.4 py37h516909a_0 conda-forge
multipledispatch 0.6.0 py_0 conda-forge
mypy 0.770 py_0 conda-forge
mypy_extensions 0.4.3 py37hc8dfbb8_1 conda-forge
natsort 7.0.1 py_0 conda-forge
nbconvert 5.6.1 py37_0 conda-forge
nbformat 5.0.4 py_0 conda-forge
nbsphinx 0.5.1 py_0 conda-forge
ncls 0.0.51 py37h516909a_0 bioconda
ncurses 6.1 hf484d3e_1002 conda-forge
netcdf4 1.5.3 nompi_py37hd35fb8e_102 conda-forge
networkx 2.4 py_0 conda-forge
notebook 6.0.3 py37_0 conda-forge
nspr 4.25 he1b5a44_0 conda-forge
nss 3.47 he751ad9_0 conda-forge
numba 0.48.0 py37hb3f55d8_0 conda-forge
numcodecs 0.6.4 py37he1b5a44_0 conda-forge
numexpr 2.7.1 py37hb3f55d8_0 conda-forge
numpy 1.17.5 py37h95a1406_0 conda-forge
numpydoc 0.8.0 py_1 conda-forge
olefile 0.46 py_0 conda-forge
openjdk 8.0.192 h516909a_1004 conda-forge
openjpeg 2.3.1 h981e76c_3 conda-forge
openpyxl 3.0.3 py_0 conda-forge
openssl 1.1.1d h516909a_0 conda-forge
opt_einsum 3.1.0 py_0 conda-forge
packaging 20.1 py_0 conda-forge
palettable 3.3.0 py_0 conda-forge
pandas 1.0.0 py37hb3f55d8_0 conda-forge
pandoc 2.9.1.1 0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
pango 1.42.4 ha030887_1 conda-forge
papermill 1.2.1 py37_0 conda-forge
param 1.9.3 py_0 conda-forge
paramiko 2.7.1 py37_0 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.6.1 py_0 conda-forge
partd 1.1.0 py_0 conda-forge
pathlib2 2.3.5 py37_0 conda-forge
pathspec 0.7.0 py_0 conda-forge
patsy 0.5.1 py_0 conda-forge
pcre 8.43 he1b5a44_0 conda-forge
perl 5.26.2 h516909a_1006 conda-forge
pexpect 4.8.0 py37_0 conda-forge
pickleshare 0.7.5 py37_1000 conda-forge
pillow 7.0.0 py37hefe7db6_0 conda-forge
pip 20.0.2 py37_1 conda-forge
pixman 0.38.0 h516909a_1003 conda-forge
pkg-config 0.29.2 h516909a_1006 conda-forge
plotly 4.5.0 py_0 conda-forge
plotly-orca 1.2.1 1 plotly
plotnine 0.6.0 py_0 conda-forge
pluggy 0.12.0 py_0 conda-forge
pomegranate 0.12.0 py37hc9558a2_0 conda-forge
poyo 0.5.0 py_0 conda-forge
prettytable 0.7.2 py_3 conda-forge
prometheus_client 0.7.1 py_0 conda-forge
prometheus_flask_exporter 0.13.0 py_0 conda-forge
prompt_toolkit 3.0.3 py_0 conda-forge
protobuf 3.8.0 py37he1b5a44_2 conda-forge
psutil 5.6.7 py37h516909a_0 conda-forge
pthread-stubs 0.4 h14c3975_1001 conda-forge
ptyprocess 0.6.0 py_1001 conda-forge
py 1.8.1 py_0 conda-forge
py-spy 0.3.3 pypi_0 pypi
py-xgboost 0.90 py37_4 conda-forge
py4j 0.10.7 py_1 conda-forge
pyarrow 0.15.1 py37h8b68381_1 conda-forge
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.7 py_0 conda-forge
pybind11 2.4.3 py37hc9558a2_1 conda-forge
pycairo 1.19.0 py37h438ddbb_0 conda-forge
pycodestyle 2.5.0 py_0 conda-forge
pycparser 2.19 py37_1 conda-forge
pyct 0.4.6 py_0 conda-forge
pyct-core 0.4.6 py_0 conda-forge
pyfaidx 0.5.8 pypi_0 pypi
pyflakes 2.1.1 py_0 conda-forge
pygments 2.5.2 py_0 conda-forge
pygpu 0.7.6 py37hc1659b7_1000 conda-forge
pygraphviz 1.3 py37h14c3975_1
pymongo 3.10.1 py37he1b5a44_0 conda-forge
pynacl 1.3.0 py37h516909a_1001 conda-forge
pyopenssl 19.1.0 py37_0 conda-forge
pypandoc 1.4 py_1 conda-forge
pyparsing 2.4.6 py_0 conda-forge
pyqt 5.12.3 py37hcca6a23_1 conda-forge
pyqt5-sip 4.19.18 pypi_0 pypi
pyqtwebengine 5.12.1 pypi_0 pypi
pyranges 0.0.71 py_0 bioconda
pyrle 0.0.29 py37h516909a_0 bioconda
pyrsistent 0.15.7 py37h516909a_0 conda-forge
pysam 0.15.3 py37hbcae180_3 bioconda
pysftp 0.2.9 py_1 conda-forge
pysocks 1.7.1 py37_0 conda-forge
pyspark 2.4.5 py_0 conda-forge
pytables 3.6.1 py37h9f153d1_1 conda-forge
pytest 5.3.5 py37_0 conda-forge
pytest-cov 2.8.1 py_0 conda-forge
python 3.7.6 h357f687_2 conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python-editor 1.0.4 py_0 conda-forge
python-igraph 0.7.1.post7 py37h516909a_0 conda-forge
python-irodsclient 0.8.2 py_0 conda-forge
python_abi 3.7 1_cp37m conda-forge
pytz 2019.3 py_0 conda-forge
pyviz_comms 0.7.3 py_0 conda-forge
pywavelets 1.1.1 py37hc1659b7_0 conda-forge
pyyaml 5.3 py37h516909a_0 conda-forge
pyzmq 18.1.1 py37h1768529_0 conda-forge
qt 5.12.5 hd8c4c69_1 conda-forge
querystring_parser 1.2.4 py_0 conda-forge
ratelimiter 1.2.0 py37_1000 conda-forge
ray 0.8.2 pypi_0 pypi
re2 2020.01.01 he1b5a44_0 conda-forge
readline 8.0 hf8c457e_0 conda-forge
redis 3.4.1 pypi_0 pypi
regex 2020.2.20 py37h8f50634_1 conda-forge
related 0.7.2 pypi_0 pypi
requests 2.22.0 py37_1 conda-forge
retrying 1.3.3 py_2 conda-forge
rhash 1.3.6 h14c3975_1001 conda-forge
rsa 4.0 py_0 conda-forge
s3transfer 0.3.2 py37_0 conda-forge
scanpy 1.4.4.post1 py_0 bioconda
scikit-allel 1.2.1 py37hb3f55d8_1 conda-forge
scikit-image 0.16.2 py37hb3f55d8_0 conda-forge
scikit-learn 0.22.1 py37hcdab131_1 conda-forge
scipy 1.4.1 py37h921218d_0 conda-forge
seaborn 0.10.0 py_0 conda-forge
send2trash 1.5.0 py_0 conda-forge
setuptools 45.1.0 py37_0 conda-forge
simplejson 3.17.0 py37h516909a_0 conda-forge
six 1.14.0 py37_0 conda-forge
smmap2 2.0.5 py_0 conda-forge
snakemake 5.10.0 0 bioconda
snakemake-minimal 5.10.0 py_0 bioconda
snappy 1.1.8 he1b5a44_1 conda-forge
snowballstemmer 2.0.0 py_0 conda-forge
sorted_nearest 0.0.31 py37hc1659b7_0 bioconda
sortedcontainers 2.1.0 py_0 conda-forge
soupsieve 2.0 pypi_0 pypi
sphinx 2.0.1 py_0 conda-forge
sphinxcontrib-applehelp 1.0.2 py_0 conda-forge
sphinxcontrib-devhelp 1.0.2 py_0 conda-forge
sphinxcontrib-htmlhelp 1.0.3 py_0 conda-forge
sphinxcontrib-jsmath 1.0.1 py_0 conda-forge
sphinxcontrib-qthelp 1.0.3 py_0 conda-forge
sphinxcontrib-serializinghtml 1.1.4 py_0 conda-forge
sqlalchemy 1.3.13 py37h516909a_0 conda-forge
sqlite 3.30.1 hcee41ef_0 conda-forge
sqlparse 0.3.1 py_0 conda-forge
statsmodels 0.11.0 py37h516909a_0 conda-forge
tabulate 0.8.6 py_0 conda-forge
tbb 2018.0.5 h2d50403_0 conda-forge
tblib 1.6.0 py_0 conda-forge
tenacity 6.0.0 py37_0 conda-forge
tensorboard 1.15.0 py37_0 conda-forge
tensorboardx 2.0 pypi_0 pypi
tensorflow 1.15.0 gpu_py37h0f0df58_0
tensorflow-base 1.15.0 gpu_py37h9dcbed7_0
tensorflow-estimator 1.15.1 pyh2649769_0
tensorflow-gpu 1.15.0 h0d30ee6_0
termcolor 1.1.0 py_2 conda-forge
terminado 0.8.3 py37_0 conda-forge
testpath 0.4.4 py_0 conda-forge
textwrap3 0.9.2 py_0 conda-forge
theano 1.0.4 py37he1b5a44_1001 conda-forge
threadpoolctl 2.0.0 pypi_0 pypi
thrift-cpp 0.12.0 hf3afdfd_1004 conda-forge
tiledb 1.7.4 hcde45ca_0 conda-forge
tiledbvcf 0.1.0 pypi_0 pypi
tinydb 3.15.2 pypi_0 pypi
tk 8.6.10 hed695b0_0 conda-forge
toml 0.10.0 py_0 conda-forge
toolz 0.10.0 py_0 conda-forge
toposort 1.5 py_3 conda-forge
tornado 6.0.3 py37h516909a_0 conda-forge
tqdm 4.42.0 py_0 conda-forge
traitlets 4.3.3 py37_0 conda-forge
typed-ast 1.4.1 py37h516909a_0 conda-forge
typeguard 2.5.0 pypi_0 pypi
typing_extensions 3.7.4.1 py37hc8dfbb8_1 conda-forge
umap-learn 0.3.10 py37_1 conda-forge
uriparser 0.9.3 he1b5a44_1 conda-forge
urllib3 1.25.7 py37_0 conda-forge
wcwidth 0.1.8 py_0 conda-forge
webencodings 0.5.1 py_1 conda-forge
websocket-client 0.57.0 py37hc8dfbb8_1 conda-forge
werkzeug 0.16.1 py_0 conda-forge
wheel 0.34.2 py37_0 conda-forge
whichcraft 0.6.1 py_0 conda-forge
widgetsnbextension 3.5.1 py37_0 conda-forge
wrapt 1.11.2 py37h516909a_0 conda-forge
xarray 0.15.0 py_0 conda-forge
xgboost 0.90 py37he1b5a44_4 conda-forge
xlrd 1.2.0 py_0 conda-forge
xmlrunner 1.7.7 py_0 conda-forge
xorg-fixesproto 5.0 h14c3975_1002 conda-forge
xorg-inputproto 2.3.2 h14c3975_1002 conda-forge
xorg-kbproto 1.0.7 h14c3975_1002 conda-forge
xorg-libice 1.0.10 h516909a_0 conda-forge
xorg-libsm 1.2.3 h84519dc_1000 conda-forge
xorg-libx11 1.6.9 h516909a_0 conda-forge
xorg-libxau 1.0.9 h14c3975_0 conda-forge
xorg-libxdmcp 1.1.3 h516909a_0 conda-forge
xorg-libxext 1.3.4 h516909a_0 conda-forge
xorg-libxfixes 5.0.3 h516909a_1004 conda-forge
xorg-libxi 1.7.10 h516909a_0 conda-forge
xorg-libxpm 3.5.13 h516909a_0 conda-forge
xorg-libxrender 0.9.10 h516909a_1002 conda-forge
xorg-libxt 1.1.5 h516909a_1003 conda-forge
xorg-libxtst 1.2.3 h516909a_1002 conda-forge
xorg-recordproto 1.14.2 h516909a_1002 conda-forge
xorg-renderproto 0.11.1 h14c3975_1002 conda-forge
xorg-xextproto 7.3.0 h14c3975_1002 conda-forge
xorg-xproto 7.0.31 h14c3975_1007 conda-forge
xz 5.2.4 h14c3975_1001 conda-forge
yaml 0.2.2 h516909a_1 conda-forge
yarl 1.3.0 py37h516909a_1000 conda-forge
zarr 2.4.0 py_0 conda-forge
zeromq 4.3.2 he1b5a44_2 conda-forge
zict 1.0.0 py_0 conda-forge
zipp 2.1.0 py_0 conda-forge
zlib 1.2.11 h516909a_1006 conda-forge
zstd 1.4.4 h3b9ef0a_1 conda-forge
My spark settings are the following:
PYSPARK_SUBMIT_ARGS='--packages io.projectglow:glow_2.11:0.3.0,org.bdgenomics.adam:adam-assembly-spark2_2.12:0.29.0,io.delta:delta-core_2.11:0.5.0 --driver-memory 367001600k --executor-memory 367001600k pyspark-shell'
spark = (
SparkSession.builder
.appName('abc')
.config("spark.local.dir", os.environ.get("TMP"))
.config("spark.sql.execution.arrow.enabled", "true")
.config("spark.sql.shuffle.partitions", "2001")
.config("spark.databricks.io.cache.enabled", "true")
.config("spark.databricks.io.cache.maxDiskUsage", "500G")
.config("spark.driver.maxResultSize", "%dk" % MEM)
.getOrCreate()
)
glow.register(spark)
sparkfrom glow.
karenfeng
commented on June 9, 2024
Thanks for the detail @Hoeze!
The issue most likely comes down to a dependency conflict in Maven artifact resolution, as you're also pulling in ADAM. Both Glow and ADAM depend on the HTSJDK, but on different versions (2.20.3 and 2.19.0 respectively). Spark can only pull in one version of the HTSJDK, and unluckily you seem to have pulled in 2.19.0. To avoid this, you can directly depend on the HTSJDK by adding com.github.samtools:htsjdk:2.20.3 to the --packages list; this should override the transitive dependency.
FYI, it'd also be safer if you depended on org.bdgenomics.adam:adam-assembly-spark2_2.11:0.29.0 rather than the Scala 2.12 version.
from glow.
Hoeze
commented on June 9, 2024
Thank you @karenfeng, this makes more sense and it is working now.
Actually, htslib has no effect after changing it back to 1.9 now...
from glow.
Related Issues (20)
- VCF Infinity/NaN values are not handled according to VCF spec
- PySpark 3.3 support HOT 8
- Load glow mave package from Python HOT 5
- spark.read.format("vcf").load() fails for vcf.bgz files HOT 3
- Cannot write INFO fields with LongType to VCF HOT 2
- Python tests fail with KeyError: '_glow_regression_values' HOT 1
- Logistic regression ValueError: Null fit failed! HOT 2
- AnalysisException: Column 'num_workers' does not exist. HOT 1
- pipe transformer support for file input/output for command line apps HOT 2
- Interaction Tests with GLOW HOT 1
- Spark version upgrade 3.4, 3.3 HOT 5
- Cannot install+use the python glow package any more HOT 2
- Regular updates to work with cloud providers HOT 1
- Problem importing Glow HOT 3
- glow.normalize_variant fails with NullPointerException in NormalizeVariantExpr.scala:55 HOT 4
- Unexpected behaviour when using the SplitMultiallelics Transformer with unbounded INFO fields HOT 2
- import glow throws error: module 'numpy' has no attribute 'unicode'. Did you mean: 'unicode_'? HOT 2
- Feature Request: add regenie's gene-based testing
- Comparision of performance of regenie WGR and glow WGR HOT 2
- Pandas should be pinned <2.0.0 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from glow.