Giter Club home page Giter Club logo

htslib-plugins's Introduction

Plugins for use with HTSlib.

Built plugins need to be on HTSlib's plugin search path so that HTSlib and programs using it can find them. This can be arranged in either of the following ways:

  • Copy them to HTSlib's built-in search path, which typically includes /usr/local/libexec/htslib and similar directories. (If building HTSlib yourself, use ./configure --with-plugin-path=…:DIR:… to include directories of your choice in the path.)

  • Alternatively, set the HTS_PATH environment variable to include the directory containing the built plugins.

EGA-style encrypted (.cip) files

The hfile_cip plugin provides access to files encrypted with the European Genome-Phenome Archive's AES-CTR scheme, which usually have the extension .cip. The en-/decryption key is taken from the $HTS_CIP_KEY environment variable.

iRODS

The hfile_irods plugin provides access to remote data stored in iRODS. It can be built for iRODS 3.x, 4.1.x, or 4.2 onwards, and the resulting plugins can be renamed with version numbers so that they can be installed alongside each other.

When built against iRODS 4.1.x, the plugin is incompatible with HTSlib 1.3.1 and earlier as it needs to be loaded with RTLD_GLOBAL. The hfile_irods_wrapper plugin can be installed in the same directory as hfile_irods to work around this problem and enable the iRODS plugin to be used with these earlier versions of HTSlib.

Memory-mapped local files

The hfile_mmap plugin provides access to local files via mmap(2).

htslib-plugins's People

Contributors

jmarshall avatar jrandall avatar pd3 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

wtsi-hgi liyaoyao0324 tneymanov tiledb-inc dkj aerhufker whitwham

htslib-plugins's Issues

irods init_client_api_table deprecation

init_client_api_table appears to be deprecated and replaced by load_client_api_plugins.

index 092c575..0b4e05a 100644
--- a/hfile_irods.c
+++ b/hfile_irods.c
@@ -56,9 +56,9 @@ DEALINGS IN THE SOFTWARE.  */
 #define PRIORITY (10 * IRODS_VERSION_MAJOR) + IRODS_VERSION_MINOR
 #else
 #define PRIORITY 30
-// For iRODS 3.x, there is no init_client_api_table() to call and clientLogin()
+// For iRODS 3.x, there is no load_client_api_plugins() to call and clientLogin()
 // has fewer parameters.  Define wrappers so the 4.x-style code below compiles.
-static void init_client_api_table() { }
+static void load_client_api_plugins() { }
 #define clientLogin(conn, x, y) (clientLogin((conn)))
 #endif

@@ -135,7 +135,7 @@ static int irods_init()
     // state (by default, termination; or as already set by our caller).
     pipehandler_ret = sigaction(SIGPIPE, NULL, &pipehandler);

-    init_client_api_table();
+    load_client_api_plugins();
     irods.conn = rcConnect(irods.env.rodsHost, irods.env.rodsPort,
                            irods.env.rodsUserName, irods.env.rodsZone,
                            NO_RECONN, &err);```

undefined symbol: hfile_plugin_init for hfile_irods.so in 1.11

iRODS support in Dockerfile below works for the version 1.10.2, but not for 1.11.

ARG htsversion=1.10.2
ARG samversion=1.10
ARG bcfversion=1.10.2

Specifically, when loading an irods file with

[D::init_add_plugin] Loaded "knetfile"
[D::init_add_plugin] Loaded "mem"
[D::init_add_plugin] Loaded "crypt4gh-needed"
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_cip.so"
[M::hfile_crypt4gh.init] version 015c142
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_crypt4gh.so"
[M::hfile_gcs.init] version 1.11
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_gcs.so"
[W::load_plugin] can't load plugin "/usr/local/libexec/htslib/hfile_irods.so": /usr/local/libexec/htslib/hfile_irods.so: undefined symbol: hfile_plugin_init
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_libcurl.so"
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_mmap.so"
[M::hfile_s3.init] version 1.11
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_s3.so"
[M::hfile_s3_write.init] version 1.11
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_s3_write.so"
[W::hts_path_itr] can't scan directory "/usr/libexec/htslib": No such file or directory
[E::hts_open_format] Failed to open file "irods:/seq/pacbio/r64125e_20210218_150723/1_A01/demultiplex.bc1016_BAK8B_OA--bc1016_BAK8B_OA.bam" : Protocol not supported

My suspicion is perhaps samtools/htslib@9dfbf8d7 from @daviesrob and then some change now required here in htslib-plugins??

The above logging looks like this with 1.10.2:

[D::init_add_plugin] Loaded "knetfile"
[D::init_add_plugin] Loaded "mem"
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_cip.so"
[M::hfile_crypt4gh.init] version 015c142
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_crypt4gh.so"
[M::hfile_gcs.init] version 1.10.2
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_gcs.so"
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_irods.so"
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_libcurl.so"
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_mmap.so"
[M::hfile_s3.init] version 1.10.2
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_s3.so"
[M::hfile_s3_write.init] version 1.10.2
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_s3_write.so"
[W::hts_path_itr] can't scan directory "/usr/libexec/htslib": No such file or directory
[M::hfile_irods.init] version d3ad3cc-dirty built against rods4.2.8(d)

The Dockerfile is a modification of one from @jmarshall 's gist (https://gist.github.com/jmarshall/adc75969306354f4398896f1fc1f4865)

ARG  ubuntu_version=18.04
FROM ubuntu:${ubuntu_version}

USER root

# ubuntu18 as no irods packages available for later versions yet
# gnuplot for plot-bamstats
# python/matplotlib/texlive for plot-vcfstats

RUN apt-get update && DEBIAN_FRONTEND="noninteractive" apt-get install -y \
        locales \
        build-essential \
        curl \
        git \
        libbz2-dev \
        libcurl4-openssl-dev \
        libgsl0-dev \
        liblzma-dev \
        libncurses5-dev \
        libperl-dev \
        libssl-dev \
        zlib1g-dev \
        gnuplot \
        python3-matplotlib \
        python3-numpy \
        texlive-latex-base \
        texlive-fonts-recommended \
        texlive-fonts-extra \
        texlive-latex-extra \
        libsodium-dev \
        autoconf \
        automake \
        lsb-release \
    && curl -L https://packages.irods.org/irods-signing-key.asc | apt-key add - && \
        echo "deb [arch=amd64] https://packages.irods.org/apt/ $(lsb_release -sc) main" | tee /etc/apt/sources.list.d/renci-irods.list && \
        apt-get update && \
        apt-get install -y irods-dev irods-externals-* irods-auth-plugin-krb \
    && sed -i '/^#.* en_GB.UTF-8 /s/^#//' /etc/locale.gen && locale-gen && update-locale LC_ALL="en_GB.UTF-8" \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /tmp

ARG htsversion=1.11
ARG samversion=1.11
ARG bcfversion=1.11
ARG libdeflateversion=1.7

# htslib-plugins hack is issue https://github.com/samtools/htslib-plugins/issues/4
RUN curl -L https://github.com/ebiggers/libdeflate/archive/v${libdeflateversion}.tar.gz | tar xz && \
    (cd libdeflate-${libdeflateversion} && make install) && \
    curl -L https://github.com/samtools/htslib/releases/download/${htsversion}/htslib-${htsversion}.tar.bz2 | tar xj && \
    (cd htslib-${htsversion} && ./configure --with-libdeflate --enable-plugins --with-plugin-path='$(libexecdir)/htslib:/usr/libexec/htslib' && make install) && \
    ldconfig && \
    curl -L https://github.com/samtools/samtools/releases/download/${samversion}/samtools-${samversion}.tar.bz2 | tar xj && \
    (cd samtools-${samversion} && ./configure --with-htslib=system && make install) && \
    curl -L https://github.com/samtools/bcftools/releases/download/${bcfversion}/bcftools-${bcfversion}.tar.bz2 | tar xj && \
    (cd bcftools-${bcfversion} && ./configure --enable-libgsl --enable-perl-filters --with-htslib=system && make install) && \
    git clone --depth 1 git://github.com/samtools/htslib-plugins && \
    (cd htslib-plugins && sed -i 's/init_client_api_table/load_client_api_plugins/g' hfile_irods.c && make install) && \
    git clone --depth 1 git://github.com/samtools/htslib-crypt4gh && \
    (cd htslib-crypt4gh && autoheader && autoconf -Wno-syntax && ./configure --with-htslib=system && cp config.h include/ && make install) && \
    rm -rf /tmp/*

WORKDIR /

Plugin issue with iRODS >4.3.0

samtools 1.18
htslib-plugins: 201712

Running the following (which is from one of our test suites which passes on iRODS 4.2.7, 4.2.11 and 4.2.12) fails in iRODS >4.3.0

samtools view -C -T ./t/data/aln_data_object/test_ref.fa -o irods:/testZone/home/irods/Illumina::AlnDataObjectTest.53621.19/7915_5\#0.cram ./t/data/aln_data_object/7915_5\#0.sam

We see:

'samtools: /opt/irods-externals/boost1.81.0-0/include/boost/smart_ptr/shared_ptr.hpp:783: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<irods::network>::operator->() const [T = irods::network]: Assertion `px != 0' failed.'

The client and server are both >4.3.0, built from the 4-3-stable iRODS branch. We have Docker containers of the client/server combo exhibiting the issue, should anyone want to reproduce this:

ghcr.io/wtsi-npg/ub-22.04-irods-4.3-nightly:4.3
ghcr.io/wtsi-npg/ub-22.04-irods-clients-4.3-nightly:4.3

These contain unofficial Debian package builds created with RENCI's iRODS development environment which we're using to get ahead of potential issues with future iRODS releases, before they impact us.

Thanks.

Plugin segfault issue with iRODS 4.3.1 release

samtools 1.18
htslib-plugins: 201712

Running the following (which is from one of our test suites which passes on iRODS 4.2.7, 4.2.11 and 4.2.12) fails on the current 4.3.1 release of iRODS. This is similar to, but not quite the same error as in #6, which is from a 4-3-stable iRODS branch prior to the release of 4.3.1.

The command producing the error is the same as #6 :

samtools view -C -T ./t/data/aln_data_object/test_ref.fa -o irods:/testZone/home/irods/Illumina::AlnDataObjectTest.53621.19/7915_5\#0.cram ./t/data/aln_data_object/7915_5\#0.sam

Reading symbols from samtools...
(gdb) run
Starting program: /usr/local/bin/samtools view -C -T ./t/data/aln_data_object/test_ref.fa -o irods:/testZone/home/irods/Illumina::AlnDataObjectTest.5733.19/7915_5\#0.cram ./t/data/aln_data_object/7915_5\#0.sam
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7a50617 in std::__1::pair<boost::unordered::detail::iterator_detail::iterator<boost::unordered::detail::node<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, boost::any>, void*>, boost::unordered::detail::bucket<boost::unordered::detail::node<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, boost::any>, void*>, void*> >, bool> boost::unordered::detail::table<boost::unordered::detail::map<std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, boost::any> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, boost::any, irods::irods_string_hash, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >::try_emplace_unique<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) () from /lib/libirods_client.so.4.3.1
(gdb) bt
#0  0x00007ffff7a50617 in std::__1::pair<boost::unordered::detail::iterator_detail::iterator<boost::unordered::detail::node<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, boost::any>, void*>, boost::unordered::detail::bucket<boost::unordered::detail::node<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, boost::any>, void*>, void*> >, bool> boost::unordered::detail::table<boost::unordered::detail::map<std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, boost::any> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, boost::any, irods::irods_string_hash, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >::try_emplace_unique<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) () from /lib/libirods_client.so.4.3.1
#1  0x00007ffff7abf8db in irods::plugin_base::call<char const*, BytesBuf const*, BytesBuf const*, BytesBuf const*, int, iRODSProtocol>(RsComm*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::shared_ptr<irods::first_class_object>, char const*, BytesBuf const*, BytesBuf const*, BytesBuf const*, int, iRODSProtocol)::{lambda(irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, char const*, BytesBuf const*, BytesBuf const*, BytesBuf const*, int, iRODSProtocol)#1}::operator()(irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, char const*, BytesBuf const*, BytesBuf const*, BytesBuf const*, int, iRODSProtocol) const () from /lib/libirods_client.so.4.3.1
#2  0x00007ffff7abe543 in irods::error irods::plugin_base::call<char const*, BytesBuf const*, BytesBuf const*, BytesBuf const*, int, iRODSProtocol>(RsComm*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::shared_ptr<irods::first_class_object>, char const*, BytesBuf const*, BytesBuf const*, BytesBuf const*, int, iRODSProtocol) () from /lib/libirods_client.so.4.3.1
#3  0x00007ffff7aba66b in sendRodsMsg(boost::shared_ptr<irods::network_object>, char const*, BytesBuf const*, BytesBuf const*, BytesBuf const*, int, iRODSProtocol) () from /lib/libirods_client.so.4.3.1
#4  0x00007ffff7ab9a5e in sendStartupPack () from /lib/libirods_client.so.4.3.1
#5  0x00007ffff7ab89a5 in connectToRhost () from /lib/libirods_client.so.4.3.1
#6  0x00007ffff7a9f3c4 in _rcConnect () from /lib/libirods_client.so.4.3.1
#7  0x00007ffff7a9f102 in rcConnect () from /lib/libirods_client.so.4.3.1
#8  0x00007ffff7b42b00 in irods_init () at hfile_irods.c:139
#9  hopen_irods (filename=0x5555556113f0 "irods:/testZone/home/irods/Illumina::AlnDataObjectTest.5733.19/7915_5#0.cram", mode=0x7fffffffe0d0 "wc") at hfile_irods.c:267
#10 0x00007ffff7ee0cf8 in hopen (fname=fname@entry=0x5555556113f0 "irods:/testZone/home/irods/Illumina::AlnDataObjectTest.5733.19/7915_5#0.cram", mode=mode@entry=0x7fffffffe0d0 "wc") at hfile.c:1274
#11 0x00007ffff7eeada9 in hts_open_format (fn=fn@entry=0x5555556113f0 "irods:/testZone/home/irods/Illumina::AlnDataObjectTest.5733.19/7915_5#0.cram", mode=mode@entry=0x7fffffffe37c "wc", fmt=fmt@entry=0x7fffffffe220) at hts.c:908
#12 0x0000555555568652 in main_samview (argc=7, argv=0x7fffffffe4e0) at sam_view.c:1194
#13 0x00007ffff7bb5d90 in __libc_start_call_main (main=main@entry=0x5555555608b0 <main>, argc=argc@entry=8, argv=argv@entry=0x7fffffffe4d8) at ../sysdeps/nptl/libc_start_call_main.h:58
#14 0x00007ffff7bb5e40 in __libc_start_main_impl (main=0x5555555608b0 <main>, argc=8, argv=0x7fffffffe4d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe4c8) at ../csu/libc-start.c:392
--Type <RET> for more, q to quit, c to continue without paging--
#15 0x00005555555611a5 in _start ()

Full samtools details:

samtools 1.18
Using htslib 1.18
Copyright (C) 2023 Genome Research Ltd.

Samtools compilation details:
    Features:       build=configure curses=no 
    CC:             gcc
    CPPFLAGS:       -I/usr/include/irods
    CFLAGS:         -Wall -g -O2
    LDFLAGS:        
    HTSDIR:         
    LIBS:           
    CURSES_LIB:     

HTSlib compilation details:
    Features:       build=configure libcurl=yes S3=yes GCS=yes libdeflate=no lzma=yes bzip2=yes plugins=yes plugin-path=/usr/local/libexec/htslib: htscodecs=1.5.1
    CC:             gcc
    CPPFLAGS:       -I/usr/include/irods
    CFLAGS:         -Wall -g -O2 -fvisibility=hidden
    LDFLAGS:        -fvisibility=hidden -rdynamic

HTSlib URL scheme handlers present:
    built-in:    preload, data, file
    Google Cloud Storage:        gs+http, gs+https, gs
    S3 Multipart Upload:         s3w, s3w+https, s3w+http
    Amazon S3:   s3+https, s3+http, s3
    mmap:        mmap
    libcurl:     imaps, pop3, gophers, http, smb, gopher, sftp, ftps, imap, smtp, smtps, rtsp, scp, ftp, telnet, mqtt, rtmp, ldap, https, ldaps, smbs, tftp, pop3s, dict
    iRODS:       irods4.3.1, irods
    crypt4gh-needed:     crypt4gh
    mem:         mem

clang compilation warning/error

When compiling with clang, I get a warning (by default promoted to an error):

hfile_irods.c:149:23: warning: address of array 'irods.env.rodsHost' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if (irods.env.rodsHost && irods.env.rodsPort)
            ~~~~~~~~~~^~~~~~~~ ~~

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.