danpovey / lilcom Goto Github PK

View Code? Open in Web Editor NEW

32.0 32.0 10.0 19.01 MB

Small compression utility

License: Other

Makefile 0.62% Python 9.74% C++ 57.39% CMake 31.13% Shell 1.12%

lilcom's People

Contributors

Stargazers

Watchers

Forkers

funcwj entn-at alok-binary songmeixu aadps csukuangfj marcinwitkowski shushanxingzhe deeplearningsprint videodanchik

lilcom's Issues

Error when installing lhotse

Hi, I am trying to install lhoste in docker, and I was unable to install lilcom. I tried to install it separately and I tried 1.3, 1.4 and 1.5 and none worked. Would you mind take a look at this error log. Thanks!

Collecting lilcom>=1.1.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/00/93/13666e5015803fc10ee755838f7d8a8b0b0643377d3a82a7e613afe1a819/lilcom-1.5.tar.gz (45 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.8/45.8 kB 837.0 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):
        File "<string>", line 36, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-jbrwvxxq/lilcom_d57f9e34639a480ba981ac5731a289d8/setup.py", line 73, in <module>
          "License :: OSI Approved :: MIT License",
        File "/opt/anaconda3/lib/python3.7/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/opt/anaconda3/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 147, in setup
          _setup_distribution = dist = klass(attrs)
        File "/opt/anaconda3/lib/python3.7/site-packages/setuptools/dist.py", line 479, in __init__
          for k, v in attrs.items()
        File "/opt/anaconda3/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 283, in __init__
          self.finalize_options()
        File "/opt/anaconda3/lib/python3.7/site-packages/setuptools/dist.py", line 898, in finalize_options
          for ep in sorted(loaded, key=by_order):
        File "/opt/anaconda3/lib/python3.7/site-packages/setuptools/dist.py", line 897, in <lambda>
          loaded = map(lambda e: e.load(), filtered)
        File "/opt/anaconda3/lib/python3.7/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 196, in load
          return functools.reduce(getattr, attrs, module)
      AttributeError: type object 'Distribution' has no attribute '_finalize_feature_opts'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Speeding up lilcom decompression

As lilcom is often used through Lhotse, several Lhotse users have reported that the decompression is computationally expensive; some of my colleagues performed dataloader profiling and found it takes 50% of the time in their data pipeline. I am not familiar with the details of lilcom’s algorithm, but perhaps it could be optimized somehow?

Protocol "https" not supported or disabled in libcurl

When I install lilcom from PyPI, pybind11 can't be downloaded because of that error.
Here is the log:

error: downloading 'https://github.com/pybind/pybind11/archive/v2.9.2.tar.gz' failed
                status_code: 1
                status_string: "Unsupported protocol"
                log:
                --- LOG BEGIN ---
                Protocol "https" not supported or disabled in libcurl

But when I execute curl -V, it has https:

curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

Import issues after upgrading to version 1.6

Hello,

after upgrading to 1.6 we started to observe really weird import issues (when executing tests via pytest), like this:

try:
  from _k2 import DeterminizeWeightPushingType
 E       ImportError: libpython3.10.so.1.0: cannot open shared object file: No such file or directory

The file in question does exist, but for some reason it is simply not found.

torch==1.13.0
k2==1.22.dev20221121+cuda11.6.torch1.13.0

After pinning lilcom to version 1.5.1, these import errors went away. Unfortunately, I can't really offer a reproduction: manually importing pytorch and _k2 from python shell results in the ImportError above with both versions of lilcom. However, the tests that execute the import do not throw this error if licom is pinned to 1.5.1. The tests are executed in a ubuntu:20.04 container with python 3.10.

I apologize that the description is so vague and for not bringing a reproducible case. I hope you might be able to identify what's going on but if not, I am happy to provide further info.

num-args

Guys, we need to target python3 here, not python2. As far as I can tell, in python3, these c functions are supposed to take 2 args, not 3, i.e. in

static PyObject * compress16i_8i(PyObject * self, PyObject * args, PyObject * keywds)

we should really have, I think,

static PyObject * compress16i_8i(PyObject * args, PyObject * keywds)

Lilcom numpy version requirements.

Lilcom restricts numpy<1.23 for python 3.10 see

lilcom/setup.py

Line 48 in 6860da1

"numpy<1.23.0; python_version >= '3.8' and python_version <= '3.10'",

Is there any particular reason for this, because it dumps my numpy version during my docker environment build when pip resolves version conflicts?

requirements, why?

@csukuangfj why does our requirements file have detailed requirements on the numpy and associated python versions?

"Cython; sys_platform=='win32' and python_version >= '3.10'",
"numpy<=1.19.5; python_version >= '3.6' and python_version < '3.7'",
"numpy<=1.21.6; python_version >= '3.7' and python_version < '3.8'",
"numpy<1.23.0; python_version >= '3.8' and python_version < '3.10'",
"numpy; python_version >= '3.10'",

Is this just what we've tested? I think we are basically only using the dlpack or whatever, no detailed features, I think?
It seems to sometimes cause problems e.g. when lhotse wants to update numpy, and it suggests a version that isn't within this requirements matrix.

Error when installing lhotse

Hi, I found errors when installing lhoste. Do you know anything about that?
Thanks.

(k2) hwang258@c07:~$ pip install lhotse
Collecting lhotse
  Downloading lhotse-1.5.0-py3-none-any.whl (490 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 490.9/490.9 kB 5.8 MB/s eta 0:00:00
Collecting intervaltree>=3.1.0
  Downloading intervaltree-3.1.0.tar.gz (32 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy>=1.18.1 in ./anaconda3/envs/k2/lib/python3.8/site-packages (from lhotse) (1.23.1)
Collecting lilcom>=1.1.0
  Downloading lilcom-1.2.1.tar.gz (44 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.5/44.5 kB 3.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting SoundFile>=0.10
  Using cached SoundFile-0.10.3.post1-py2.py3-none-any.whl (21 kB)
Collecting dataclasses
  Downloading dataclasses-0.6-py3-none-any.whl (14 kB)
Collecting packaging
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting audioread>=2.1.9
  Downloading audioread-3.0.0.tar.gz (377 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 377.0/377.0 kB 7.1 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting click>=7.1.1
  Downloading click-8.1.3-py3-none-any.whl (96 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.6/96.6 kB 5.5 MB/s eta 0:00:00
Collecting pyyaml>=5.3.1
  Downloading PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 701.2/701.2 kB 6.7 MB/s eta 0:00:00
Collecting cytoolz>=0.10.1
  Downloading cytoolz-0.12.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 6.1 MB/s eta 0:00:00
Collecting tqdm
  Using cached tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
Collecting toolz>=0.8.0
  Downloading toolz-0.12.0-py3-none-any.whl (55 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.8/55.8 kB 3.0 MB/s eta 0:00:00
Collecting sortedcontainers<3.0,>=2.0
  Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Collecting numpy>=1.18.1
  Downloading numpy-1.22.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.9/16.9 MB 5.0 MB/s eta 0:00:00
Collecting cffi>=1.0
  Downloading cffi-1.15.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (442 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 442.7/442.7 kB 7.1 MB/s eta 0:00:00
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Building wheels for collected packages: audioread, intervaltree, lilcom
  Building wheel for audioread (setup.py) ... done
  Created wheel for audioread: filename=audioread-3.0.0-py3-none-any.whl size=23703 sha256=f0c797d34b5ae92038b1b584ee2b8d0e23faa738fa4963f1a0b5dd0336c4ebe2
  Stored in directory: /home/hwang258/.cache/pip/wheels/0a/ed/be/49df2538fca496690a024a4374455584d65c2afd6fc3d6e9c7
  Building wheel for intervaltree (setup.py) ... done
  Created wheel for intervaltree: filename=intervaltree-3.1.0-py2.py3-none-any.whl size=26100 sha256=3a236d5c8a30c6123e645d2f4067219a56c055175a9ec9b16c416ea34645bdf0
  Stored in directory: /home/hwang258/.cache/pip/wheels/45/23/de/5789a92962483fd33cb06674792b9697c1b3766d7c7742830e
  Building wheel for lilcom (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [79 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-38
      creating build/lib.linux-x86_64-cpython-38/lilcom
      copying lilcom/lilcom_interface.py -> build/lib.linux-x86_64-cpython-38/lilcom
      copying lilcom/__init__.py -> build/lib.linux-x86_64-cpython-38/lilcom
      running build_ext
      Setting PYTHON_EXECUTABLE to /home/hwang258/anaconda3/envs/k2/bin/python
      For fast compilation, run:
      export LILCOM_MAKE_ARGS="-j"; python setup.py install
      build command is:
      
                      cd build/temp.linux-x86_64-cpython-38
      
                      cmake -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=/home/hwang258/anaconda3/envs/k2/bin/python /tmp/pip-install-vi92662e/lilcom_11a4eed184dc472abb6a870ef0757b02
      
      
                      make  lilcom_extension
      
      CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
        CMake 3.8 or higher is required.  You are running version 3.7.2
      
      
      -- Configuring incomplete, errors occurred!
      make: *** No rule to make target 'lilcom_extension'.  Stop.
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-vi92662e/lilcom_11a4eed184dc472abb6a870ef0757b02/setup.py", line 52, in <module>
          setuptools.setup(
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 973, in run_commands
          self.run_command(cmd)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
          cmd_obj.run()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
          self.run_command('build')
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
          self.distribution.run_command(command)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
          cmd_obj.run()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/build.py", line 24, in run
          super().run()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 132, in run
          self.run_command(cmd_name)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
          self.distribution.run_command(command)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
          cmd_obj.run()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
          _build_ext.run(self)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
          self.build_extensions()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 466, in build_extensions
          self._build_extensions_serial()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 492, in _build_extensions_serial
          self.build_extension(ext)
        File "/tmp/pip-install-vi92662e/lilcom_11a4eed184dc472abb6a870ef0757b02/cmake/cmake_extension.py", line 101, in build_extension
          raise Exception(
      Exception:
      Build lilcom failed. Please check the error message.
      You can ask for help by creating an issue on GitHub.
      
      Click:
          https://github.com/danpovey/lilcom/issues/new
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for lilcom
  Running setup.py clean for lilcom
Successfully built audioread intervaltree
Failed to build lilcom
Installing collected packages: sortedcontainers, dataclasses, tqdm, toolz, pyyaml, pyparsing, pycparser, numpy, intervaltree, click, audioread, packaging, lilcom, cytoolz, cffi, SoundFile, lhotse
  Attempting uninstall: numpy
    Found existing installation: numpy 1.23.1
    Uninstalling numpy-1.23.1:
      Successfully uninstalled numpy-1.23.1
  Running setup.py install for lilcom ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for lilcom did not run successfully.
  │ exit code: 1
  ╰─> [83 lines of output]
      running install
      /home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-38
      creating build/lib.linux-x86_64-cpython-38/lilcom
      copying lilcom/lilcom_interface.py -> build/lib.linux-x86_64-cpython-38/lilcom
      copying lilcom/__init__.py -> build/lib.linux-x86_64-cpython-38/lilcom
      running build_ext
      Setting PYTHON_EXECUTABLE to /home/hwang258/anaconda3/envs/k2/bin/python
      For fast compilation, run:
      export LILCOM_MAKE_ARGS="-j"; python setup.py install
      build command is:
      
                      cd build/temp.linux-x86_64-cpython-38
      
                      cmake -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=/home/hwang258/anaconda3/envs/k2/bin/python /tmp/pip-install-vi92662e/lilcom_11a4eed184dc472abb6a870ef0757b02
      
      
                      make  lilcom_extension
      
      CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
        CMake 3.8 or higher is required.  You are running version 3.7.2
      
      
      -- Configuring incomplete, errors occurred!
      make: *** No rule to make target 'lilcom_extension'.  Stop.
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-vi92662e/lilcom_11a4eed184dc472abb6a870ef0757b02/setup.py", line 52, in <module>
          setuptools.setup(
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 973, in run_commands
          self.run_command(cmd)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
          cmd_obj.run()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/install.py", line 68, in run
          return orig.install.run(self)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/install.py", line 692, in run
          self.run_command('build')
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
          self.distribution.run_command(command)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
          cmd_obj.run()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/build.py", line 24, in run
          super().run()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 132, in run
          self.run_command(cmd_name)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
          self.distribution.run_command(command)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
          cmd_obj.run()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
          _build_ext.run(self)
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
          self.build_extensions()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 466, in build_extensions
          self._build_extensions_serial()
        File "/home/hwang258/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 492, in _build_extensions_serial
          self.build_extension(ext)
        File "/tmp/pip-install-vi92662e/lilcom_11a4eed184dc472abb6a870ef0757b02/cmake/cmake_extension.py", line 101, in build_extension
          raise Exception(
      Exception:
      Build lilcom failed. Please check the error message.
      You can ask for help by creating an issue on GitHub.
      
      Click:
          https://github.com/danpovey/lilcom/issues/new
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> lilcom

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Testing multi-platform reproducibility

@soroushzargar can you please make it so that your end-to-end reconstruction-test also prints out some kind of hash values of the compressed arrays? (E.g. could just be the result of casting to int32 and then summing all the elements).

What we need is to make sure that this gives exactly the same results (bitwise) on different platforms. If there are inter-platform differences it can render the files non-decompressible.
(This is true at the int16 level; differences in how we round the floats to int16 would be OK.)

liblto_plugin.so not found

When I install the lilcom, the error occured...
python setup.py bdist_wheel
g++: fatal error: -fuse-linker-plugin, but liblto_plugin.so not found
compilation terminated.

Do you know how to solve the problem？

Bug: Fresh installation is missing numpy

Hi, when installing this package in a fresh virtual environment, the following error eccurs:

  Downloading https://files.pythonhosted.org/packages/8b/53/43f601d93e9f33da1c9d6909c1bf94390666726dbd2d18e76c377a5a1a9a/lilcom-1.1.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-szfvvzdg/lilcom/setup.py", line 26, in <module>
        import numpy
    ModuleNotFoundError: No module named 'numpy'

By installing numpy beforehand, it is then possible to install lilcom. Could this be mentioned in the README?

Finalization for first release

Here are things we are supposed to do currently to have a tested, completed working package release:

Inside lilcom_c_extension.c: There shouldn't be any miss-calculation on parameter passing and procedure inside extension functions. Following notes should be considered;

There are no default value and optional arguments passed to function. Any of these should be postponed to the wrapper module.
Return codes from both lilcom_c_extension.c and lilcom.c must be documented in the readme.md
Error codes must be the same as what is commented inside the code. All error values are better to be more than 100 in order not to be mistaken with conversion exponent.

The wrapper:

Quoted from @danpovey:

I want to have the top-level interface callable from c as just lilcom.compress() and lilcom.decompress(), you can add import statements in the init file to make this possible.
Also, please test with close-to-denormal and close-to-infinite numbers.

Double-checking the code to check whether all conditions of the input array are handled.

Test Procedure: These are the main testing procedures we might design:

Testing all possible workflows which call any function inside lilcom_c_extension.c.
Testing all possible workflows calling the wrapper
Quoted by @danpovey:

Please test with close-to-denormal and close-to-infinite numbers. I suggest to generate random normal data and then multiply by 2^n for n in, say, [-40, 40]. It should work as long as, when converted to single precision float, there is no inf or nan.

I'd like some graphs of SNR versus lpc_order for some real speech or music data; and also time taken versus lpc_order (to check it's roughly linear).

This is what I am going to figure out.

After doing these we are ready to go for a finalized distribution.

Build lilcom failed. Please check the error message. -- solved!

I am trying to install lhotse
and I am getting this error:
Build lilcom failed. Please check the error message.

Code I used:
pip install git+https://github.com/lhotse-speech/lhotse

Please help.

Big-endian system test

I'd like to find a way to test this on a big-endian system. @funcwj or @soroushzargar do you have any ideas on this? A system emulator should be OK.

System compatibility is going to be a big deal here because of the nature of the algorithm (it requires perfectly identical computations on the system where it was compressed and where it's decompressed; that's why everything is done in integer arithmetic).

lilcom fails to compress Fortran contiguous arrays.

Hi, it seems that lilcom fails to compress Fortran contiguous arrays and it's probably ok, as numpy preserves C contiguity in the majority operations, but some of the common operators like numpy.transpose() changes the contiguity.

To reproduce:

>>> import lilcom
>>> import numpy as np
>>> value = np.array([[0.1, 0.2], [0.3, 0.4]])
>>> lilcom.compress(value)
b'L\x00g\xc2\x0f\x13\x08\xc0\x04\x02\x10\x8dd2\x9b\x01'
>>> lilcom.compress(value.T)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/videodanchik/anaconda3/lib/python3.8/site-packages/lilcom/lilcom_interface.py", line 38, in compress
    raise RuntimeError("Something went wrong in compression, return value was ",
RuntimeError: ('Something went wrong in compression, return value was ', None)

Just in case, this error I got in the following environment:

Python 3.8.3 (default, Jul  2 2020, 16:21:59) 
[GCC 7.3.0] :: Anaconda, Inc. on linux

Some Questions on the Procedures

Is there any reason you called compress inside? I changed it, but I thought there might be something I didn't figure out.

lilcom/lilcom/lilcom_c_extension.c

Line 213 in 1b4cf23

int ret = lilcom_compress(dim, input_data, input_stride,

Release

@funcwj can you please let me know what I have to do to publish this on PyPi?
I think it might be just about ready to release.

Compression fails on non-contigous feats

Hi, I am getting a stride error when running compression on non-contiguous tensors:

lilcom: compression error: last stride should be 1, got 253

The error message is a bit confusing and its not clear wht the problem is. Some solutions could be:

warn the user that inputs should be contiguous and make the tensor contiguous or
fail with a message that will tell the user to make their tensor contiguous.

Checking Python style guide

I'm checking the Python codes against the style conventions in PEP 8 using the pycodestyle tool.

Make sure reconstruction test is runnable end to end

@soroushzargar perhaps you could do this.. it would be better if the reconstruction-test were easier to run. For example, have a script that downloads and suitably formats the appropriate data and then runs the test. Right now there is some hacky stuff going on with a temp dir that probably isn't in the source tarball.

I was only printing out the bits in the file as a debugging step. But I think we should be displaying the file sizes or bits per unit time in some form alongside the various compression methods; this will make comparisons more meaningful.

I just set the bits per sample to 6 in reconstruction-test.py but we may want to test various settings. What I would really like is some info about PSNR's and file sizes formatted in a way that's easily readable. IMO a simple Python program that prints out a summary would be preferable to this CSV stuff, in that it would be clearer. But as long as you do print out something readable at some point (maybe as a post processing step that might be in a separate script) it's OK with me. Make sure it's all documented with a top-level run.sh or something.

Issue installing

Do you know how to fix it?
Thanks!

(icefall_armory_june22) hwang258@b01:~/codes/icefall/egs/yesno/ASR$ python
Python 3.8.13 (default, Mar 28 2022, 11:38:47)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lilcom
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hwang258/anaconda3/envs/icefall_armory_june22/lib/python3.8/site-packages/lilcom/__init__.py", line 2, in <module>
    from .lilcom_interface import compress, decompress, get_shape
  File "/home/hwang258/anaconda3/envs/icefall_armory_june22/lib/python3.8/site-packages/lilcom/lilcom_interface.py", line 3, in <module>
    import lilcom_extension
ImportError: libpython3.8.so.1.0: cannot open shared object file: No such file or directory

ValueError: lilcom: Length of string was too short

I am using lilcom compress and decompress the numpy data, however I got the below error:

WARNING:root:len(f.read()) is 277089
WARNING:root:len(f.read()) is 906037
Traceback (most recent call last):
  File "train.py", line 492, in <module>
    main()
  File "train.py", line 488, in main
WARNING:root:len(f.read()) is 777463
WARNING:root:len(f.read()) is 558017
    train(a.local_rank, cfg, a)
  File "train.py", line 221, in train
    for i, batch in pb:
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/deepspeed/runtime/dataloader.py", line 90, in __next__
    return next(self.data)
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/deepspeed/runtime/dataloader.py", line 108, in <genexpr>
WARNING:root:len(f.read()) is 231040
    self.data = (x for x in self.dataloader)
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1376, in _next_data
    return self._process_data(data)
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
    data.reraise()
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/_utils.py", line 461, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/mntnfs/lee_data1/maduo/w_2022/librispeech/transfusion-asr/transfusion/dataset.py", line 58, in __getitem__
    arr = lilcom.decompress(f.read())
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/lilcom/lilcom_interface.py", line 99, in decompress
    shape = lilcom_extension.get_float_matrix_shape(byte_string)
ValueError: lilcom: Length of string was too short

Could you help me？

In file included from lilcom/lilcom_extension.cc:15: In file included from lilcom/compression.h:6: lilcom/int_stream.h:7:10: fatal error: 'vector' file not found #include <vector> ^~~~~~~~ 1 error generated. error: command 'gcc' failed with exit status 1

python3 setup.py install --user
running install
error: [Errno 13] Permission denied: '/Users/wuxb/Library/Python/3.7/lib'
wuxb@wuxbdeMacBook-Pro lilcom % sudo python3 setup.py install --user
Password:
running install
running bdist_egg
running egg_info
writing lilcom.egg-info/PKG-INFO
writing dependency_links to lilcom.egg-info/dependency_links.txt
writing requirements to lilcom.egg-info/requires.txt
writing top-level names to lilcom.egg-info/top_level.txt
reading manifest file 'lilcom.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'lilcom.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.6-intel/egg
running install_lib
running build_py
running build_ext
building 'lilcom.lilcom_extension' extension
gcc -fno-strict-aliasing -Wsign-compare -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch i386 -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/core/include -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c lilcom/lilcom_extension.cc -o build/temp.macosx-10.6-intel-3.7/lilcom/lilcom_extension.o -g -Wall -UNDEBUG -Wno-c++11-compat-deprecated-writable-strings
clang: warning: include path for libstdc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
clang: warning: include path for libstdc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
In file included from lilcom/lilcom_extension.cc:15:
In file included from lilcom/compression.h:6:
lilcom/int_stream.h:7:10: fatal error: 'vector' file not found
#include
^~~~~~~~
1 error generated.
error: command 'gcc' failed with exit status 1

gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.0 (clang-1200.0.32.21)
Target: x86_64-apple-darwin19.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

can't be installed

why I cann't install it.