Giter Club home page Giter Club logo

miurahr / pyppmd Goto Github PK

View Code? Open in Web Editor NEW
8.0 5.0 3.0 1.68 MB

pyppmd provides classes and functions for compressing and decompressing text data, using PPM (Prediction by partial matching) compression algorithm variation H and I.2. It provide an API similar to Python's zlib/bz2/lzma modules.

Home Page: https://pyppmd.readthedocs.io/en/latest/

License: GNU Lesser General Public License v2.1

CMake 2.52% C 74.20% Python 23.28%
compression ppmd python

pyppmd's Introduction

PyPPMd

https://readthedocs.org/projects/pyppmd/badge/?version=latest https://img.shields.io/conda/vn/conda-forge/pyppmd

Introduction

pyppmd module provides classes and functions for compressing and decompressing text data, using PPM(Prediction by partial matching) compression algorithm which has several variations of implementations. PPMd is the implementation by Dmitry Shkarin. PyPPMD use Igor Pavlov's range coder introduced in 7-zip.

Give Up GitHub

This project has given up GitHub. (See Software Freedom Conservancy's Give Up GitHub site for details)

You can now find this project at https://codeberg.org/miurahr/pyppmd instead.

pyppmd's People

Contributors

cielavenir avatar dependabot[bot] avatar miurahr avatar sarthakpati avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

pyppmd's Issues

OSX build issues on conda

Describe the bug
Conda is unable to build the OSX version of the new version.

Related issue
(if exist)

To Reproduce
See build here: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=364581&view=logs&j=15f8dc28-9f6f-5e3a-5b3c-2b169071e5be&t=63fa899e-c160-5185-75c4-c5fd30c90374&l=257

  lib2/threading.c:113:5: warning: implicit declaration of function 'clock_gettime' is invalid in C99 [-Wimplicit-function-declaration]
      clock_gettime(CLOCK_REALTIME, &abstime);
      ^
  lib2/threading.c:113:19: error: use of undeclared identifier 'CLOCK_REALTIME'

Expected behavior
The build should succeed.

Environment (please complete the following information):

  • OS: macOS
  • Python 3.6-3.9
  • project version: 1.60.0

Test data(please attach in the report):
N.A.

Additional context
N.A.

build failure on macOS

Describe the bug
Installing pyppmd via pip fails with:

ValueError: invalid pyproject.toml config: `project`

To Reproduce
I can only reproduce this via the macos-latest runner on GitHub Actions. It works without issues on Linux and Windows, in the GitHub Actions CI and locally.

Expected behavior
Installation should not fail.

Environment (please complete the following information):

  • OS: macOS from GitHub Actions
  • Python 3.10
  • project version: 0.17.4

Additional context

Full log of error in the CI: pyppmd_pip_error.txt

Fuzz on Windows fails

Describe the bug
When running fuzzing test on Windows, hypothesis fails.
It seems multi-threading problem.

It is happened with C implementation.
CFFI works fine.

Expected behavior
pass fuzzing test on windows.

Environment:

  • OS: Windows 10
  • Python CPython 3.6-3.9
  • project version: v0.10.0
______________________________ test_ppmd7_fuzzer ______________________________
  
  self = <hypothesis.core.StateForActualGivenExecution object at 0x00000225E5946D68>
  data = ConjectureData(INTERESTING, 58 bytes, frozen)
  
      def _execute_once_for_engine(self, data):
          """Wrapper around ``execute_once`` that intercepts test failure
          exceptions and single-test control exceptions, and turns them into
          appropriate method calls to `data` instead.
      
          This allows the engine to assume that any exception other than
          ``StopTest`` must be a fatal error, and should stop the entire engine.
          """
          try:
              trace = frozenset()
              if (
                  self.failed_normally
                  and Phase.explain in self.settings.phases
                  and sys.gettrace() is None
                  and not PYPY
              ):  # pragma: no cover
                  # This is in fact covered by our *non-coverage* tests, but due to the
                  # settrace() contention *not* by our coverage tests.  Ah well.
                  tracer = Tracer()
                  try:
                      sys.settrace(tracer.trace)
                      result = self.execute_once(data)
                      if data.status == Status.VALID:
                          self.explain_traces[None].add(frozenset(tracer.branches))
                  finally:
                      sys.settrace(None)
                      trace = frozenset(tracer.branches)
              else:
  >               result = self.execute_once(data)
  
  .tox\py36\lib\site-packages\hypothesis\core.py:683: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  
  self = <hypothesis.core.StateForActualGivenExecution object at 0x00000225E5946D68>
  data = ConjectureData(INTERESTING, 58 bytes, frozen), print_example = False
  is_final = False, expected_failure = None
  
      def execute_once(
          self, data, print_example=False, is_final=False, expected_failure=None
      ):
          """Run the test function once, using ``data`` as input.
      
          If the test raises an exception, it will propagate through to the
          caller of this method. Depending on its type, this could represent
          an ordinary test failure, or a fatal error, or a control exception.
      
          If this method returns normally, the test might have passed, or
          it might have placed ``data`` in an unsuccessful state and then
          swallowed the corresponding control exception.
          """
      
          data.is_find = self.is_find
      
          text_repr = [None]
          if self.settings.deadline is None:
              test = self.test
          else:
      
              @proxies(self.test)
              def test(*args, **kwargs):
                  self.__test_runtime = None
                  initial_draws = len(data.draw_times)
                  start = time.perf_counter()
                  result = self.test(*args, **kwargs)
                  finish = time.perf_counter()
                  internal_draw_time = sum(data.draw_times[initial_draws:])
                  runtime = datetime.timedelta(
                      seconds=finish - start - internal_draw_time
                  )
                  self.__test_runtime = runtime
                  current_deadline = self.settings.deadline
                  if not is_final:
                      current_deadline = (current_deadline // 4) * 5
                  if runtime >= current_deadline:
                      raise DeadlineExceeded(runtime, self.settings.deadline)
                  return result
      
          def run(data):
              # Set up dynamic context needed by a single test run.
              with local_settings(self.settings):
                  with deterministic_PRNG():
                      with BuildContext(data, is_final=is_final):
      
                          # Generate all arguments to the test function.
                          args, kwargs = data.draw(self.search_strategy)
                          if expected_failure is not None:
                              text_repr[0] = arg_string(test, args, kwargs)
      
                          if print_example or current_verbosity() >= Verbosity.verbose:
                              output = StringIO()
      
                              printer = RepresentationPrinter(output)
                              if print_example:
                                  printer.text("Falsifying example:")
                              else:
                                  printer.text("Trying example:")
      
                              if self.print_given_args:
                                  printer.text(" ")
                                  printer.text(test.__name__)
                                  with printer.group(indent=4, open="(", close=""):
                                      printer.break_()
                                      for v in args:
                                          printer.pretty(v)
                                          # We add a comma unconditionally because
                                          # generated arguments will always be kwargs,
                                          # so there will always be more to come.
                                          printer.text(",")
                                          printer.breakable()
      
                                      for i, (k, v) in enumerate(kwargs.items()):
                                          printer.text(k)
                                          printer.text("=")
                                          printer.pretty(v)
                                          printer.text(",")
                                          if i + 1 < len(kwargs):
                                              printer.breakable()
                                  printer.break_()
                                  printer.text(")")
                              printer.flush()
                              report(output.getvalue())
                          return test(*args, **kwargs)
      
          # Run the test function once, via the executor hook.
          # In most cases this will delegate straight to `run(data)`.
  >       result = self.test_runner(data, run)
  
  .tox\py36\lib\site-packages\hypothesis\core.py:619: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  
  data = ConjectureData(INTERESTING, 58 bytes, frozen)
  function = <function StateForActualGivenExecution.execute_once.<locals>.run at 0x00000225E5A2DB70>
  
      def default_new_style_executor(data, function):
  >       return function(data)
  
  .tox\py36\lib\site-packages\hypothesis\executors.py:52: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  
  data = ConjectureData(INTERESTING, 58 bytes, frozen)
  
      def run(data):
          # Set up dynamic context needed by a single test run.
          with local_settings(self.settings):
              with deterministic_PRNG():
                  with BuildContext(data, is_final=is_final):
      
                      # Generate all arguments to the test function.
                      args, kwargs = data.draw(self.search_strategy)
                      if expected_failure is not None:
                          text_repr[0] = arg_string(test, args, kwargs)
      
                      if print_example or current_verbosity() >= Verbosity.verbose:
                          output = StringIO()
      
                          printer = RepresentationPrinter(output)
                          if print_example:
                              printer.text("Falsifying example:")
                          else:
                              printer.text("Trying example:")
      
                          if self.print_given_args:
                              printer.text(" ")
                              printer.text(test.__name__)
                              with printer.group(indent=4, open="(", close=""):
                                  printer.break_()
                                  for v in args:
                                      printer.pretty(v)
                                      # We add a comma unconditionally because
                                      # generated arguments will always be kwargs,
                                      # so there will always be more to come.
                                      printer.text(",")
                                      printer.breakable()
      
                                  for i, (k, v) in enumerate(kwargs.items()):
                                      printer.text(k)
                                      printer.text("=")
                                      printer.pretty(v)
                                      printer.text(",")
                                      if i + 1 < len(kwargs):
                                          printer.breakable()
                              printer.break_()
                              printer.text(")")
                          printer.flush()
                          report(output.getvalue())
  >                   return test(*args, **kwargs)
  
  .tox\py36\lib\site-packages\hypothesis\core.py:615: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  
  obj = b'\x07\x15\xa5\x07\x99BR\xa4*\xbf):kc=\x0e\xb1L\xa1', max_order = 62
  mem_size = 4140799450
  
      @given(
  >       obj=st.binary(min_size=5),
          max_order=st.integers(min_value=2, max_value=64),
          mem_size=st.integers(min_value=1 << 11, max_value=MAX_SIZE),
      )
      def test_ppmd7_fuzzer(obj, max_order, mem_size):
  
  tests\test_ppmd7_fuzzer.py:14: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  
  args = (b'\x07\x15\xa5\x07\x99BR\xa4*\xbf):kc=\x0e\xb1L\xa1', 62, 4140799450)
  kwargs = {}, initial_draws = 1, start = 0.5331072
  
      @proxies(self.test)
      def test(*args, **kwargs):
          self.__test_runtime = None
          initial_draws = len(data.draw_times)
          start = time.perf_counter()
  >       result = self.test(*args, **kwargs)
  
  .tox\py36\lib\site-packages\hypothesis\core.py:557: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  
  obj = b'\x07\x15\xa5\x07\x99BR\xa4*\xbf):kc=\x0e\xb1L\xa1', max_order = 62
  mem_size = 4140799450
  
      @given(
          obj=st.binary(min_size=5),
          max_order=st.integers(min_value=2, max_value=64),
          mem_size=st.integers(min_value=1 << 11, max_value=MAX_SIZE),
      )
      def test_ppmd7_fuzzer(obj, max_order, mem_size):
  >       enc = pyppmd.Ppmd7Encoder(max_order=max_order, mem_size=mem_size)
  E       SystemError: <class '_ppmd.Ppmd7Encoder'> returned NULL without setting an error
  
  tests\test_ppmd7_fuzzer.py:19: SystemError
  
  During handling of the above exception, another exception occurred:
  
      @given(
  >       obj=st.binary(min_size=5),
          max_order=st.integers(min_value=2, max_value=64),
          mem_size=st.integers(min_value=1 << 11, max_value=MAX_SIZE),
      )
      def test_ppmd7_fuzzer(obj, max_order, mem_size):
  
  tests\test_ppmd7_fuzzer.py:14: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  .tox\py36\lib\site-packages\hypothesis\core.py:735: in _execute_once_for_engine
      data.mark_interesting(interesting_origin)
  .tox\py36\lib\site-packages\hypothesis\internal\conjecture\data.py:1056: in mark_interesting
      self.conclude_test(Status.INTERESTING, interesting_origin)
  .tox\py36\lib\site-packages\hypothesis\internal\conjecture\data.py:1052: in conclude_test
      self.freeze()
  .tox\py36\lib\site-packages\hypothesis\internal\conjecture\data.py:980: in freeze
      self.observer.conclude_test(self.status, self.interesting_origin)
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  
  self = <hypothesis.internal.conjecture.datatree.TreeRecordingObserver object at 0x00000225E59A8F98>
  status = Status.INTERESTING
  interesting_origin = (<class 'SystemError'>, 'D:\\a\\pyppmd\\pyppmd\\tests\\test_ppmd7_fuzzer.py', 19, ())
  
      def conclude_test(self, status, interesting_origin):
          """Says that ``status`` occurred at node ``node``. This updates the
          node if necessary and checks for consistency."""
          if status == Status.OVERRUN:
              return
          i = self.__index_in_current_node
          node = self.__current_node
      
          if i < len(node.values) or isinstance(node.transition, Branch):
              inconsistent_generation()
      
          new_transition = Conclusion(status, interesting_origin)
      
          if node.transition is not None and node.transition != new_transition:
              # As an, I'm afraid, horrible bodge, we deliberately ignore flakiness
              # where tests go from interesting to valid, because it's much easier
              # to produce good error messages for these further up the stack.
              if isinstance(node.transition, Conclusion) and (
                  node.transition.status != Status.INTERESTING
                  or new_transition.status != Status.VALID
              ):
                  raise Flaky(
                      "Inconsistent test results! Test case was %r on first run but %r on second"
  >                   % (node.transition, new_transition)
                  )
  E               hypothesis.errors.Flaky: Inconsistent test results! Test case was Conclusion(status=Status.INTERESTING, interesting_origin=(<class 'SystemError'>, 'D:\\a\\pyppmd\\pyppmd\\tests\\test_ppmd7_fuzzer.py', 23, ())) on first run but Conclusion(status=Status.INTERESTING, interesting_origin=(<class 'SystemError'>, 'D:\\a\\pyppmd\\pyppmd\\tests\\test_ppmd7_fuzzer.py', 19, ())) on second
  
  .tox\py36\lib\site-packages\hypothesis\internal\conjecture\datatree.py:413: Flaky
  =========================== short test summary info ===========================
  FAILED tests/test_ppmd7_fuzzer.py::test_ppmd7_fuzzer - hypothesis.errors.Flak...
  ========================= 1 failed, 5 passed in 0.74s =========================
  ERROR: InvocationError for command 'D:\a\pyppmd\pyppmd\.tox\py36\Scripts\python.EXE' -m pytest -vv -s (exited with code 1)

About API

If there are some variants of PPMd, could consider an unified API suitable for these.

PPMd was designed for text compression, so pyppmd can provide some text-related APIs, such as encoding/decoding, line break, etc.

If need help, you may @ me.

Better license

Describe the bug
There are mixture of licenses, Public Domain or LGPLv2+ (c code from p7zip, that derivative from 7-zip) and BSD-3.

It is better to change to LGPLv2+ that is clean.

Drop endmark feature

Is your feature request related to a problem? Please describe.
Now ppmd8 decoder can recognize EOF position from original stream.
We don't need user marker of eof.

Describe the solution you'd like
drop endmark feature

related
#34

pip install fails in GitHub CI with metadata-generation-failed

Describe the bug
I'm still not 100% sure if it's a bug in this repo or a standards change, but I'm getting a failure to install this library in our CI (private, log below) that I can't reproduce locally. I don't 100% understand pyproject.toml stuff to be able to debug this myself, but one thing I found is this part of the log:

      configuration error: `project` must contain ['version'] properties
      DESCRIPTION:
          version should be statically defined in the ``version`` field

Related issue
I also found #95 as a PR in this repo and pypa/setuptools#3199 as potential related issues given the timing of when this started failing for us.

Unfortunately, I'm not 100% sure what the actual fix is here or how related those related issues are. Please let me know if I can help you get more info.

Environment (please complete the following information):

  • OS: Mac OS X 11.6.5 (Github CI)
  • Python 3.10.3
  • project version: 0.18.1
Run jurplel/install-qt-action@v3
  with:
    version: 5.15.2
    py7zrversion: >=0.18.0
    target: desktop
    install-deps: true
    cached: false
    setup-python: true
    set-env: true
    tools-only: false
    aqtversion: ==2.0.0
Successfully setup CPython (3.10.3)
brew install p7zip
Warning: p7zip 17.04 is already installed and up-to-date.
To reinstall 17.04, run:
  brew reinstall p7zip
python3 -m pip install setuptools wheel
Requirement already satisfied: setuptools in /Users/runner/hostedtoolcache/Python/3.10.3/x64/lib/python3.10/site-packages (58.1.0)
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.37.1
python3 -m pip install py7zr>=0.18.0
Collecting py7zr>=0.18.0
  Downloading py7zr-0.18.3-py3-none-any.whl (76 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.5/76.5 KB 6.7 MB/s eta 0:00:00
Collecting pyppmd<0.19.0,>=0.18.1
  Downloading pyppmd-0.18.1.tar.gz (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 34.8 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [107 lines of output]
      /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-bh_hq6sv/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py:102: _ExperimentalProjectMetadata: Support for project metadata in `pyproject.toml` is still experimental and may be removed (or change) in future releases.
        warnings.warn(msg, _ExperimentalProjectMetadata)
      configuration error: `project` must contain ['version'] properties
      DESCRIPTION:
          version should be statically defined in the ``version`` field
      
      GIVEN VALUE:
          {
              "name": "pyppmd",
              "requires-python": ">=3.6",
                      "sphinx>=2.3",
                      "sphinx_rtd_theme"
                  ],
                  "fuzzer": [
                      "atheris",
                      "hypothesis"
                  ]
              },
              "urls": {
                  "Source": "https://github.com/miurahr/pyppmd"
              }
          }
      
      OFFENDING RULE: 'required'
      
      DEFINITION:
          {
              "required": [
                  "version"
              ]
          }
      Traceback (most recent call last):
        File "/Users/runner/hostedtoolcache/Python/3.10.3/x64/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/Users/runner/hostedtoolcache/Python/3.10.3/x64/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/runner/hostedtoolcache/Python/3.10.3/x64/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-bh_hq6sv/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 188, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-bh_hq6sv/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 174, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 84, in <module>
          setup(
        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-bh_hq6sv/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-bh_hq6sv/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 122, in setup
          dist.parse_config_files()
        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-bh_hq6sv/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 854, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-bh_hq6sv/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 54, in apply_configuration
          config = read_configuration(filepath, True, ignore_option_errors, dist)
        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-bh_hq6sv/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 120, in read_configuration
          validate(subset, filepath)
        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-bh_hq6sv/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 43, in validate
          raise error from None
      ValueError: invalid pyproject.toml config: `project`
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Error: The process 'python3' failed with exit code 1

Build fails with MinGW/MSYS2

Describe the bug
I'm trying to install aqtinstall package on MSYS2, which depends on pyppmd, and it fails at compilation.

To Reproduce
Try to install pyppmd on MSYS2 with the MINGW64 shell.

Expected behavior
That it builds.

Environment (please complete the following information):

  • OS: Windows 10 / MSYS2
  • Python 3.9.6

Additional context
Error log:

$ python -m pip install aqtinstall
Collecting aqtinstall
  Using cached aqtinstall-2.0.0-py2.py3-none-any.whl (48 kB)
Requirement already satisfied: semantic-version in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from aqtinstall) (2.8.5)
Requirement already satisfied: py7zr>=0.15.1 in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from aqtinstall) (0.16.1)
Requirement already satisfied: requests in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from aqtinstall) (2.26.0)
Requirement already satisfied: texttable in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from aqtinstall) (1.6.4)
Requirement already satisfied: patch>=1.16 in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from aqtinstall) (1.16)
Collecting bs4
  Using cached bs4-0.0.1-py3-none-any.whl
Requirement already satisfied: brotli>=1.0.9 in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from py7zr>=0.15.1->aqtinstall) (1.0.9)
Collecting pyppmd>=0.14.0
  Downloading pyppmd-0.17.1.tar.gz (462 kB)
     |################################| 462 kB 192 kB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: pycryptodomex>=3.6.6 in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from py7zr>=0.15.1->aqtinstall) (3.10.1)
Requirement already satisfied: pyzstd<0.15.0,>=0.14.4 in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from py7zr>=0.15.1->aqtinstall) (0.14.4)
Requirement already satisfied: multivolumefile>=0.2.3 in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from py7zr>=0.15.1->aqtinstall) (0.2.3)
Requirement already satisfied: bcj-cffi<0.6.0,>=0.5.1 in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from py7zr>=0.15.1->aqtinstall) (0.5.1)
Collecting beautifulsoup4
  Using cached beautifulsoup4-4.10.0-py3-none-any.whl (97 kB)
Requirement already satisfied: idna<4,>=2.5 in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from requests->aqtinstall) (3.2)
Requirement already satisfied: charset-normalizer~=2.0.0 in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from requests->aqtinstall) (2.0.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from requests->aqtinstall) (1.26.6)
Requirement already satisfied: certifi>=2017.4.17 in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from requests->aqtinstall) (2021.5.30)
Requirement already satisfied: cffi>=1.14.0 in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from bcj-cffi<0.6.0,>=0.5.1->py7zr>=0.15.1->aqtinstall) (1.14.6)
Collecting soupsieve>1.2
  Using cached soupsieve-2.2.1-py3-none-any.whl (33 kB)
Requirement already satisfied: pycparser in c:/projets/softs/msys2/mingw64/lib/python3.9/site-packages (from cffi>=1.14.0->bcj-cffi<0.6.0,>=0.5.1->py7zr>=0.15.1->aqtinstall) (2.20)
Building wheels for collected packages: pyppmd
  Building wheel for pyppmd (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: C:/Projets/Softs/msys2/mingw64/bin/python.exe C:/Projets/Softs/msys2/mingw64/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel C:/Users/Lynix/AppData/Local/Temp/tmph60mq72p
       cwd: C:/Users/Lynix/AppData/Local/Temp/pip-install-49z0wcvx/pyppmd_412da10fe35543508a445f0a0087f5e6
  Complete output (45 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.mingw_x86_64-3.9
  creating build/lib.mingw_x86_64-3.9/pyppmd
  copying src/pyppmd/__init__.py -> build/lib.mingw_x86_64-3.9/pyppmd
  package init file 'src/pyppmd/c/__init__.py' not found (or not a regular file)
  creating build/lib.mingw_x86_64-3.9/pyppmd/c
  copying src/pyppmd/c/c_ppmd.py -> build/lib.mingw_x86_64-3.9/pyppmd/c
  copying src/pyppmd/__init__.pyi -> build/lib.mingw_x86_64-3.9/pyppmd
  copying src/pyppmd/py.typed -> build/lib.mingw_x86_64-3.9/pyppmd
  running build_ext
  dllwrap: ATTENTION: dllwrap est d▒pr▒ci▒, utilisez plut▒t gcc -shared ou ld -shared

  building 'pyppmd.c._ppmd' extension
  creating build/temp.mingw_x86_64-3.9
  creating build/temp.mingw_x86_64-3.9/lib
  creating build/temp.mingw_x86_64-3.9/lib2
  creating build/temp.mingw_x86_64-3.9/src
  creating build/temp.mingw_x86_64-3.9/src/ext
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe -Ilib -Ilib2 -IC:/Projets/Softs/msys2/mingw64/include/python3.9 -c lib/Ppmd7.c -o build/temp.mingw_x86_64-3.9/lib/Ppmd7.o
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe -Ilib -Ilib2 -IC:/Projets/Softs/msys2/mingw64/include/python3.9 -c lib/Ppmd7Dec.c -o build/temp.mingw_x86_64-3.9/lib/Ppmd7Dec.o
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe -Ilib -Ilib2 -IC:/Projets/Softs/msys2/mingw64/include/python3.9 -c lib/Ppmd7Enc.c -o build/temp.mingw_x86_64-3.9/lib/Ppmd7Enc.o
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe -Ilib -Ilib2 -IC:/Projets/Softs/msys2/mingw64/include/python3.9 -c lib/Ppmd8.c -o build/temp.mingw_x86_64-3.9/lib/Ppmd8.o
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe -Ilib -Ilib2 -IC:/Projets/Softs/msys2/mingw64/include/python3.9 -c lib/Ppmd8Dec.c -o build/temp.mingw_x86_64-3.9/lib/Ppmd8Dec.o
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe -Ilib -Ilib2 -IC:/Projets/Softs/msys2/mingw64/include/python3.9 -c lib/Ppmd8Enc.c -o build/temp.mingw_x86_64-3.9/lib/Ppmd8Enc.o
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe -Ilib -Ilib2 -IC:/Projets/Softs/msys2/mingw64/include/python3.9 -c lib2/Buffer.c -o build/temp.mingw_x86_64-3.9/lib2/Buffer.o
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe -Ilib -Ilib2 -IC:/Projets/Softs/msys2/mingw64/include/python3.9 -c lib2/ThreadDecoder.c -o build/temp.mingw_x86_64-3.9/lib2/ThreadDecoder.o
  lib2/ThreadDecoder.c:36:8: error: unknown type name 'DWORD'
     36 | static DWORD nsec_to_ms(unsigned long nsec) {
        |        ^~~~~
  lib2/ThreadDecoder.c: In function 'nsec_to_ms':
  lib2/ThreadDecoder.c:37:5: error: unknown type name 'DWORD'
     37 |     DWORD t;
        |     ^~~~~
  lib2/ThreadDecoder.c:40:16: error: 'INFINITE' undeclared (first use in this function)
     40 |         return INFINITE;
        |                ^~~~~~~~
  lib2/ThreadDecoder.c:40:16: note: each undeclared identifier is reported only once for each function it appears in
  lib2/ThreadDecoder.c: In function 'ppmd_timedwait':
  lib2/ThreadDecoder.c:54:7: warning: implicit declaration of function 'SleepConditionVariableCS' [-Wimplicit-function-declaration]
     54 |  if (!SleepConditionVariableCS(cond, mutex, nsec_to_ms(nsec))) return ETIMEDOUT;
        |       ^~~~~~~~~~~~~~~~~~~~~~~~
  error: command 'C:\\Projets\\Softs\\msys2\\mingw64\\bin/gcc.exe' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for pyppmd
Failed to build pyppmd
ERROR: Could not build wheels for pyppmd, which is required to install pyproject.toml-based projects

It worked before and started to fail on my MSYS2 CI a few weeks ago.

Access violation on 32bit python on Windows

Describe the bug
Decompression on 32bit python on Windows causes access violation

Related issue
#86

To Reproduce
Run tox test on 32bit python on windows

Expected behavior
It should be no exception

Environment (please complete the following information):

  • OS: Windows Server 2022
  • Python 3.7 x86
  • project version: HEAD on main

Test data(please attach in the report):

tests/test_fuzzer.py::test_ppmd8_fuzzer Windows fatal exception: access violation


GLOB sdist-make: D:\a\pyppmd\pyppmd\setup.py
py37 create: D:\a\pyppmd\pyppmd\.tox\py37
tox: py37
  py37 inst: D:\a\pyppmd\pyppmd\.tox\.tmp\package\1\pyppmd-0.1.dev31.zip
  py37 installed: atomicwrites==1.4.0,attrs==21.4.0,colorama==0.4.4,coverage==6.3.2,hypothesis==6.39.3,importlib-metadata==4.11.2,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,py==1.11.0,py-cpuinfo==8.0.0,pyparsing==3.0.7,pyppmd @ file:///D:/a/pyppmd/pyppmd/.tox/.tmp/package/1/pyppmd-0.1.dev31.zip,pytest==7.0.1,pytest-benchmark==3.4.1,pytest-cov==3.0.0,pytest-timeout==2.1.0,sortedcontainers==2.4.0,tomli==2.0.1,typing_extensions==4.1.1,zipp==3.7.0
  py37 run-test-pre: PYTHONHASHSEED='125'
  py37 run-test: commands[0] | python -m pytest -vv -s
  ============================= test session starts =============================
  platform win32 -- Python 3.7.9, pytest-7.0.1, pluggy-1.0.0 -- D:\a\pyppmd\pyppmd\.tox\py37\Scripts\python.EXE
  cachedir: .tox\py37\.pytest_cache
  hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('D:\\a\\pyppmd\\pyppmd\\.hypothesis\\examples')
  benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
  rootdir: D:\a\pyppmd\pyppmd, configfile: pyproject.toml, testpaths: tests
  plugins: hypothesis-6.39.3, benchmark-3.4.1, cov-3.0.0, timeout-2.1.0
  collecting ... collected 27 items
  
  tests/test_benchmark.py::test_benchmark_text_compress[PPMd H-7-6-16777216] SKIPPED
  tests/test_benchmark.py::test_benchmark_text_compress[PPMd I-8-8-8388608] SKIPPED
  tests/test_benchmark.py::test_benchmark_text_decompress[PPMd H-7-6-16777216] SKIPPED
  tests/test_benchmark.py::test_benchmark_text_decompress[PPMd I-8-8-8388608] SKIPPED
  tests/test_fuzzer.py::test_ppmd7_fuzzer PASSED
  tests/test_fuzzer.py::test_ppmd8_fuzzer Windows fatal exception: access violation
  
  Current thread 0x000010f0 (most recent call first):
    File "D:\a\pyppmd\pyppmd\tests\test_fuzzer.py", line 42 in test_ppmd8_fuzzer
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\core.py", line 575 in test
    File "D:\a\pyppmd\pyppmd\tests\test_fuzzer.py", line 31 in test_ppmd8_fuzzer
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\core.py", line 633 in run
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\executors.py", line 47 in default_new_style_executor
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\core.py", line 637 in execute_once
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\core.py", line 699 in _execute_once_for_engine
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\internal\conjecture\engine.py", line 184 in __stoppable_test_function
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\internal\conjecture\engine.py", line 208 in test_function
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\internal\conjecture\engine.py", line 1055 in cached_test_function
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\internal\conjecture\engine.py", line 809 in generate_mutations_from
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\internal\conjecture\engine.py", line 729 in generate_new_examples
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\internal\conjecture\engine.py", line 876 in _run
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\internal\conjecture\engine.py", line 470 in run
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\core.py", line 776 in run_engine
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\hypothesis\core.py", line 1159 in wrapped_test
    File "D:\a\pyppmd\pyppmd\tests\test_fuzzer.py", line 31 in test_ppmd8_fuzzer
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\python.py", line 192 in pytest_pyfunc_call
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\python.py", line 1718 in runtest
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\runner.py", line 168 in pytest_runtest_call
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\runner.py", line 261 in <lambda>
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\runner.py", line 340 in from_call
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\runner.py", line 261 in call_runtest_hook
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\runner.py", line 221 in call_and_report
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\runner.py", line 132 in runtestprotocol
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\runner.py", line 113 in pytest_runtest_protocol
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\main.py", line 347 in pytest_runtestloop
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\main.py", line 322 in _main
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\main.py", line 268 in wrap_session
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\main.py", line 315 in pytest_cmdline_main
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\config\__init__.py", line 166 in main
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\_pytest\config\__init__.py", line 188 in console_main
    File "D:\a\pyppmd\pyppmd\.tox\py37\lib\site-packages\pytest\__main__.py", line 5 in <module>
    File "c:\hostedtoolcache\windows\python\3.7.9\x86\lib\runpy.py", line 85 in _run_code
    File "c:\hostedtoolcache\windows\python\3.7.9\x86\lib\runpy.py", line 193 in _run_module_as_main

ppmd8dec: Access violation is reported on Windows

Describe the bug
When test ppmd8 decompression on windows, access violation is reported.

Related issue
#32 #33 #34 #44

To Reproduce
CI reported on windows tox test

Environment (please complete the following information):

  • OS: Windows 10
  • Python: CPython 3.6, 3.7, 3.8, 3.9, 3.10
  • project version: main branch[db70d43]

Additional context
This may happened because pthread wrapper for windows seems incomplete, and break synchronization of threads.
By investigation, Ppmd8T_decode() is blocked in line 1 where taking mutex lock, and worker thread is live and write to output buffer that has not configured yet. Status variable indicate worker thread is finished, but it is wrongly live.

Ppmd8_DecodeSymbol should be received by int, not unsigned char. Then stop decoding if <0

Describe the bug

https://github.com/miurahr/pyppmd/blob/v0.15.2/src/ext/_ppmdmodule.c#L1451 Ppmd8_DecodeSymbol should be received by int, not unsigned char

if Ppmd8_DecodeSymbol returns <0, decoding should terminate immediately. and char -1 and int -1 are different (this is similar to that fgetc() should be received by int, not char).

it will help:

  • to identify eof when -1 is returned
  • to identify data error when -2 is returned

Additional context

  • you can determine eof from Ppmd8_DecodeSymbol return-value; do you really need "end marker"?
  • as you will rewrite buffering, I'm glad if you take a look at this.

Additional context 2

Actually I wrote ppmd handler for zipfile but additional end-marker creates incompatible stream.

Fuzzer test report access violation for PPMd8 on Windows

https://github.com/miurahr/pyppmd/pull/33/checks?check_run_id=3268207157

Windows
Python 3.9

  py39 run-test-pre: PYTHONHASHSEED='2'
  py39 run-test: commands[0] | python -m pytest -vv -s14
  ============================= test session starts =============================15
  platform win32 -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- D:\a\pyppmd\pyppmd\.tox\py39\Scripts\python.EXE
  cachedir: .tox\py39\.pytest_cache
  hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('D:\\a\\pyppmd\\pyppmd\\.hypothesis\\examples')
  benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
  rootdir: D:\a\pyppmd\pyppmd, configfile: tox.ini
  plugins: hypothesis-6.14.5, benchmark-3.4.1, cov-2.12.1
  collecting ... collected 22 items
  
  tests/test_benchmark.py::test_benchmark_text_compress[PPMd H-7-6-16777216] SKIPPED
  tests/test_benchmark.py::test_benchmark_text_compress[PPMd I-8-8-8388608] SKIPPED
  tests/test_benchmark.py::test_benchmark_text_decompress[PPMd H-7-6-16777216] SKIPPED
  tests/test_benchmark.py::test_benchmark_text_decompress[PPMd I-8-8-8388608] SKIPPED
  tests/test_fuzzer.py::test_ppmd7_fuzzer PASSED
  tests/test_fuzzer.py::test_ppmd8_fuzzer Windows fatal exception: access violation
  

Windows access violation error

Describe the bug
pytest on windows sometimes produce the error Windows fatal exception: access violation

Related issue

To Reproduce
It happened sometimes, not constant.

Expected behavior
Not happened

Environment (please complete the following information):

  • OS: Window 10
  • Python CPython
  • project version: HEAD of main observed at 5758c3e

Additional context
When adding -p nofaulthander to pytest reduce an error.
It is caused by DLL/extension binary.

consumer-producer problem

Describe the bug

Decompressor become wrong status when memory_size is smaller than file size.

Related issue
#25

To Reproduce

The test has already merged into main branch.

Expected behavior

PyPPMd should handle it correctly.

Environment:

  • OS: no dependency
  • Python: all python versions
  • project version: all released versions, main branch.

Additional context

The root cause is the producer-consumer problem. Now producer use Python buffer protocol that provide a buffer, but consumer expect it can read arbitrary data from buffer.
The solution is to run the library code,consumer, in another thread and stop and wait when input buffer is exhausted.
When returns to caller code by need-input-data status, it go back to user python code and wait a next call, and when input is given, it wake the library code to continue.

Add conda recipe

Is your feature request related to a problem? Please describe.
Working towards making py7zr available on conda.

Describe the solution you'd like
Add a conda recipe.

Describe alternatives you've considered
N.A.

Additional context
N.A.

PPMd8 decompress generates data size just one-byte less than expected

Describe the bug
PPMd8 decompression, with restore_method == CUTOFF, on python 3.8 on Windows sometimes generate one-byte smaller than expected

To Reproduce
https://github.com/miurahr/pyppmd/runs/3322119476?check_suite_focus=true

Environment (please complete the following information):

  • OS: Windows 10, Linux
  • Python: CPython 3.7
  • project version: v0.16.0

Additional context

2021-08-13T12:53:50.1591006Z tests/test_ppmd8.py::test_ppmd8_encode_decode[1048576-1] FAILED
2021-08-13T12:53:50.1591880Z 
2021-08-13T12:53:50.1592431Z ================================== FAILURES ===================================
2021-08-13T12:53:50.1593036Z _____________________ test_ppmd8_encode_decode[1048576-1] _____________________
2021-08-13T12:53:50.1593498Z 
2021-08-13T12:53:50.1594801Z tmp_path = WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-unknown/pytest-0/test_ppmd8_encode_decode_104851')
2021-08-13T12:53:50.1596160Z mem_size = 1048576, restore_method = 1
2021-08-13T12:53:50.1597050Z 
2021-08-13T12:53:50.1598047Z     @pytest.mark.parametrize(
2021-08-13T12:53:50.1598706Z         "mem_size, restore_method",
2021-08-13T12:53:50.1599210Z         [
2021-08-13T12:53:50.1599804Z             (8 << 20, pyppmd.PPMD8_RESTORE_METHOD_RESTART),
2021-08-13T12:53:50.1600803Z             (8 << 20, pyppmd.PPMD8_RESTORE_METHOD_CUT_OFF),
2021-08-13T12:53:50.1601474Z             (1 << 20, pyppmd.PPMD8_RESTORE_METHOD_RESTART),
2021-08-13T12:53:50.1602274Z             (1 << 20, pyppmd.PPMD8_RESTORE_METHOD_CUT_OFF),
2021-08-13T12:53:50.1602817Z         ],
2021-08-13T12:53:50.1603210Z     )
2021-08-13T12:53:50.1603705Z     @pytest.mark.timeout(20)
2021-08-13T12:53:50.1604464Z     def test_ppmd8_encode_decode(tmp_path, mem_size, restore_method):
2021-08-13T12:53:50.1605133Z         length = 0
2021-08-13T12:53:50.1605742Z         m = hashlib.sha256()
2021-08-13T12:53:50.1606433Z         with testdata_path.joinpath("10000SalesRecords.csv").open("rb") as f:
2021-08-13T12:53:50.1607227Z             with tmp_path.joinpath("target.ppmd").open("wb") as target:
2021-08-13T12:53:50.1608142Z                 enc = pyppmd.Ppmd8Encoder(6, mem_size, restore_method=restore_method, endmark=True)
2021-08-13T12:53:50.1608921Z                 data = f.read(READ_BLOCKSIZE)
2021-08-13T12:53:50.1609443Z                 while len(data) > 0:
2021-08-13T12:53:50.1609930Z                     m.update(data)
2021-08-13T12:53:50.1610522Z                     length += len(data)
2021-08-13T12:53:50.1611141Z                     target.write(enc.encode(data))
2021-08-13T12:53:50.1611786Z                     data = f.read(READ_BLOCKSIZE)
2021-08-13T12:53:50.1612414Z                 target.write(enc.flush())
2021-08-13T12:53:50.1613540Z         shash = m.digest()
2021-08-13T12:53:50.1614660Z         m2 = hashlib.sha256()
2021-08-13T12:53:50.1615351Z         assert length == 1237262
2021-08-13T12:53:50.1615937Z         length = 0
2021-08-13T12:53:50.1616543Z         with tmp_path.joinpath("target.ppmd").open("rb") as target:
2021-08-13T12:53:50.1617283Z             with tmp_path.joinpath("target.csv").open("wb") as out:
2021-08-13T12:53:50.1618171Z                 dec = pyppmd.Ppmd8Decoder(6, mem_size, restore_method=restore_method, endmark=True)
2021-08-13T12:53:50.1619007Z                 data = target.read(READ_BLOCKSIZE)
2021-08-13T12:53:50.1619588Z                 while len(data) > 0 or not dec.eof:
2021-08-13T12:53:50.1620140Z                     res = dec.decode(data)
2021-08-13T12:53:50.1620632Z                     m2.update(res)
2021-08-13T12:53:50.1621116Z                     out.write(res)
2021-08-13T12:53:50.1621603Z                     length += len(res)
2021-08-13T12:53:50.1622161Z                     data = target.read(READ_BLOCKSIZE)
2021-08-13T12:53:50.1622711Z >       assert length == 1237262
2021-08-13T12:53:50.1623150Z E       assert 1237261 == 1237262
2021-08-13T12:53:50.1623560Z E         +1237261
2021-08-13T12:53:50.1623925Z E         -1237262
2021-08-13T12:53:50.1624222Z 
2021-08-13T12:53:50.1624726Z tests\test_ppmd8.py:105: AssertionError

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.