Comments (6)
From version https://github.com/openpreserve/jpylyzer/releases/tag/1.15.0 onward jpylyzer uses memory mapping. This doesn't solve this issue completely, as memory errors may still occur (also depending on hardware/OS setup).
from jpylyzer.
Especially 32 bit operating systems (Linux / Windows in i386 mode) cannot mmap more than 4 GiB (minus system and program memory, so the real limit might be lower than 2 GiB, on Windows even lower than 1 GiB). For those systems, tests and code fixes to handle mmap errors gracefully will still be needed. These tests can be done with any kind of file (no JP2 needed), for example with DVD images or video files, because only the file size matters.
On 64 bit operating systems, even very large files (much larger than RAM size) should work thanks to mmap.
from jpylyzer.
On 64 bit operating systems, even very large files (much larger than RAM size) should work thanks to mmap.
I tried this, but I couldn't get a 6.7 GB image working under 64-biy Linux Mint with 4 GB RAM. See also the comments I made here:
http://jpylyzer.openpreservation.org/2015/12/03/Release-of-jpylyzer-1-15-1/
However my impression is that things only go wrong after the image data are read/mapped, as in the case of the 6.7 GB image jpylyzer would spit out an "unknown box" warning before the error occurred. That might also explain why a test with e.g. a large DVD image doesn't result in the error, because jpylyzer cannot really parse those file for much beyond the start of the file anyway (unlike a JP2).
from jpylyzer.
OK, so I think I found a way to make jpylyzer handle any memory errors gracefully without crashing. Coming up shortly!
from jpylyzer.
First stab here:
https://github.com/openpreserve/jpylyzer/tree/improvedExceptionHandling
- In checkOneFile I created a handler for any exceptions that may occur while reading/analysing a file
- Added a statusInfo element to the output which contains a Boolean success field, and (in case an exception occurred) a failureMessage with a user-friendly message of what went wrong
- In addition a warning is printed to screen for each exception.
This should prevent any crashes in case of memory and I/O errors. If multiple files are processed at once, jpylyzer will keep running in case of an exception.
Also updated the XSD schema accordingly a (using 1-1 version number, so previously created jpylyzer output will still validate); published schema at http://jpylyzer.openpreservation.org/jpylyzer-v-1-1.xsd
Two examples of output files here:
https://gist.github.com/bitsgalore/a76a64804f772a569299
This needs some further testing (also under Windows).
from jpylyzer.
Changes added to 1.16.0 release: https://github.com/openpreserve/jpylyzer/releases/tag/1.16.0
from jpylyzer.
Related Issues (20)
- foundExpectedNumberOfTileParts clarification HOT 2
- Feature Request: validate RSIZ profile: cinema and IMF HOT 1
- Add documentation + update XSD schema for PLM, PLT marker output HOT 3
- E ModuleNotFoundError: No module named 'six' HOT 4
- Finalize/review unit tests that use jpylyzer test corpus
- Refactor findFiles function HOT 1
- Remove support for Python 2.7
- Show warnings for deprecated features HOT 1
- Remove --wrapper and --legacyout options HOT 1
- Unexpected behaviour running tests on openjpeg-data HOT 1
- Feature request: validate TLM marker HOT 1
- Example of corrupt PLT markers HOT 1
- feature request : abbreviate </plt> marker output for large images HOT 8
- jpylyzer failing on compressed CMYK file with valid enumcs value of 12 HOT 2
- Garbled sot info HOT 2
- Finalize 2.1 release
- Hard-coded path in test_test.py (PY27 support and Travis questions) HOT 3
- Travis CI fails on pre-commit (Py 3.5 build) HOT 2
- JPEG 2000 files downloaded from USGS are marked as invalid HOT 5
- Travis fails on pre-commit command HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jpylyzer.