Comments (6)
I think this is a side effect of often needing to keep the file open for either the update
method or memmapping the arrays. If we close the file in either of those cases, it will go "boom". I suspect this is resolvable, but one does end up with behavior that's a little surprising, such as this not working:
with AsdfFile.read('masked.asdf') as f2:
pass
x = f2.tree['test']
from asdf.
Actually, are you sure this isn't working? I can't reproduce, and it should (and seemingly does) close the file after each with block is exited.
from asdf.
With the exact script above, I get:
Traceback (most recent call last):
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/numpy/ma/core.py", line 2709, in __new__
mask = np.array(mask, copy=copy, dtype=mdtype)
TypeError: __array__() takes 1 positional argument but 2 were given
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "ttt.py", line 12, in <module>
np.sum(f2.tree['test'])
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/numpy/core/fromnumeric.py", line 1716, in sum
sum = a.sum
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/tags/core/ndarray.py", line 309, in __getattr__
return getattr(self._make_array(), attr)
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/tags/core/ndarray.py", line 223, in _make_array
self._array = self._apply_mask(self._array, self._mask)
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/tags/core/ndarray.py", line 228, in _apply_mask
return ma.array(array, mask=mask)
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/numpy/ma/core.py", line 5883, in array
fill_value=fill_value, ndmin=ndmin, shrink=shrink)
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/numpy/ma/core.py", line 2712, in __new__
mask = np.array([tuple([m] * len(mdtype)) for m in mask],
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/numpy/ma/core.py", line 2712, in <listcomp>
mask = np.array([tuple([m] * len(mdtype)) for m in mask],
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/tags/core/ndarray.py", line 312, in __getitem__
return self._make_array()[item]
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/tags/core/ndarray.py", line 221, in _make_array
shape, self._dtype, block.data,
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/block.py", line 776, in data
self.data_offset, self._size)
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/generic_io.py", line 605, in memmap_array
self._fd, mode=mode, offset=offset, shape=size)
File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/numpy/core/memmap.py", line 270, in __new__
self.filename = os.path.abspath(filename.name)
File "/Users/tom/miniconda3/envs/production/lib/python3.4/posixpath.py", line 361, in abspath
cwd = os.getcwd()
OSError: [Errno 24] Too many open files
This is on MacOS X with Python 3, in case this matters?
from asdf.
Ok -- I see it now. The file must contain a masked array in order to reproduce. (This was hidden a bit because the example writes to a file called 'test.asdf', but reads a file called 'masked.asdf'). With that, I can reproduce it just about anywhere. I fear this may be a reference bug in np.ma
, but I'm not done investigating yet.
from asdf.
See #117. The issue is not that the context manager is not closing the file -- passing a memmapped array as a mask to a numpy MaskedArray causes the memmap reference to never be freed, thus keeping the file open. Making a view of the mask array seems to resolve this (not quite sure why though).
from asdf.
@mdboom - that was a pretty subtle issue - good catch! :)
from asdf.
Related Issues (20)
- asdf odd handling of `namedtuple` that prevents implementing a `Converter`
- ASDF swallows warnings raised by converters HOT 6
- `load_schema` does not resolve local references
- asdf pytest plugin is incompatible with pytest 8.1.dev
- `add_history_entry` validates the entire tree
- Objects that implement `__asdf_traverse__` are expected to have a `_tag`
- Raise a warning type specific to software version mismatches when wrong version of packages is installed
- Add schema manifest providing package version to asdf metadata and check schema packages on read
- Update `setuptools_scm` `write_to`
- Retire the "stable" branch HOT 4
- Combine package and build workflows
- masked arrays do not roundtrip with all false masks
- `AsdfSpec` misses expected match
- deprecate `AsdfSpec` and `format_tag`
- Tracking `sunpy` 6.0 and ASDF 1.6.0 HOT 1
- Old (<2.14) versions of asdf do not fully support ASDF standard 1.6.0
- `AsdfFile` instances are not pickleable HOT 1
- Chunking support HOT 2
- Investigate enabling `validate_checksum` as default `True`
- Investigate returning `ndarray` when `lazy_load=False` HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from asdf.