Giter Club home page Giter Club logo

Comments (6)

mdboom avatar mdboom commented on June 16, 2024

I think this is a side effect of often needing to keep the file open for either the update method or memmapping the arrays. If we close the file in either of those cases, it will go "boom". I suspect this is resolvable, but one does end up with behavior that's a little surprising, such as this not working:

with AsdfFile.read('masked.asdf') as f2:
    pass
x = f2.tree['test']

from asdf.

mdboom avatar mdboom commented on June 16, 2024

Actually, are you sure this isn't working? I can't reproduce, and it should (and seemingly does) close the file after each with block is exited.

from asdf.

astrofrog avatar astrofrog commented on June 16, 2024

With the exact script above, I get:

Traceback (most recent call last):
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/numpy/ma/core.py", line 2709, in __new__
    mask = np.array(mask, copy=copy, dtype=mdtype)
TypeError: __array__() takes 1 positional argument but 2 were given

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ttt.py", line 12, in <module>
    np.sum(f2.tree['test'])
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/numpy/core/fromnumeric.py", line 1716, in sum
    sum = a.sum
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/tags/core/ndarray.py", line 309, in __getattr__
    return getattr(self._make_array(), attr)
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/tags/core/ndarray.py", line 223, in _make_array
    self._array = self._apply_mask(self._array, self._mask)
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/tags/core/ndarray.py", line 228, in _apply_mask
    return ma.array(array, mask=mask)
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/numpy/ma/core.py", line 5883, in array
    fill_value=fill_value, ndmin=ndmin, shrink=shrink)
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/numpy/ma/core.py", line 2712, in __new__
    mask = np.array([tuple([m] * len(mdtype)) for m in mask],
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/numpy/ma/core.py", line 2712, in <listcomp>
    mask = np.array([tuple([m] * len(mdtype)) for m in mask],
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/tags/core/ndarray.py", line 312, in __getitem__
    return self._make_array()[item]
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/tags/core/ndarray.py", line 221, in _make_array
    shape, self._dtype, block.data,
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/block.py", line 776, in data
    self.data_offset, self._size)
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/pyasdf-0.0.dev426-py3.4.egg/pyasdf/generic_io.py", line 605, in memmap_array
    self._fd, mode=mode, offset=offset, shape=size)
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/numpy/core/memmap.py", line 270, in __new__
    self.filename = os.path.abspath(filename.name)
  File "/Users/tom/miniconda3/envs/production/lib/python3.4/posixpath.py", line 361, in abspath
    cwd = os.getcwd()
OSError: [Errno 24] Too many open files

This is on MacOS X with Python 3, in case this matters?

from asdf.

mdboom avatar mdboom commented on June 16, 2024

Ok -- I see it now. The file must contain a masked array in order to reproduce. (This was hidden a bit because the example writes to a file called 'test.asdf', but reads a file called 'masked.asdf'). With that, I can reproduce it just about anywhere. I fear this may be a reference bug in np.ma, but I'm not done investigating yet.

from asdf.

mdboom avatar mdboom commented on June 16, 2024

See #117. The issue is not that the context manager is not closing the file -- passing a memmapped array as a mask to a numpy MaskedArray causes the memmap reference to never be freed, thus keeping the file open. Making a view of the mask array seems to resolve this (not quite sure why though).

from asdf.

astrofrog avatar astrofrog commented on June 16, 2024

@mdboom - that was a pretty subtle issue - good catch! :)

from asdf.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.