adata.write errors about anndata HOT 23 CLOSED

Hi all, i have been using scanpy library for three months now, just discovered that adata.write will give
"ValueError: name already used as a name or title" error if adata.obs contains a column name 'index'.

Just sharing in case anyone else has this problem too.

from anndata.

Hi, for me it has something to do with duplicated column ( or duplication between column and index), you might want to remove that

from anndata.

Attaching a demo dataset with test conversion script.
issue_52_demo.tar.gz

For me, it errors like this:

$ ./demo_load_maggie.py 
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
/opt/Python-3.7.0/lib/python3.7/site-packages/anndata/readwrite/write.py:301: UserWarning: Could not save field with key = "uns/condition_categories" to hdf5 file.
  'to hdf5 file.'.format(key))

from anndata.

Hi, @huidongchen.

Do you write with sparse adata.X or dense?
Do you work in backed or in-memory mode?

from anndata.

Hi ,

sorry about the slow response. I'm using dense matrix and in-memory mode.

It seems that it has something to do with 'uns' annotation. If I feed an anndata object to the unstructured annotation, when i save it, the writing process will get stuck and report error in the end.

I'm attaching a test script here. Hope it will help.

Thanks again.
test.ipynb.zip

from anndata.

Hi,

I'm just following up on adata.write. 'uns' annotation doesn't seemingly work for rpy2 object either. I got the following warning when trying to write adata to .h5ad-formatted hdf5 file. Is a python object required in order to write anndata object to files?

UserWarning: Could not save field with key = "uns/X_r_obj" to hdf5 file.

I will appreciate your help.

from anndata.

Hi!

The line

adata.uns['raw'] = adata.copy()

is invalid. But unfortunately, this doesn't throw an error. The .uns attribute allows for simple data types, numpy arrays, sparse matrices and simple dictionaries.

Use the .raw attribute of AnnData for doing what you want to do

adata.raw = adata

Let me think about whether there is an easy way of testing the type upon assignment.

Alex

from anndata.

well, of course there is, right?

def check_simple_type(obj):
    return any(check(obj) for check in checks)

def check_simple_dict(obj):
    return isinstance(x, dict) and all(
        check_simple_type(o)
        for seq in [x.keys(), x.values()]
        for o in seq
    )

checks = [
    lambda x: isinstance(x, int),
    lambda x: isinstance(x, str),
    ...,
    np.isarray,
    sp.sparse.issparse,
    check_simple_dict,
]

from anndata.

Thanks a lot @Koncopd @falexwolf . It's very helpful. I'm closing the issue.

from anndata.

Yes, of course, in a custom class - but that involves subclassing OrderedDict. It might be that this is not such a trouble as for DataFrames (also for arrays it wasn't trivial, as we saw). But that's the thing to be investigated before wrapping .uns's __setitem__ as you prototyped, right?

from anndata.

Hi, sorry that i have to reopen the issue. I'm having some trouble again when writing anndata object to a file.

I'm using dense matrix and in-memory mode.

When I ran adata.write(results_file), i got the error ''RuntimeError: Unable to create link (message type not found)"

5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5o.pyx in h5py.h5o.link()

RuntimeError: Unable to create link (message type not found)`

But if I run adata.write_loom(results_file), it works well.

from anndata.

I'm sorry but with this amount of information, I cannot say anything.

from anndata.

Hi Alex,

I'm sorry about the limited information. I was running a big dataset so it was a bit difficult to provide an easily reproducible script.

But I guess i finally figured it out. I stored a dictionary into anndata unstructured annotation .uns. One keyword contains a slash symbol '/' , which resulted in the error 'RuntimeError: Unable to create link (message type not found)'`

'repr'+'eval' strategy helps solve this issue.

from anndata.

Ah, thank you for this information! Indeed, HDF5 has problems with slashes. I'll reopen this issue so that we can print out a warning in the future.

from anndata.

I ran into this too:

/opt/Python-3.7.0/lib/python3.7/site-packages/anndata/readwrite/write.py:301: UserWarning: Could not save field with key = "uns/cell_type_categories" to hdf5 file.
  'to hdf5 file.'.format(key))

from anndata.

Mine actually happened because there were non-unicode values in the cells of the cell_type column. Just a note.

from anndata.

Interesting, how come you have non-unicode values? Do you need them? Unicode is already super rich. A previous version of anndata only supported ASCII, which was limiting in some cases...

from anndata.

We absolutely don't need them, no. It happened by chance because user copy/pasted a block from a web site into their input. I corrected this and we were fine.

from anndata.

Can you please give a reproducible example? Python does unicode by default, there should be almost no effort to fix this.

The only reason we don’t have tests for this is that I wouldn’t think it’d ever be a problem.

from anndata.

I get a write error:
Above error raised while writing key 0 of <class 'h5py._hl.group.Group'> from /.
if a name of the index to the var table is an int. Setting the name to a string fixes this.

from anndata.

Yeah, this is not supported. We should improve the error in this case though

from anndata.

I'm going to close this, since new writing problems should open up new issues.

from anndata.

I get a write error:
Above error raised while writing key 0 of <class 'h5py._hl.group.Group'> from /.
if a name of the index to the var table is an int. Setting the name to a string fixes this.

I have the same problem. How did you solve it? Thanks in advance!

from anndata.