Large Data Objects Not Always Saving Correctly about qs HOT 5 CLOSED

There are 4 magic bytes written at the start of every qs file, 0xBEAC.

That error message means that those bytes are not seen. It could be user error, it's hard to see how it could be a bug but I guess it could be ;)

Can you upload one of the files with an error?

Some possibilities on your end:

If you try to qsave to the same file at the same time from different threads, this could happen.

If you forgot to change your function, e.g. you did something like saveRDS(mydata, "myfile.qs")

from qs.

If you try to qsave to the same file at the same time from different threads, this could happen.

I think it might be related to this comment. I will explore further. I will try to make a short script defining some random data that displays this behavior.

Thanks for your speedy response!

from qs.

Hey @RSchwinn, was there any further insight from your exploration? If it's possible to send me one of the broken files, I might be able to figure out what happened.

from qs.

Unfortunately, I cannot share the files. But the good news is that I’ve only been able to reproduce the issue once. I think it occurs when overwriting an existing files that is also being read by another process. So, user error.

Do you have any plans to extend compatibility to Python? QS far outperforms parquet and all other comparable formats, it should be the ZIP for data.

from qs.

Thanks, I do have plans to extend to python, but it's a ways off. I'll have to think very carefully how to do this since data in python and R are quite different.

from qs.