UnicodeDecodeError when using MsgPackSerializer about aiocache HOT 7 OPEN

By the way, I also tried to use encoding = None when initializing MemcachedCache (or RedisCache) which works, but reading from the cache gives back bytes instead of strings (which is kind of annoying for most use-cases).

from aiocache.

Possibly the same as #515?

If not, then it'd probably be a good idea to create a test in a PR that reproduces the problem in CI.

from aiocache.

Possibly the same as #515?

Yeah, #515 is probably related, but UnicodeDecodeError isn't raised when using PickleSerializer instead of MsgPackSerializer. See for yourself:

import asyncio

from aiocache               import MemcachedCache
from aiocache.serializers   import PickleSerializer

async def test():
    cache = MemcachedCache(
        serializer = PickleSerializer(),
    )

    await cache.set("test_case", { "key1": "test", "key2": 3 })

    test_case = await cache.get("test_case")
    print(test_case)

if __name__ == "__main__":
    asyncio.run(test())

This code works fine.

If not, then it'd probably be a good idea to create a test in a PR that reproduces the problem in CI.

I'll try and add that.

from aiocache.

Acceptance tests for MsgPackSerializer added (#836).

Please let me know if you have any insight on the issue, I'd be glad to help.

from aiocache.

I guess the design currently is that a serialiser should use strings? Though I don't understand how this serialiser got added when it doesn't work with anything... (the original tests don't actually test it with any caches: https://github.com/aio-libs/aiocache/pull/370/files#diff-14d53e5086ad36720b5073972f662da656b46898f2f8bdcdd06bb2fd9c2df9f5R86).

I guess the simple fix for now is to add .encode()/.decode() into the serialiser? So that it works with strings.

Then maybe a better refactoring can be done to avoid the overhead in future, though it's probably best we get #512 done before tackling that (which also requires some other tasks to be complete before that can be done: https://github.com/aio-libs/aiocache/milestone/8).

from aiocache.

I tried adding the following to MsgPackSerializer's dumps/loads methods, but the UnicodeDecodeError is still raised. Is that the workaround you had in mind?

diff --git a/aiocache/serializers/serializers.py b/aiocache/serializers/serializers.py
index 39a67e6..a3fcce9 100644
--- a/aiocache/serializers/serializers.py
+++ b/aiocache/serializers/serializers.py
@@ -184,7 +184,7 @@ class MsgPackSerializer(BaseSerializer):
         :param value: obj
         :returns: bytes
         """
-        return msgpack.dumps(value)
+        return msgpack.dumps(value).decode("utf-8")

     def loads(self, value):
         """
@@ -196,4 +196,4 @@ class MsgPackSerializer(BaseSerializer):
         raw = False if self.encoding == "utf-8" else True
         if value is None:
             return None
-        return msgpack.loads(value, raw=raw, use_list=self.use_list)
+        return msgpack.loads(value.encode("utf-8"), raw=raw, use_list=self.use_list)

from aiocache.

Yeah, that's what I was thinking. Thinking about it more, the problem is probably that we are decoding the raw bytes first, and that is actually in some binary form, so the decode operation won't work. I guess we'd need some way to disable the decoding (or make the decoding the responsibility of the serialiser maybe?). I suspect such changes won't be backwards compatible and will need to go to v1.

from aiocache.