Missing API implementations for working with Sets about cachemanager HOT 5 CLOSED

The intention was to use CacheManager for caching only. And therefore you don't really need more than key/values.
Also, Redis is the only distributed store which has those advanced features and the common interface from CacheManager provides only the subset which is supported by all of them.

I don't want an interface which gets implemented only partially or where other handles throw "NotSupported" exceptions!

That being said, if you have an idea of how we could extend the functionality for Redis only without breaking above mentioned pattern, let me know :)

from cachemanager.

I wonder, could we have interceptors for set and get calls, to specific backends? This would be an optional hook that could be added to the pipeline, which would be application specific (as each application using this library could write its own implementation; If it became commonly used it could be thrown into the library)

I actually have a work requirement for this, and am going to have to add it myself if it's not already in this project. Here's my scenario:

I will be using the first level cache as an in-memory cache, the second level cache as Redis, and using Redis as a backplane between two servers in a webfarm. We have an in-house ORM, which is essentially a glorified SQL constructor. Before sending off the query to the database, I'd like to modify it and only request the records I don't have in cache.

This is my desired code for the cache checker:

public class CacheChecker
{

    public ICacheManager Cache {get;set;}

    private HashSet<T> GetIdTrackerForType<T>(){

        Type type = typeof(T);

        return (HashSet<T>)Cache.Get(@"{type.AssemblyQualifiedName}--IdsCached");
    }

    public IEnumerable<T> RecordsAreInCache<T>(IEnumerable<T> recordIds) {
        var recordsInCache = new List<T>();

        Type type = typeof(T);

        var typeName = type.AssemblyQualifiedName;


        var idsCached = GetIdTrackerForType<T>();

        foreach (var id in recordIds) {
            if (idsCached.Contains(id)){
                recordsInCache.Add(id);
            }
        }

        return recordsInCache;
    }

}

The way I understand CacheManager, each HashSet sent to Redis will be serialized using either JSON or Protobuf. However, if I am using the HashSet to keep track of what Ids I have in cache, I may have 100, 1000, or 10,000 entries in the set. Serialization on each change, is going to be a performance hit. I could do this without a HashSet, but then I'll be engineering a system where I'm assuming to get cache misses. I would much rather be able to use a HashSet, and then for Redis, use the built in Set data type.

If we could have an API similar to this, we could solve this problem:

// My understanding of the workings of CacheManager are not the best, but I think we'd do something along these lines:
public interface ICacheTypeSerializer
{
    bool CanHandle<TCacheValue>(BaseCacheHandle<TCacheValue> cacheBackend);

    TCacheValue Get<TCacheValue>(String key);
    TCacheValue Put<TCacheValue>(String key, TCacheValue v);
    TCacheValue Update<TCacheValue>(String key, TCacheValue v);
    TCacheValue Add<TCacheValue>(String key, TCacheValue v);
}

// Partial implementation of the above interface just to demonstrate the point:
public class RedisHashSetSerializer : ICacheTypeSerializer {

    public bool CanHandle<TCacheValue>(BaseCacheHandle<TCacheValue> cacheBackend){
        if (typeof(TCacheValue) is HashSet<string> && 
            cacheBackend is RedisCacheBackend<TCacheValue>) {
            return true;    
        }
        return false;
    }

        // Update, etc. could use StackExchange.Redis commands for Redis Sets.
}

We would define a new class which would be used for each type and back-end that we wanted to provide custom serialization for.

Then, when doing Update, Put, Get, Set etc, the CacheManager could check if any ICacheTypeSerializer, has been registered for that particular type, and particular cache.

using System.Collections.Concurrent;
using System.Collections.Generic;


class BaseCacheManager
{
    private ConcurrentDictionary<Tuple<Type, Type>, ICacheTypeSerializer> _cacheTypeSerializersForTypes;
    private IEnumerable<ICacheTypeSerializer> _cacheTypeSerializers;

    // Efficiently retrieve the correct serializer
    private ICacheTypeSerializer GetSerializerForCacheHandle<TCacheValue>(BaseCacheHandle<TCacheValue> handle)
    {
        var typeTuple = new Tuple<Type, Type>(typeof (TCacheValue), handle.GetType);
        ICacheTypeSerializer serializer = null;
        if (_cacheTypeSerializersForTypes.TryGetValue(typeTuple, serializer))
        {
            return serializer;
        }
        else
        {
            foreach (var s in _cacheTypeSerializers)
            {
                if (s.CanHandle(handle))
                {
                    serializer = s;
                    break;
                }
            }
            _cacheTypeSerializersForTypes.TryAdd(typeTuple, serializer);
        }

        return serializer;
    }
}

My implementation might be off, but I feel like this would add more flexibility for this library. Other custom types could be serialized as well (for example, using the Hash type with Redis, to efficiently update values of large dictionaries, etc.)

from cachemanager.

Yes you are right, serializing the whole list every time to update just one item would be problematic.

In those cases you should maybe store each item of the list separately (in a region maybe) with a unique key and not as one single (large) object.

I was thinking about adding some support for lists but couldn't yet figure out a simple/good concept

Tbh, I don't really get how your idea would work/solve the issue.
How would the client side (CacheManager) know which items of your HashSet are already stored in the Redis list for example?

I think it also has not really something to-do with serialization

from cachemanager.

I was thinking about that as well. This is my current implementation:

• All entries of a given type, are stored with a key along the lines of {NameOfType}:Id:{Id}
• There is a HashSet stored as well, with the key {NameOfType}:RecordsInCache

  When I need to pull 1, 2, and 3 (for example) from the set, I first check {NameOfType}:RecordsInCache. Really, this library lacks a Has function, and I'm trying to come up with a way to overcome that lack.

  Is there a reason that that isn't implemented? Seems to me like Has could be implemented by doing something very similar to what I'm doing, but all you'd need would be a HashSet<String> (since everything is stored by a key after all). Add, Put, Remove operations would just need to modify this single HashSet. Perhaps an ImmutableHashSet for concurrency issues?

from cachemanager.

Closing this off as Exists is now in the API for some time and there is another discussion around having an equivalent to Redis's Keys or Scan operation integrated into CacheManager, see #161

from cachemanager.