Fulfil the open-source checklist about nebulex HOT 10 CLOSED

A quick comparison with libraries like https://github.com/zackehh/cachex ?

from nebulex.

@cohawk I just added a simple benchmarks, using benchfella, but those are very simple/naive tests, but I think at least it is a good starting point. These tests runs either for the local generational adapter and the distributed adapter, spawning three nodes, and setting up 1000 cache entries (this applies either for the local and distributed cache). My goal is to do some load/stress tests as well, in order to have a better and more real idea about the throughput and latency under heavy load, but I'll open a separated issue for that.

Meanwhile you can run the simple benchmarks:

mix nebulex.bench

Overall, IMHO I think results looks pretty pretty good :), but as it is recommended: "you should do and run your own performance and/or load tests". Please try to run the bench and let me know your thoughts!

On other hand, remember one of the main features of Nebulex is its extensibility, you are able to add your own adapters, for example an adapter for Redis, or an adapter for Cachex in order to have other option for local cache and use the distributed adapter on top of it, and actually this adapter is in the roadmap, the idea is not to compete with Cachex, the idea is they can complement each other. Anyways, this is where the @princemaple suggestion comes in – blog post coming!

from nebulex.

Is cachex distributed though? I'd like to see (or plan to setup) some performance benchmarks.
Might be able to remove my Redis dependency.

edit: you should also post on https://elixirforum.com. I found Nebulex from the Elixir Radar newletter

from nebulex.

Is cachex distributed though?

exactly. comparison :)

from nebulex.

@princemaple yes, good point, I'll include this as part of the open source list (maybe write a blog post about it).

@cohawk performance benchmark is actually one the requirements (highest priority) before to launch the v1.0.0, and I think it will come very very soon! And it should be included in this list as well, thanks! BTW, I'll post it on elixir forum as well :) !!

from nebulex.

@princemaple Cachex no longer support distributed cache

Caches issue relared

from nebulex.

@cabol I'm curious why an issue was never opened on the Cachex repository to introduce your ideas in the existing codebase, rather than jumping to starting new project? Especially seeing reimplementation of existing Cachex features inside this project (hooks, stats, etc). I still have intentions of bringing distribution back as a first class feature in Cachex, and I'm curious whether we can merge ideas somehow to maintain the Cachex feature set whilst providing the distribution aspects.

@WolfDan & @cohawk: to clarify on Cachex's "support" a little; distribution is only no longer a first class feature. If you were trying to replace Redis (or whatever), you could just have a master node in your cluster which hosts a Cachex cache in memory and have the other nodes RPC over to it to retrieve their data - that's not something which has to be explicitly "supported", so to speak.

from nebulex.

@whitfin first of all thanks for your thoughts!

I'm curious why an issue was never opened on the Cachex repository to introduce your ideas in the existing codebase, rather than jumping to starting new project?

That is a good question, my plan with Nebulex wasn't only to provide another Cache, but provide a Caching framework or wrapper that allows you to implement different types of Cache, different eviction algorithms, caching topologies, etc. For that reason, I took Ecto as inspiration, the API and its pluggable design based on adapters. Based on these ideas:

1. It would have been very disruptive to build that such of features on top of an existing Cache library like Cachex (widely used), perhaps it would have meant to refactor the existing API, implementation, etc., and maybe for the current users it wouldn't have been too good (IMHO).
2. As far as I know and I saw from Cachex (ofcourse maybe I'm wrong), it was implemented as a local LRU cache, so it is attached to that particular implementation, which is good, but in my case I wanted provide something more flexible, especially for eviction algorithms, cache topologies, a different alternative for local cache (generational cache – and more in the roadmap), etc.
3. When I started Nebulex, my intention never was to create other cache to compete with Cachex or ConnCache, totally opposite, for me they can complement each other, for instance, in the roadmap, I have in mind to implement an adapter for Cachex, in order to be able to use Cachex from Nebulex, and on top of this to use the distributed part, that would be great.

Especially seeing reimplementation of existing Cachex features inside this project (hooks, stats, etc)

In terms of additional and/or complementary features (Nice-To-Have features) like hooks, stats, etc., you are right, these things are very similar. But, in terms of core features in a cache, like backend, eviction, generational cache, distribution, topologies, etc., Nebulex is not re-implementing Cachex at all, in that sense I think Nebulex is different.

I still have intentions of bringing distribution back as a first class feature in Cachex, and I'm curious whether we can merge ideas somehow to maintain the Cachex feature set whilst providing the distribution aspects.

That sounds great, I'm open to discuss this alternative in details. As I mentioned before, one of the ideas I have in mind, is to implement an adapter for Cachex, for me this can be a very nice feature, because we would be combining the best of both, Cachex as a local cache backend plus the distribution features of Nebulex, be able to implement the same caching topologies but having Cachex as a backend. Of course, if you have other ideas we can discuss them, and l'll glad to help :) !!

Let's try to have a more detailed talk so we can cross our ideas and maybe come up with something nicer!!!

Thanks!

from nebulex.

@cabol

perhaps it would have meant to refactor the existing API

That's fair (and it would have).

As far as I know and I saw from Cachex (ofcourse maybe I'm wrong), it was implemented as a local LRU cache, so it is attached to that particular implementation, which is good, but in my case I wanted provide something more flexible, especially for eviction algorithms, cache topologies, a different alternative for local cache (generational cache – and more in the roadmap), etc.

Cachex is a little more than LRU (in fact, technically it's not LRU at all). Eviction is driven by expiration on keys, but size bounds can be provided using policies (Cachex.Policy) to enable implementing your own algorithms.

When I started Nebulex, my intention never was to create other cache to compete with Cachex or ConnCache, totally opposite, for me they can complement each other, for instance, in the roadmap, I have in mind to implement an adapter for Cachex, in order to be able to use Cachex from Nebulex, and on top of this to use the distributed part, that would be great.

Sure, and even if that was your intention, competition isn't a bad thing :) I'd be interested in assisting you writing an adapter, if it comes to it.

Of course, if you have other ideas we can discuss them, and l'll glad to help :) !!

The next step with Cachex now that v3 is out of the way, is actually a v3.1 which uses a sharding algorithm to shard across nodes in a cluster. This would be opt in and totally up to the user to control their node connections (since that's too much outside of the scope of Cachex), but essentially you would pass nodes: [ :a, :b, :c ] or whatever at startup and your data would shard automatically.

It'll be a basic algorithm which controls what goes where, no load balancing etc, but it provides the basic interaction a Cachex cache would need in a distributed state. Perhaps this is where we could focus the Cachex adapter pieces for Nebulex, as it can control the node management and then delegate the caching to Cachex.

from nebulex.

@whitfin I like the idea, actually Cachex might be the default cache backend, I like that, I'll start thinking on it, the simplest approach would be start writing the adapter mapping its callbacks to Cachex functions, but I'd like to review Cachex more in details because it has a lot of features we cannot afford to loose, so this mapping should be designed very carefully.

Of course, the other option is to use the Cachex API, but I think it migh be a bit more tricky, in that case it mighr be better to take the distributed part of Nebulex and make the necessary fixes to integrate it within Cachex, or maybe in a separated repo. Anyways, I think we should consider to implement both, just to give more flexibility to the users, using the Cachex or the Nebulex API.

So, I'll start working on the Cachex adapter for Nebulex and of course I'll be requesting your assistance :) – BTW let me know if you have something in mind already or some work already done that we can reuse as base.

Thanks and let's keep in touch!!

from nebulex.