Comments (39)
Tried using benchfella to benchmark the cache.
I was wondering why this 500,000 does not change on a laptop 2 cores or a desktop 4+ cores.
https://github.com/fire/database_cache_benchmark/blob/master/bench/database_bench.exs#L13
## BasicBench
benchmark name iterations average time
cached select 500000 5.12 µs/op
from nebulex.
This is using cockroachdb, but it should be compatible with postgresql.
from nebulex.
@fire thanks for the bench! I had added a simple bench to Nebulex using the same library benchfella, but only for local and distributed adapter, without Ecto, so it is an interesting and complementary bench test to Nebulex.
On other hand, I don't know much about benchfella, at least not in terms of its implementation, so I'm not sure if benchfella is suitable to do load tests, spawning a bunch of processes to hit the cache and then be able to measure not only latency but also throughput (how many writes and reads per second is able to handle). For example, I did some load tests for shards using basho_bench, and that is precisely what I want to be able to do – check out this blog post, the performance tests section.
from nebulex.
Are you familiar with using basho_bench to test ecto (with postgresql / cockroachdb) driver?
from nebulex.
I'm familiar with basho_bench, but the tricky part is to integrate Elixir and Ecto within the project, since basho_bench doesn't works with rebar3 (AFAIK), then you cannot use the rebar3_elixir_compile plugin, and that makes it more difficult. However, I think it would be easy to add rebar3, and I'd also remove unnecessary dependencies, try to prune the project a bit, anyways, we should try to do it.
from nebulex.
There's two parts to my benchmark. A)The ecto schema / ecto caching and B) the phoenix json api.
Only the ecto schema part is used in the benchfella benchmark. So I can delete the phoenix framework, but wish keep it because one of the goals is to either use thrift (elixir-thrift) or phoenix to provide access to the ecto database / caching.
from nebulex.
Absolutely, that's fine, you should keep everything that make sense to you. When I said "try to prune the project a bit" I was referring to basho_bench, because it brings with a lot of dependencies, which are not necessary to do the test. What I'd like to do is try to use basho_bench against Nebulex, I'll try to make it soon.
from nebulex.
Are you using a fork of basho_bench? I could look at it after work hours.
from nebulex.
Yes I use a fork but it is outdated and it hasn't Nebulex included yet, I used that fork to test shards
from nebulex.
Have you looked at basho bench lately? There are compatibility fixes I believe to erlang 20 or at least there's a riak working group now.
from nebulex.
@fire no, but I will, it is a very good news, I'll take a look at it and try to do this load testing soon. Thanks :) !!
from nebulex.
If you had problems, this is the one that has erlang 19 compatibility. https://github.com/mrallen1/basho_bench/blob/mra-rebar3/README.org#quick-start
from nebulex.
Ping!
from nebulex.
Hi @fire, my plan is start work on this issue this week (actually is the next one in my list), and of course I'll start checking out the link you sent me, as soon as I have some result I'll let you know. BTW, have you tried it out already?
from nebulex.
Was able to build it in Elixir and erlang 20, but didn't find time understanding how the drivers worked. That's where I stopped.
from nebulex.
Ok, basically you have to provide two things to basho_bench
:
- As you mentioned, the driver, so in the
src
you can create a module like this:
-module(basho_bench_nebulex).
-export([
new/1,
run/4
]).
-define(nbx_cache, 'Elixir.NbxCache').
new(_State) ->
{ok, #{}}.
run(get, KeyGen, _ValueGen, State) ->
Key = KeyGen(),
_ = ?nbx_cache:get(Key),
{ok, State};
run(put, KeyGen, ValueGen, State) ->
?nbx_cache:set(KeyGen(), ValueGen()),
{ok, State}.
In this driver we are testing two operations, get
and set
, and the run/4
function is executed depending on you config file.
- The config file, so in the folder
examples
we can add out config filebasho_bench_nebulex.config
:
{mode, max}.
{duration, 5}.
{concurrent, 8}.
{driver, basho_bench_driver_shards}.
{operations, [{put, 1}, {get, 1}]}.
{key_generator, {int_to_bin_bigendian, {uniform_int, 1000}}}.
{value_generator, {fixed_bin, 100000}}.
See that {operations, [{put, 1}, {get, 1}]}
match with the run/4
functions within the driver.
Now, the tricky part is, we have to create an elixir app for the Nebulex cache (NbxCache
), and in order to be able to compile nebulex
dep and our Cache NbxCache
within the basho_bench
project we need to use the rebar3_elixir_compile
plugin.
I was able to compile basho_bench
using rebar3 and with OTP 20, but now I'm struggling with the rebar3_elixir_compile
plugin, for some reason the dependencies are not included in the escript so it fails. So I'll continue working on this and I'll let you know any progress. Now, if you can try as well that would be great, and let me know any issue or progress :) !!
from nebulex.
When I do a iex --werl -S mix run inside of https://github.com/fire/database_cache_benchmark I'm able to access the basho_bench functions.
Inside of the elixir shell I have the ability to use both ecto and basho_bench. I'm not sure what do though.
from nebulex.
# Windows BASH or Linux
iex -S mix
:basho_bench.start()
:basho_bench.setup_benchmark([])
:basho_bench.run_benchmark(['./src/basho_bench_nebulex.config'])
Seems to work, but it crashes on {driver, basho_bench_driver_shards}.
from nebulex.
Added https://github.com/fire/database_cache_benchmark/blob/master/src/basho_bench_driver_shards.erl but still failing:
:basho_bench.start()
:basho_bench.setup_benchmark([])
:basho_bench.run_benchmark(['./src/basho_bench_nebulex.config'])
iex(4)> :basho_bench.run_benchmark(['./src/basho_bench_nebulex.config'])
12:39:49.792 [info] No dimension available for key generator: {int_to_bin_bigendian,{uniform_int,1000}}
12:39:49.796 [debug] Supervisor basho_bench_run_sup started basho_bench_duration:start_link() at pid <0.414.0>
12:39:49.845 [info] module=basho_bench_stats_writer_csv event=start stats_sink=csv
12:39:49.846 [debug] Supervisor basho_bench_run_sup started basho_bench_stats:start_link() at pid <0.415.0>
12:39:49.856 [info] Random source: calling crypto:strong_rand_bytes(100663296) (override with the 'value_generator_source_size' config option
12:39:51.926 [info] Random source: finished crypto:strong_rand_bytes(100663296)
12:39:51.931 [error] Failed to initialize driver basho_bench_driver_shards: {'EXIT',{badarg,[{ets,lookup,[xyz,state],[]},{shards_state,get,1,[{file,"/mnt/g/database_cache_benchmark/deps/shards/src/shards_state.erl"},{line,166}]},{basho_bench_driver_shards,new,1,[{file,"src/basho_bench_driver_shards.erl"},{line,9}]},{basho_bench_worker,worker_idle_loop,1,[{file,"/mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench_worker.erl"},{line,229}]}]}}
12:39:51.931 [error] CRASH REPORT Process <0.419.0> with 0 neighbours exited with reason: {init_driver_failed,{'EXIT',{badarg,[{ets,lookup,[xyz,state],[]},{shards_state,get,1,[{file,"/mnt/g/database_cache_benchmark/deps/shards/src/shards_state.erl"},{line,166}]},{basho_bench_driver_shards,new,1,[{file,"src/basho_bench_driver_shards.erl"},{line,9}]},{basho_bench_worker,worker_idle_loop,1,[{file,"/mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench_worker.erl"},{line,229}]}]}}} in basho_bench_worker:init/1 line 152
12:39:51.932 [error] Supervisor basho_bench_worker_sup had child basho_bench_worker_1 started with basho_bench_worker:start_link(basho_bench_worker_1, {single_worker,1,1}, []) at undefined exit with reason {init_driver_failed,{'EXIT',{badarg,[{ets,lookup,[xyz,state],[]},{shards_state,get,1,[{file,"/mnt/g/database_cache_benchmark/deps/shards/src/shards_state.erl"},{line,166}]},{basho_bench_driver_shards,new,1,[{file,"src/basho_bench_driver_shards.erl"},{line,9}]},{basho_bench_worker,worker_idle_loop,1,[{file,"/mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench_worker.erl"},{line,229}]}]}}} in context start_error
** (exit) exited in: :gen_server.call({:global, :basho_bench_stats}, :run)
** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
(stdlib) gen_server.erl:206: :gen_server.call/2
(basho_bench) /mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench.erl:98: :basho_bench.run_benchmark/1
from nebulex.
Loading https://github.com/fire/database_cache_benchmark:
# Windows BASH or Linux
iex -S mix
:basho_bench.start()
:basho_bench.setup_benchmark([])
:basho_bench.run_benchmark(['./src/basho_bench_nebulex.config'])
Will insert and fetch random uuids.
How do you use key_gen and value_gen in def run(:put, key_gen, value_gen, state)
?
from nebulex.
https://github.com/fire/database_cache_benchmark/blob/master/lib/basho_bench/basho_bench_nebulex_driver.ex is the nebulex driver written in Elixir.
from nebulex.
fire/database_cache_benchmark@eca59d2 Has my last work on trying to get uuid_v4 generation.
Reminder:
# Setup Postgresql / Cockroachdb config
:basho_bench.start()
:basho_bench.setup_benchmark([])
:basho_bench.run_benchmark(['./src/basho_bench_nebulex.config'])
from nebulex.
Error log:
iex(4)> :basho_bench.run_benchmark15:10:01.647 [debug] Lager installed handler lager_backend_throttle into lager_event
15:10:01.691 [info] No dimension available for key generator: uuid_v4
15:10:01.696 [debug] Supervisor basho_bench_run_sup started basho_bench_duration:start_link() at pid <0.421.0>
15:10:01.741 [info] module=basho_bench_stats_writer_csv event=start stats_sink=csv
15:10:01.742 [debug] Supervisor basho_bench_run_sup started basho_bench_stats:start_link() at pid <0.422.0>
15:10:01.749 [error] CRASH REPORT Process <0.426.0> with 0 neighbours crashed with reason: no function clause matching basho_bench_worker:init_generators(false, uuid_v4, 1, basho_bench_keygen) line 406
15:10:01.750 [error] Supervisor basho_bench_worker_sup had child basho_bench_worker_1 started with basho_bench_worker:start_link(basho_bench_worker_1, {single_worker,1,1}, []) at undefined exit with reason no function clause matching basho_bench_worker:init_generators(false, uuid_v4, 1, basho_bench_keygen) line 406 in context start_error
15:10:01.750 [error] Supervisor basho_bench_run_sup had child basho_bench_worker_sup started with basho_bench_worker_sup:start_link() at undefined exit with reason {shutdown,{failed_to_start_child,basho_bench_worker_1,{function_clause,[{basho_bench_worker,init_generators,[false,uuid_v4,1,basho_bench_keygen],[{file,"/mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench_worker.erl"},{line,406}]},{basho_bench_worker,add_generators,1,[{file,"/mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench_worker.erl"},{line,399}]},{basho_bench_worker,init,1,[{file,"/mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench_worker.erl"},{line,...}]},...]}}} in context start_error
15:10:01.762 [warning] No data for op: {put,put}
15:10:01.762 [warning] No data for op: {get,get}
15:10:01.763 [info] No Errors.
15:10:01.763 [info] module=basho_bench_stats_writer_csv event=stop stats_sink=csv
** (exit) exited in: :gen_server.call({:global, :basho_bench_stats}, :run)
** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
(stdlib) gen_server.erl:206: :gen_server.call/2
(basho_bench) /mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench.erl:98: :basho_bench.run_benchmark/1
from nebulex.
Hi @fire, well actually you did a good progress, you took other approach (which I like it), instead of add nebulex
and the cache to basho_bench
, you added basho_bench
as dependency of your elixir bench project, so I'll work on that, I'll try to reproduce the error you got, and try to move forward – BTW Thanks :) !!
from nebulex.
@fire good news, I was able to run the basho_bench tests with your repo (database_cache_benchmark), I found some issues:
- This line: https://github.com/fire/database_cache_benchmark/blob/master/src/basho_bench_nebulex.config#L11
Replace {key_generator, uuid_v4}.
by {key_generator, {int_to_bin_bigendian, {uniform_int, 1000}}}.
- There are some errors here: https://github.com/fire/database_cache_benchmark/blob/master/lib/basho_bench/basho_bench_nebulex_driver.ex – change the
ok
by:ok
It would be something like:
defmodule BashoBench.Nebulex.Driver do
def new(_state) do
{:ok, {}}
end
def run(:put, key_gen, value_gen, state) do
changeset = Hello.World.changeset(%Hello.World{}, %{id: key_gen.(), random_number: value_gen.()})
case Hello.CacheableRepo.insert(changeset) do
{:ok, _schema} -> {:ok, state}
{:error, reason} -> {:error, reason, state}
end
end
def run(:get, key_gen, value_gen, state) do
_ = Hello.CacheableRepo.get(Hello.World, key_gen.())
{:ok, state}
end
end
Try that and let me know if it works for you, but I was able to run load tests against Nebulex finally, thanks to your help :) !!! I got with a humble machine (Mac OSX Sierra, Dual Core, 4 GB) about ~150K ops/seg, which is great!!
Well, try to run your test and let me know how it goes!
from nebulex.
Nebulex ran on my machine but I need to generate uuids.
07:16:49.070 [debug] Driver 'Elixir.BashoBench.Nebulex.Driver' crashed: {#{'__exception__' => true,'__struct__' => 'Elixir.Ecto.Query.CastError',message => <<"deps/ecto/lib/ecto/repo/queryable.ex:322: value `<<0, 0, 1, 244>>` cannot be dumped to type :binary_id in query:\n\nfrom w in Hello.World,\n where: w.id == ^<<0, 0, 1, 244>>,\n select: w\n">>,type => binary_id,value => <<0,0,1,244>>},[{'Elixir.Enum','-reduce/3-lists^foldl/2-0-',3,[{file,"lib/enum.ex"},{line,1826}]},{'Elixir.Enum','-map_reduce/3-lists^mapfoldl/2-0-',3,[{file,"lib/enum.ex"},{line,1372}]}]}
from nebulex.
Instructions:
mix ecto.create -r Hello.Repo
mix ecto.migrate -r Hello.Repo
iex -S mix
:basho_bench.start()
:basho_bench.setup_benchmark([])
:basho_bench.run_benchmark(['./src/basho_bench_nebulex.config'])
from nebulex.
I modified the schema so it works with integer keys, but I wanted uuid results.
fire/database_cache_benchmark@735852b
from nebulex.
Results on Windows and my desktop. Do not use this for scientific reference.
{operations, [{put, 1}, {get, 1}]}.
Currently trying:
{operations, [{put, 2}, {get, 8}]}.
from nebulex.
{mode, max}.
{duration, 5}.
{concurrent, 8}.
{driver, 'Elixir.BashoBench.Nebulex.Driver'}.
{operations, [{put, 2}, {get, 8}]}.
{key_generator, {pareto_int, 1000}}.
{value_generator, {fixed_bin, 100000}}.
Results on Windows and my desktop. Do not use this for scientific reference.
from nebulex.
Previous results are wrong, the puts were failing.
from nebulex.
{mode, max}.
{duration, 5}.
{concurrent, 36}.
{driver, 'Elixir.BashoBench.Nebulex.Driver'}.
{operations, [{put, 1}, {get, 1}]}.
{key_generator, {pareto_int, 1000}}.
{value_generator, {uniform_int, 10000}}.
Edit: Note that these are using the on_conflict database functionality.
from nebulex.
These were my results, but only testing the cache, no Ecto (no DB access), I wanted to get some metrics from the Nebulex cache itself first. Keep in mind that you are testing Ecto with Nebulex, which means that for reads you fetch the data drom the DB sometimes, and for writes you always hit both, the cache and the DB.
But well, with your tests we have metrics from only nebulex
and nebulex_ecto
:)
Results (Mac OSX 10.11.4, 2.2 GHz Intel Core i7, RAM 16 GB):
from nebulex.
I recommend we pick a few databases with standard hardware try out nebulex_ecto.
from nebulex.
@fire 100% agreed, that would be great. Meanwhile, I'll create a separated project for these load tests (only for nebulex
) – probably within nebulex_examples
I'll create a new one, e.g.: nebulex_bench
. Then, we can use your project database_cache_benchmark
for nebulex_ecto
load tests.
from nebulex.
I've created a new project to run bench/load tests against Nebulex
: nebulex_bench – the project also contains some initial graphs with results for local cache.
from nebulex.
Interesting results trying to copy ycsb's
Workload B: Read mostly workload
This workload has a 95/5 reads/write mix. Application example: photo tagging; add a tag is an update, but most operations are to read tags.
Hint. It mostly gets cached.
from nebulex.
@fire based on you suggestion (pick a few databases with standard hardware try out nebulex_ecto
), I just opened a new issue on nebulex_ecto in order to continue the load tests but with nebulex_ecto
. So I think this issue can be closed, let me know your thoughts!!
from nebulex.
That's a good suggestion. You can close this issue.
from nebulex.
Related Issues (20)
- Docs on Migrating to v2 from Nebulex.Adapters.Dist.Cluster HOT 5
- `Nebulex.Cache` callbacks mention "Shared Options" section that do not exist
- Support for functions that can set TTL in Decorator similar to Match HOT 9
- Where to use load/2, dump/2 HOT 4
- random :erpc, :timeout with partitioned get HOT 2
- Is there any way to get the size of the cache? HOT 4
- Seeing Nebulex.RPCError during deployments with partitioned adapter HOT 4
- Nebulex.RegistryLookupError HOT 9
- Ambiguity regarding ttl and gc_interval relation. HOT 4
- Replicated adapter syncing during rolling deployment HOT 5
- Bug on Local adapter when using `delete_all` and keys are nested tuples: not a valid match specification HOT 3
- is allocated_memory param global on per node when using Nebulex.Adapters.Partitioned with :shards HOT 2
- Dangling reference on `cache_evict` when using referenced keys HOT 4
- Regression on decorated functions and Elixir 1.16 HOT 3
- Empty arguments list passed to `generate/3` in Elixir 1.16 HOT 3
- `Multilevel` inclusive cache doesn't duplicate entries backwards on `get_all/2` HOT 3
- Fix compatibility with Elixir 1.15 and 1.16 HOT 3
- Support cache refreshing in the background HOT 1
- Having a cache per user ID... HOT 4
- 2.6.2 is missing from tags and releases on github HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nebulex.