Load/Stress Tests about nebulex HOT 39 CLOSED

Tried using benchfella to benchmark the cache.

I was wondering why this 500,000 does not change on a laptop 2 cores or a desktop 4+ cores.

https://github.com/fire/database_cache_benchmark/blob/master/bench/database_bench.exs#L13

## BasicBench
benchmark name iterations   average time
cached select      500000   5.12 µs/op

from nebulex.

This is using cockroachdb, but it should be compatible with postgresql.

from nebulex.

@fire thanks for the bench! I had added a simple bench to Nebulex using the same library benchfella, but only for local and distributed adapter, without Ecto, so it is an interesting and complementary bench test to Nebulex.

On other hand, I don't know much about benchfella, at least not in terms of its implementation, so I'm not sure if benchfella is suitable to do load tests, spawning a bunch of processes to hit the cache and then be able to measure not only latency but also throughput (how many writes and reads per second is able to handle). For example, I did some load tests for shards using basho_bench, and that is precisely what I want to be able to do – check out this blog post, the performance tests section.

from nebulex.

Are you familiar with using basho_bench to test ecto (with postgresql / cockroachdb) driver?

from nebulex.

I'm familiar with basho_bench, but the tricky part is to integrate Elixir and Ecto within the project, since basho_bench doesn't works with rebar3 (AFAIK), then you cannot use the rebar3_elixir_compile plugin, and that makes it more difficult. However, I think it would be easy to add rebar3, and I'd also remove unnecessary dependencies, try to prune the project a bit, anyways, we should try to do it.

from nebulex.

There's two parts to my benchmark. A)The ecto schema / ecto caching and B) the phoenix json api.

Only the ecto schema part is used in the benchfella benchmark. So I can delete the phoenix framework, but wish keep it because one of the goals is to either use thrift (elixir-thrift) or phoenix to provide access to the ecto database / caching.

from nebulex.

Absolutely, that's fine, you should keep everything that make sense to you. When I said "try to prune the project a bit" I was referring to basho_bench, because it brings with a lot of dependencies, which are not necessary to do the test. What I'd like to do is try to use basho_bench against Nebulex, I'll try to make it soon.

from nebulex.

Are you using a fork of basho_bench? I could look at it after work hours.

from nebulex.

Yes I use a fork but it is outdated and it hasn't Nebulex included yet, I used that fork to test shards

from nebulex.

Have you looked at basho bench lately? There are compatibility fixes I believe to erlang 20 or at least there's a riak working group now.

from nebulex.

@fire no, but I will, it is a very good news, I'll take a look at it and try to do this load testing soon. Thanks :) !!

from nebulex.

If you had problems, this is the one that has erlang 19 compatibility. https://github.com/mrallen1/basho_bench/blob/mra-rebar3/README.org#quick-start

from nebulex.

Ping!

from nebulex.

Hi @fire, my plan is start work on this issue this week (actually is the next one in my list), and of course I'll start checking out the link you sent me, as soon as I have some result I'll let you know. BTW, have you tried it out already?

from nebulex.

Was able to build it in Elixir and erlang 20, but didn't find time understanding how the drivers worked. That's where I stopped.

from nebulex.

Ok, basically you have to provide two things to basho_bench:

1. As you mentioned, the driver, so in the src you can create a module like this:

-module(basho_bench_nebulex).

-export([
  new/1,
  run/4
]).

-define(nbx_cache, 'Elixir.NbxCache').

new(_State) ->
  {ok, #{}}.

run(get, KeyGen, _ValueGen, State) ->
  Key = KeyGen(),
  _ = ?nbx_cache:get(Key),
  {ok, State};

run(put, KeyGen, ValueGen, State) ->
  ?nbx_cache:set(KeyGen(), ValueGen()),
  {ok, State}.

In this driver we are testing two operations, get and set, and the run/4 function is executed depending on you config file.

2. The config file, so in the folder examples we can add out config file basho_bench_nebulex.config:

{mode, max}.

{duration, 5}.

{concurrent, 8}.

{driver, basho_bench_driver_shards}.

{operations, [{put, 1}, {get, 1}]}.

{key_generator, {int_to_bin_bigendian, {uniform_int, 1000}}}.

{value_generator, {fixed_bin, 100000}}.

See that {operations, [{put, 1}, {get, 1}]} match with the run/4 functions within the driver.

Now, the tricky part is, we have to create an elixir app for the Nebulex cache (NbxCache), and in order to be able to compile nebulex dep and our Cache NbxCache within the basho_bench project we need to use the rebar3_elixir_compile plugin.

I was able to compile basho_bench using rebar3 and with OTP 20, but now I'm struggling with the rebar3_elixir_compile plugin, for some reason the dependencies are not included in the escript so it fails. So I'll continue working on this and I'll let you know any progress. Now, if you can try as well that would be great, and let me know any issue or progress :) !!

from nebulex.

When I do a iex --werl -S mix run inside of https://github.com/fire/database_cache_benchmark I'm able to access the basho_bench functions.

Inside of the elixir shell I have the ability to use both ecto and basho_bench. I'm not sure what do though.

from nebulex.

# Windows BASH or Linux
iex -S mix
:basho_bench.start()
:basho_bench.setup_benchmark([])
:basho_bench.run_benchmark(['./src/basho_bench_nebulex.config'])

Seems to work, but it crashes on {driver, basho_bench_driver_shards}.

from nebulex.

Added https://github.com/fire/database_cache_benchmark/blob/master/src/basho_bench_driver_shards.erl but still failing:

:basho_bench.start()
:basho_bench.setup_benchmark([])
:basho_bench.run_benchmark(['./src/basho_bench_nebulex.config'])
iex(4)> :basho_bench.run_benchmark(['./src/basho_bench_nebulex.config'])
12:39:49.792 [info] No dimension available for key generator: {int_to_bin_bigendian,{uniform_int,1000}}
12:39:49.796 [debug] Supervisor basho_bench_run_sup started basho_bench_duration:start_link() at pid <0.414.0>
12:39:49.845 [info] module=basho_bench_stats_writer_csv event=start stats_sink=csv
12:39:49.846 [debug] Supervisor basho_bench_run_sup started basho_bench_stats:start_link() at pid <0.415.0>
12:39:49.856 [info] Random source: calling crypto:strong_rand_bytes(100663296) (override with the 'value_generator_source_size' config option
12:39:51.926 [info] Random source: finished crypto:strong_rand_bytes(100663296)
12:39:51.931 [error] Failed to initialize driver basho_bench_driver_shards: {'EXIT',{badarg,[{ets,lookup,[xyz,state],[]},{shards_state,get,1,[{file,"/mnt/g/database_cache_benchmark/deps/shards/src/shards_state.erl"},{line,166}]},{basho_bench_driver_shards,new,1,[{file,"src/basho_bench_driver_shards.erl"},{line,9}]},{basho_bench_worker,worker_idle_loop,1,[{file,"/mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench_worker.erl"},{line,229}]}]}}
12:39:51.931 [error] CRASH REPORT Process <0.419.0> with 0 neighbours exited with reason: {init_driver_failed,{'EXIT',{badarg,[{ets,lookup,[xyz,state],[]},{shards_state,get,1,[{file,"/mnt/g/database_cache_benchmark/deps/shards/src/shards_state.erl"},{line,166}]},{basho_bench_driver_shards,new,1,[{file,"src/basho_bench_driver_shards.erl"},{line,9}]},{basho_bench_worker,worker_idle_loop,1,[{file,"/mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench_worker.erl"},{line,229}]}]}}} in basho_bench_worker:init/1 line 152
12:39:51.932 [error] Supervisor basho_bench_worker_sup had child basho_bench_worker_1 started with basho_bench_worker:start_link(basho_bench_worker_1, {single_worker,1,1}, []) at undefined exit with reason {init_driver_failed,{'EXIT',{badarg,[{ets,lookup,[xyz,state],[]},{shards_state,get,1,[{file,"/mnt/g/database_cache_benchmark/deps/shards/src/shards_state.erl"},{line,166}]},{basho_bench_driver_shards,new,1,[{file,"src/basho_bench_driver_shards.erl"},{line,9}]},{basho_bench_worker,worker_idle_loop,1,[{file,"/mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench_worker.erl"},{line,229}]}]}}} in context start_error
** (exit) exited in: :gen_server.call({:global, :basho_bench_stats}, :run)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (stdlib) gen_server.erl:206: :gen_server.call/2
    (basho_bench) /mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench.erl:98: :basho_bench.run_benchmark/1

from nebulex.

Loading https://github.com/fire/database_cache_benchmark:

# Windows BASH or Linux
iex -S mix
:basho_bench.start()
:basho_bench.setup_benchmark([])
:basho_bench.run_benchmark(['./src/basho_bench_nebulex.config'])

Will insert and fetch random uuids.

How do you use key_gen and value_gen in def run(:put, key_gen, value_gen, state) ?

from nebulex.

https://github.com/fire/database_cache_benchmark/blob/master/lib/basho_bench/basho_bench_nebulex_driver.ex is the nebulex driver written in Elixir.

from nebulex.

fire/database_cache_benchmark@eca59d2 Has my last work on trying to get uuid_v4 generation.

Reminder:

# Setup Postgresql / Cockroachdb config
:basho_bench.start()
:basho_bench.setup_benchmark([])
:basho_bench.run_benchmark(['./src/basho_bench_nebulex.config'])

from nebulex.

Error log:

iex(4)> :basho_bench.run_benchmark15:10:01.647 [debug] Lager installed handler lager_backend_throttle into lager_event

15:10:01.691 [info] No dimension available for key generator: uuid_v4
15:10:01.696 [debug] Supervisor basho_bench_run_sup started basho_bench_duration:start_link() at pid <0.421.0>
15:10:01.741 [info] module=basho_bench_stats_writer_csv event=start stats_sink=csv
15:10:01.742 [debug] Supervisor basho_bench_run_sup started basho_bench_stats:start_link() at pid <0.422.0>
15:10:01.749 [error] CRASH REPORT Process <0.426.0> with 0 neighbours crashed with reason: no function clause matching basho_bench_worker:init_generators(false, uuid_v4, 1, basho_bench_keygen) line 406
15:10:01.750 [error] Supervisor basho_bench_worker_sup had child basho_bench_worker_1 started with basho_bench_worker:start_link(basho_bench_worker_1, {single_worker,1,1}, []) at undefined exit with reason no function clause matching basho_bench_worker:init_generators(false, uuid_v4, 1, basho_bench_keygen) line 406 in context start_error
15:10:01.750 [error] Supervisor basho_bench_run_sup had child basho_bench_worker_sup started with basho_bench_worker_sup:start_link() at undefined exit with reason {shutdown,{failed_to_start_child,basho_bench_worker_1,{function_clause,[{basho_bench_worker,init_generators,[false,uuid_v4,1,basho_bench_keygen],[{file,"/mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench_worker.erl"},{line,406}]},{basho_bench_worker,add_generators,1,[{file,"/mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench_worker.erl"},{line,399}]},{basho_bench_worker,init,1,[{file,"/mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench_worker.erl"},{line,...}]},...]}}} in context start_error
15:10:01.762 [warning] No data for op: {put,put}
15:10:01.762 [warning] No data for op: {get,get}
15:10:01.763 [info] No Errors.
15:10:01.763 [info] module=basho_bench_stats_writer_csv event=stop stats_sink=csv
** (exit) exited in: :gen_server.call({:global, :basho_bench_stats}, :run)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (stdlib) gen_server.erl:206: :gen_server.call/2
    (basho_bench) /mnt/g/database_cache_benchmark/deps/basho_bench/src/basho_bench.erl:98: :basho_bench.run_benchmark/1

from nebulex.

Hi @fire, well actually you did a good progress, you took other approach (which I like it), instead of add nebulex and the cache to basho_bench, you added basho_bench as dependency of your elixir bench project, so I'll work on that, I'll try to reproduce the error you got, and try to move forward – BTW Thanks :) !!

from nebulex.

@fire good news, I was able to run the basho_bench tests with your repo (database_cache_benchmark), I found some issues:

1. This line: https://github.com/fire/database_cache_benchmark/blob/master/src/basho_bench_nebulex.config#L11

Replace {key_generator, uuid_v4}. by {key_generator, {int_to_bin_bigendian, {uniform_int, 1000}}}.

2. There are some errors here: https://github.com/fire/database_cache_benchmark/blob/master/lib/basho_bench/basho_bench_nebulex_driver.ex – change the ok by :ok

It would be something like:

defmodule BashoBench.Nebulex.Driver do
  def new(_state) do
    {:ok, {}}
  end

  def run(:put, key_gen, value_gen, state) do
    changeset = Hello.World.changeset(%Hello.World{}, %{id: key_gen.(), random_number: value_gen.()})
    case Hello.CacheableRepo.insert(changeset) do
      {:ok, _schema} -> {:ok, state}
      {:error, reason} -> {:error, reason, state}
    end
  end

  def run(:get, key_gen, value_gen, state) do
   _ = Hello.CacheableRepo.get(Hello.World, key_gen.())
    {:ok, state}
  end
end

Try that and let me know if it works for you, but I was able to run load tests against Nebulex finally, thanks to your help :) !!! I got with a humble machine (Mac OSX Sierra, Dual Core, 4 GB) about ~150K ops/seg, which is great!!

Well, try to run your test and let me know how it goes!

from nebulex.

Nebulex ran on my machine but I need to generate uuids.

07:16:49.070 [debug] Driver 'Elixir.BashoBench.Nebulex.Driver' crashed: {#{'__exception__' => true,'__struct__' => 'Elixir.Ecto.Query.CastError',message => <<"deps/ecto/lib/ecto/repo/queryable.ex:322: value `<<0, 0, 1, 244>>` cannot be dumped to type :binary_id in query:\n\nfrom w in Hello.World,\n  where: w.id == ^<<0, 0, 1, 244>>,\n  select: w\n">>,type => binary_id,value => <<0,0,1,244>>},[{'Elixir.Enum','-reduce/3-lists^foldl/2-0-',3,[{file,"lib/enum.ex"},{line,1826}]},{'Elixir.Enum','-map_reduce/3-lists^mapfoldl/2-0-',3,[{file,"lib/enum.ex"},{line,1372}]}]}

from nebulex.

Instructions:

mix ecto.create -r Hello.Repo
mix ecto.migrate -r Hello.Repo
iex -S mix
:basho_bench.start()
:basho_bench.setup_benchmark([])
:basho_bench.run_benchmark(['./src/basho_bench_nebulex.config'])

from nebulex.

I modified the schema so it works with integer keys, but I wanted uuid results.
fire/database_cache_benchmark@735852b

from nebulex.

Results on Windows and my desktop. Do not use this for scientific reference.

{operations, [{put, 1}, {get, 1}]}.

Currently trying:

{operations, [{put, 2}, {get, 8}]}.

from nebulex.

{mode, max}.

{duration, 5}.

{concurrent, 8}.

{driver, 'Elixir.BashoBench.Nebulex.Driver'}.

{operations, [{put, 2}, {get, 8}]}.

{key_generator, {pareto_int, 1000}}.

{value_generator, {fixed_bin, 100000}}.

Results on Windows and my desktop. Do not use this for scientific reference.

from nebulex.

Previous results are wrong, the puts were failing.

from nebulex.

{mode, max}.

{duration, 5}.

{concurrent, 36}.

{driver, 'Elixir.BashoBench.Nebulex.Driver'}.

{operations, [{put, 1}, {get, 1}]}.

{key_generator, {pareto_int, 1000}}.

{value_generator, {uniform_int, 10000}}.

Edit: Note that these are using the on_conflict database functionality.

from nebulex.

These were my results, but only testing the cache, no Ecto (no DB access), I wanted to get some metrics from the Nebulex cache itself first. Keep in mind that you are testing Ecto with Nebulex, which means that for reads you fetch the data drom the DB sometimes, and for writes you always hit both, the cache and the DB.

But well, with your tests we have metrics from only nebulex and nebulex_ecto :)

Results (Mac OSX 10.11.4, 2.2 GHz Intel Core i7, RAM 16 GB):

from nebulex.

I recommend we pick a few databases with standard hardware try out nebulex_ecto.

from nebulex.

@fire 100% agreed, that would be great. Meanwhile, I'll create a separated project for these load tests (only for nebulex) – probably within nebulex_examples I'll create a new one, e.g.: nebulex_bench. Then, we can use your project database_cache_benchmark for nebulex_ecto load tests.

from nebulex.

I've created a new project to run bench/load tests against Nebulex: nebulex_bench – the project also contains some initial graphs with results for local cache.

from nebulex.

Interesting results trying to copy ycsb's

Workload B: Read mostly workload
This workload has a 95/5 reads/write mix. Application example: photo tagging; add a tag is an update, but most operations are to read tags.

Hint. It mostly gets cached.

from nebulex.

@fire based on you suggestion (pick a few databases with standard hardware try out nebulex_ecto), I just opened a new issue on nebulex_ecto in order to continue the load tests but with nebulex_ecto. So I think this issue can be closed, let me know your thoughts!!

from nebulex.

That's a good suggestion. You can close this issue.

from nebulex.