Comments (8)
PragTob
commented on June 9, 2024
1
I had a minor hope that #408 would maybe put a dent into this one, but it doesn't which makes sense as the inputs used here are relatively small. That said I haven't forgotten about this one, but the other ones have been more prevalent 😅
from benchee.
PragTob
commented on June 9, 2024
👋
Thanks for reporting an issue! 💚
Sorry for the long silence!
Also, yes that's likely - they run so fast that so many measurements are generated in memory that it breaks depending on how much RAM you got.
Doesn't mean we can't be better (this has happened before), we could offer something like max_measurements to avoid and short cut these kinda problems 🤔
I'll run it later just to make sure. Ok ran it now. From the looks yeah it's definitely just too many samples:
The remedy right now is to just reduce the time:
inputs = %{
"large" => Map.new(1..33, fn n -> {:"key_#{n}", []} end),
"single" => %{key_1: []}
}
Benchee.run(
%{
"body_match" => fn map ->
%{key_1: value} = map
value
end,
"case" => fn map ->
case map do
%{key_1: value} -> value
end
end,
"dot" => fn map ->
map.key_1
end,
"head_match" => fn
%{key_1: value} ->
value
end
},
inputs: inputs,
pre_check: true,
time: 1,
warmup: 0.1,
formatters: [{Benchee.Formatters.Console, extended_statistics: true}]
)Which gets me this output:
Operating System: Linux
CPU Information: AMD Ryzen 9 5900X 12-Core Processor
Number of Available Cores: 24
Available memory: 31.25 GB
Elixir 1.15.4
Erlang 26.0.2
Benchmark suite executing with the following configuration:
warmup: 100 ms
time: 1 s
memory time: 0 ns
reduction time: 0 ns
parallel: 1
inputs: large, single
Estimated total run time: 8.80 s
Benchmarking body_match with input large ...
Benchmarking body_match with input single ...
Benchmarking case with input large ...
Benchmarking case with input single ...
Benchmarking dot with input large ...
Benchmarking dot with input single ...
Benchmarking head_match with input large ...
Benchmarking head_match with input single ...
##### With input large #####
Name ips average deviation median 99th %
head_match 31.18 M 32.07 ns ±37.93% 30 ns 40 ns
case 30.84 M 32.43 ns ±37.84% 30 ns 40 ns
dot 30.81 M 32.46 ns ±139.96% 30 ns 50 ns
body_match 30.13 M 33.19 ns ±38.29% 30 ns 40 ns
Comparison:
head_match 31.18 M
case 30.84 M - 1.01x slower +0.36 ns
dot 30.81 M - 1.01x slower +0.39 ns
body_match 30.13 M - 1.04x slower +1.13 ns
Extended statistics:
Name minimum maximum sample size mode
head_match 20 ns 4590 ns 2.22 M 30 ns
case 20 ns 4810 ns 2.22 M 30 ns
dot 20 ns 64761 ns 2.22 M 30 ns
body_match 20 ns 4780 ns 2.22 M 30 ns
##### With input single #####
Name ips average deviation median 99th %
body_match 35.04 M 28.54 ns ±42.13% 30 ns 40 ns
head_match 34.98 M 28.59 ns ±49.44% 30 ns 40 ns
case 34.97 M 28.60 ns ±42.11% 30 ns 40 ns
dot 34.80 M 28.74 ns ±43.26% 30 ns 40 ns
Comparison:
body_match 35.04 M
head_match 34.98 M - 1.00x slower +0.0430 ns
case 34.97 M - 1.00x slower +0.0530 ns
dot 34.80 M - 1.01x slower +0.197 ns
Extended statistics:
Name minimum maximum sample size mode
body_match 20 ns 4510 ns 2.22 M 30 ns
head_match 20 ns 10491 ns 2.23 M 30 ns
case 20 ns 4491 ns 2.22 M 30 ns
dot 20 ns 10200 ns 2.23 M 30 ns
You can see that's 2.22M samples per scenario and 8 scenarios oh my. The time it actually blows up is during the statistics calculation I think (with the longer times) and part of that is that instead of implementing the super hard way to get some stats the list is actually sorted 😅 So, could also try make that better although benchmarks say that for normal cases it's fine.
Again, thanks for the report!
from benchee.
lud-wj
commented on June 9, 2024
Hey, thanks for the confirmation!
max_measurements could be nice but really my benchmark does not do any actual work, I'm not sure that would be a really useful feature :)
from benchee.
PragTob
commented on June 9, 2024
I mean it has happened before and you ran into an issue because of it - so I think benchee as a tool can do something better and so I'll see if I can easily implement it which should be possible. Might need to start tracking the measurement count but that shouldn't be tooo bad.
from benchee.
lud-wj
commented on June 9, 2024
Alright! I don't know the benchee code but indeed it should be simple enough :)
from benchee.
PragTob
commented on June 9, 2024
Looking at the code, part of the problem is probably also that we calculate all the statistics in parallel for each scenario 😂
def statistics(suite) do
percentiles = suite.configuration.percentiles
update_in(suite.scenarios, fn scenarios ->
Parallel.map(scenarios, fn scenario ->
calculate_scenario_statistics(scenario, percentiles)
end)
end)
endWhich... for a normal number of scenarios this is probably fine but overall not the greatest and may be good to optionally turn off 😂
from benchee.
PragTob
commented on June 9, 2024
Same thing for formatters :sweat-smile:
from benchee.
lud-wj
commented on June 9, 2024
If there is a max_concurrency option or something like that it could be nice to expose it indeed.
from benchee.
Related Issues (20)
- Trying to round :infinity HOT 11
- Sugguestion: Express time in `h:m:s` HOT 4
- Promote use of keywords lists instead maps for job list HOT 3
- Is it worth releasing a new version? HOT 2
- Is there an option to try all possible scenarios once to know early if some would raise ? HOT 2
- Suggestion: Option to kill long-running scenarios early HOT 2
- Is it possible to benchmark a subprocess call like `python -c "8*9"`? HOT 5
- Question: Benchmark function operations ? HOT 1
- Warn if the benchmarking function is evaluating HOT 2
- Benchee throws an exception when running in the Livebook-App HOT 3
- Benchee warning when used in mix task HOT 2
- Unpredictable experiment order in output HOT 1
- Elixir provide HOT 1
- Built in outlier detection
- suggestion HOT 1
- (Protocol.UndefinedError) protocol List.Chars not implemented for {:error, :bad_name} HOT 5
- Top level functions slower than module functions in Elixir <1.16 HOT 7
- Avoid handing all of inputs to every scenario process
- Add Solaris Support to Benchee.System HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from benchee.