Benchmarking is flawed and nano-memoize is much slower in realistic scenarios about nano-memoize HOT 5 CLOSED

________________________________ I From: Thomas Földi ***@***.***> Sent: Tuesday, March 14, 2023, 3:08 PM To: anywhichway/nano-memoize ***@***.***> Cc: Subscribed ***@***.***> Subject: [anywhichway/nano-memoize] Benchmarking is flawed and nano-memoize is much slower in realistic scenarios (Issue #54) The benchmarking in this repository only tests for the memoized function to be called with the same arguments millions of times. That is not a realistic scenario and one would just put the return value of the original function into a variable. If you test for a realistic scenario, with say, the function being called with 5000 different values the results for nano-memoize look very different. Here are the numbers for a more realistic scenario I ran with the benchmarking: Name Ops / sec Relative margin of error Sample size lodash 4,231 ± 0.45% 97 underscore 3,677 ± 0.29% 99 iMemoized 2,237 ± 0.29% 97 memoizee 2,228 ± 0.39% 97 addy-osmani 1,163 ± 0.87% 97 micro-memoize 784 ± 0.36% 95 moize 734 ± 0.19% 96 fast-memoize 433 ± 0.31% 93 memoizerific 44 ± 8.67% 62 nano-memoize 28 ± 2.23% 51 lru-memoize 15 ± 8.55% 44 Here is the code I used for the more realistic benchmarking. You can just copy / paste it into your repo and run it: function dateToWeekDay(year, month, day) { const d = new Date(Date.UTC(year, month, day)); return d.getDay(); } const days = []; const date = new Date(Date.UTC(2023, 0, 1)); for (let i = 0; i < 5000; i++) { date.setDate(date.getDate() + 1); days.push([date.getFullYear(), date.getMonth(), date.getDate()]); } const runRealisticScenario = () => { const suite = new Benchmark.Suite("Realistic scenario"); const mUnderscore = underscore(dateToWeekDay); const mLodash = lodash(dateToWeekDay); // const mRamda = ramda(dateToMonth); const mMemoizee = memoizee(dateToWeekDay); const mFastMemoize = fastMemoize(dateToWeekDay); const mAddyOsmani = addyOsmani(dateToWeekDay); const mMemoizerific = memoizerific(Infinity)(dateToWeekDay); const mLruMemoize = lruMemoize(Infinity)(dateToWeekDay); const mMoize = moize(dateToWeekDay); const mMicroMemoize = microMemoize(dateToWeekDay); const mIMemoized = iMemoized.memoize(dateToWeekDay); const mNano = nanomemoize(dateToWeekDay); return new Promise((resolve) => { suite .add("nano-memoize", () => { for (let i = 0; i < days.length; i++) { mNano(...days[i]); } }) .add("addy-osmani", () => { for (let i = 0; i < days.length; i++) { mAddyOsmani(...days[i]); } }) .add("lodash", () => { for (let i = 0; i < days.length; i++) { mLodash(...days[i]); } }) .add("lru-memoize", () => { for (let i = 0; i < days.length; i++) { mLruMemoize(...days[i]); } }) .add("memoizee", () => { for (let i = 0; i < days.length; i++) { mMemoizee(...days[i]); } }) .add("memoizerific", () => { for (let i = 0; i < days.length; i++) { mMemoizerific(...days[i]); } }) /*.add('ramda', () => { mRamda(fibonacciNumber); })*/ .add("underscore", () => { for (let i = 0; i < days.length; i++) { mUnderscore(...days[i]); } }) .add("iMemoized", () => { for (let i = 0; i < days.length; i++) { mIMemoized(...days[i]); } }) .add("micro-memoize", () => { for (let i = 0; i < days.length; i++) { mMicroMemoize(...days[i]); } }) .add("moize", () => { for (let i = 0; i < days.length; i++) { mMoize(...days[i]); } }) .add("fast-memoize", () => { for (let i = 0; i < days.length; i++) { mFastMemoize(...days[i]); } }) .on("start", () => { console.log(""); // eslint-disable-line no-console console.log("Starting cycles for functions with realistic scenario..."); // eslint-disable-line no-console results = []; spinner.start(); }) .on("cycle", onCycle) .on("complete", () => { onComplete(); resolve(); }) .run({ async: true, }); }); }; — Reply to this email directly, view it on GitHub<#54>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABF2USZHNMEKEKPJYMEMVXLW4DT5JANCNFSM6AAAAAAV3AP7VU>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

________________________________ From: Thomas Földi ***@***.***> Sent: Tuesday, March 14, 2023, 3:59 PM To: anywhichway/nano-memoize ***@***.***> Cc: Simon Y. Blackwell ***@***.***>; Comment ***@***.***> Subject: Re: [anywhichway/nano-memoize] Benchmarking is flawed and nano-memoize is much slower in realistic scenarios (Issue #54) I am happy to help :) I just figured out that the test I wrote is flawed too, since lodash does not memoize functions with multiple arguments correctly out of the box, but just uses the first argument for a key. That might be the case for some more of these libs I put into the test. So it's not really a realistic test afterall and I guess it takes some time to get into the details and get everything figured out. — Reply to this email directly, view it on GitHub<#54 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABF2USYTBLEITJW7EV6PBFTW4DZ4ZANCNFSM6AAAAAAV3AP7VU>. You are receiving this because you commented.Message ID: ***@***.***>