Performance Benchmark about lc-core HOT 11 CLOSED

This is great! Tons of things to analyse.

• Can you upload the test script to gist.github.com?
• Let's fix on 1000 concurrent users @ 100 users/second for now.
• We need to ensure that each test begins with a clean and consistent state. For example, by removing the database, and populating it fresh every time. Can you write a script for the test? It should roughly do the following:

rm << captcha database files >>
sbt runMain << classNameForPopulatingDB >>
run locust

• Before we analyse the performance, we need to address the failures. For the most common failure, can you raise a separate issue in this repo, with the stack trace and other details included?

from lc-core.

Will be using locust for load testing.

from lc-core.

Looks good. You could simulate a swarm of users that:

1. Get a captcha challenge
2. Submit a solution for it after 5 to 10 seconds

Let's see how many concurrent uses can do that.

from lc-core.

Load test data for 1000 concurrent users spawning at a rate of 10 users/second.

Statistics for the test

Total requests per second

Response times

Number of users

from lc-core.

Load test data for 1000 concurrent users spawning at a rate of 100 users/second.

Statistics for the test

Total requests per second

Response times

Number of users

Failures

from lc-core.

Load test data for 2000 concurrent users spawning at a rate of 10 users/second.

Statistics for the test

Total requests per second

Response times

Number of users

Failures

from lc-core.

I also observed this unusual spike in failures for a few Milliseconds when the test starts.

From 17:34:15 to 17:34:30

from lc-core.

Load test data for 1000 concurrent users spawning at a rate of 10 users/second after DB upgrade and DB sync

Statistics for the test

Total requests per second

Response times

Number of users

Failures

from lc-core.

Load test data for 1000 concurrent users spawning at a rate of 100 users/second after DB upgrade and DB sync

Statistics for the test

Total requests per second

Response times

Number of users

Failures

Note: The majority of failures on /captcha endpoint are because of the rate limiter. It gives back a 400 error when the limit is reached, as seen in failures.

from lc-core.

Performance results for CRUD app

Config

• OpenJDK 11
• locust 100 users (each user contributes about 31 requests per second)
• Each locust user calls create then increment : read is in the ratio 1 : 8
• Xodus 1.3.232
• h2-1.4.200.jar
• Locust and CRUD server running on the same DO droplet
• application compiled and assembled, then run standalone

ulimit

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3832
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 40000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3832
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

RPS

For Xodus, peak RPS is 2150
For H2, peak RPS is 2400

Response times were sub 1ms for both

from lc-core.

After multiple tests, we decided to retain H2 as the database, and added some tweaks to the schema.

from lc-core.