Sharing the report here for discussion before I make a PR with test scripts etc.

Netsim Replication Benchmarks

Full Sync Two SSB Nodes

All tests were run with the same fixtures consisting of 100,000 messages from 20 unique authors. Two identities were chosen: one is labelled as server and the other as peer. The skipoffset peer netsim command was used to ensure the peer starts with 0 messages. The peer is connected to the server and we time how long it takes for peer to replicate the full feed of server (4219 messages). A mutual follow is established before connecting the peer to the server (hence the 4220 message count in the test output). Replication rate is calculated as the total peer messages divided by the run time.

See the bottom of the page for the full test script.

Summary

JS - JS (db)

# Total time: 16.891932964s
# Active time: 4.391932964s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server            16.291s       3.791s         4220
# peer              14.203s       3.703s         4220
# 
# Started timers & final elapsed time
# Label                Time
# run                 2.65s

Replication rate (msgs / second): 1592

JS - JS (db2)

# Total time: 21.009802043s
# Active time: 8.509802043s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server            20.668s       8.168s         4220
# peer              18.328s       7.828s         4220
# 
# Started timers & final elapsed time
# Label                Time
# run                7.239s

Replication rate (msgs / second): 583

Go - Go

# Total time: 14.898761001s
# Active time: 2.398761001s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server             13.78s        1.28s         4220
# peer              12.799s       2.299s         4220
# 
# Started timers & final elapsed time
# Label                Time
# run                 509ms

Replication rate (msgs / second): 8291

Go - JS (db)

server : go ... peer : js

# Total time: 16.877485387s
# Active time: 4.377485387s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server            16.332s       3.832s         4220
# peer              14.767s       4.267s         4220
# 
# Started timers & final elapsed time
# Label                Time
# run                2.831s

Replication rate (msgs / second): 1491

server : js ... peer : go

# Total time: 2m54.868505121s
# Active time: 2m32.368505121s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server          2m53.837s    2m31.337s         4220
# peer            2m52.303s    2m31.803s           76
# 
# Started timers & final elapsed time
# Label                Time
# run             2m40.596s

Replication rate (msgs / second): 0.5

Go - JS (db2)

server : go ... peer : js

# Total time: 17.870999912s
# Active time: 5.370999912s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server            17.577s       5.077s         4219
# peer              15.767s       5.267s         4220
# 
# Started timers & final elapsed time
# Label                Time
# run                3.948s

Replication rate (msgs / second): 1069

server : js ... peer : go

# Total time: 2m54.645813867s
# Active time: 2m32.145813867s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server          2m53.614s    2m31.114s         4220
# peer             2m51.95s     2m31.45s            1
# 
# Started timers & final elapsed time
# Label                Time
# run             2m40.597s

Replication rate (msgs / second): 0

Test Script

The only changes to this script between runs are the start server go-sbot and start peer go-sbot commands; go-sbot is substituted for ssb-server or ssb-server-db1 when required.

comment [ configuring peer ]
enter peer
hops peer 3
load peer @sWJqvI4hdv96aRMk1pQOJo1frXRFr7oBB5/r/jBMcxw=.ed25519
skipoffset peer

comment [ configuring server ]
enter server
hops server 3
load server @NZpjhmgTaHOHFZI6G6dIuWwa3UqwSpfRpMbY5ue6vAI=.ed25519

start server go-sbot
start peer go-sbot

follow server peer
wait 4000
follow peer server
wait 4000

timerstart run

connect peer server
waituntil peer server@latest

timerstop run

disconnect peer server
stop server
stop peer

Netsim Partial Versus Full Replication Benchmarks

Goal

The motivation behind this batch of tests was to ascertain the efficacy and efficiency of partial replication (using index feeds), as compared with full replication.

Dataset

All tests were run with the same fixtures consisting of 100,000 classic messages and 10,935 index messages (about & contact) from 100 unique authors. The complete fixtures comprise 500 feeds. The following command was used to generate the fixtures:

npx ssb-fixtures@latest --progress --messages=10000 --authors=100 --followGraph=true --allkeys=true --indexFeeds=100 --seed=ritual11 --outputDir=100k-100auth-indexed-fixtures

Note: ssb-fixtures@latest was 3.0.0 at the time of command execution.

Run Command

netsim run --spec fixtures-output/netsim-test-js-db2-indexes.txt --fixtures 100k-100auth-indexed-fixtures/fixtures-output ssb-server-db2

Note: the exact test file and implementation(s) passed to netsim differ slightly for each test variant. In the above command, we are testing full replication using a JS db2 implementation.

Simulation Setup

Two identities were chosen: one is labelled as server (@fa+bdOd11etLngOjltjPx/wtQTbCX6lE9vhnQY7zGX8=.ed25519) and the other as peer (@pqRSz6qLrlPo8TH4txzSxcXwtlEUj6pkkrJ066+Xfpg=.ed25519). The peer instance started with 562 messages and 1 feed. The alloffsets server netsim command was used to ensure the server started with all 110935 messages and 500 feeds. The peer was connected to the server and we measured how many messages were replicated from server to peer in 6 seconds and 25 seconds respectively. A mutual follow was established before connecting the peer to the server. The disconnect command was used at the end of the test script to ensure the peer instance was disconnected from server before netsim performed the message count; failing to pass this command can result in inaccurate counts, since replication may continue after the timed window.

Each test was run three times and averages were calculated for run time, feed counts and message counts. Replication rates are calculated as (the total peer messages minus 562 & the total peer feeds minus 1) divided by the run time (either 6 or 25 seconds).

Summary

All values are calculated as an average across three test runs for each implementation.

Six Seconds of Replication

Twenty Five Seconds of Replication

Test Script

comment [ six second sync two js db2 ssb servers with indexes 100k msgs ]

comment [ configuring peer ]
enter peer
hops peer 3
load peer @pqRSz6qLrlPo8TH4txzSxcXwtlEUj6pkkrJ066+Xfpg=.ed25519

comment [ configuring server ]
enter server
hops server 3
load server @fa+bdOd11etLngOjltjPx/wtQTbCX6lE9vhnQY7zGX8=.ed25519
alloffsets server

comment [ starting ssb-server instances ]
start server ssb-server-db2
wait 10000

start peer ssb-server-db2
wait 5000

follow server peer
follow peer server

comment [ connecting peer to server ]
timerstart index_run
connect peer server
wait 6000
timerstop index_run

comment [ disconnecting peer instance ]
disconnect peer server