Trying to understand inconsistent disk usage between cluster nodes about typesense HOT 6 OPEN

@jasonbosco Thanks for the quick response. I've updated to Typesense 26.0 and included the --db-compaction-interval=21600 in the arguments. Disk usage has held around 8-9GB since then but that seems quite high for the number of documents. It could just be leftover cruft from 0.25.2. I've wiped the persistent storage, restarted the cluster and started a fresh index and will let it stew over the weekend to see what happens.

from typesense.

We've improved disk usage in v26.0, for this specific write pattern of creating new timestamped collections and deleting the old one, like how the scraper does.

Could you try upgrading to it, and then setting db-compaction-interval = 21600 as a Typesense server parameter, to see if that helps?

from typesense.

After rebuilding all the data under 26.0 the disk usage under /usr/share/typesense/data is much more steady and consistent across cluster nodes. We are noticing that one node seems to be keeping everything in ./db while the other 2 appear to be moving data over to ./state/snapshot despite all containers being started with identical parameters. Any ideas why that may be?

typesense-0:

typesense-0:/$ cd /usr/share/typesense/data
typesense-0:/usr/share/typesense/data$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme2n1    9.8G  1.5G  8.3G  16% /usr/share/typesense/data
typesense-0:/usr/share/typesense/data$ du -h --max-depth=2
4.0K    ./models
236K    ./state/log
1.5G    ./state/snapshot
8.0K    ./state/meta
1.5G    ./state
16K     ./lost+found
3.9M    ./db/archive
30M     ./db
4.0K    ./meta/archive
5.3M    ./meta
1.5G    .

typesense-1:

typesense-1:/$ cd /usr/share/typesense/data
typesense-1:/usr/share/typesense/data$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme4n1    9.8G  1.6G  8.2G  16% /usr/share/typesense/data
typesense-1:/usr/share/typesense/data$ du -h --max-depth=2
236K    ./state/log
8.0K    ./state/meta
1.6G    ./state/snapshot
1.6G    ./state
4.0K    ./models
16K     ./lost+found
3.9M    ./db/archive
30M     ./db
4.0K    ./meta/archive
5.3M    ./meta
1.6G    .

typesense-2:

typesense-2:/$ cd /usr/share/typesense/data
typesense-2:/usr/share/typesense/data$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme3n1    9.8G  1.5G  8.3G  16% /usr/share/typesense/data
typesense-2:/usr/share/typesense/data$ du -h --max-depth=2
16K     ./lost+found
3.9M    ./db/archive
1.5G    ./db
4.0K    ./models
4.0K    ./meta/archive
5.3M    ./meta
236K    ./state/log
680K    ./state/snapshot
8.0K    ./state/meta
928K    ./state
1.5G    .

from typesense.

Can you tell me what type of disk you are using for the data directory?

When a write arrives, it is written to the raft log and also written to the store in db directory. Every 1 hour, a snapshot happens where the contents of the db directory is hard linked within the state/snapshot directory (hard linking is like soft link but happens at the inode level so that data is not duplicated). When Typesense is restarted, we replace the db directory with the contents of the db from state/snapshot . You can confirm this behavior in a Typesense server on your localhost.

So ideally, the db directory should be more or less have the same data in state/snapshot unless a lot of writes have happened before a snapshot occurs.

from typesense.

from typesense.