This repo contains my experimental implementation of a POSIX(FUSE) based distributed filesystem that automatically de-duplicates by managing data blocks by hashes.

It contains executables for

1. direct connection to the TiKV cluster
2. GRPC client+server to avoid many long round-trip times when operating on slow internet connection.

Features:

• deduplication by using hashed blocks
• reference counting on the bashed blocks
• provides access to the internally computed hashes by special automatically listed hash-files.
• vectored upload and download of blocks to speedup transfer
• very basic snapshot mechanism (no write-protection yet)

Experiences:

• despite some optimizations, the block-reference counting brings some overhead that significantly slows down the write operations. For me it wasn't possible to get much more than 20 MBit/s on Gigabit Ethernet (LAN).

TiFS is a distributed POSIX filesystem based on TiKV, with partition tolerance and strict consistency. The remaining part of this README is still original from TiFS.

• Linux libfuse and build-essential are required, in ubuntu/debian:

sudo apt install -y libfuse-dev libfuse3-dev build-essential

• macOS

brew install --cask osxfuse

• Linux fuse3 and openssl are required, in ubuntu/debian:

sudo apt-get install -y libfuse3-dev fuse3 libssl-dev

• macOS

brew install --cask osxfuse

In Catalina or former version, you need to load osxfuse into the kernel:

/Library/Filesystems/osxfuse.fs/Contents/Resources/load_osxfuse

You can use the image on docker hub or build from the Dockerfile.

mkdir tmp
cd tmp
wget https://github.com/Hexilee/tifs/releases/download/v0.2.1/tifs-linux-amd64.tar.gz
tar -xvf tifs-linux-amd64.tar.gz
sudo ./install.sh

The install.sh may fail in macOS Catalina or Big Sur because of the SIP.

You can just use the target/release/tifs to mount tifs.

Example

target/release/tifs tifs:127.0.0.1:2379 ~/mnt

git clone https://github.com/Hexilee/tifs.git
cd tifs
sudo make install

You need a tikv cluster to run tifs. tiup is convenient to deploy one, just install it and run tiup playground.

docker run -d --device /dev/fuse \
    --cap-add SYS_ADMIN \
    -v <mount point>:/mnt:shared \
    hexilee/tifs:0.2.2 --mount-point /mnt --pd-endpoints <endpoints>

You need ca.crt, client.crt and client.key to access TiKV cluster on TLS.

It will be convenient to get self-signed certificates by sign-cert.sh(based on the easy-rsa).

You should place them into a directory and execute following docker command.

docker run -d --device /dev/fuse \
    --cap-add SYS_ADMIN \
    -v <cert dir>:/root/.tifs/tls \
    -v <mount point>:/mnt:shared \
    hexilee/tifs:0.3.1 --mount-point /mnt --pd-endpoints <endpoints>

mkdir <mount point>
mount -t tifs tifs:<pd endpoints> <mount point>

mount -t tifs -o tls=<tls config file> tifs:<pd endpoints> <mount point>

By default, the tls-config should be located in ~/.tifs/tls.toml, refer to the tls.toml for detailed configuration.

Enable global direct io, to avoid page cache.

mount -t tifs -o direct_io tifs:<pd endpoints> <mount point>

The block size, 64KiB by default, could be human-readable.

mount -t tifs -o blksize=512 tifs:<pd endpoints> <mount point>

The quota of fs capacity, could be human-readable.

mount -t tifs -o maxsize=1GiB tifs:<pd endpoints> <mount point>

cargo build
mkdir ~/mnt
RUST_LOG=debug target/debug/tifs --mount-point ~/mnt

Then you can open another shell and play with tifs in ~/mnt.

Maybe you should enable user_allow_other in /etc/fuse.conf.

for developing under FreeBSD, make sure the following dependencies are met.

pkg install llvm protobuf pkgconf fusefs-libs3 cmake

for now, user_allow_other and auto unmount does not work for FreeBSD, using as root and manually umount is needed.

Please refer to the design.md

There are little docs about FUSE, refer to the example for the meaning of FUSE API.

Please refer to the tikv-deploy.md.

• FUSE API

  • init
  • lookup
  • getattr
  • setattr
  • readlink
  • readdir
  • open
  • release
  • read
  • write
  • mkdir
  • rmdir
  • mknod
  • lseek
  • unlink
  • symlink
  • rename
  • link
  • statfs
  • create
  • fallocate
  • getlk
  • setlk

• Testing and Benchmarking

  • pjdfstest
  • fio

• Real-world usage

  • vim
  • emacs
  • git
  • gcc
  • rustc
  • cargo build
  • npm install
  • sqlite
  • tikv on tifs
  • client runs on FreeBSD: simple case works