c-fs / cfs Goto Github PK

A network file system interface for building the cloud

License: Apache License 2.0

Go 89.09% Protocol Buffer 7.61% Shell 1.64% Makefile 1.65%

cfs's Introduction

#Cloud File System

Overview

cfs (cloud file system) is a low-level filesystem interface for building the cloud. cfs is not a distributed file system. It is designed to be the lowest building block of cloud infrastructure that is available on all machines. The design goal of cfs is reliability, simplicity and usability.

Status

Current status is pre-alpha

TODO:

basic file system interface (write, read, remove, mkdir, rename)
copy from another cfs server
bit-matrix based data reconstruction
expose basic run time metrics
basic resource enforcement
container deployment

alpha

TODO:

profiling and benchmark
per disk metadata store
simple ACL support
user level buffer
tons of tests

We will BREAK everything (data format, API, command line tool, etc.) before 1.0.

Install

cfs depends on GF-complete and Jerasure for erasure coding.

cfs requires that Go version must be 1.5 at minimum.

For now, to install these packages:

go get github.com/c-fs/vendor
go get -u github.com/c-fs/Jerasure
export GO15VENDOREXPERIMENT=1
make build

Play with cfs

Run Server

cd server

go build

./server

cfs listens on 15524 by default. The root path of cfs is the current directory by default.

cfs creates a /cfs0 disk with root path /cfs0000 by default.

TODO: make all these configurable.

Run Server in a Docker Container

You can run a cfs to provide file service in container easily:

make docker

sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/tmp:/tmp:rw \
  --volume=${PWD}/cfs0000:/cfs0000
  --volume=${PWD}/cfs0001:/cfs0001
  --publish=15524:15524 \
  --detach=true \
  --name=cfs \
  c-fs/cfs

cfs is now running in the background on http://localhost:15524, and it uses ./cfs0000/ and ./cfs0001/ as data volume. For development, you can replace --detach=true with -ti to let it run as an interactive process.

Write and Read file

cd cfsctl

go build

cfsctl write --name="cfs0/foo" --data="bar"
2015/05/24 11:16:48 3 bytes written to foo at offset 0

cfsctl read --name="cfs0/foo" --length=100
2015/05/24 11:17:34 bar

Rename and Remove file

cfsctl rename --oldname="cfs0/foo" --newname="cfs0/food"
2015/05/28 15:20:22 rename cfs0/foo into cfs0/food

cfsctl remove --name="cfs0/food"
2015/05/28 15:20:43 deletion succeeded

Read a corrupted file

echo "corrupt" -> ./server/cfs0000/foo

Try to read out the file again

cfsctl read --name="cfs0/foo" --length=100
2015/05/24 11:18:56

Nothing is read out.

We can see the log output from the server side

2015/05/24 11:18:56 server: read error disk: not a valid CRC

cfs's People

Contributors

Stargazers

Watchers

Forkers

yichengq tnachen yifan-gu a6802739 read-later yutongp missmayo x140cc

cfs's Issues

disk.ReadAt not return EOF

disk.WriteAt("cfs0/newfile", []byte("123"), 0)

buf := make([]byte, 3)
disk.ReadAt("cfs0/newfile", buf, 0)  return 3, nil
disk.ReadAt("cfs0/newfile", buf, 3)  return 0, nil   // actual expected 0, io.EOF?

When disk.Readat wrap to io.Reader interface, then as a read param for io.Copy, it will dead loop.

use subcommand in ctl tool

what will happen while update partial block？

通过 writeBlock(ws io.WriteSeeker, index, bs int64, p []byte)，更新一个block。
当len(p)小于block size时，是什么语义？

算错误，直接返回
更新前len(p)字节，并重新计算整个block的crc

# github.com/c-fs/cfs/erasure
./erasure.go:45:31: warning: unknown attribute '__gcc_struct__' ignored [-Wattributes]
        } __attribute__((__packed__, __gcc_struct__)) *a = v;
                                     ^
./erasure.go:65:31: warning: unknown attribute '__gcc_struct__' ignored [-Wattributes]
        } __attribute__((__packed__, __gcc_struct__)) *a = v;
                                     ^
./erasure.go:79:31: warning: unknown attribute '__gcc_struct__' ignored [-Wattributes]
        } __attribute__((__packed__, __gcc_struct__)) *a = v;
                                     ^
./erasure.go:100:31: warning: unknown attribute '__gcc_struct__' ignored [-Wattributes]
        } __attribute__((__packed__, __gcc_struct__)) *a = v;
                                     ^
./erasure.go:117:31: warning: unknown attribute '__gcc_struct__' ignored [-Wattributes]
        } __attribute__((__packed__, __gcc_struct__)) *a = v;
                                     ^
5 warnings generated.
# github.com/c-fs/cfs/erasure
clang: warning: argument unused during compilation: '-pthread'
# github.com/c-fs/cfs/erasure
clang: warning: argument unused during compilation: '-pthread'

@yunxing could you take a look?
And close this issue if we don't need to worry about it.

on Mac with:

Apple LLVM version 6.0 (clang-600.0.54) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin14.3.0
Thread model: posix