leo-project / leofs Goto Github PK

View Code? Open in Web Editor NEW

1.5K 86.0 155.0 30.72 MB

The LeoFS Storage System

Home Page: https://leo-project.net/leofs/

License: Apache License 2.0

Makefile 0.59% Shell 8.42% Erlang 87.67% Batchfile 0.75% Java 0.17% Roff 0.55% RPC 1.33% Dockerfile 0.54%

leofs erlang distributed-storage distributed-file-system s3-storage s3 nfs-server nfs datalake

leofs's Introduction

LeoFS - A Storage System for a Data Lake and the Web

Overview

LeoFS is an Enterprise Open Source Storage, and it is a highly available, distributed, eventually consistent object/blob store. If you are seeking a storage system that can store huge amount and various kind of files such as photo, movie, log data and so on, LeoFS is suitable for that.

LeoFS is supporting the following features:

Multi Protocol
- S3-API Support
  - LeoFS is an Amazon S3 compatible storage system.
  - Switch to LeoFS to decrease your cost from more expensive public-cloud solutions.
- REST-API Support
  - To easily access LeoFS with REST-API
- NFS Support
  - NFS support was provided from LeoFS v1.1, the current status of which is beta.
Large Object Support
- LeoFS covers handling large size objects.
Multi Data Center Replication
- LeoFS is a highly scalable, fault-tolerant distributed file system without SPOF.
- LeoFS's cluster can be viewed as a huge capacity storage. It consists of a set of loosely connected nodes.
- We can build a global scale storage system with easy operations

Architecture

LeoFS consists of three core components - LeoStorage, LeoGateway and LeoManager which depend on Erlang.

LeoGateway handles http-request and http-response from any clients when using REST-API OR S3-API. Also, it is already built in the object-cache mechanism (memory and disk cache).

LeoStorage handles GET, PUT and DELETE objects as well as metadata. Also, it has replicator, recoverer and queueing mechanism in order to keep running a storage node and realise eventual consistency.

LeoManager always monitors LeoGateway and LeoStorage nodes. The main monitoring status are Node status and RING’s checksum in order to realise to keep high availability and keep data consistency.

You can access a LeoFS system using Amazon S3 clients and the SDK.

Slide

The presentation - Scaling and High Performance Storage System: LeoFS was given at Erlang User Conference 2014 in Stockholm on June 2014

GOALs

LeoFS has been aiming to provide high reliability, high scalability, and high cost performance ratio:
- HIGH Reliability
  - Nine nines - Operating ratios is 99.9999999%
- High Scalability
  - Build huge-cluster at low cost
- HIGH Cost Performance
  - Fast - Over 10Gbps
  - A lower cost than other storage
  - Provide easy management and easy operation

Further Reference

LeoFS Documentation.

Build LeoFS with LeoFS Packages

LeoFS packages have been already provided on the Web. You're able to easily install LeoFS on your environments.

LeoProject
- CentOS 6.x, 7.x
- Ubuntu 14.04, 16.04
Community
- FreeBSD

Here is the installation manual.

Build LeoFS From Source (For Developers)

Here, we explain how to build LeoFS from source code. First, you have to install the following packages to build Erlang and LeoFS.

Build Dependencies

## [CentOS]
$ sudo yum install cmake check-devel gcc gcc-c++ make
## [Ubuntu]
$ sudo apt-get install gcc g++ cmake make check libtool
### For Docker
$ apt-get install lsb-release

Install Erlang

You can install Erlang with kerl.

Install kerl

$ curl -O https://raw.githubusercontent.com/kerl/kerl/master/kerl
$ chmod a+x kerl
$ mkdir -p ~/bin
$ mv kerl ~/bin/
$ echo "export PATH=$PATH:~/bin" >> ~/.bashrc
$ source ~/.bashrc

Install Erlang (Erlang/OTP 19.3)

$ kerl build 19.3 19.3
$ kerl list builds
19.3,19.3

$ kerl install 19.3 /path/to/19.3
$  kerl list installations
19.3 /path/to/19.3

$ source /path/to/19.3/activate
$ kerl active
The current active installation is:
/path/to/19.3

Install LeoFS

Then, clone source of LeoFS and libraries from GitHub.

$ git clone https://github.com/leo-project/leofs.git
$ cd leofs
$ git checkout -b develop remotes/origin/develop
$ ./rebar get-deps
$ ./git_checkout.sh develop

Then, build LeoFS with the following commands.

$ make && make release_for_test

Now, you can find the LeoFS package as follow.

$ ls package/
leo_gateway/  leo_manager_0/  leo_manager_1/  leo_storage/  README.md

Then, we can start and access LeoFS with the following commands. Also, you're able to easily operate LeoFS with leofs-adm script.

$ package/leo_manager_0/bin/leo_manager start
$ package/leo_manager_1/bin/leo_manager start
$ package/leo_storage/bin/leo_storage start
$ package/leo_gateway/bin/leo_gateway start
$ ./leofs-adm status
 [System Confiuration]
-----------------------------------+----------
 Item                              | Value
-----------------------------------+----------
 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.3.4
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 1
          number of successes of R | 1
          number of successes of W | 1
          number of successes of D | 1
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------
 Multi DC replication settings
-----------------------------------+----------
 [mdcr] max number of joinable DCs | 2
 [mdcr] total replicas per a DC    | 1
 [mdcr] number of successes of R   | 1
 [mdcr] number of successes of W   | 1
 [mdcr] number of successes of D   | 1
-----------------------------------+----------
 Manager RING hash
-----------------------------------+----------
                 current ring-hash |
                previous ring-hash |
-----------------------------------+----------

 [State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | [email protected]      | attached     |                |                | 2017-06-02 14:59:20 +0900
-------+--------------------------+--------------+----------------+----------------+----------------------------

$ ./leofs-adm start
OK

$ ./leofs-adm status
 [System Confiuration]
-----------------------------------+----------
 Item                              | Value
-----------------------------------+----------
 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.3.4
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 1
          number of successes of R | 1
          number of successes of W | 1
          number of successes of D | 1
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------
 Multi DC replication settings
-----------------------------------+----------
 [mdcr] max number of joinable DCs | 2
 [mdcr] total replicas per a DC    | 1
 [mdcr] number of successes of R   | 1
 [mdcr] number of successes of W   | 1
 [mdcr] number of successes of D   | 1
-----------------------------------+----------
 Manager RING hash
-----------------------------------+----------
                 current ring-hash | 433fe365
                previous ring-hash | 433fe365
-----------------------------------+----------

 [State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | [email protected]      | running      | 433fe365       | 433fe365       | 2017-06-02 15:00:10 +0900
  G    | [email protected]      | running      | 433fe365       | 433fe365       | 2017-06-02 15:00:12 +0900
-------+--------------------------+--------------+----------------+----------------+----------------------------

Build a LeoFS Cluster

You can easily build a LeoFS cluster. See here.

Configure LeoFS

About the configuration of LeoFS, See here.

Benchmarking

You can benchmark LeoFS with Basho Bench, and here is a documentation to benchmark LeoFS.

Integration Test

You can test LeoFS with leofs_test whether LeoFS has issues or not before getting installed LeoFS in your dev/staging/production environment(s).

Milestones

Version 1

DONE - v1.0
- Multi Data Center Replication
- Increase compatibility S3-APIs#5
  - Other bucket operations
DONE - v1.1
- NFS v3 Support (alpha)
- Improve Web GUI Console (Option)
DONE - v1.2
- NFS v3 Support (beta)
- Watchdog
- Automated data-compaction
DONE - v1.3
- NFS v3 Support (stable)
- Improve compatibility S3-APIs#6
  - AWS Signature v4 support (v1.3.0)
  - Custom Metadata Support (v1.3.1)
DONE - v1.4
- Improvement of the core features
- Integration with distributed computing frameworks#1
  - Hadoop integration
  - Spark integration

Version 2

WIP - v2.0
- Erasure Code
- Improve Data Security for GDPR and Enterprise Storages
- Improve compatibility S3-APIs#7
  - Server Side Object Encryption
- NFS v3 Support (stable)
  - Improve performance of the list objects, the ls command
- Improvement of the Multi Data Center Replication
- Searching objects by a custom-metadata
v2.1
- Hinted Hand-off
- Improve compatibility S3-APIs#8
  - Objects Expiration into a Bucket
  - Object Versioning
- Kubernetes Persistent Volumes Support
- Integration with distributed computing frameworks#2
  - Hadoop integration
  - Spark integration
- Improve Web GUI console, LeoFS Center (option)
v2.2
- Data Deduplication
- Improve compatibility S3-APIs#9

Versioning Policy

LeoFS adheres to the versioning policy from v1.3.3.

Licensing

LeoFS is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

leofs's People

Contributors

Stargazers

Watchers

Forkers

byplayer marutha rakutentech matsue yamt seanjensengrey kthguru cy-satoshi-nakamura alepharchives koomee kengon mikcire mykook huainansto thinkhero weifengyao feiquanbifeng arikachen eagle518 statc0der atula project-fifo xokao zhanglei yoya rr-paras-patel xiufan usingsystem doubaokun chinnurtb akiradeveloper trociny e42s hungld yunlongcn kozlov-v gitinsky atenz maxel5 firejun rgordeev llsdimple xubingyue stamhe fysoft2006 sxhao hunterfu bindu21 windkit gitssie linearregression witeman hjwsm1989 chintran27 cloudxtreme dawnlab st3alth guangminglion gitter-badger whidbey hiroaki-iwase teotikalki uzzz sneal rowhit bhyvex walthinking loulancn getong jegade jacktang joe2hpimn emrul hbcbh1999 singularitymatrix neuroradiology gaoxiaojun yut148 ghosthamlet balajirajan zachlungu openaccess solertis yosukehara satishgoda mocchira sunsparc64 w796933 a29107a henrypfhu yucl80 vstax chenokay eminarcissus xiaofengliao githubch rathishvbr lifeinmirrors cn-hup practise2017

leofs's Issues

Has omissions an object of rebalance

Write "s3cmd" setup document

Related issues:
- #19
- #20

Improve re-launch storage process

When regularly stop the storage-process, It writes current status in a file. Then it restarts the storage-process when reading the file.

Retrieve total number of messages into the queue (manager-console)

Able to recognize and expect when consumption of queue's message completed for administrators
- This function can use from manager-console
  - command: status $node

Write Java's S3 client code examples

Survey F2FS(Flash Friendly File System)

Currently we'd recommend LeoFS users using xfs where to store raw data files,
but F2FS might become more suitable one when using SSD.
let's check it out!

links
- http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/filesystems/f2fs.txt;h=dcf338e62b71108a3a02e0198edf0eb564930121;hb=refs/heads/master

Bucket's data type replace from string to binary

NOT able to remove an endpoint

NOT handle over 32bit(4GB) values in leo-gateways's snmp

Output value with 'MB'

Support for deletion of chunked object

Deletion chunked objects are set as the object of a compaction.

Support for "Range request API" for a large object

Able to retrieve a part of an (large) object from the storage-cluster
- Make available log-data analysis, etc
Reference:
- GET Object

Support for deletion of objects in the buckets

Fix and Improve cache functions in Gateway

Make compaction's control(suspend/resume) finer granularities

ASIS: per AVS(a raw file on OS filesystem)
TOBE: per some time unit ( ex. 5sec/1min ... )
TODO
- To prevent gen_server from blocking, must do compaction on another process.
- To realize time unit controlable, a compaction process must invoke receive regulaly.
Files to be modified
- leo_object_storage_server
- leo_compaction_manager_fsm
- leo_object_storage_api

Able to add a bucket with NOT exists access-key

Improve configuration

Object of Configs
- Application.config (app.config)
- VM.config

Add VM-values for the monitor and administrators

Elements:
- vm_version
- process_limit
- kernel_poll
- thread_pool_size

Improve synchronization of s3-endpoint between manager and gateway

Currently, s3-endpoint set up on manger before launch gateway

Improve compaction mechanism

Able to gradually compact data
- Assignable data-compaction target raw-file(s)
Able to suspend/resume compaction
- Need to implement commands of compact status, compact suspend, compact resume and compact start in LeoFS-Manager

Import LeoFS-related all source codes

Improve compaction performance

While doing compaction, Since a src file will be read by ONLY sequential read AND a dst file will be written by ONLY sequential write, So we expect to improve compaction performance by using posix_fadvise.

links
- http://erlang.org/doc/man/file.html#advise-4

Implement termination of storages for the restriction of file-destruction

Target libs
- leo_object_storage
- leo_backend_db

Modify object-cache related documents

Need to modify by changed object-cache configuration (gateway) from v0.14
Easily understand that for administrators and developpers

build error following quick start

I am writing an installer for my dev env using information found on the quickstart guide.

#!/bin/bash
#http://www.leofs.org/docs/getting_started.html
#http://www.erlang.org/download_release/16
#https://github.com/leo-project/leofs

sudo apt-get install libncurses5-dev
http://www.erlang.org/download/otp_src_R14B04.tar.gz
tar xzf otp_src_R14B04.tar.gz
cd otp_src_R14B04
./configure --prefix=/usr/local/erlang/R14B04 --enable-smp-support --enable-m64-build --enable-halfword-emulator --enable-kernel-poll --without-javac --disable-native-libs --disable-hipe --disable-sctp --enable-threads
make
sudo make install

git clone https://github.com/leo-project/leofs.git
cd leofs
make
make release

the second last line of code produces this output

==> leo_commons (get-deps)
==> proper (get-deps)
==> jiffy (get-deps)
==> leo_logger (get-deps)
WARNING: deprecated port_envs option used
Option 'port_envs' has been deprecated
in favor of 'port_env'.
'port_envs' will be removed soon.

==> meck (get-deps)
WARNING: deprecated port_envs option used
Option 'port_envs' has been deprecated
in favor of 'port_env'.
'port_envs' will be removed soon.

==> bitcask (get-deps)
WARNING: deprecated port_envs option used
Option 'port_envs' has been deprecated
in favor of 'port_env'.
'port_envs' will be removed soon.

==> eleveldb (get-deps)
==> leo_backend_db (get-deps)
==> leo_object_storage (get-deps)
==> leo_mq (get-deps)
==> leo_redundant_manager (get-deps)
==> bear (get-deps)
==> folsom (get-deps)
==> leo_statistics (get-deps)
==> leo_s3_libs (get-deps)
==> leo_manager (get-deps)
==> lz4 (get-deps)
==> leo_ordning_reda (get-deps)
==> leo_storage (get-deps)
WARNING: deprecated port_envs option used
Option 'port_envs' has been deprecated
in favor of 'port_env'.
'port_envs' will be removed soon.

==> cherly (get-deps)
==> ecache (get-deps)
==> cowboy (get-deps)
==> leo_gateway (get-deps)
==> rel (get-deps)
==> leofs (get-deps)
==> leo_commons (compile)
ERROR: OTP release R15B01 does not match required regex R14B04|R15B02|R15B03
ERROR: compile failed while processing /home/paul/leofs/deps/leo_commons: rebar_abort
make: *** [compile] Error 1

Write quick-start for building storage-cluster

Support for "Multi-Part API" for a large object

Reference is "Amazon S3: Multipart Upload".

basho_bench

Could you detail a bit how you got the bash_bench driver working?

ibrowse is up and running
basho_bench http raw driver works

but the leofs driver gives me:

10:21:59.096 [debug] Supervisor folsom_sup started folsom_sample_slide_sup:start_link() at pid <0.70.0>
10:21:59.097 [debug] Supervisor folsom_sup started folsom_meter_timer_server:start_link() at pid <0.71.0>
10:21:59.098 [debug] Supervisor folsom_sup started folsom_metrics_histogram_ets:start_link() at pid <0.72.0>
10:21:59.098 [info] Application folsom started on node nonode@nohost
10:21:59.111 [debug] Supervisor basho_bench_sup started basho_bench_stats:start_link() at pid <0.65.0>
10:21:59.114 [debug] ID 1 generating range 0 to 20833
10:21:59.209 [error] Failed to initialize driver basho_bench_driver_leofs: {'EXIT',{undef,[{basho_bench_driver_leofs,new,[1],[]},{basho_bench_worker,worker_idle_loop,1,[{file,"src/basho_bench_worker.erl"},{line,201}]}]}}

Refactor method of retrieving an object AND input an object

Implement 'synchronize' function in Manager

Synchronized object(s) by "unit of a file" and "unit of storage-node"
- Able to recover manually

Handle invalid parameter on manager-console

The correct processing of commands which does not need parameter(s)

Avoid polluting OS page caches during compaction

The characteristic of compaction causes polluting OS page caches.
The solution is already here(see below link)

links
- https://issues.apache.org/jira/browse/CASSANDRA-1470
- http://www.erlang.org/doc/man/file.html#advise-4 (use dont_need flag)

Wrong list of objects in the bucket

Exists plural the same filename objects when using "get-objects"

NOT include a large message in a log

It has occurred to degradation of temporary performance.

Refactor "repairer" from gen_server to common-module

NOT allow duplication registration of 'user_id' into the s3-credential

Deprecate parameterized module

Target module is 'leo_object_haystack'
- reference: http://www.erlang.org/news/35

Gateway always responds with 403

Can't get a simple developer model deployment to run. Everything comes up, but spits out a 403 Forbidden when I access using s3cmd or DragonFS. Any logs I can look at to figure why it's failing ? I'm new to Erlang, but have loads of dev experience otherwise.

For the secure application

Support for compressor/decompressor replacement from snappy to lz4

Change type of key from 'string' to 'binary' in storage-engine and object-storage

We reduce cost of "list_to_binary/1" for improve performances.

Support for "s3cmd (s3-client/tool)"

Results:

    New settings:
      Access Key: 05236
      Secret Key: 802562235
      Encryption password: 
      Path to GPG program: /usr/bin/gpg
      Use HTTPS protocol: False
      HTTP Proxy server name: 
      HTTP Proxy server port: 0

    Test access with supplied credentials? [Y/n] Y
    Please wait, attempting to list all buckets...
    ERROR: Test failed: no element found: line 1, column 0

Adjust the start timing of RPC

Benchmark docs unclear

A bit unsure on how to explain these things so I'll put the few issues we discussed here:

The bucket must be created before tests are ran, using the special test user (command and key should be given from the documentation directly)
The bucket name cannot contain _ (underscore), so the "test_bucket" provided as example plain doesn't work, and it's not obvious from this page alone that it isn't
The benchmark file given as an example at the end doesn't work without extra configuration, you have to edit the file to set the right port and bucket name first

Unify name of LeoFS's processes

We are difficult to recognize the LeoFS's processes on Erlang's console.

This function implement for leo-tamer (Web GUI console)

Protect raw files against any operation mistakes by `Mandatory lock`

ONLY OS processes of LeoFS Storage should be able to handle raw files.

NOTE
As you may know, there are some limitations on the linux Implementations(see second link's chapter 0). But in our usecases, we have no effect with those limitations

links
- http://www.thegeekstuff.com/2012/04/linux-file-locking-types/
- http://kernel.org/doc/Documentation/filesystems/mandatory-locking.txt