dbiir / paraflow Goto Github PK

Write a detailed documentation on project aims, installation, usage guides and configuration explanations.

Segments are cached in memory before flushing to HDFS.
Distribtued locks are required to safely flushing in-memory segments without interrupting running queries.

Pixels is a new 'smart' columnar format developed by us, and it will be open source soon.

We need add support for Pixels, and integrate our optimizations over columnar storage into this project.

paraflow-presto module is for executing SQL queries using Presto.
This is a list of all related issues of this module, this can be seen as a mini task board.

paraflow-loader-producer module is for loading data into Kafka cluster.
This is a list of all related issues of this module, this can be seen as a mini task board.

1. Issue #45

Kafka AdminClient seems not be thread safe. When several clients are trying to create a topic at the same time, exceptions will be thrown.

Define LoaderProducer api.

Add validation support for configuration.

1. Check if lib dir exists under the directory of ParaFlow/dist/ParaFlow-xxx/.
2. If not exists, create the lib dir.
3. Copy jar files from RealTimeAnalysis/dist/bin/ to lib.
4. Copy jar files from Presto/presto-server/target/lib to lib.
5. Package the ParaFlow-xxx/ dir to a tar file as ParaFlow-xxx.tar.
6. scp tar file to each server specified in servers file.
7. Untar the tar file to specified dir in each server.

paraflow-tools is for convenient tools to compile, test and deploy paraflow system.
This is a list of all related issues of this module, this can be seen as a mini task board.

1. compile and test tool
2. deployment tool

Provide an open website using Github Pages. The website contains:

1. binary package download
2. usage documentation
3. project details.

paraflow-commons module is for common classes shared by several modules.
This is a list of all related issues of this module, this can be seen as a mini task board.

1. #19 Logger system
2. ConfigurationFactory
3. Exceptions

Support message filters and transformations before loading into Kafka.

TO BE PROCESSED LATER

A producer client for loading data into Kafka.
The client provides following apis:

1. send()
2. createTopic()
3. createUser()
4. createDatabase()
5. createFiberTable()
6. createRegularTable()
7. createFiberFunc()
8. registerFilter()
9. registerTransformer()

We shall design a configuration server, which is a centralized service for all collectors and loaders in the cluster. Each collector or loader listens on this service to get latest configuration parameters.

In this way, we can avoid copying the same configuration files all over the cluster, and tune the cluster on the fly without halting.

setup and learn a little about etcd. Intend to use it for leader election, metadata cache, etc.

Refactor MetaServer:

1. Service. Defines services provided to client in several ways:
   1. params. Necessary parameters to call this service.
   2. execution flow. Each service consists of a list of actions executing from first to the end.
   3. transaction. Deal with transaction based on needs.
2. Action. Defines basic unit of execution. It can be reused in different services.
3. ConnectionPool. A connection factory to get connection instance.
4. Connection. Handles low level execution. Each connection is bind to a TransactionController.
5. TransactionController. Responsible for connection transaction commit and rollback.
6. ParaFlowException. Describe all kinds of exceptions in MetaServer. It has five levels:
   1. DEBUG. This is used only for debugging by developers.
   2. INFO. Useful information provided back to clients.
   3. WARNING. Exceptions trigged by inappropriate user behaviours.
   4. ERROR. System errors which should be fixed, however, the system can still run.
   5. FATAL. Fatal errors. The system can no longer continue running.
7. ExceptionHandler. Define handler for exceptions, and convert exception to status.

Add models in paraflow-metaserver.

List all public interfaces of MetaServer.

Sub task of #22

API:

• List<String> listDatabases()
• List<String> listTables(String database)
• Database getDatabase(String database)
• Table getTable(String database, String table)
• Column getColumn(String database, String table, String column)
• Status createDatabase(Database database)
• Status createTable(Table table)
• Status deleteDatabase(String database)
• Status deleteTable(String database, String table)
• Status renameDatabase(String oldName, String newName)
• Status renameTable(String database, String oldName, String newName)
• Status renameColumn(String database, String table, String oleName, String newName)
• Status createFiber(String database, String table, long value)
• List<Long> listFiberValues(String database, String table, long value)
• Status addBlockIndex(String database, String table, long fiberV, String beginTime, String endTime, String path)
• List<String> filterBlockPaths(String database, String table, String timeLow, String timeHigh)
• List<String> filterBlockPaths(String database, String table, long fiberV, String timeLow, String timeHigh)

Models:

1. Database
   String name;
String locationUri;
User user;
2. Table
   Database database;
long creationTime;
long lastAccessTime;
User owner;
String tableName;
String tableLocationUri;
List<Column> columns;
3. User
   String userName;
String userPass;
String roleName;
long creationTime;
long lastVisitTime;
4. Column
   Table table;
String colName;
String dataType;
int colIndex;
5. Status (Enum type)

• OK
• DATABASE_ALREADY_EXISTS
• DATABASE_NOT_FOUND
• TABLE_ALREADY_EXISTS
• TABLE_NOT_FOUND
• COLUMN_ALREADY_EXISTS
• COLUMN_NOT_FOUND
• FIBER_ALREADY_EXISTS
• BLOCK_INDEX_ERROR

Implement a Web-based user interface, including the front-end and the back-end.
The Web UI will interact with existing Paraflow modules to provide a user-friendly layout optimization service.

Follow this PR in Presto to replace existing Parquet writer with a native one.

Currently, we only support Parquet.
ORC is as popular as Parquet, and it has well integrated with Presto.

We need add support for ORC.

Main loop for meta server start and stop

1. List all possible exceptions in MetaServer.
2. Define status code return to client.

MetaServer uses gRPC as the service definition tool.

1. Design the rpc service interface.
2. Add service implementation.

There are many useless css files in the paraflow-http-server module. Remove and compress them as much as possible.

A cache for metadata.

MetaServer is a standalone server for meta data management.
It keeps all metadata inside, and provide powerful and rich rpc interfaces.
This is a list of all related issues of this module, this can be seen as a mini task board.

1. Issue #5
2. Issue #20
3. Issue #21
4. Issue #27
5. Issue #42
6. Issue #43

basic kafka consumer client to pull messages from specified topic and partitions.

Implement paraflow-commons logger system.

paraflow-loader-consumer module is for loading data from Kafka into file system.
This is a list of all related issues of this module, this can be seen as a mini task board.

In ct.sh, testRP function should cd to corresponding dir and then execute mvn command.

I find some segments are staged on local tmpfs forever, and it seems no one is responsible for flushing them to HDFS any more.