Giter Club home page Giter Club logo

tsung-wei-huang / dtcraft Goto Github PK

View Code? Open in Web Editor NEW
143.0 16.0 25.0 46.43 MB

A High-performance Cluster Computing Engine

Home Page: https://tsung-wei-huang.github.io/DtCraft/

License: MIT License

Shell 5.03% Makefile 2.01% M4 0.83% C++ 91.47% HTML 0.04% CSS 0.02% JavaScript 0.03% CMake 0.01% C 0.57%
distributed-systems parallel-computing cluster-computing event-driven-programming reactor streaming dataflow-programming stream-processing-engine network-programming

dtcraft's Introduction

What's DtCraft?

DtCraft is a general-purpose distributed programming system based on data-parallel streams. It offers a new powerful programming model called stream graph to build parallel and distributed workloads. Once an application is cast into this framework, the kernel transparently performs job distribution for you. You don't have to worry about system programming and can focus on high-level development!

Unified Framework Stream Graph

Whether you are application developers or domain-specific engineers, you can use DtCraft for:

  • Distributed computing
  • Event-driven programming
  • Network programming
  • Data stream processing

To get your first DtCraft program up and running, visit QuickStart.

Design Goals

The goal of DtCraft is to help you write simple, easy, and effective code at cluster scale. The example below demonstrates a simple application to run two programs on two machines sending each other a message.

#include <dtc/dtc.hpp>
  
using namespace std::literals;  // for the use of string literal
using namespace dtc::literals;  // for the use of memory literal

int main(int argc, char* argv[]) {

  dtc::Graph G;

  auto A = G.vertex();
  auto B = G.vertex();

  auto lambda = [] (dtc::Vertex& v, dtc::InputStream& is) {
    if(std::string s; is(s) != -1) {
      std::cout << "Received: " << s << '\n';
      return dtc::Event::REMOVE;
    }
    return dtc::Event::DEFAULT;
  };

  auto AB = G.stream(A, B).on(lambda);
  auto BA = G.stream(B, A).on(lambda); 

  A.on([&AB] (dtc::Vertex& v) { (*v.ostream(AB))("hello world from A"s); });  
  B.on([&BA] (dtc::Vertex& v) { (*v.ostream(BA))("hello world from B"s); });
  
  G.container().add(A).cpu(1).memory(1_GB);
  G.container().add(B).cpu(1).memory(1_GB);

  dtc::Executor(G).run();
}

There are myriads of cluster computing frameworks such as Hadoop MapReduce, Apache Spark, Dryad, and Ray. We believe each has its reason to exist. DtCraft targets at:

  • Programming model. DtCraft leverages modern C++17 to offer a new programming model for cluster computing. Our model is very general and can implement most distributed computing patterns.

  • Performance. DtCraft is designed completely from the ground up using advanced software techniques in order to deliver the best performance.

  • Productivity. DtCraft allows you to easily bring up a parallel and distributed workload in only a few lines of code. It takes only a few steps to set up a cluster to get things up and running.

System Requirements

To install and run DtCraft, you only need the following:

  • 64-bit Linux machine(s) with Kernel 3.8 or higher
  • GNU C++ Compiler G++ v7.2 with -std=c++1z
  • GNU Autotool (autoconf, automake, libtool)

Learn More

Get Involved

dtcraft's People

Contributors

clin99 avatar gguo4 avatar tsung-wei-huang avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dtcraft's Issues

Control group support

The current master branch is adding the support for Linux control group. Launching agent requires sudo permission.

Ver 0.2.2 error in single-host distributed mode

Updated to ver. 0.2.2. Local mode works fine. But the encountered following error in single-host distributed mode. Error message in the master:

frankliu@arlz010:~/Projects/DtCraft$ sudo ./bin/dtc-master           
I  6016 2018-05-29 20:22:09 master.cpp:200] Master @127.0.0.1 [agent:9909|graph:9910|webui:9912]                                          
I  6016 2018-05-29 20:22:42 master.cpp:271] Agent 0 connected @127.0.0.1 [cpu:0|mem:135081242624|disk:17340207104]                        
I  6016 2018-05-29 20:23:20 master.cpp:300] Graph 0 connected @arlz010 [vertex:4|stream:6|container:4]                                    
W  6016 2018-05-29 20:23:20 master.cpp:303] Graph 0 doesn't fit with available resources                                                  
I  6016 2018-05-29 20:23:20 master.cpp:142] Graph 0 is removed from the master                                                            
I  6016 2018-05-29 20:23:30 master.cpp:300] Graph 1 connected @arlz010 [vertex:2|stream:2|container:2]                                    
W  6016 2018-05-29 20:23:30 master.cpp:303] Graph 1 doesn't fit with available resources                                                  
I  6016 2018-05-29 20:23:30 master.cpp:142] Graph 1 is removed from the master       

Error message in the submission window:

frankliu@arlz010:~/Projects/DtCraft$ sbin/submit.sh --master=127.0.0.1 /home/frankliu/Projects/DtCraft/example/hello_world                
I  1920 2018-05-29 20:26:33 executor.cpp:159] Executor @arlz010 [stdout:38211|stderr:38045]                                               
I  1920 2018-05-29 20:26:33 executor.cpp:161] Submit graph to master @127.0.0.1:9910                                                      
I 55040 2018-05-29 20:26:33 executor.cpp:173] Solution received      
[Graph 4]                         
+----+-----+------+-----------+-------------------+                  
|Task|Agent|Status|Elapsed (s)|Memory (peak/limit)|                  
+----+-----+------+-----------+-------------------+                  
Graph finished with 1 error(s): Resource request doesn't fit in cluster

OS: Ubuntu 17.10

Wrong status message in cluster mode

Sequence of commands:

frankliu@arlz010:~/Projects/DtCraft$ ./sbin/kernel.sh --action=start
start master @frankliu@arlz010:/home/frankliu/Projects/DtCraft [OK]
start agent @frankliu@arlz010:/home/frankliu/Projects/DtCraft [OK]
start agent @frankliu@arlz009:/home/frankliu/Projects/DtCraft [OK]

But returned results from ping are wrong:

frankliu@arlz010:~/Projects/DtCraft$ ./sbin/kernel.sh --action=ping
master @frankliu@arlz010:/home/frankliu/Projects/DtCraft [STOPPED]
agent @frankliu@arlz009:/home/frankliu/Projects/DtCraft [RUNNING]
agent @frankliu@arlz010:/home/frankliu/Projects/DtCraft [STOPPED]

The cluster appears to be alive and functional:

frankliu@arlz010:~/Projects/DtCraft$ ./sbin/submit.sh --master=arlz010 /home/frankliu/Projects/DtCraft/example/pi
I 47872 2018-05-29 20:46:30 executor.cpp:186] Executor @arlz010 [stdout:36041|stderr:34575]
I 47872 2018-05-29 20:46:30 executor.cpp:191] Submitting graph to master @arlz010:9910
pi estimation: 3.14146
I 59136 2018-05-29 20:46:32 executor.cpp:200] Solution received
[Graph 5]
+--------+------+-------+
|Topology|Status|Message|
+--------+------+-------+
|5-3     |Exit 0|Success|
+--------+------+-------+
|5-2     |Exit 0|Success|
+--------+------+-------+
|5-0     |Exit 0|Success|
+--------+------+-------+
|5-1     |Exit 0|Success|
+--------+------+-------+
Graph finished with 0 error(s)

Tried on the other node, works fine!

Fail to run master/agent under gcc 7.1.0 in centos 7

By following the instruction of DtCraft Distributed Mode (Single Host),
I tried to run bin/master and bin/agent, however, it crashed.

$ ./master
terminate called after throwing an instance of 'std::system_error'
what(): unshare failed: Invalid argument
Abort (core dumped)

Support for Vertex Program

The vertex will support running external program (exec) in the release 0.2.2. Example usage is like :

G.vertex().on([](dtc::Vertex& v) { std::cout << "ready for ls\n"; }).program("/bin/ls / -al");

The external program will have DTC_BRIDGES in its environment variable to store the file descriptor mapping for its input/output stream connections with other vertices or so.

unittest/archive.sh fail

Centos7, Linux 3.10.0-514.21.1.el7.x86_64
gcc 7.1.0

=====================================

DtCraft 0.2.1: ./test-suite.log

TOTAL: 9

PASS: 8

SKIP: 0

XFAIL: 0

FAIL: 1

XPASS: 0

ERROR: 0

.. contents:: :depth: 2

FAIL: unittest/archive.sh

0.010 s: ArchiverTest.BinaryPOD
0.020 s: ArchiverTest.BinaryStruct
0.023 s: ArchiverTest.BinaryBasicString
1.854 s: ArchiverTest.BinaryVector
1.155 s: ArchiverTest.BinaryDeque
1.573 s: ArchiverTest.BinaryList
1.573 s: ArchiverTest.BinaryForwardList
2.374 s: ArchiverTest.BinaryMap
2.053 s: ArchiverTest.BinaryUnorderedMap
1.608 s: ArchiverTest.BinarySet
1.524 s: ArchiverTest.BinaryArray
0.002 s: ArchiverTest.BinaryVariant
0.003 s: ArchiverTest.SmartPointer
0.002 s: ArchiverTest.ErrorCode

lt-archive is a Catch v2.0.1 host application.
Run with -? for options
-------------------------------------------------------------------------------
ArchiverTest.BinaryAtomicity
-------------------------------------------------------------------------------
unittest/archive.cpp:764
...............................................................................

unittest/archive.cpp:764: FAILED:
  {Unknown expression after the reported line}
due to a fatal error condition:
  SIGTERM - Termination request signal

0.000 s: ArchiverTest.BinaryAtomicity
===============================================================================
test cases:    15 |    14 passed | 1 failed
assertions: 98226 | 98225 passed | 1 failed

dtcraft web engr illinois edu website down

Describe the bug
http://dtcraft.web.engr.illinois.edu account isn't active

To Reproduce
Steps to reproduce the behavior:

  1. Go to git hub page of the project clicking on the DtCraft link in the top left
  2. Click on Quickstart
  3. See "This account isn't active." error

Expected behavior
To get your first DtCraft program up and running, visit QuickStart and Write the First DtCraft Program.

Hopefully it is just a misunderstanding with the managers of illinois.edu

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.