Giter Club home page Giter Club logo

bluebridge's Introduction

DOI

BlueBridge

An (aspiring) distributed shared memory system intended to simplify large-scale computing for scientists and researchers. In BlueBridge, remote memory pages are identified on the basis of IPv6 addresses. Access and data transfer is connectionless and can easily be transferred across machines. BlueBridge relies on a virtual or SDN switch to route and manages requests.

BlueBridge has several features:

  • A memory daemon which exposes remote memory to clients in the form of a managed virtual address cache or simply as userspace block device.
  • A P4 template implementing a BlueBridge memory protection and cache coherence protocol directly in the switch.
  • A specialized RPC framework on the basis of Apache Thrift which uses shared global memory instead of transporting data over TCP streams.
  • BlueBridge supports a high-performance linux networking as well as a userspace DPDK stack for fast data transmission.

To build BlueBridge, simply run sudo ./install.sh and choose the configuration you want to run.

The bluebridge application directory (ip6/apps/bin) contains several toy applications and benchmarks:

  • testing A simple benchmarking application that tests all functions of the basic BlueBridge API. From the ip6 directory it can be run as: ./apps/bin/testing -c CONFIG -i ITERATIONS -t THREADS [-b] A sample configuration file can be found in ip6/distmem_client.cnf. ITERATIONS specifies the number of roundtrips per API call and the overall size of the memory allocation (beware of exceeding the memory limit of your local machine), THREADS the number of threads to use (default is one), and -b wether to use batch mode (higher throughput) or single block transmissions (low latency).
  • bb_disk bb_disk uses BUSE and NBD to expose BlueBridge as a userspace block device to the operating system. This block device can be formatted as any file system or used as swap device.
  • event_server The event server is the simple BlueBridge memory server. It also requires a configuration file, which is given as distmem_server.cnf. The server can be launched using sudo ./apps/bin/event_server -c [CONFIG]
  • thrift BlueBridge also contains a modified Thrift version which uses shared global BlueBridge memory instead of directly transporting data. To run thrift benchmarks run: ./apps/thrift/tcp_client -c [CONFIG] or ./apps/thrift/ddc_client -c [CONFIG] -i [ITERATIONS] Further information can be found in the thrift application and source repository.
  • server (DEPRECATED) An old implementation of the BlueBridge server. Instead of using a specialized server networking backend, it uses the generic client send and receive calls. It is no longer maintained.
  • rmem_test (DEPRECATED) A toy application to verify the functionality of the BlueBridge virtual memory backend. This file runs several small test computations which uses a managed mmap space and cache layer. It supports multithreading and several different virtual backends. The virtual memory system is currently not maintained and not stable. To run the client page system: ./apps/bin/rmem_test use: {CONFIG] [PAGES] [FRAMES] [MEMORYTYPE] [PROGRAM]" Options for the memory type are: disk, rmem, and mem Options for the program are: wc (wordcount) and wc_t (threaded wordcount)

To run tests, the ip6 repository provides a Mininet emulator. It can be launched using the sudo python run_emulator.py command in the ip6 folder. Mininet also creates a sample configuration file, which can be found in ip6/tmp/config/distMem.conf after launching the Mininet emulator. To test thrift, run the emulator with the --thrift-tcp or thrift-udp option. It will start Thrift DDC/TCP servers in addition to the BlueBridge memory servers.

bluebridge's People

Contributors

acarb95 avatar bbouchrd avatar clementfung avatar fruffy avatar wantonsolutions avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

zhilongzheng

bluebridge's Issues

ToDos

  • Move BlueBridge to physical hardware.
  • Get Mininet RTT latency down to <=10 us.
  • Get hardware RTT latency down to <=10 us.
  • Integrate the Intel DPDK.
  • Implement client-side multithreading and sharing support (IP-Header IPs).
  • Implement better paging strategy (FIFO -> LRU).
  • Switch to AF_XDP sockets instead of PACKET_MMAP
  • Test Userspace DMA.
  • Add a dispatcher layer for multithreaded client read/write.
  • Add an event-based server-model.
  • Add the ability of batched writes (over MTCP integration or IP fragmentation).
  • Add the ability of batched reads
  • Use a P4 mininet switch instead of ovs (tighter networking integration).
  • Use a custom, optimised P4 switch.
  • Build a RocksDB/ArangoDB benchmarking infrastructure.
  • Implement native migration support of pages.
  • Implement a switch-level TLB and caching strategy.
  • Figure out a good fault-tolerance policy.
  • Add retransmits
  • Add congestion control
  • Add raiding support to the system.
  • Implement raid support on the switch and servers.
  • (Use userfaultfd instead of mmap segfaults.)
  • (Figure out a compute prefetching and scheduling policy.)
  • Compile BlueBridge into a linkable library.
  • Make the library compilable with C++
  • Add an actual testing framework.
  • Create a BlueBridge socket info struct which is passed along functions.
  • Implement a netmap backend.
  • Switch from makefiles to meson or cmake.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.