Giter Club home page Giter Club logo

distributed-computing's Introduction

Distributed Computing

Distributed computing refers to a network of computers working together to solve a problem or complete a task. Instead of a single computer handling all the processing, the workload is divided among multiple machines, allowing for faster and more efficient processing.


โš ๏ธ This repository contains submodules, add --recurse-submodules when cloning

Scope

This is a basic implementation of a generic distributed computing system that can be used to distribute any task across a network of computers. The clients and servers can be any type of computer, including desktops, servers, and even phones. There are no limitations (I hope) to the types of systems that can be used, and the network can be heterogeneous, meaning that different systems with different architectures and specifications can be used together.

Implementation

Introduction

The server is responsible for managing the tasks and distributing them to the workers. The workers, also known as clients, are computers that are waiting for tasks to be assigned to them. There unique role is to execute tasks.

Connections

Server and clients are connected through bidirectional sockets. Moreover, a web server is running on the server.

How it works

When the server receives a new task, it checks to see if there are any available workers. If there is at least one, it sends the task ID to one worker through the socket. The worker receives the task and may need additional data to complete it. To obtain this data, it fetches the webpage associated with the task. Once it has all the necessary data, it runs the task, sends the results back to the server through the socket, and waits for another task.

Running the system

  • Start your server (check the examples below for more information)
  • The server is now ready to establish connections, and clients can be set up by running the os-image on their machines.

Build the lib

To build the lib, just run the appropriate script.

Build the clients' os-image

To build the ISO file for the clients, navigate to the /os and run make os-image, following the instructions provided. Note that you will need to download a Linux kernel and busybox.

Examples

A template program can be found in /examples/template/, which provides a minimal implementation. Other examples are forthcoming.

Warnings

This very basic distributed computing program is unstable. I am still working on. There are a lot of issues: it is not efficient, it is not user-friendly, etc ... but it should works. I am open to any advice/helps/contributions, so fill free to help the project !

TODO

  • To test. Do I really need to set up different local IP addresses for each client? I guess yes
  • Implement a "delete client" function and don't crash on socket's client interruption
  • Change my dirty system of mutex
  • Implement the ability to stop workers if necessary
  • Display statistics on the index file
  • Memory leak
  • Remove the maximum length limit for task character length and client return data
  • Use the same connection/socket between the client and webserver (Connection keep-alive)

distributed-computing's People

Contributors

abadiet avatar

Watchers

 avatar

distributed-computing's Issues

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.