Giter Club home page Giter Club logo

central-orchestrator's Introduction

Central Orchestrator

Central Orchestrator collects all other robolaunch components under the same roof. Makes robolaunch accessible and usable by the end users.

Central Orchestrator integrates other robolaunch components, and generates an end user consumable endpoints.

Table of Contents

Overview

The aim of this project is to make robolaunch accessible and usable for everyone.

  • Kubernetes Integration
  • User Management
  • Resource Management
  • Observability & Monitoring
  • Robot Management

Quick Start

After cloning this repository:

  1. Build the application with Maven:
mvn clean install
  1. Install and configure kogito components using this docker-compose file.

  2. Start the Central Orchestrator in development mode:

mvn quarkus:dev

Aims & Roadmap

  • Reducing the entry barrier for robotics
  • Make robotics development and deployment easier and faster.

Contributing

Please see this guide if you want to contribute.

central-orchestrator's People

Contributors

mert18 avatar tunahanertekin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

central-orchestrator's Issues

Hybrid robot - create physical instance script

What would you like to be added?

Fill out the correct script format to create a physical instance on a robotics cloud.

Why is this needed?

To create hybrid robots, physical instance is needed.

GraphQL integration

What would you like to be added?

Integrate graphql data with the application to check some resources.

Why is this needed?

To see all the available resources and operate on them.

Fix all potential null access warnings

What would you like to be added?

Fix all potential null access warning in kubernetes java client using Optional class.

Why is this needed?

To make code more clear and readable, null pointer error safe.

user operations refactor

What would you like to be added?

Refactor the all user, organization, and team operations.

Why is this needed?

Ensurance

Efficiently create ApiClients

What would you like to be added?

Create as less as possible Kubernetes ApiClients.

Why is this needed?

Due to performance considerations.

Buffering logic to dynamic super clusters

What would you like to be added?

Implement a dynamic buffering logic for super clusters.

Why is this needed?

To have distributed buffered clusters, ready to instantiate and use.

Pricing method

What would you like to be added?

Implement a pricing method.

Why is this needed?

In order to track usage of resources, and billing them.

Trim Subnet CIDR While Creating Virtual Link

What happened?

While creating a cloud instance, a virtual link job is being executed with manifest:

apiVersion: batch/v1
kind: Job
metadata:
  name: add-vl
  labels:
    app: add-vl
spec:
  template:
    spec:
      # robolaunch tenancy labels
      nodeSelector:
        robolaunch.io/cloud-instance: <CLOUD-INSTANCE>
        robolaunch.io/department: <DEPARTMENT>
        robolaunch.io/organization: <ORGANIZATION>
        robolaunch.io/super-cluster: <SUPER-CLUSTER>
      containers:
      - image: nicolaka/netshoot:latest
        command: [ "/bin/bash", "-c", "--" ]
        #args: [ "ip link add subnet-3 type dummy; ip addr add 10.10.103.0/32 brd + dev subnet-3 label subnet-3:0; ip link set dev subnet-3 up;" ]
        args: [ "ip link add <subnet-id> type dummy; ip addr add <subnet-network-address>/32 brd + dev <subnet-id> label <subnet-id>:0; ip link set dev <subnet-id> up;" ]
        imagePullPolicy: IfNotPresent
        name: add-vl
        securityContext:
          capabilities:
            add: ["NET_ADMIN", "SYS_ADMIN"]
          runAsUser: 0
      restartPolicy: Never
      hostNetwork: true

Since the input <subnet-network-address> is gathered from cloud instance's subnet's spec.cidrBlock field, orchestrator sets the value wrong such as 10.10.1.0/24/32. Job seems completed but it returns error log.

What did you expect to happen?

If the cloud instance cluster CIDR is X.Y.Z.T/24, I would expect orchestrator to set <subnet-network-address> X.Y.Z.T/32. (/32 is static postfix) And the job should be completed successfully.

How can we reproduce it (as minimally and precisely as possible)?

After cloud instance creation, see logs of virtual link job:

kubectl logs -f pod/add-vl-<POD-POSTFIX>

Anything else we need to know?

No response

Implement Physical Instance Connection Hub Integration Flow

What would you like to be added?

For complete flow, reference this tutorial in Connection Hub Wiki.

To connect a physical instance (connection hub member) to a cloud instance (connection hub host), these steps should be followed respectively:

  • Setting up connection hub in cloud instance
    • by installing operator
    • by creating connection hub resource
    • continue if connection hub is ready to operate
  • Registering a physical instance to cloud instance
    • by creating physicalinstances.connection-hub.roboscale.io/v1alpha1 custom resource
  • Sending installer script to physical instance and executing
  • Waiting for physical instance to be connected to cloud instance
    • by watching connectionhubs.connection-hub.roboscale.io/v1alpha1 custom resource
    • continue if connection hub is connected over multicast and waiting for credentials
  • Add physical instance credentials to physicalinstances.connection-hub.roboscale.io/v1alpha1 custom resource in cloud instance

Why is this needed?

It's needed to connect a physical instance to a cloud instance. (Related to robolaunch/fleet-operator#3)

Memory usage increases non-stop and causes application to restart

What happened?

After starting the main kogito processes (provider, region and super cluster flows), memory usage starts increasing and does not stop. After some time, infinispan restarts and caches are cleaned. Processes keep showing in dataindex but not in infinispan. And they are not accessible anymore.

What did you expect to happen?

.

How can we reproduce it (as minimally and precisely as possible)?

Start the application and start non-dying processes.

Anything else we need to know?

No response

Sign in with Google

What would you like to be added?

Sign in with Google

Why is this needed?

Registration process takes time, some users may want to use Google.

Keycloak dynamically arranging the realms

What would you like to be added?

Users must only be able to connect to realm they are belong to. Their realms must validate the tokens, not the main realm. In order to set up this, we will find a way to create dynamic configurations

Why is this needed?

Seperation of concerns.

Create organization refactor

What would you like to be added?

Re-configuring requests & responses and models in creating organization

Why is this needed?

request and response types are not as expected.

Robot Operations

What would you like to be added?

Add CRUD operations for both cloud and hybrid robots.

Why is this needed?

Robots are the most important part in the project. Committing operations on them should be easy and fast.

Fleet endpoints

What would you like to be added?

create fleet endpoints for

  • Organization
  • Team
  • Robotics cloud

Why is this needed?

.

Policies, DMN integration

What would you like to be added?

Add DMN policies for needed operations.

Why is this needed?

To make sure the policies are enforced correctly.

Protect admin endpoints

What would you like to be added?

Protect admin endpoints.

Why is this needed?

To prevent any security problem.

Add more details to robotics clouds.

What would you like to be added?

Add detailed information when returning robotics clouds.

Why is this needed?

To provide more information about robotics clouds.

Follow Platform Component's Information From Platform Map

What would you like to be added?

Details about platform components will be available here. Orchestrator should use latest production release while creating & operating robolaunch Robotics Cloud.

Why is this needed?

It's needed to track platform dependencies and provide consistency between Robotics Cloud components.

Handle All exceptions

What would you like to be added?

Currently not all exceptions are caught. Catch and handle all exceptions as possible.

Why is this needed?

It is a necessity for error management.

Marketplace public robots

What would you like to be added?

While creating robot, user selects if the robot will be public or not. If the robot is public, the robot yaml will be pushed to minio, public robots folder(under marketplace bucket).

Why is this needed?

Thus the robots can be made public, available for everyone.

Manage freeIPA admin cookies

What would you like to be added?

manage and delete them when the operation finished.
We have many operations that does it, but not every request. Need to add to each request.

Why is this needed?

to ensure collecting garbages efficiently.

Namespace integration

What would you like to be added?

Create CRUD operations for Namespaces.

Why is this needed?

In order to seperate resources, we need namespaces.

Add extra information for user.

What would you like to be added?

Create Phone number,
allow communication,
allow marketing fields on freeipa

also reserve fields for

General Notifications

Billing Updates

Project Updates

Newsletter

enter email and phone numbers for all of them.

Why is this needed?

add extra information about user.

Dynamic configs, master config processes on kogito

What would you like to be added?

In order to create configs and update them dynamically. Add a master config processes for those who need it.

Why is this needed?

To change configurations dynamically.

Update Tenancy Labels

What would you like to be added?

Duplicate of robolaunch/robot-operator#62.

Current labels for tenancy:

  • Organization: robolaunch.io/organization
  • Team: robolaunch.io/team
  • Region: robolaunch.io/region
  • Buffer Instance: robolaunch.io/buffer-instance
  • Cloud Instance: robolaunch.io/cloud-instance
  • Physical Instance: robolaunch.io/physical-instance

Necessary labels for tenancy:

  • Organization: robolaunch.io/organization
  • Team: robolaunch.io/team
  • Region: robolaunch.io/region
  • Cloud Instance: robolaunch.io/cloud-instance
  • Cloud Instance Alias: robolaunch.io/cloud-instance-alias
  • Physical Instance: robolaunch.io/physical-instance

Why is this needed?

It was an early assumption that cloud instance name matches with the domain name of the cluster but it doesn't in virtual cluster architecture. Since the most of the logical flow uses cloud instance name, it will be guaranteed that it will match with the cluster domain name instead of using buffer instances.

robot operations refactor

What would you like to be added?

Refactoring robot operations.

Why is this needed?

enhancement. robolaunch/robolaunch#11

Implement Watch API

What would you like to be added?

Some watch operations are done manually, by polling. Make them watch with K8s API.

Why is this needed?

To produce more clean and readable code.

Connect to super cluster with admin user

What would you like to be added?

Admin kubeconfig is currently taken from local directories. In order to make this work, we need to create ApiClients with keycloak admin credentials.

Why is this needed?

We need this to be able to connect to super cluster.

Test robotics cloud operations

What would you like to be added?

test start, stop, and terminate

Why is this needed?

ensure system works as expected.

Fix Kogito MGMT and Task Consoles

What happened?

Kogito processes cannot get the process nodes.

What did you expect to happen?

Admin must be able to trigger nodes on the mgmt and task console.

How can we reproduce it (as minimally and precisely as possible)?

f

Anything else we need to know?

No response

Cloud Instances Response

What would you like to be added?

Frontend receives "Cloud Instances" with their kogito process ids.

Why is this needed?

To send cloud instance resource to frontend.

Add more namespaces to subnet

What would you like to be added?

Add the following namespaces on subnet create:

  • submariner-k8s-broker
  • connection-hub-system

Why is this needed?

.

Kubernetes RBAC

What would you like to be added?

User must be able to connect to a cluster with its keycloak profile. Thus, cluster-wide security will be established. In order to achieve this issue, we must see that user performs operations on cluster with their context.

Why is this needed?

With the security concerns in mind, user must only be able to perform actions that they are authorized.

User policies on FreeIPA

What would you like to be added?

Add user policies to FreeIPA fields.

Why is this needed?

To check how many resources can a user create etc.

Convert all "teamName"s to "teamId"

What would you like to be added?

There are two fields in FreeIPA: "teamName" and "teamId". We are currently showing the teamName to the end user. But the unique one is the teamId. All the incoming requests must use teamId instead of teamName. There are some parts using teamName, and some others use teamId in the code. Fix this inconsistency.

Why is this needed?

To make code as consistent as possible.

Generate Hybrid Robot Scripts

What would you like to be added?

Generate hybrid robot scripts so minio became accessible from robot,
add minio multi-tenancy
aws authorization with tokens

Why is this needed?

Authorize robots to minio.
Perform physical instance operations

Get User's robotics clouds

What would you like to be added?

Get rc's that user created.

Why is this needed?

non-specific group members wiill only see their robotics clouds.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.