"This Project has been archived by the owner, who is no longer providing support. The project remains available to authorized users on a "read only" basis."
A Large-scale Cloud Emulator provides ability to
- emulate data center physical topologies and network devices including hosts, switches, and routers.
- emulate a large volume of compute nodes (more than 100K) with limited physical hardware resources.
- conduct the performance test for a target project's (e.g., Alcor) control plane with a large-size of VPC having more than 1M VMs.
- automatically create and conduct different performance test scenarios and collect results.
There are many different hardware resource management platform in the field, currently we choose two platforms to investigate and create our prototype:
- Kubernetes cluster with Meshnet CNI
- Distrinet with LXD containers
The following diagram illustrate the high-level architecture of Merak on a kubernetes cluster using Meshnet CNI and the basic workflow to emulate Alcor's control plane for creating VMs in the emulated compute nodes.
- Scenario Manager: create the required topology and test scenarios.
- K8S-Topo: deploy pods with the given topology.
- Merak Network: create network infrastructure resources, e.g., vpcs, subnets, and security groups.
- Merak Compute: register compute nodes informantion, create VMs and collect test results from merak agents.
- Merak Agent: create virtual network devices (bridges, tap devices and veth pairs) and network namespace for VMs, collect test results and send the results back to merak compute.
In order to provide more virtual and emulated resources with limited hardware resources, three possible solutions are investigated and developed in this project:
- Docker-in-Docker
- Kubernetes-in-Kubernetes (KinK)
- Kubernetes cluster in virtual machines
For more detail design and information, please refer to the docs folder in this repository.
Kind: Simple Deployment and E2E Test
This test will bring up Merak and Alcor in a single master node Kind Kubernetes cluster.
-
Minimum Machine Requirements (Our tests were ran on AWS t2.2xlarge ec2 instances)
- 16GB RAM
- 8 Core CPU
-
Update
sudo apt-get update
sudo apt-get install make
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh
export PATH=$PATH:/home/ubuntu/.linkerd2/bin
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.17.0/kind-linux-amd64 && chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind
curl -LO https://dl.k8s.io/release/v1.26.0/bin/linux/amd64/kubectl && sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
sudo apt-get install docker.io
- Add current user to docker group (for running docker without sudo)
sudo groupadd docker
sudo gpasswd -a $USER docker
newgrp docker
You can deploy Merak and Alcor in Kind with the command below.
git clone https://github.com/futurewei-cloud/merak.git
cd merak
make kind-alcor
Please wait for all pods to be in running state as shown in the picture below before proceeding to the next step. This should take approximately 5 minutes.
You can use the prebuilt test tool as shown below.
./tools/teste2e/bin/teste2e
This will create 5 hosts with 10 VM each. Once everything is created, you can test network connnectivity as shown below.
-
Run
kubectl get pods -Ato see all vhost pods.
-
Merak uses network namespaces to emulate VMs, run
kubectl exec -it -n <namespace of the pod> vhost-0 ip netns exec v000 ip ato get the IP address of the emulated VMv000inside of the emulated hostvhost-0.
-
Ping the VM
v000onvhost-0from a different VM onvhost-1with the following commandkubectl exec -it -n <namespace of the pod> vhost-1 ip netns exec v000 ping (IP address from step 2)
Run the command below to clean up the Kind environment.
kind delete cluster
To build this project, please make sure the following things are installed:
- Go: https://go.dev/doc/install
- Protobuf: https://grpc.io/docs/protoc-installation/
- Go plugins for gRPC: https://grpc.io/docs/languages/go/quickstart/
- Docker: https://docs.docker.com/engine/install/
Then, the project can be built with:
makeBefore deploying Merak with Alcor, you will need the following.
- A Kubernetes cluster with flannel installed
- Helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
- Needed for Alcor
- Linkerd installed on the cluster
- openvswitch-switch installed on every node (
apt install openvswitch-switch)
NOTE: Please wait for all LinkerD pods and containers to be fully running before moving on to the steps below.
Once your cluster is ready, you can deploy the latest small scale development (one replica for every service) build of Merak and Alcor with the command below.
kubectl kustomize https://github.com/futurewei-cloud/merak/deployments/kubernetes/alcor --enable-helm | kubectl apply -f -
A successful deployment should take roughly 5 minutes for all pods to come to running state.
The deployed components are as follows:
- Merak Microservices and their Redis instances
- Merak Scenario Manager
- Merak Topology
- Merak Network
- Merak Compute
- Meshnet CNI
- Temporal
- Prometheus
- Alcor Microservices and their Ignite instances
- Port Manager
- Network Config Manager
- API Manager
- EIP Manager
- Dataplane Manager
- IP Manager
- Mac Manager
- Node Manager
- Quota Manager
- Route Manager
- Security Group Manager
- Subnet Manager
- VPC Manager
- LinkerD
The deployment settings such as container image and replicas can be changed by editing the kustomize file under deployments/kubernetes/alcor/kustomization.yaml and redeploying with
kubectl kustomize deployments/kubernetes/alcor --enable-helm | kubectl apply -f -
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent