scp756-221 / term-project-cloudriven Goto Github PK

• Perform Gatling load experiments on the following simulations:

1. Light Load (s1/s2/s3 : 10/10/10 requests per second)
2. Medium Load (s1/s2/s3 : 40/40/40 requests per second)
3. Heavy Load (s1/s2/s3 : 80/80/80 requests per second)

• With the default system configuration set up:
  • Node Type: t3.medium
  • Number of EKS nodes deployed (default): 2
  • Number of replicas deployed (default):
  - S1: 1 S2: 1 S3: 1 DB:1
  • DynamoDB Read/Write Capacity (default):
  - ReadCapacityUnits for respective tables: 5

As a user,
I want to create/maintain/remove my playlists
So that I can listen to my favourite songs segregated by user-defined categories.

Based on the defined schema of the "playlist" table, create "playlist.csv" file which will be used for Gatling load test.

In order for the team to come up with a concrete plan, we need to understand how Gatling is used for load testing and how Grafana and Prometheus allow us to monitor performance/load metrics.

As a user
I want to be able to create and delete and retrieve my account information.
So that we can have a list of users

Before we can get the ball rolling, we need to have a clear understanding of how S1/S2 micro-services are communicating with db microservice to create/modify/delete users and music data on DynamoDB.

@bingsoorim @rezvanm

Go through Guide 3 to understand how Service Mesh (Istio) visualizes traffic movement within the system.

The workflow of deploying the new version:
Develop and prepare the deployment YAML file for the new version (update)
Using istio VirtualService‘s DestinationRule feature to split the traffic between the old version and a new version

1. Deploying the new update
2. If there is no bug move to Step 8
3. Reroute the traffic to the old version
4. Fix the bug and redeploy a new version
5. Router part of traffic to new service
6. If there is a bug move to Step 4
7. Delete old version
8. The process of deploying a new version is finished.
   @bingsoorim @beinfluential88 @rezvanm @BagheriReza

System Stress Test with Heavy Load
· Max number of requests to be served per second: 1,000
· Service Level Objective (SLO):
o There should be less than 10 errors in any period of 30 seconds.
o 90 percent of requests for all the 3 services should be served within 100ms.

EKS Cluster Configuration set for the target load test for User service (1,000 users per second)
· Node Type (default): t3.medium
· Number of EKS nodes deployed (scaled progressively up to): 17
· Number of replicas deployed (scaled progressively up to):
o s1: 85 s2: 1 s3: 1 db:40
· DynamoDB Read/Write Capacity (scaled progressively up to):
o ReadCapacityUnits for User table: 150
· Target Load introduced for respective services (per sec):
o User: 1000

EKS Cluster Configuration set for the target load test - COMBINED (200 users + 500 requests to music service + 300 requests to playlist)
· Node Type (scaled): t3.xlarge
· Number of EKS nodes deployed (scaled progressively up to): 10
· Number of replicas deployed (scaled progressively up to):
o s1: 50 s2: 150 s3:60 db:200
· DynamoDB Read/Write Capacity (scaled progressively up to):
13
o ReadCapacityUnits for User table: 50 Music table: 200 Playlist table: 100
· Target Load introduced for respective services (per sec):
o User (s1): 200 Music (s2): 500 Playlist (s3): 300

Implement the following simulations in Scala.

• UpdateUserSim (s1) ⇒ a new simulation that continuously calls update_user() with a specific interval. (HTTP PUT requests)
• WriteMusicSim (S2) ⇒ a new simulation that constantly calls create_song() and delete_song() in sequence with a specific interval. (HTTP POST/DELETE requests)
• WriteBothVaryingSim (S1/S2) ⇒ a new simulation that constantly calls the above 3 user/music functions simultaneously with a varying interval. (Each interval is randomly selected between 1 and 60 seconds)
• GetPlaylistSim (S3) ⇒ a new simulation that constantly calls get_playlist() with a pacific interval. (HTTP GET requests)
• WritePlaylistSim (S3) ⇒ a new simulation that constantly calls create_playlist() and delete_playlist() with a pacific interval. (HTTP POST/DELETE requests)
• PlaylistVaryingSim (S3) ⇒ a new simulation that constantly calls the above 3 playlist functions simultaneously with a varying interval. (Each interval is randomly selected between 1 and 60 seconds)

In order for the team to come up with a concrete plan, we need to understand how Gatling is used for load testing and how Grafana and Prometheus allow us to monitor performance/load metrics.

• Modify api-tpl.mak file

Overview of our microservice-based application

Adding the circuit breaker

Two make files are added to make automation for creating the cluster environment ( init.mak ) and building workload ( workload.mak ).

Preparing the cluster environment

make init.mak

Deleting the cluster environment from AWS

$ make -f init.mak stop

Gatling workload tests

The workload.mak is built to create automation for analyzing the workload for the user and music services.
Example of creating a workload for user service:

$ make -f workload.mak user number_user=5

Example of creating a workload for music service:

make -f workload.mak music number_user=5

The workload defines by the number_user.

The command to kill all the Gatling workload

make -f workload.mak stop

Implement Playlist service features:

• A function to get the playlist based on its id
• A function to create the playlist
• A function to delete the playlist
• A function to add songs to the current playlist
• A function to delete a song from the current playlist
• A function to edit the name of the playlist
• A function to edit the order of the songs in the playlist (Nice to have)

TASKS:

• add edit_playlist_name function
• add delete_song function
• add add_song function
• add delete_playlist function
• add create_playlist function
• add get_playlist function
• add list_all_playlists

Initialize the GitHub repository and move some files over.

The repo should include:

• Readme
• Front-end code (mcli)
• Back-end code (microservices)

Before we can get the ball rolling, we need to have a clear understanding of how S1/S2 micro-services are communicating with db microservice to create/modify/delete users and music data on DynamoDB.

Clean up some unnecessary files and functions from the Music service

Discovering configuration YAML files enable us to reimplement current services and figure out which kind of changes is necessary to be done on the cluster for adding a new service.
@bingsoorim @rezvanm @beinfluential88 @BagheriReza

DynamoDB not updating automatically

Based on a scrum meeting held on March 11:

Our application persists its data to AWS DynamoDB. We might use additional data stores (e.g, AWS RDS, Kafka, etc).

I just renamed issue19-s3 branch to dev. I believe the name of the branches should not be based on issues because that might lead to confusion and errors. To see the change in your local repo, please run these commands

$ git branch -m issue19-s3 dev
$ git fetch origin
$ git branch -u origin/dev dev
$ git remote set-head origin -a

This will safely rename the branch name in your local repo. I think it's also a good idea to merge issue26-s3 with dev.

• debug s3/app.py and POST endpoint

As a user,
I want to create, delete, query, and add songs and artists.
So that we can have a list of songs and their artist.

The playlist structure needs to be discussed.
@bingsoorim

To encounter the problem of scalability, we need to calculate the resources required to enable us to handle the workload.
We need to calculate the maximum capacity of services based on current configuration and resources. Then based on this calculation we can calculate the number of replicas that are essential to handle a required number of requests.

In case of failure, by utilizing the retry feature of Istio. It is possible to add mechanisms to retry requests independent of services.
https://istio.io/latest/docs/reference/config/networking/virtual-service/#HTTPRetry

In-service 3 which is a playlist, we need to create the table in Dynamodb.

• debug s3/app.py and /all endpoint
• debug db/app.py and /read_all endpoint

Based on a scrum meeting held on March 11:

Kubernetes is an open-source tool that manages the containers on a cluster; we need to choose the public cloud to run within this managed Kubernetes cluster.

EKS (Elastic Kubernetes Service) is Amazon’s managed Kubernetes service, and it simplifies the use of Kubernetes by taking care of the installation and operation of Kubernetes on a group of machines.

1. gatling--user.sh ⇒ To be used to start Gatling with a pre-defined number of users that will perform “ReadUserSim” simulations.
2. gatling--music.sh ⇒ To be used to start Gatling with a pre-defined number of users that will perform “ReadMusicSim” simulations.
3. gatling--playlist.sh ⇒ To be used to start Gatling with a pre-defined number of users that will perform “GetPlaylistSim” simulations.

1. Git commit message & Git issue title convention

   • Here I found a good example: https://gist.github.com/joshbuchea/6f47e86d2510bce28f8e7f42ae84c716

2. Git branch naming convention

   • Branch name should include the issue id
   • For instance, the branch name for issue #19 should be something like: issue19-s3

• Fix the Error: flags cannot be placed before plugin name: -n
• Loader log says: error: container cmpt756loader is not valid for pod cmpt756loader-ph56x