Giter Club home page Giter Club logo

scalable-microservice-demo's Introduction

Scalable Microservice Infrastructure Example with K8s Istio Kafka

Medium article: https://medium.com/@wuestkamp/scalable-microservice-demo-k8s-istio-kafka-344a2610eba3?sk=7404e77f2a42d21261707794afaed58d

Setup project

Install:

Then:

cp .env .env.local
# fill .env.local with correct values

./run/install.sh

Run

Edit infrastructure/k8s/rest-request.yaml deployment to fetch the correct Istio Gateway IP. Then slowly raise its replicas and watch the metrics.

Monitor Cluster

Grafana

Run Grafana and then import the ./infrastructure/grafana/dashboard.json.

./run/grafana.sh # admin:admin

Kiali

./run/kiali.sh # admin:admin

Use/build Services manually

Operation Service

build and run

cd services/operation

# pass env file
docker build -t operation-service . && \
    docker run --env-file ../../auth/kafka.env \
    --env-file ../../auth/mongodb.env \
    -p 80:5000 \
    operation-service


# build and pass all parameters manually
docker build -t operation-service . && \
    docker run -e "KAFKA_BOOTSTRAP_SERVERS=server:9092" \
    -e "KAFKA_SASL_USERNAME=XXX" \
    -e "KAFKA_SASL_PASSWORD=XXX" \
    -e "MONGODB_CONNECTION_STRING=mongodb://" \
    -p 80:5000 \
    operation-service

call and use

# add user
curl -X POST \
    -H 'Content-Type: application/json' \
    -d '{"name": "hans"}' \
    "http://localhost:80/operation/create/user-create"

curl -X POST -H 'Content-Type: application/json' -d '{"name": "hans"}' "http://localhost:80/operation/create/user-create"

# get operation status
curl "http://localhost:80/operation/get/6d232092-dceb-419f-bff1-2d686eace56c"

User Service

build and run

cd services/user

docker build -t user-service . && \
    docker run --env-file ../../auth/kafka.env \
    --env-file ../../auth/mongodb.env \
    user-service

# build and pass all env variables manually
docker build -t user-service . && \
    docker run -e "KAFKA_BOOTSTRAP_SERVERS=server:9092" \
    -e "KAFKA_SASL_USERNAME=XXX" \
    -e "KAFKA_SASL_PASSWORD=XXX" \
    -e "MONGODB_CONNECTION_STRING=mongodb://" \
    user-service

User Approval Service

build and run

docker build -t user-approval-service . && \
    docker run --env-file ../../auth/kafka.env \
    user-approval-service

Setup project step by step

Terraform

https://docs.microsoft.com/en-us/azure/terraform/terraform-create-k8s-cluster-with-tf-and-aks

Plugins

install Terraform ConfluenceCloud plugin

TF_PLUGIN_VERSION=0.0.1
mkdir ~/.terraform.d/plugins/darwin_amd64
cd ~/tmp
wget "https://github.com/Mongey/terraform-provider-confluent-cloud/releases/download/v${TF_PLUGIN_VERSION}/terraform-provider-confluent-cloud_${TF_PLUGIN_VERSION}_darwin_amd64.tar.gz"
tar xzf terraform-provider-confluent-cloud_${TF_PLUGIN_VERSION}_darwin_amd64.tar.gz
mv terraform-provider-confluent-cloud_v${TF_PLUGIN_VERSION} ~/.terraform.d/plugins/darwin_amd64/

Init

terraform init

Plan & Apply

cp .env .env.local
# fill .env.local with correct values

source .env.local
export TF_VAR_azure_client_id TF_VAR_azure_client_secret CONFLUENT_CLOUD_USERNAME CONFLUENT_CLOUD_PASSWORD

cd infrastructure/terraform
terraform apply

Connect

echo "$(terraform output azure-k8s-cluster_kube_config)" > ./kube_config
export KUBECONFIG=$(PWD)/kube_config
kubectl get node

Istio

Istio is already persisted at infrastructure/k8s/istio-system.yaml. It was generated using:

istioctl manifest generate --set values.kiali.enabled=true --set values.tracing.enabled=true --set values.grafana.enabled=true --set values.prometheus.enabled=true

Apply K8s Resources

kubectl apply -f infrastructure/k8s

Install Helm Charts

If your k8s provider doesn't install metrics-server you need to enable in it infrastructure/helm/helmfile.yaml and the namespace in infrastructure/k8s/namespaces.yaml.

cd infrastructure/helm

# as of now I had to specify the path to helm3 manually because I also had helm2 installed
helmfile --helm-binary "/usr/local/opt/helm@3/bin/helm" diff
helmfile --helm-binary "/usr/local/opt/helm@3/bin/helm" sync

view Terraform credentials

terraform refresh

Kafka commands

Apache Kafka CLI

Use Apache Kafka CLI tools with ConfluentCloud: https://www.confluent.io/blog/using-apache-kafka-command-line-tools-confluent-cloud

First create the file kafka.config.properties and add the API key and token.

kafka-console-producer --broker-list pkc-e8mp5.eu-west-1.aws.confluent.cloud:9092 --producer.config kafka.config.properties --topic user-create
kafka-console-consumer --bootstrap-server pkc-e8mp5.eu-west-1.aws.confluent.cloud:9092 --consumer.config kafka.config.properties --topic user-create

Confluent Cloud CLI

ccloud api-key create --resource lkc-1j98z
ccloud api-key use XXX --resource lkc-1j98z

ccloud kafka topic create test-topic
ccloud kafka topic produce test-topic
ccloud kafka topic consume test-topic

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.