Giter Club home page Giter Club logo

mendix-kubernetes-azure's Introduction

Disclaimers:

1) This is not an official Mendix product, see Support Status

2) The recommended, productized way to run Mendix on Azure is using Mendix for Private Cloud / Mendix Operator for Kubernetes

3) Parts of below how-to are now incorrect due to changes in Azure (DevOps). It is left as an exercise for the reader to work around those. I will not update this how-to anymore.

Mendix on Azure Kubernetes Service

This how-to outlines how to deploy a scalable, production-ready Kubernetes cluster for hosting Mendix apps on Microsoft Azure. The solution includes all components necessary to succesfully build and operate Mendix apps on Azure and consists of the following components:

Mendix on Azure

Features of the solution:

  • Host Mendix apps in a secure, scalable and managed environment.
  • Utilize standard Azure services to ensure stability, support and cost-effectiveness.
  • Utilize Azure DevOps for automated deployments, fully in control of your team.

How to deploy the solution

Prerequisites

  • Account with Owner role assignment on the target Azure Subscription.
  • Azure DevOps or Azure DevOps Server administrator access (needed to install Extensions)
  • App DNS records and SSL certificates (can be wildcards)
  • Make sure all Azure Resource Providers used in this how-to are enabled on the subscription. More info about this here.

Step A. Deploying Container Platform (Azure Kubernetes Service)

The Mendix apps will run in Docker containers which will be orchestrated using Kubernetes. For this purpose we are going to deploy Azure Kubernetes Service, which will provide us with a managed Kubernetes cluster to host our app containers.

Deploying Azure Kubernetes Service

  1. Sign in to the Azure Portal (https://portal.azure.com).
  2. Start the wizard to create an Azure Kubernetes Service.
  3. Fill out the basic information:
    • When choosing node size: keep in mind that Mendix containers typically need relatively more memory vs. CPU. So choosing instance sizes with a higher memory to CPU ratio tends to be more cost efficient (e.g. E2s_v3). Create Kubernetes cluster
  4. Fill out the scaling information:
    • With regards to enabling Virtual Nodes: The Virtual nodes option allows containers to be directly scheduled on Azure Container Instances. We will not use this option in this how-to. Since Mendix containers typically run 24/7, using VMs as dedicated agent nodes is typically more cost-effective.
    • With regards to enabling VM Scale Sets: VM Scale Sets can safely be enabled as they allow more flexibility in scaling the cluster at a later stage.
  5. Fill out the authentication information:
    • With regards to enabling RBAC: Role-Based Access Control (RBAC) allows you to define security roles within the cluster and assign different cluster permissions to different groups of users. Enabling this is required in order to run a secure cluster. Authentication options
  6. Fill out the network information:
    • HTTP Application routing: we disable this as we will deploy an NGINX ingress controller that supports HTTPS later.
    • Network configuration: we will select Basic to deploy to a newly created VNet. Specifying an Advanced network configuration is required when we want to deploy to a custom VNet (e.g. a VNet routable over an ExpressRoute). Networking options
  7. Fill out the monitoring information
    • Enable this to use the built-in cluster monitoring features of Azure. Monitoring options
  8. Optionally, fill out tags (e.g. for cost tracking)
  9. Confirm your choices to start deployment of the cluster

Connecting to the Kubernetes cluster

  1. After the deployment of the Kubernetes Service, a resource group will have been created containing the cluster object:

Monitoring options

  1. Open the Kubernetes Dashboard
  2. The cluster has been deployed successfully and can be managed from your workstation!

Step B. Deploying the cluster load balancer (NGINX Ingress Controller)

Web traffic has to flow from outside the cluster towards the right Mendix containers in the cluster. In the Kubernetes world, this is handled by an ingress controller. We will deploy the NGINX ingress controller into the cluster. We will do this using Helm, the package manager for Kubernetes.

Installing Helm

  1. Install helm: https://helm.sh/

  2. Add helm to your system PATH (see https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/ for instructions on Windows)

  3. Create a service account and intialize Tiller, as described here

Installing NGINX Ingress Controller

  1. Install NGINX into the cluster by running:
helm install stable/nginx-ingress

Helm output

  1. (Optional) In case you deployed on an internal VNet, be sure to add the service.beta.kubernetes.io/azure-load-balancer-internal: "true" annotation as outlined here_:_ https://docs.microsoft.com/en-us/azure/aks/internal-lb
  2. After a few minutes, run:

kubectl get services –output wide

  1. The output should contain the external IP address of the load balancer

Helm output

  1. By opening this URL in your browser, you can verify communication from your workstation to the cluster is fully functioning:

working connection

Step C. Setting up build and release pipelines using Azure DevOps

We will use Azure Devops to automatically deploy the following components:

  • Azure SQL Server & Elastic Pool
  • Azure Container Registry
  • Azure Keyvaults (for master secrets and app-specific secrets)
  • An app onboarding pipeline (for adding new apps to the cluster, it will deploy per app: databases, Azure Blob Storage Accounts,key vaults, build & release pipelines)

Setting up the Mendix Azure DevOps project

In this how-to we will use Azure DevOps as a CI/CD solution to execute the pipelines. If you know what you are doing, any other CI/CD solution (e.g. Jenkins, Gitlab) can be used instead. But this how-to accompanies fully-working templates for Azure DevOps. A free version of Azure Devops can be obtained by creating a new account at https://dev.azure.com .

Installing required extensions and configuring Service Connections:

  1. Login to your Azure DevOps environment and create a new project called Mendix.
  2. We need to install some extensions which we will use in our pipelines. This has to be done in the organizational settings pane of Azure Devops. Please install the following extensions from the Azure DevOps marketplace:

Azure DevOps extensions

  1. Next, we need to create several Service Connections in our Azure DevOps project and write down the corresponding GUIDs so that they can be leveraged in the pipelines.

Azure Resource Manager Service Connection

Create a service connection of the type Azure Resource Manager, connecting Azure DevOps to your Azure subscription. Record the GUID of the service connection, which you will find in the address bar of your browser when the service connection is selected:

Select type of Service Connection ARM SC form Get GUID

Azure DevOps REST API Service Connections

The pipelines we will create in the upcoming steps will use the Azure DevOps REST API to interface with Azure DevOps itself. We need to create two Service Connections (of type Generic) for this. Before we can do that, we need to create a Personal Access Token (PAT) for our current Azure DevOps. Follow the instructions here to create a PAT with full access (all scopes) and note down the PAT.

We need to create two Generic Service Connections:

Azure DevOps Azure DevOps REST API

The URL of this Service Connection is the project URL appended with _apis:

When configuring the Service Connection, you can leave username empty and use the PAT as Password/Token Key.

Azure DevOps Release Mgmt Azure DevOps Release Mgmt REST API

The configuration of this Service Connection is identical, with one exception: the hostname in the URL is vsrm.dev.azure.com instead of dev.azure.com:

Kubernetes Service Connection

Finally, we need to create a Service Connection that allows Azure DevOps to connect to our Kubernetes cluster. Create a Service Connection of the type Kubernetes. The field KubeConfig should contain the contents of the kubectl config file which you can find in "your homeprofile directory/.kube/config":

AKS SC form

Note down the GUIDs of all four created Service Connections. you will use them in the next step

Importing the initial deployment pipelne

Download and save the initial deployment pipeline to your computer.

In order for the Azure DevOps UI to show us the option of importing a Release pipeline, we need to add an empty Release pipeline. Go ahead and create an empty pipeline:

Create empty pipeline

Now import the initial deployment pipeline file you downloaded from this repository. During the import process, you will have to manually configure it to use the Azure Pipelines agent pool and the ubuntu agent specification.

Create empty pipeline

Executing the initial deployment pipeline

Now execute the pipeline by creating a release and filling out the correct parameter values, as per the table below:

Variable Description
Azure_DataResourceGroup Name of the resource group in which to deploy databases, key vaults and storage accounts
Azure_DataResourceGroup_Region Azure Region of the data resourcegroup (this is the resource group in which all database-, storage- and key vault resources will be created), e.g. westeurope
Azure_Subscription GUID of the Azure Resource Manager Service Connection, created in the previous section. Be careful to use the GUID of the Service Connection, not of the subscription itself
AzureDevOps_API_Endpoint GUID of the Azure DevOps API Service Connection, created in the previous section
AzureDevOps_Release_API_Endpoint GUID of the Azure DevOps Release Management API Service Connection, created in the previous section
kubernetes_cluster GUID of the Kubernetes cluster Service Connection, created in the previous section
orgname Select a short organizational name (<10 characters, only lowercase, only letters) that will be used in the various deployed resources (to ensure global uniqueness on Azure)
AzurePipelines_Pool_QueueID The queue ID of the "Azure Pipelines" agent pool*
  • Queue IDs of Agent Pools can be derived by hovering over the Queue in the Agent Pool settings (located in the Project Settings tab).

Find Azure Pipelines queue ID

Correctly filled out, it should look like this:

Release pipeline settings

Executing this pipeline will:

  • Deploy a master key vault to store service credentials
  • Deploy an Azure SQL Server and Elastic Pool to host app databases.
  • Deploy Azure Container Registry to host app Docker images
  • Deploy a "New App Onboarding pipeline" which will allow you to add new Mendix apps to your environment.

Onboarding a new app

Onboard a new app by creating a new release of the "New app onboarding pipeline". It takes the following arguments:

Variable Description
appfqdn_accp The FQDN for the acceptance environment of the app
appfqdn_accp_leader The FQDN for targetting only the leader container of the acceptance environment (e.g. for reaching the debugger endpoint consistently)
appfqdn_prod The FQDN for the production environment of the app
appfqdn_prod_leader The FQDN for targetting only the leader container of the production environment
appname Application Name (should be short, < 8 characters)
ts_password Mendix Team Server password to be used to retrieve the app model
ts_url URL of app repository on the Mendix Team Server (can be fetched from the Team Server tab on the project page at the Mendix Developer portal/Sprintr)
ts_username Mendix Team Server username to be used to retrieve the app model

Correctly filled out it should look like:

New app Release pipeline settings

Executing this release will:

  • Provision app-specific Key Vaults
  • Provision app-specific databases
  • Provision app-specific (Blob) Storage Accounts
  • Provision Kubernetes namespaces for acceptance and production environments
  • Provision apps-specific build & release pipelines

Step D. Adding extended logging & metrics using Datadog

  1. Create a Datadog (trial) account at http://www.datadoghq.com and get the API key.
  2. Download the Datadog templates from manifests/kubernetes/datadog in this repository.
  3. Insert the Datadog API key in line 26 of datadog.yaml
  4. Apply both manifests to your clusters

kubectl create -f datadog.yaml
kubectl create -f datadog-rbac.yaml

  1. Within minutes, logging and metrics should start appearing in Datadog.

Known issues

Frequently Asked Questions

Support status

  • Questions / issues with regards to this reference implementation can be raised in the Issues section of this Github repository, they will be answered on a best-effort basis.
  • The templates published in this repository are not a supported Mendix product (although they are in use by several major Mendix customers). I maintain them on a best-effort basis.

Roadmap of this solution

  • Document custom runtime settings, license activation etc.
  • Document how to use cert-manager for automatic provisioning of SSL certificates using Let's Encrypt
  • Document how to use Azure Insights for monitoring.
  • Document how to use pipelines with other Kubernetes clusters (e.g. AWS EKS).
  • Document cluster configuration backups using Velero

mendix-kubernetes-azure's People

Contributors

kamysz avatar mxclyde avatar nmkleiberg avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

robojobobobo s-ahuja kamysz rajithaed iuliantoader koranit martijn-de-kam deadlyembrace misraa-zz jacobsmendixcoderepo charlie13 sathishphcl pablogd-mx todorovc

mendix-kubernetes-azure's Issues

Error building package.

Getting the following error when trying to build in Devops, any suggestions?

Build Agent = Ubuntu 18.04
Using the docker-mendix-buildpack-telegrafext-nonroot build pack.

2019-10-25T15:03:12.0971652Z Step 16/29 : WORKDIR /buildpack 2019-10-25T15:03:12.3028897Z ---> Running in 49c82ca107b4 2019-10-25T15:03:13.1061786Z Removing intermediate container 49c82ca107b4 2019-10-25T15:03:13.1063443Z ---> db26a0f26b8e 2019-10-25T15:03:13.1064238Z Step 17/29 : RUN "/buildpack/compilation" /build /cache && rm -fr /cache /tmp/javasdk /tmp/opt && useradd -r -U -u 1050 -d /root mendix && chown -R mendix /buildpack /build /.java /root 2019-10-25T15:03:13.3101294Z ---> Running in 335f977bd175 2019-10-25T15:03:13.7595624Z �[91m/bin/sh: 1: /buildpack/compilation: Permission denied 2019-10-25T15:03:14.1249185Z �[0mThe command '/bin/sh -c "/buildpack/compilation" /build /cache && rm -fr /cache /tmp/javasdk /tmp/opt && useradd -r -U -u 1050 -d /root mendix && chown -R mendix /buildpack /build /.java /root' returned a non-zero code: 126 2019-10-25T15:03:14.1390865Z ##[error]The command '/bin/sh -c "/buildpack/compilation" /build /cache && rm -fr /cache /tmp/javasdk /tmp/opt && useradd -r -U -u 1050 -d /root mendix && chown -R mendix /buildpack /build /.java /root' returned a non-zero code: 126 2019-10-25T15:03:14.1424510Z ##[error]The process '/opt/hostedtoolcache/docker-stable/17.9.0-ce/x64/docker' failed with exit code 126

Error: No such container:path: mendixapp:/build/model/metadata.json

We started getting the below error since 26 November across all the application landscape (DEV, QA and PROD - QA and PROD were taken from this git repository):

Error: No such container:path: mendixapp:/build/model/metadata.json
/usr/bin/docker failed with return code 1

It is failing on this step:
Failed_Step

This is running Ubuntu 18.04.

Any idea/pointers to what may be causing this?

Initial Template Doesnt work

When running the initial template deploy i get errors on 'Add Task Group'. Gives unhandled error with code 401. If I copy out the section for adding a group and try to import the task group manually i get the following error: Import of task group failed with error TypeError: n.metaTaskInputs.map is not a function.

"Add Task Group: Create New App Onboarding Release pipeline changes" needs some updates

Few updates can be made to the "Add Task Group: Create New App Onboarding Release pipeline"

  1. The definition id passed is 2 assuming that the definition id 2 exists since 1 is the empty release pipeline and 2 is the Initial Mendix setup pipeline. This might not always be the case if it is a project which had these release pipelines cleaned up and newer ones created. Azure Dev Ops doesn't claim a deleted definition id it just does max+1. In this case it would throw a 404, release pipeline doesn't exist error.
  2. How about passing 0 based on the Azure Dev Ops recommendation as listed in their documentation https://docs.microsoft.com/en-us/rest/api/azure/devops/release/definitions/create?view=azure-devops-rest-5.1 . This will then auto-generate a defintion with the next sequence number for definition id
  3. This I am not too sure but from what I understand we don't need to pass the url and links (which references the wrong release pipeline) as this would be generated as a part of response "Release definition"
	"url": "https://vsrm.dev.azure.com/mxclydedemo/01a29e90-f89b-4f9f-b1ea-2e845b5f3cb2/_apis/Release/definitions/2",
	"_links": {
		"self": {
			"href": "https://vsrm.dev.azure.com/mxclydedemo/01a29e90-f89b-4f9f-b1ea-2e845b5f3cb2/_apis/Release/definitions/2"
		},
		"web": {
			"href": "https://dev.azure.com/mxclydedemo/01a29e90-f89b-4f9f-b1ea-2e845b5f3cb2/_release?definitionId=2"
		}
	}
  1. Finally it is no longer in preview so instead of passing /release/definitions?api-version=5.1-preview.3 we can pass /release/definitions?api-version=5.1 but Azure Dev Ops would still handle the redirection even if we did pass the preview version

Reference - Line No 807 https://github.com/MXClyde/mendix-kubernetes-azure/blob/master/manifests/azuredevops/Initial%20Mendix%20Setup.json#L807

Thanks again Clyde for this awesome piece of documentation

Abe

DevOps Task ids not valid anymore

mendix-kubernetes-azure/manifests/azuredevops/Initial Mendix Setup.json

Line 503 in 2367214

"taskId": "6f9cdff3-a6b9-47e1-90a8-6590ba11e157",

The ID mentioned for this (and other similar tasks) is not valid anymore.
The old id 6f9cdff3-a6b9-47e1-90a8-6590ba11e157 needs to be replaced with ffd54dce-ce4f-4080-bdda-a110f334feff which is a task JSON to variable with below specs

Task : JSON to Variable
Description : Task that will read a JSON file and create build and/or release variables available to other tasks for the data in the JSON
Version : 1.0.21
Author : Jeff Przylucki
Help :

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.