The gitops-energy-usage-notes from nikimanoledaki

Install Kepler, Prometheus and Grafana with Flux

This ticket is to ensure that Flux can watch and reconcile Prometheus and Grafana from manifests in ./clusters.

Provision KVM instance to use Kepler

What is Kepler?

Kepler (Kubernetes-based Efficient Power Level Exporter) uses eBPF to probe energy related system stats and exports as Prometheus metrics

It can help us get energy consumption metrics from a cluster.

Kepler requirements as outlined here:

Support for cgroup v2
Support for kernel-devel extensions
Provide the kernel headers (required by eBPF)
Kernel with eBPF support

For example, Kepler on Openshift has already been integrated and works. This integration can be used as an example of what configuration is needed, e.g.:

  kernelArguments:
    - systemd.unified_cgroup_hierarchy=1
    - cgroup_no_v1="all"
  extensions:
  - kernel-devel

TODO (before KubeCon)

#8
#6

Long-term TODO:

#7

Deployment is unmarshalled with empty/null fields

Current Deployment that is created looks like this:

typemeta:
  kind: Deployment
  apiversion: apps/v1
objectmeta:
  name: 09-09-2022-18-17-02
  generatename: ""
  namespace: ""
  selflink: ""
  uid: ""
  resourceversion: ""
  generation: 0
  creationtimestamp:
    time: {}
  deletiontimestamp: null
  deletiongraceperiodseconds: null
  labels:
    app: date-setter
  annotations: {}
  ownerreferences: []
  finalizers: []
  managedfields: []
spec:
  replicas: 1
  selector:
    matchlabels:
      app: date-setter
    matchexpressions: []
  template:
    objectmeta:
      name: ""
      generatename: ""
      namespace: ""
      selflink: ""
      uid: ""
      resourceversion: ""
      generation: 0
      creationtimestamp:
        time: {}
      deletiontimestamp: null
      deletiongraceperiodseconds: null
      labels:
        app: date-setter
      annotations: {}
      ownerreferences: []
      finalizers: []
      managedfields: []
    spec:
      volumes: []
      initcontainers: []
      containers:
      - name: date-setter
        image: docker.io/niki2401/date-setter:latest
        command: []
        args: []
        workingdir: ""
        ports:
        - name: ""
          hostport: 0
          containerport: 8080
          protocol: ""
          hostip: ""
        envfrom: []
        env: []
        resources:
          limits: {}
          requests: {}
        volumemounts: []
        volumedevices: []
        livenessprobe: null
        readinessprobe: null
        startupprobe: null
        lifecycle: null
        terminationmessagepath: ""
        terminationmessagepolicy: ""
        imagepullpolicy: ""
        securitycontext: null
        stdin: false
        stdinonce: false
        tty: false
      ephemeralcontainers: []
      restartpolicy: ""
      terminationgraceperiodseconds: null
      activedeadlineseconds: null
      dnspolicy: ""
      nodeselector: {}
      serviceaccountname: ""
      deprecatedserviceaccount: ""
      automountserviceaccounttoken: null
      nodename: ""
      hostnetwork: false
      hostpid: false
      hostipc: false
      shareprocessnamespace: null
      securitycontext: null
      imagepullsecrets: []
      hostname: ""
      subdomain: ""
      affinity: null
      schedulername: ""
      tolerations: []
      hostaliases: []
      priorityclassname: ""
      priority: null
      dnsconfig: null
      readinessgates: []
      runtimeclassname: null
      enableservicelinks: null
      preemptionpolicy: null
      overhead: {}
      topologyspreadconstraints: []
      sethostnameasfqdn: null
      os: null
      hostusers: null
  strategy:
    type: ""
    rollingupdate: null
  minreadyseconds: 0
  revisionhistorylimit: null
  paused: false
  progressdeadlineseconds: null
status:
  observedgeneration: 0
  replicas: 0
  updatedreplicas: 0
  readyreplicas: 0
  availablereplicas: 0
  unavailablereplicas: 0
  conditions: []
  collisioncount: null

Setup local dev env

Create a cluster on KVM for local dev env
Setup Kepler, Prometheus, Grafana
Meet requirements for Kepler on KVM
Get cluster-wide metrics locally

Create Dev & Production env using Liquid Metal on Equinix

This is to create a dev env using Liquid Metal on Equinix.

Requirements:

Setup Liquid Metal https://github.com/weaveworks-liquidmetal/getting-started/blob/main/docs/intro.md#terraform-an-environment-on-equinix
Setup Kubernetes cluster
- CAPMVM https://github.com/weaveworks-liquidmetal/getting-started/blob/main/docs/create.md

[EPIC] Design energy-consumption tests

`date-setter` Go app

Go server that takes current date, sets it as a deployment.yaml metadata name, and marshals it into YAML

GitOps arch

Script to bootstrap Flux to watch this repo's deployment.yaml
#4

Snowflake arch

#9

Setup energy metrics

#6

Cron job

Schedule job that runs date-setter app and should trigger Flux reconciliation & GitHub Action workflow

Run on EC2 instance

Acceptance Criteria

This is to find an AWS instance that meets the following requirements:

will run KVM
can be used as instances to run a Kubernetes cluster, ideally in a EKS self-managed nodegroup using eksctl

Suggestions

Check out this demo on installing KVM on AWS.
On the other hand, the demo linked above recommends i3.metal. Try eksctl create cluster with self-managed nodegroups and specify i3.metal as the instance.
Question: Is baremetal necessary to run KVM? Can these reqs be met on non-baremetal instances? They can be quite expensive so it would be nice to use something cheaper.
EKS works with a m5.metal instance so we could use that.
If nothing else works, here is a list of EC2 instances (search those with .metal)

Setup GitHub Action workflow for Snowflake Cluster

Acceptance Criteria

Setup GitHub Action workflow for Snowflake Cluster with:
- A step that sets up Kepler to get cluster-wide metrics in Snowflake clutser
- A step that does kubectl apply -f deployment.yaml
Somehow measure CI pipeline energy metrics
(OPTIONAL) Use Liquid Metal as CI platform :)
(OPTIONAL) Could test with https://github.com/Green-Software-Foundation/Carbon_CI_Pipeline_Tooling

Add Kepler prerequisites on Flintlock Vagrantfile

Add the following to this Vagrantfile:

Support for cgroup v2 - done by installing Ubuntu 22.04 which comes with cgroups v2 support.
Provide the kernel headers https://www.tecmint.com/install-kernel-headers-in-ubuntu-and-debian/
Kernel with eBPF support https://github.com/iovisor/bcc/blob/master/INSTALL.md#kernel-configuration

Example from Openshift:

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 50-master-kernel-devel-cgroupv2
  labels:
    machineconfiguration.openshift.io/role: master
spec:
  kernelArguments:
    - systemd.unified_cgroup_hierarchy=1
    - cgroup_no_v1="all"
  extensions:
  - kernel-devel

Add Helm chart for Kepler

Open an issue / PR to add that to their repo :)
Perhaps include instructions on how to use Flux to add Kepler to a cluster. ✨

nikimanoledaki / gitops-energy-usage-notes Goto Github PK

gitops-energy-usage-notes's Introduction

Energy-efficient Kubernetes

[Update] The latest version of this project can be found here: https://github.com/nikimanoledaki/sustainability-journey-with-gitops

Resources

Get Started with Local Development

OS

Linux

MacOS

Start a Kubernetes cluster with minikube

Get Started in a Cloud Environment

gitops-energy-usage-notes's People

Contributors

Stargazers

Watchers

gitops-energy-usage-notes's Issues

date-setter Go app

GitOps arch

Snowflake arch

Setup energy metrics

Cron job

Acceptance Criteria

Suggestions

Acceptance Criteria

Recommend Projects

Recommend Topics

Recommend Org

`date-setter` Go app