Giter Club home page Giter Club logo

ironic-standalone-operator's Introduction

Metal3 Ironic Operator

Operator to maintain an Ironic deployment for Metal3. See architecture for a quick overview of the deployed architecture.

This project is EXPERIMENTAL. It's not currently integrated will with other Metal3 components, not fully tested in the CI and is not ready to be used in production.

Usage

Let's assume we're creating an Ironic deployment called ironic in the namespace test, and that BMO will access it via https://ironic.test.svc.

Start with creating a TLS certificate and a secret for it. In this example, I'll use a self-signed certificate, you may want to sign it with some authority, e.g. using built-in Kubernetes facilities.

openssl req -x509 -new -subj "/CN=ironic.test.svc" -addext \
        "subjectAltName = DNS:ironic.test.svc" -newkey ec -pkeyopt \
        ec_paramgen_curve:secp384r1 -nodes -keyout tls.key -out tls.crt
kubectl create secret tls ironic-tls -n test --key="tls.key" --cert="tls.crt"

Now create the Ironic configuration:

---
apiVersion: metal3.io/v1alpha1
kind: Ironic
metadata:
  name: ironic
  namespace: test
spec:
  tlsRef:
    name: ironic-tls
  ramdiskSSHKey: "<YOUR SSH PUBLIC KEY HERE>"
kubectl create -f ironic.yaml

After some time, you can check the outcome:

$ kubectl describe ironic -n test ironic
...
Status:
  Conditions:
    Last Transition Time:  2023-08-25T13:05:35Z
    Message:               ironic is available
    Observed Generation:   2
    Reason:                DeploymentAvailable
    Status:                True
    Type:                  Ready

Now you can see the service that can be used to access Ironic (it has the same name as the Ironic object):

$ kubectl get service -n test ironic
NAME     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
ironic   ClusterIP   10.96.107.239   <none>        443/TCP   156m
$ # From inside the cluster, ignoring TLS certificates for now:
$ curl -k https://10.96.107.239/

The autogenerated credentials secret is linked in the spec:

$ IRONIC_SECRET=$(kubectl get ironic -n test ironic -o \
    jsonpath='{.spec.credentialsRef.name}')
$ IRONIC_USER=$(kubectl get secret -n test "$IRONIC_SECRET" -o \
    jsonpath="{.data.username}" | base64 --decode)
$ IRONIC_PASSWORD=$(kubectl get secret -n test "$IRONIC_SECRET" -o \
    jsonpath="{.data.password}" | base64 --decode)
$ # From inside the cluster, accessing an authenticated endpoint:
$ curl -k -u "$IRONIC_USER:$IRONIC_PASSWORD" https://10.96.107.239/v1/drivers

By its nature, Ironic listens on a host network and thus is reachable on one of the Kubernetes nodes, even from the outside (note the port):

$ kubectl get pod -n test -l \
    metal3.io/ironic-standalone-operator=ironic-service -o wide
NAME                              READY   STATUS    RESTARTS   AGE    IP            NODE                 NOMINATED NODE   READINESS GATES
ironic-service-698899755d-6xzxl   4/4     Running   0          152m   10.89.0.2     kind-control-plane   <none>           <none>
$ curl -k -u "$IRONIC_USER:$IRONIC_PASSWORD" https://10.89.0.2:6385/v1/drivers

More detailed example

In this example, a MariaDB database is used instead of SQLite, and a provisioning network is configured.

You will need to generate your TLS certificates with one more subjectAltName in the format <database name>-database.<namespace>.svc (in this example, ironic-database.test.svc). See architecture guide for details.

Then another resource needs to be created for the database itself:

---
apiVersion: metal3.io/v1alpha1
kind: IronicDatabase
metadata:
  name: ironic
  namespace: test
spec:
  tlsRef:
    name: ironic-tls
---
apiVersion: metal3.io/v1alpha1
kind: Ironic
metadata:
  name: ironic
  namespace: test
spec:
  databaseRef:
    name: ironic
  tlsRef:
    name: ironic-tls
  networking:
    ipAddress: 10.89.0.2
    dhcp:
      networkCIDR: 10.89.0.1/24
      rangeBegin: 10.89.0.10
      rangeEnd: 10.89.0.100
  ramdiskSSHKey: "<YOUR SSH PUBLIC KEY HERE>"

This example also shows configuring DHCP for network booting.

OpenShift notes

  • Running a database requires the user to have nonroot or similar SCC.
  • Running Ironic requires nonroot and hostnetwork.

ironic-standalone-operator's People

Contributors

dtantsur avatar metal3-io-bot avatar tuminoid avatar rozzii avatar lentzi90 avatar

Stargazers

 avatar  avatar  avatar Jan Schoone avatar Dmitri Fedotov avatar

Watchers

 avatar  avatar Riccardo Pittau avatar Huy Mai avatar  avatar

ironic-standalone-operator's Issues

RFE: Avoid host networking for Ironic

Exposing Ironic on the host networking is far from ideal. For instance, if we do so, we're going to expose JSON RPC. There may be other internal traffic that we don't want everyone to see. In theory, only dnsmasq must be on host networking.

So, why does Metal3 use host networking at all?

  • DNSmasq serves as DHCP and TFTP servers. Both are UDP based and hard to route. DHCP involves broadcasts.
  • When booting over iPXE, hosts need to download iPXE scripts and kernel/initramfs from an HTTP server. This server is local to the Ironic instance that handles this host. Since the host is not yet a part of the cluster network, it cannot use the cluster service DNS name or IP.
  • IPA needs to reach back to the Ironic API (any of the running instances, optimally to the one handling the host). Still no cluster networking at this point.

One complication is supporting the bootstrap scenario. While most Metal3 consumers bootstrtap their production clusters by establishing a temporary cluster with Metal3 and then pivoting, OpenShift has an important limitation: the bootstrap cluster only provisions control plane hosts. Thus, it cannot rely on any components that won't come up without workers, including e.g. Ingress.

RFE: image cache resource

Currently, the operator runs IPA downloader as an init container. This has a few problems:

  • It's not trivial to opt out of using it (OpenShift, for instance, does not use IPA downloader)
  • Downloading IPA takes time which delays starting Ironic (although IPA is not actually required on start-up)
  • It's not possible to cache IPA for longer time or reuse it between separate Ironic deployments.
  • Agent download parameters complicate the already large Ironic CRD
  • In the future, we may want to provide an ability to also cache or even build instance images

Let us have a new CRD IronicImageCache that will download IPA in an init container and serve it via httpd.

type IronicImageCacheSpec struct {
	// AgentBranch is the branch of IPA to download. The main branch is used by default.
	// +optional
	AgentBranch string `json:"agentBranch,omitempty"`

	// AgentDownloadURL is the base URL from which IPA should be downloaded.
	// The default value should be good for most users.
	// +optional
	AgentDownloadURL string `json:"agentDownloadURL,omitempty"`

	// AgentDownloaderImage is the image to be used at pod initialization to download the IPA ramdisk.
	// +kubebuilder:default=quay.io/metal3-io/ironic-ipa-downloader
	// +optional
	AgentDownloaderImage string `json:"agentDownloaderImage,omitempty"`

        // AgentMinimumModificationTime is the time after which the IPA image must be produced.
        // If not provided, a hardcoded version-specific default is used.
	// +optional
        AgentMinimumModificationTime *metav1.Time `json:"agentMinimumModificationTime,omitempty"`
}

In the initial version, upgrades will rely on AgentMinimumModificationTime. Every release of the operator, we will update both the Ironic image version and this date. This version will be passed down to IPA downloader as an environment variable, causing it to re-run on changes. Users may update this value to force an upgrade.

Find a way to run online_data_migrations

With a separate MariaDB, it is possible that Ironic is upgraded without restarting the MariaDB pod. Make sure that online_data_migrations are run after Ironic is started (so, it cannot be done in an init container).

RFE: finish Distributed architecture support

The fact that we only run 1 Ironic instance is somewhat unfortunate: Ironic has a built-in active/active HA that spreads the load evenly in the cluster by assigning each Node to one Ironic instance. Ironic also has a take over process making sure that Nodes never go orphaned. On the other hand, due to its usage of green threads, each Ironic instance only uses 1 CPU core. Having 3 instances will improve CPU utilizations. For instance, CERN manages around 10k of Nodes through its 9 (at some point, not sure about the current state) Ironic conductors.

An easy step to do is to use a DaemonSet for Ironic instead of the Deployment. We will need to drop Inspector because, unlike Ironic, it's not HA ready. The new inspection implementation won't have this issue. I believe that will give us virtual media deployments without a provisioning network right away. We will need to sort our JSON RPC since all Ironic instances need to talk to each other. If we use the pods' cluster IPs, TLS may be an issue since they're not quite predictable.

DHCP is also a problem. If we run one dnsmasq, we won't be able to direct a Node to the iPXE server of the Ironic instance that handles it (Ironic API itself is fine: any Ironic will respond correctly for any Node, redirecting the request internally through the RPC). Not without changes to Ironic, at least. If we run several dnsmasq instances, that's still a problem: the request will land on the random one. Also, the networking configuration will be a challenge.

  • Figure out how to run dnsmasq
  • How to make iPXE to talk to the right instance?
  • Make sure the host it set correctly for JSON RPC (possibly my-pod.my-namespace.pod to make TLS usable)
  • Make sure JSON RPC has authentication
  • Make sure JSON RPC has TLS (do we need a reverse proxy to avoid eventlet?)

Conditions Available/Progressing -> Ready

Available and Progressing and OpenShift-isms, it seems. The logic around progressing is not super useful now, let's only leave Ready and see if anyone complains.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.