flowfuse / driver-k8s Goto Github PK

Current Behavior

Symptoms

After Project is created/started: project is indicated as up an running but it is unreachable via Editor link.
For ops its very obvious because they can see the ingresses and simply detect the problem. But this is not obvious given with FlowFuse interface only where the project is stated as running and the link to the Editor is available but never get to work.

Problems description

There are 2 problems with the way how k8s resources are provisioned and deprovisioned, for the Project:

1. Dependent provisioning operations are done concurrently instead of sequentially.
2. In deprovision scenario (e.g. "suspend"), the state of the project is changed optimistically w/o guarantee (and handling) of the actual state of the resources it is bound to, which allows the clients to change the state of the project further while the previous operations are still in progress.

Problems detailed description

1. In "createProject", it's a race condition between the operations that are almost simultaneously pushed onto concurrent promises list which allows them to execute in any order and priority. Whilst creation of a service depends on creation of deployment, and creation of ingress depends on creation of a service. This can lead to situation when the ingress won't be created in first place because service creation is yet not started or in flight, and k8s API will return, in attempt to create the ingress for service, "Not found" error.
   E..g instead of dealing with a list of concurrent promises

   return Promise.all(promises).then(async () => {...}

   must be smth like

   await this._k8sAppApi.createNamespacedDeployment(...);
   // poll, verify it's there
   await this._k8sApi.createNamespacedService(...);
   // poll, verify it's there
   await this._k8sNetApi.createNamespacedIngress(...);
   // poll, verify it's there
   // Done with resources
   this._app.log.debug(`[k8s] Container ${project.id} started`)
   project.state = 'running'
   await project.save()
   this._projects[project.id].state = 'starting'

2. In "suspend" project handler, the operations of deprovisioning service and ingress are not guaranteed to be accomplished as a result of the handler execution (however the project's state is moved to "suspended" which unlocks consequential state change actions on it, by the clients).
   This problem is introduced by this commit which aims to deprovision the networking resources associated with the project when the project is suspended.
   k8s operations that provision/deprovision/modify resource (e.g. 'delete service', 'delete ingress') are "promises" for the client of this API. They just send the request onto queue of requests for the relevant k8s service. Thus in order to guarantee the operation was completed, corresponding resources should be polled in order to ensure they are left in desired state.
   This polling is already done in "suspend" handler for deployments but not for service and ingress resources.

Additional approach when concurrently work with [k8s] resources is the "[Kubernetes] Resource Versioning" so that the originator of the change would know if the change it conducted was applied ('no-lock write attempt').

Expected Behavior

• Operations of provisioning of k8s resources are done with respect to the dependency between the resources
• Whenever Project's operation can assume concurrent modification of k8s resources, such operations side effects should be guaranteed, e.g. they have to be "mutex-ed".
• Project's state reflects the actual status of the deployment, including the bound [k8s] resources.

Steps To Reproduce

This is especially simple to reproduce when you try to change the stack for the project.
Starting from recently, changing the stack automatically suspends and then starts the project.

Above steps 100% lead to the Project unreachability via ingress after attempt to change the stack.

To recover, one has to suspend the instance, wait for a bit and then start it again.

Environment

• FlowForge version: 1.11.3 (kuber driver 1.11.0)
• Node.js version: any (in our case its 18.16.0)
• npm version: any (9.5.1)
• Platform/OS: any OS.
• Browser: any

k8s setup

We currently create a k8s Deployment when deploying a Node-RED instance in the platform.

As part of the roadmap to HA (see FlowFuse/flowfuse#1920) we need to migrate to using StatefulSets. This will allows us to run two copies of the instance in an active/stand-by configuration.

We only recently moved to using Deployments rather than bare pods, so already have some code in place to deal with migration as instances are suspended/resumed.

For this task the requirements are:

• Any new instance should be created as a StatefulSet
• Existing instances, whether still a bare pod or deployment, should get recreated as a StatefulSet when it is next suspended/resumed

Current Behavior

Cannot start instance with error:

{"statusCode":400,"body":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Deployment in version \"v1\" cannot be handled as a Deployment: json: cannot unmarshal number into Go struct field EnvVar.spec.template.spec.containers.env.value of type string","reason":"BadRequest","code":400}

K8s Deployment ENV

"env": [
              {
                "name": "TZ",
                "value": "Europe/London"
              },
              {
                "name": "FORGE_CLIENT_ID",
                "value": "ffp_1gkCsIqdZKFW8AF-5-1n0BgtFdvnktW64cKwzR7tJ94"
              },
              {
                "name": "FORGE_CLIENT_SECRET",
                "value": "GjML-nMDCIRxvFEN3a4fnjG4w3imSHzpbs2xW56hURRtApOBTxYrreswDd0RpJLs"
              },
              {
                "name": "FORGE_URL",
                "value": "http://forge.flowforge"
              },
              {
                "name": "BASE_URL",
                "value": "https://precious-purple-sandpiper-7561.forge.k8s.xxx"
              },
              {
                "name": "FORGE_TEAM_ID",
                "value": "34qaN4xX5v"
              },
              {
                "name": "FORGE_PROJECT_ID",
                "value": "8dcb5b43-06fe-4541-ae3f-2695027c9ec0"
              },
              {
                "name": "FORGE_PROJECT_TOKEN",
                "value": "fft_R0qEj6XaI-UN4uPjJNhu2xR8xKtNgvWPfDaUFnOzOnE"
              },
              {
                "name": "FORGE_NR_NO_TCP_IN",
                "value": "true"
              },
              {
                "name": "FORGE_NR_NO_UDP_IN",
                "value": "true"
              },
              {
                "name": "FORGE_BROKER_URL",
                "value": "mqtt://flowforge-broker.flowforge:1883"
              },
              {
                "name": "FORGE_BROKER_USERNAME",
                "value": "project:34qaN4xX5v:8dcb5b43-06fe-4541-ae3f-2695027c9ec0"
              },
              {
                "name": "FORGE_BROKER_PASSWORD",
                "value": "ffbp_cU0eStkmHZnT_1TliMcMmASelIn8ZCcscXflCnLZQMQ"
              },
              {
                "name": "FORGE_LICENSE_TYPE",
                "value": "ee"
              },
              {
                "name": "FORGE_MEMORY_LIMIT",
                "value": 256
              },
              {
                "name": "FORGE_CPU_LIMIT",
                "value": 10
              },
              {
                "name": "FORGE_NR_SECRET",
                "value": "54b4a8c405a487e67d83121a58440f26ce4bddb3ddd1ab10127c31bce0b0c494"
              }
            ],

Maybe issue related to FORGE_MEMORY_LIMIT and FORGE_CPU_LIMIT with value is int instead of string

Expected Behavior

No response

Steps To Reproduce

No response

Environment

• FlowFuse version: v1.12.5
• Node.js version: v16.20.2
• npm version:
• Platform/OS: k8s
• Browser:

Description

Pull up the version to pick up dependency updates.

Will need testing

Epic/Story

No response

Current Behavior

Project pods will apply the pod affinity to nodes with the label role=project even if it is not set in the driver options

Expected Behavior

If not affinity is set then pods should be free to run on any node

Steps To Reproduce

Deploy using helm chart and override forge.projectSelector with an empty value

Environment

• FlowForge version: 1.0.0
• Node.js version: 16.x
• npm version: 8.x
• Platform/OS: K8s/linux
• Browser: na

Story

No response

Description

In 0.6, new projects will be generating their own credentialSecret. We need to ensure that gets passed through to the launcher.

See FlowFuse/driver-localfs#47 for how this is done in localfs.

Story

No response

Description

We currently represent projects as bare Kubernetes Pods

Bare Pods are effectively unmanaged by the Kubernetes environment, e.g. once created they will not be moved to a new node if the current node fails or needs to be shutdown.

A Deployment is a higher level Kubernetes entity that wraps a Pod and ensures that the required (in FF case 1) number of nodes is always available.

Discovered while testing AWS Kubernetes version upgrades

Extend the current implementation to use the k8s API to check status before making HTTP request to nr-launcher.

This will allow starting status while container is being pulled and not show a 404 during startup of project

Description:

As a FlowForge Admin
I want to be able to set the K8s Namespace that Project Pods are created in
So that I can run multiple instances of FlowForge platform on a single K8s cluster.

Currently hard coded to flowforge

Possible options

• Allow Admin to set it as part of helm chart (env var in driver)
• Generate it based on a namespace the Forge App in running in e.g if default then default-flowforge

Acceptance Criteria

• No hard coded namepsaces

Current Behavior

https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy

If using container tags for stacks then Kubernetes will not pull a new version of the same tag with the default policy

e.g. example.com/flowforge/node-red:beta will not be pulled if the node already has a local copy with that tag

Expected Behavior

Kubernetes should check the hash and pull newer versions

This can be fixed by setting the policy to Always

Steps To Reproduce

• Push a container with a tag to the docker registry
• create a stack using the new container
• Create a project with that stack
• Push an new version of the container with the same stack
• Create a new project with the same stack

New project will use first version of container

Environment

• FlowForge version: 0.4
• Node.js version: 16.x
• npm version: 8.x
• Platform/OS: K8s/Linux
• Browser: NA

Story

FlowFuse/flowfuse#324

Description

• Setup correct Ingress entries for supplied hostname
• Setup correct TLS Secret to hold certificate for supplied hostname

Current Behavior

A NodePort service is created for each instance

There are a limited number of ports available so they can be exhausted.

We also do not remove service or ingress objects for suspended instances

Expected Behavior

service and ingress removed when instance suspended

Look at using ClusterIP services

Steps To Reproduce

Create more than ~2000 instances

Environment

• FlowForge version: 1.10.0
• Node.js version: 16.x
• npm version: 8x
• Platform/OS: k8s/linux
• Browser: na

Story

Question about google home

Description

Regards!

I use the google home you wrote in node red. All due respect for that!

My question would be, how can I change integer 23 in the code below with google assistant? Thank you very much!

(I'm sorry I do not speak English. Google translator)

msg.payload = Number(msg.payload)-10;
if (msg.payload < 23) {msg.payload = "on";}
else {
{msg.payload = "off"}
}
return msg;

I made a new thermostat here: https: //googlehome.hardill.me.uk / .....
The tool appears in the home assisstant

Hi,
is there any reason that I might be overlooking in kubernetes.js for not to support TLS and WebSocket in ingressTemplate and createIngress?

In order to get the NodeRED instances working on a K8s cluster one would need to adjust the two mentioned somehow along these lines:

const ingressTemplate = {
    apiVersion: 'networking.k8s.io/v1',
    kind: 'Ingress',
    metadata: {
        // name: "k8s-client-test-ingress",
        // namespace: 'flowforge',
        annotations: {
            // ***** Added:
            // nginx.org/websocket-services: "k8s-client-test-service"
        }
    },
    spec: {
        rules: [
            {
                // host: "k8s-client-test" + "." + "ubuntu.local",
                http: {
                    paths: [
                        {
                            pathType: 'Prefix',
                            path: '/',
                            backend: {
                                service: {
                                    // name: 'k8s-client-test-service',
                                    port: { number: 1880 }
                                }
                            }
                        }
                    ]
                }
            }
        ],
        // ***** Added:
        tls: [
            {
                hosts: [
                    // "k8s-client-test" + "." + "ubuntu.local"
                ]
            }
        ]
    }
}

const createIngress = async (project, options) => {
    const prefix = project.safeName.match(/^[0-9]/) ? 'srv-' : ''

    const localIngress = JSON.parse(JSON.stringify(ingressTemplate))
    localIngress.metadata.name = project.safeName
    // ***** Added:
    localIngress.metadata.annotations['nginx.org/websocket-services'] = `${prefix}${project.safeName}`
    localIngress.spec.rules[0].host = project.safeName + '.' + options.domain
    localIngress.spec.rules[0].http.paths[0].backend.service.name = `${prefix}${project.safeName}`
    // ***** Added:
    localIngress.spec.tls[0].hosts.push(project.safeName + '.' + options.domain)

    return localIngress
}

Description

Editors: allow optional service account provisioning

Story

No response

Description

Review all calls to K8s APIs to ensure we log failures clearly.

While all the calls to create Ingress objects would have shown as succeeding for the AWS ALB limit problem we should still look to see if there are things we can do better.

Story

No response

Description

To have multiple instances running, the kube cluster should have configurable replica numbers.

Description

Scope

Ability to configure, in Stacks, images from the private artifactory which could have complex data organization structure.

Details

Existing Container Location regex works fine with regular domain/subdomain:version formats of docker image location.

However it does not work for data organization in our artifactory.
Example of the url that does work:

• dc-if/flowforge/node-red:1.13.3

Examples of the location string which does not work:

• {COMPANY}.jfrog.io/tenant-dev/dc-if/flowforge/node-red:1.13.3
• {tenant-dev/dc-if/flowforge/node-red:1.13.3

Suggestion

@hardillb could you have a look on regex like this to use?
'^([a-z0-9][a-z0-9\\-]*(\\.[a-z0-9\\-]+)*\\/)+[a-z0-9\\-]+(?:\\/[a-z0-9\\-]+)*(?::[0-9]+\\.[0-9]+\\.[0-9]+)?$'

let validate = '^([a-z0-9][a-z0-9\\-]*(\\.[a-z0-9\\-]+)*\\/)+[a-z0-9\\-]+(?:\\/[a-z0-9\\-]+)*(?::[0-9]+\\.[0-9]+\\.[0-9]+)?$';
let validator = new RegExp(validate);

let val1 = "aiola.jfrog.io/aiola-dev/dc-if/flowforge/node-red:1.13.3";
let val2 = "dc-if/flowforge/node-red:1.13.3";
let val3 = "aiola-dev/dc-if/flowforge/node-red:1.13.3";
let val4 = "flowforge/node-red:1.13.3";

console.log(validator.test(val1)); // Should be true
console.log(validator.test(val2)); // Should be true
console.log(validator.test(val3)); // Should be true
console.log(validator.test(val4)); // Should be true

Story

No response

Description

Currently node selectors are hard coded to role=management and role=projects this means that any cluster requires at least 2 nodes and they must be suitably labelled.

This should be configurable to allow for single node testing and for other node topologies.