openfaas / ingress-operator Goto Github PK

When using Traefik and the FunctionIngress object, we will get an Ingress record created, however I am not sure we will get functioning TLS with cert-manager. We do get that with Nginx presently, so this issue should be partly figuring out which additional annotations are needed and applying them.

Expected Behaviour

Add cert-manager should be supported with Traefik

Current Behaviour

Untested, it may be but I suspect it is not due to the additional annotations I found in a (French) blog post.

Currently only the annotations for cert-manager are added: https://github.com/openfaas-incubator/ingress-operator/blob/master/pkg/controller/controller.go#L538

Possible Solution

Edit the following code: https://github.com/openfaas-incubator/ingress-operator/blob/master/pkg/controller/controller.go#L514

Add these annotations:

    traefik.ingress.kubernetes.io/redirect-entry-point: https
    traefik.ingress.kubernetes.io/redirect-permanent: "true"
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/ssl-temporary-redirect: "false"

As see here: https://www.cerenit.fr/blog/kubernetes-ovh-traefik-cert-manager-secrets/

You can rebuild the controller with make and specify a TAG env-var, then tag this as your own username before pushing it. You can also run the cluster as a local Go binary.

How to test

K3s ships with Traefik running in "host mode", so this would be a very fast way of testing the fix. A VM on DO or a similar cloud will come with a public IP for the DNS certificate challenge.

You'll also need your own domain name.

1. Get a cloud VM with Ubuntu 18.x - i.e. Civo or DigitalOcean
2. Install k3s via https://k3s.io
3. Install OpenFaaS & the IngressOperator -> https://docs.openfaas.com/reference/ssl/kubernetes-with-cert-manager/#deploy-the-ingressoperator
4. Create a FunctionIngress with the Ingress type of traefik
5. Create a DNS A record for your domain and the IP of the node
6. See if you get a valid TLS cert

Context

This is an important piece of the puzzle for Traefik users.

Description

This controller is a good candidate for a multi-arch build given that it's built as a static binary.

Create cert using domain instead of function name

Expected Behaviour

If we have three FunctionIngress entries all on the same domain but with differing paths, we should create 1 cert only.

Current Behaviour

Three get created since the "name" is the function's name instead of the domain name.

Possible Solution

Create cert using domain instead of function name. This will need a change in the annotation and / or secret name.

Steps to Reproduce (for bugs)

See this example for repro - https://docs.openfaas.com/reference/ssl/kubernetes-with-cert-manager/#30-rest-style-api-mapping-for-your-functions

Expected Behaviour

spec.ingressClassName needs to be set on Ingress records created by the operator

Current Behaviour

The kubernetes.io/ingress.class annotation is still being used, and is deprecated

List All Possible Solutions and Workarounds

Update reconciliation behaviour and test out the change with Traefik and ingress-nginx separately.

Context

At some point the warning will become an error.

 go run . --kubeconfig ~/.kube/config 
I1020 14:20:08.391934 3319312 main.go:51] Starting FunctionIngress controller version: latest-dev commit: local-dev
I1020 14:20:08.394413 3319312 controller.go:70] Setting up event handlers
I1020 14:20:08.394456 3319312 core.go:72] Waiting for informer caches to sync
I1020 14:20:08.494815 3319312 core.go:77] Starting workers
I1020 14:20:08.494842 3319312 core.go:83] Started workers
I1020 14:20:26.381036 3319312 controller.go:116] Creating Ingress for: nodeinfo
W1020 14:20:26.452110 3319312 warnings.go:70] annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead

We currently let the user choose the kind of cert-manager issuer, the two available options are Issuer and ClusterIssuer. However, we don't use the value of that field when we set the annotations.

Expected Behaviour

I would expect that when I set the kind field inside issuerRef that change would be reflected on the ingress's annotations.

Current Behaviour

We can set the kind field inside issuerRef but it's a no-op.

Possible Solution

I see the following two options:

• Only allow the issuer kind and remove the kind field so that we are not misleading the user
• Use the value of kind to set the cert-manager's annotations

I've implemented the second one and test it out on my cluster. It works OK and it was a very simple change, but I don't know if this is what we want. Lets discuss 😄

Steps to Reproduce (for bugs)

1. Set the kind in issuerRef to ClusterIssuer and notice that the annotation continues to be for an issuer.

Context

I was trying to create a new function ingress using the existing ClusterIssuer in my cluster, this issuer is created by ofc-bootstrap when installing ofc in the cluster.

Your Environment

• Docker version docker version (e.g. Docker 17.0.05 ):

• What version and distriubtion of Kubernetes are you using? kubectl version

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:23:09Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-gke.8", GitCommit:"7d3d6f113e933ed1b44b78dff4baf649258415e5", GitTreeState:"clean", BuildDate:"2019-06-19T16:37:16Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}

• Operating System and version (e.g. Linux, Windows, MacOS):
  Windows

• Link to your project or a code example to reproduce issue:

• What network driver are you using and what CIDR? i.e. Weave net / Flannel
  Calico on GKE

Adding nginx.ingress.kubernetes.io/from-to-www-redirect: "true" annotations to FunctionIngress resources has no effect.

Context

My function is a website that uses the dockerfile template and is served with Nginx. I am wondering if this Nginx is causing the annotation to not be applied

Expected Behaviour

Expect annotations to be applied.

Steps to Reproduce (for bugs)

Adding from-to-www-redirect annotation.

apiVersion: openfaas.com/v1alpha2
kind: FunctionIngress
metadata:
  name: example
  namespace: openfaas
  annotations:
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
...

kubectl apply -f fn_ingress.yml

Note that I can see the annotation on functioningress resource.

$ kubectl describe functioningress.openfaas.com/example -n openfaas
Name:         example
Namespace:    openfaas
Labels:       <none>
Annotations:  nginx.ingress.kubernetes.io/from-to-www-redirect: "true"

And also in the ingress resource.

$ kubectl describe ing -n openfaas                                             
Name:             example
Namespace:        openfaas
...
Annotations:     cert-manager.io/issuer: letsencrypt-prod
                      com.openfaas.spec: {"metadata":{"name":"example","namespace":"openfaas"...
                      kubernetes.io/ingress.class: nginx
                      nginx.ingress.kubernetes.io/from-to-www-redirect: true
                      nginx.ingress.kubernetes.io/rewrite-target: /function/example/$1

Your Environment

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-21T14:51:23Z", GoVersion:"go1.14.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5", GitCommit:"e0fccafd69541e3750d460ba0f9743b90336f24f", GitTreeState:"clean", BuildDate:"2020-04-16T11:35:47Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

If I have created a function ingress resource like so:

apiVersion: openfaas.com/v1alpha2
kind: FunctionIngress
metadata:
  name: pets
  namespace: openfaas
spec:
  domain: api.petstore.com
  function: pets
  ingressType: nginx
  path: /pets/(.*)

How would I retrieve the (.*) part of the path from within the function handler? I am using the golang-http template btw.

Expected Behaviour

Update CI and Dockerfile to use Go 1.18

Current Behaviour

Still using Go 1.16 to build this project

https://github.com/openfaas/ingress-operator/blob/master/go.mod#L3
https://github.com/openfaas/ingress-operator/blob/master/Dockerfile
https://github.com/openfaas/ingress-operator/tree/master/.github/workflows

Expected Behaviour

Instead of using the custom spec.tls.* definition to generate certificates we can use the default cert-manager annotations (certmanager.k8s.io/issuer or certmanager.k8s.io/cluster-issuer), since #7 was implemented.

Current Behaviour

We need to use a custom spec.tls.* definition to enable the certificate creation.

Even though we are copying all the annotations to the Ingress, the current implementation of the controller doesn't support the default cert-manager ones.

Possible Solution

Remove the special case for TLS annotations inside makeAnnotations and change the makeTLS functions to look into the annotations instead of function.Spec.UseTLS().

I didn't try these changes, just an idea by taking a quick look into the code. It may be more complicated than that.

Context

Using the same Ingress behaviour to create TLS certificates in the FunctionIngress would require a lower learning curve, less docs and make configurations more standard across the cluster.

Expected Behaviour

All traffic destined to a domain correctly forwarded to a function regardless of the path requested.

Current Behaviour

Depending on installed version of the Traefik, you may observe different behaviors.

If your Traefik version is below v1.7.21, then your function will receive incorrect (duplicated) path, e.g. nodeinfo.example.com/version will be forwarded to gateway/function/nodeinfo/versionversion and your function will get /versionversion as the path. All requests to root path (/) will be forwarded correctly to a function.

If you have Traefik v.1.7.21 or above then you'll get infinite redirect in attempt to request any path (even root)!

The behaviors are different because of this issue that has been addressed in release v1.7.21 of Traefik

Possible Solution

1. Traefik's rewrite target annotation should be changed from
   traefik.ingress.kubernetes.io/rewrite-target: /function/FUNC_NAME/$1
   to
   traefik.ingress.kubernetes.io/rewrite-target: /function/FUNC_NAME
   where FUNC_NAME is the name of the function.
   I've created PR which fixes this issue.

2. (Nice to have) Document (where?) that Traefik v1.7.21 or above is required

Steps to Reproduce (for bugs)

Disclaimer. I tested all of the following on linux, so not really sure that this would work on Mac or Windows.

1. Create k8s cluster locally

minikube start

2. Install traefik as ingress controller. For demonstration purposes, I first install v1.7.20

helm upgrade -i traefik \
--version 1.87.1 \
--namespace kube-system \
--set imageTag=1.7.20 \
--set rbac.enabled=true \
--set dashboard.enabled=true \
--set dashboard.serviceType=NodePort \
--set dashboard.domain=traefik.local.dev \
--set dashboard.ingress.annotations."kubernetes\.io\/ingress\.class=traefik" \
--set deployment.hostPort.httpEnabled=true \
--set deployment.hostPort.dashboardEnabled=true \
--set serviceType=NodePort \
--set containerSecurityContext.capabilities.drop[0]=ALL \
--set containerSecurityContext.capabilities.add[0]=NET_BIND_SERVICE \
--set deploymentStrategy.type=Recreate \
stable/traefik

3. Install openfaas with ingress operator enabled

kubectl create ns dev-openfaas
kubectl create ns dev-openfaas-fn
helm upgrade -i openfaas \
--version 5.8.5 \
--namespace dev-openfaas \
--set functionNamespace=dev-openfaas-fn \
--set basic_auth=false \
--set generateBasicAuth=false \
--set ingressOperator.create=true \
--set ingress.enabled=true \
--set ingress.hosts[0].host=gateway.local.dev \
--set ingress.hosts[0].serviceName=gateway \
--set ingress.hosts[0].servicePort=8080 \
--set ingress.hosts[0].path=/ \
--set ingress.annotations."kubernetes\.io\/ingress\.class=traefik" \
openfaas/openfaas

4. (Optional) Add Traefik dashboard and OpenFaaS Gateway UI to your hosts if you want to access them via their domain names

echo "$(minikube ip) traefik.local.dev" | sudo tee -a /etc/hosts
echo "$(minikube ip) gateway.local.dev" | sudo tee -a /etc/hosts

or alternatively you can get their services urls to access their UIs

minikube -n kube-system service traefik-dashboard --url
minikube -n dev-openfaas service gateway-external --url

5. Deploy function

faas-cli deploy \
--image stefanprodan/podinfo:3.1.0 \
--namespace dev-openfaas-fn \
--name podinfo \
--annotation com.openfaas.health.http.initialDelay="2s" \
--annotation com.openfaas.health.http.path="/healthz" \
--env PODINFO_PORT="8080" \
--gateway $(minikube -n dev-openfaas service gateway-external --url) \
--tls-no-verify

6. Deploy FunctionIngress

cat <<EOF | kubectl apply -f -
apiVersion: openfaas.com/v1alpha2
kind: FunctionIngress
metadata:
  name: podinfo
  namespace: dev-openfaas
spec:
  function: podinfo
  domain: podinfo.local.dev
  ingressType: "traefik"
  bypassGateway: false
  tls:
    enabled: false
EOF

Now the interesting part begins

7. Let's see what routing rule we have in Traefik for the ingress the operator created

curl -s $(minikube -n kube-system service traefik-dashboard --url)/api/providers/kubernetes/frontends | jq -r '.["podinfo.local.dev/"].routes'

Output:

{
  "/": {
    "rule": "PathPrefix:/;ReplacePathRegex: ^/(.*) /function/podinfo/$1$1"
  },
  "podinfo.local.dev": {
    "rule": "Host:podinfo.local.dev"
  }
}

Note the two $1 at the end of the string with the rule. This is why our function gets duplicated path with this Traefik version.

8. Add function domain to your hosts and let's see how the routing works

echo "$(minikube ip) podinfo.local.dev" | sudo tee -a /etc/hosts
curl -s http://podinfo.local.dev

Output:

{
  "hostname": "podinfo-5569bb569c-9wvdw",
  "version": "3.1.0",
  "revision": "7b6f11780ab1ce8c7399da32ec6966215b8e43aa",
  "color": "cyan",
  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
  "message": "greetings from podinfo v3.1.0",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.13.1",
  "num_goroutine": "6",
  "num_cpu": "2"
}

And let's also request the other path

curl -s http://podinfo.local.dev/version

Output:

404 page not found

Function actually gets /versionversion as the path.
Let's also make sure that this path is handled by requesting it through gateway:

curl -s http://gateway.local.dev/function/podinfo/version

or

curl -s $(minikube -n dev-openfaas service gateway-external --url)/function/podinfo/version

Output:

{
  "commit": "7b6f11780ab1ce8c7399da32ec6966215b8e43aa",
  "version": "3.1.0"
}

9. Now let's change the annotation on the ingress to the proposed and see if it works with this version of Traefik

kubectl -n dev-openfaas annotate --overwrite ing podinfo 'traefik.ingress.kubernetes.io/rewrite-target=/function/podinfo'
curl -s http://podinfo.local.dev/version

Output:

error finding function podinfoenv.dev-openfaas-fn: server returned non-200 status code (404) for function, podinfoenv

10. Let's rollback changes to the ingress and upgrade traefik to v1.7.23 to see how things work with this version

kubectl -n dev-openfaas annotate --overwrite ing podinfo 'traefik.ingress.kubernetes.io/rewrite-target=/function/podinfo/$1'
helm upgrade traefik \
--namespace kube-system \
--reuse-values \
--set imageTag=1.7.23 \
stable/traefik
kubectl -n kube-system rollout status deploy/traefik

11. Let's request the same paths and see what we get (spoiler - infinite redirects). I intentionally limited redirects count to keep the output at acceptable level.

curl -sLv --max-redirs 1 http://podinfo.local.dev

Output:

* Rebuilt URL to: http://podinfo.local.dev/                                                                                                                                                                                                                                                                                                                          
*   Trying 192.168.39.39...                                                                                                                                                                                        
* TCP_NODELAY set                                                                                                                                                                                                  
* Connected to podinfo.local.dev (192.168.39.39) port 80 (#0)                                                                                                                                                      
> GET / HTTP/1.1                                                                                                                                                                                                   
> Host: podinfo.local.dev                                                                                                                                                                                          
> User-Agent: curl/7.58.0                                                                                                                                                                                          
> Accept: */*                                                                                                                                                                                                      
>                                                                                                                                                                                                                  
< HTTP/1.1 301 Moved Permanently                                                                                                                                                                                   
< Content-Length: 0
< Date: Thu, 18 Jun 2020 11:18:59 GMT
< Location: /function/podinfo/
< Vary: Accept-Encoding
< 
* Connection #0 to host podinfo.local.dev left intact
* Issue another request to this URL: 'http://podinfo.local.dev/function/podinfo/'
* Found bundle for host podinfo.local.dev: 0x5566c09a19d0 [can pipeline]
* Re-using existing connection! (#0) with host podinfo.local.dev
* Connected to podinfo.local.dev (192.168.39.39) port 80 (#0)
> GET /function/podinfo/ HTTP/1.1
> Host: podinfo.local.dev
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Date: Thu, 18 Jun 2020 11:18:59 GMT
< Location: /function/podinfo/function/podinfo/function/podinfo/
< Vary: Accept-Encoding
< 
* Connection #0 to host podinfo.local.dev left intact
* Maximum (1) redirects followed

curl -sLv --max-redirs 1 http://podinfo.local.dev/version

Output:

*   Trying 192.168.39.39...                                                                                                                                                                                        
* TCP_NODELAY set                                                                                                                                                                                                  
* Connected to podinfo.local.dev (192.168.39.39) port 80 (#0)                                                                                                                                                      
> GET /version HTTP/1.1                                                                                                                                                                                            
> Host: podinfo.local.dev                                                                                                                                                                                          
> User-Agent: curl/7.58.0                                                                                                                                                                                          
> Accept: */*                                                                                                                                                                                                      
>                                                                                                                                                                                                                  
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 18 Jun 2020 11:22:08 GMT
< Location: /function/podinfo/version/version
< Vary: Accept-Encoding
< 
* Connection #0 to host podinfo.local.dev left intact
* Issue another request to this URL: 'http://podinfo.local.dev/function/podinfo/version/version'
* Found bundle for host podinfo.local.dev: 0x5584618379d0 [can pipeline]
* Re-using existing connection! (#0) with host podinfo.local.dev
* Connected to podinfo.local.dev (192.168.39.39) port 80 (#0)
> GET /function/podinfo/version/version HTTP/1.1
> Host: podinfo.local.dev
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Date: Thu, 18 Jun 2020 11:22:08 GMT
< Location: /function/podinfo/function/podinfo/version/version/function/podinfo/version/version
< Vary: Accept-Encoding
< 
* Connection #0 to host podinfo.local.dev left intact
* Maximum (1) redirects followed

12. Now let's change the annotation on the ingress to the proposed and see how it works

kubectl -n dev-openfaas annotate --overwrite ing podinfo 'traefik.ingress.kubernetes.io/rewrite-target=/function/podinfo'

curl -s http://podinfo.local.dev

Output:

{
  "hostname": "podinfo-5569bb569c-9wvdw",
  "version": "3.1.0",
  "revision": "7b6f11780ab1ce8c7399da32ec6966215b8e43aa",
  "color": "cyan",
  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
  "message": "greetings from podinfo v3.1.0",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.13.1",
  "num_goroutine": "6",
  "num_cpu": "2"
}

curl -s http://podinfo.local.dev/version                                                                                                                                                                         

Output:

{                                                                                                                                                                                                                  
  "commit": "7b6f11780ab1ce8c7399da32ec6966215b8e43aa",                                                                                                                                                            
  "version": "3.1.0"                                                                                                                                                                                               
}

Those who made this far: Thanks for reading and don't forget to clean your hosts file at /etc/hosts

Context

I'm trying to use ingress operator with traefik to forward all traffic from specific domain to function.

Your Environment

• Docker version docker version (e.g. Docker 17.0.05 ):
  Docker version 19.03.11, build 42e35e61f3

• What version and distriubtion of Kubernetes are you using? kubectl version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-21T20:58:41Z", GoVersion:"go1.13.11", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Operating System and version (e.g. Linux, Windows, MacOS):
  Ubuntu 18.04.4 LTS

• Link to your project or a code example to reproduce issue:
  n/a

• What network driver are you using and what CIDR? i.e. Weave net / Flannel
  n/a

When defining the FunctionIngress you should be able to customise the host path, e.g. in a RESTful API you may want to map functions more logically (or if you were porting an existing application).

Expected Behaviour

Path maps and routes correctly

Current Behaviour

FunctionIngress only ever maps to /(.*) for each domain specified

Possible Solution

Currently, the generated ingress looks like;

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: letsencrypt-openfaas-io-prod
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /function/openfaas-insiders/$1
  name: insiders-openfaas-io-tls
spec:
  rules:
  - host: insiders.openfaas.io
    http:
      paths:
      - backend:
          serviceName: gateway
          servicePort: 8080
        path: /(.*)
  tls:
  - hosts:
    - insiders.openfaas.io
    secretName: insiders-openfaas-io-tls-cert

However, for this particular use case the end result should look like;

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    name: api-app-profiles-create
    annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/function/profiles-view/$1"
        kubernetes.io/ingress.class: "ingress-api"
    namespace: openfaas
spec:
    tls:
    - hosts:
      - api.example.dev
      secretName: example-tls
    rules:
    - host: api.example.dev
      http:
        paths:
        - path: /v1/profiles/view/(.*)
          backend:
            serviceName: gateway
            servicePort: 8080

Through some form of annotation on the CRD filter up the; spec.rules.host.http.paths.backend.path

Context

Provides RESTful like URI's through OF without the need for manually defining the ingress(s)

Currently only the Nginx ingress controller is supported. It'd be good to provide support for others, in this case Skipper.

Possible Solution

Example manual ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-openfaas
  namespace: openfaas
  annotations:
    kubernetes.io/ingress.class: skipper
    zalando.org/skipper-filter: setPath("/function/hello-openfaas")
spec:
  rules:
    - host: hello-openfaas.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: gateway
              servicePort: 8080

Context

We're not using Nginx controller in our clusters so wouldn't be able to leverage this controller. Currently we're manually creating ingress resources using the above example.

Your Environment

• What version and distriubtion of Kubernetes are you using: 1.12

• What network driver are you using and what CIDR: Weave net

Since #7 was merged, we can use ExternalDNS to automatically create DNS entries for the FunctionIngress.

We could document this behaviour, including:

• How to setup ExternalDNS and linking to their documentation
• How to verify that the DNS was created

What do you think? Any more ideas on what do document?

Expected Behaviour

It should deploy the operator

Current Behaviour

The deployment never gets ready, CrashLoopBack on pod

Possible Solution

Modify ingress-operator/artifacts/operator-amd64.yaml

command:
- ./ingress-operator
- -logtostderr
- -v=2

remove --v=2
After that it worked for me.

Steps to Reproduce (for bugs)

Context

Your Environment

• Docker version docker version (e.g. Docker 17.0.05 ):

• What version and distriubtion of Kubernetes are you using? kubectl version

1.6

• Operating System and version (e.g. Linux, Windows, MacOS):

• Link to your project or a code example to reproduce issue:

• What network driver are you using and what CIDR? i.e. Weave net / Flannel

The current instructions in the readme https://github.com/openfaas-incubator/ingress-operator#ingresscontroller-nginx

Expected Behaviour

$ helm install nginxingress stable/nginx-ingress --set rbac.create=true
WARNING: This chart is deprecated
NAME: nginxingress
LAST DEPLOYED: Sat Oct 31 17:34:38 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
*******************************************************************************************************
* DEPRECATED, please use https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx *
*******************************************************************************************************


The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w nginxingress-nginx-ingress-controller'

An example Ingress that makes use of the controller:

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

Current Behaviour

$ helm install stable/nginx-ingress --name nginxingress --set rbac.create=true
Error: unknown flag: --name

Possible Solution

Update the helm install instructions, we should also update which repo it is using.

Your Environment

$ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"dirty", GoVersion:"go1.14.3"}

Expected Behaviour

This feature would allow an Ingress record to be created that bypasses the gateway, so that an endpoint can be accessed with absolute minimal latency, by removing an additional hop.

Caveats

Today the IngressOperator, FNI definitions are created in the openfaas namespace which causes an Ingress record to be created in the openfaas namespace pointing at the gateway, which is also deployed in that namespace.

For a bypass mode we will have to deploy all resources a second time into the openfaas-fn namespace because Ingress cannot point into a different namespace - i.e. having an Ingress in openfaas which connects to a service in openfaas-fn. It only works in the first example because the gateway acts as a proxy.

See also: kubernetes/kubernetes#17088

Expected Behaviour

I expect the ingress-operator to be able to watch multiple namespaces, all namespaces, or namespaces specified by annotation. This would also include the appropriate RBAC permissions for the ingress-operator.

Current Behaviour

Currently, the ingress-operator is set to watch a specific namespace as defined in the 'function_namespace' and 'ingress_namespace' environment variables. If I want to use a different namespace, I have to manually override these values and adjust the RBAC permissions.

List All Possible Solutions and Workarounds

Adding an option to watch all namespaces in nothing was set.
Allowing the definition of multiple namespaces to watch.
Allowing the use of namespace annotations to determine which namespaces to watch.

Which Solution Do You Recommend?

I would recommend implementing the option to watch namespaces by labels(as with OpenFaas pro), as this would offer the most flexibility to users.

Steps to Reproduce (for bugs)

It's more a feature request than bug.
Anyway the steps to reproduce are simple: deploy the stack.yaml to namespace associated with openfaas.

Context

I am trying to deploy the ingress-operator in a different namespace ('demo' in my case).
However, I need to override the 'function_namespace', 'ingress_namespace' values and adjust the RBAC permissions time.

Your Environment

Less related to the issue.

• FaaS-CLI version ( Full output from: faas-cli version ):
  commit: 0074051aeb837f5f160ee8736341460468b5c190
  version: 0.15.4

• pro license.

• Docker version docker version (e.g. Docker 17.0.05 ):

• What version and distriubtion of Kubernetes are you using? kubectl version
  Server Version: v1.21.14-eks-0a21954

• Operating System and version (e.g. Linux, Windows, MacOS):
  AmazonLinux2

version: 1.0
provider:
  name: openfaas
  gateway: https://openfaas-gw.dev.local/
functions:
  next-gen:
    lang: [email protected]
    handler: .
    image: my-custom-image:tag
    secrets: []
    environment:
      fprocess: python function/main.py
    labels:
      com.openfaas.scale.min: "1"
      com.openfaas.scale.max: "5"
      com.openfaas.scale.target: "5"
      com.openfaas.scale.type: rps
      com.openfaas.scale.target-proporation: "0.80"
function-ingress:
  next-gen-ing:
    domain: "gen2.dev.local"
    function: "next-gen"
    ingressType: "nginx-general"
    annotations:
      external-dns.alpha.kubernetes.io/hostname: "gen2.dev.local"

Expected Behaviour

K8s < 1.22 and > 1.22 should work without changes.

Current Behaviour

Only < 1.22 is supported due to the API group being used going away.

List All Possible Solutions and Workarounds

Have an if statement based upon the kube API server version returned.

Steps to Reproduce (for bugs)

1. Create a new cluster with minikube or kind using 1.22
2. Deploy
3. Try to create FunctionIngress
4. See it fail

Context

Must fix to continue being used on latest K8s versions.

It will be useful if we can pass on the ingress operator the secretName on config.yaml.

At the moment it only creates a new cert from cert issuer like the cert-manager provider.

We have a case where we are using only wildcard domain and we have to edit manually each Ingress to use the same secretName like an example below.

apiVersion: openfaas.com/v1alpha2
kind: FunctionIngress
metadata:
  name: nodeinfo
  namespace: openfaas
spec:
  domain: "nodeinfo.myfaas.club"
  function: "nodeinfo"
  ingressType: "nginx"
  tls:
     enabled: true
     secretName: same-cert-secret

If it is a workaround like to automate this bit, it will very useful to know.

What do you think about it?

Expected Behaviour

It's still needed to manually create a DNS entry when using FunctionIngress. Implementing support to ExternalDNS would allow FunctionIngress to create, delete and update DNS entries automatically.

Current Behaviour

The need to create manually DNS entries each time a new FunctionIngress is deployed.

Possible Solution

Implement support for ExternalDNS to manage DNS entries automatically.

This tutorial shows how to use ExternalDNS with NGINX Ingress. The behaviour is basically synchronising the Ingress domains names with the DNS.

Also, if using with Services, ExternalDNS has annotations to indicate that the DNS entry should be manager, as shown in this example. We could try to enable/disable the ExternalDNS in FunctionIngress with these annotations.

Context

Would be amazing to eliminate one more manual process to get TLS/custom handlers to our functions ⚡️

Expected Behaviour

We could use Google's distroless container image for the runtime for this operator

Current Behaviour

It's using scratch and then we manually copy in things like certs and user files.

See also: https://github.com/alexellis/registry-creds/blob/master/Dockerfile#L28

Expected Behaviour

Convert CI to GitHub Actions

Current Behaviour

We use Travis and need to move away

Possible Solution

See the approach in https://github.com/openfaas/faas-netes/tree/master/.github

In Kubernetes 1.18 the Ingress definitions have changed slightly.

In particular the Kubernetes release notes [1] mention the change about ingressClass becoming part of the spec, instead of being an annotation.

If we can add both, perhaps that will help keep backwards compatibility for the time being?

[1] https://kubernetes.io/docs/setup/release/notes/#extending-ingress-with-and-replacing-a-deprecated-annotation-with-ingressclass

Alex

Expected Behaviour

It should deploy artifacts/operator-rbac.yaml

Current Behaviour

It deploys, but kubectl logs come back with
Failed to list *v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:openfaas:ingress-operator" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "openfaas"

Possible Solution

modify artifacts/operator-rbac.yaml

• apiGroups: ["extensions", "networking"]
  resources: ["ingresses"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

to

• apiGroups: ["extensions", "networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Context

Trying to setup custom domain for my functions

Your Environment

• Docker version docker version (e.g. Docker 17.0.05 ):

• What version and distriubtion of Kubernetes are you using? kubectl version
  1.6

• Operating System and version (e.g. Linux, Windows, MacOS):

• Link to your project or a code example to reproduce issue:

• What network driver are you using and what CIDR? i.e. Weave net / Flannel

Expected Behaviour

$ kubectl apply -f examples/tls-nodeinfo-ingress.yaml
functioningress.openfaas.com/nodeinfo-tls created

Current Behaviour

$ kubectl apply -f examples/tls-nodeinfo-ingress.yaml
The FunctionIngress "nodeinfo-tls" is invalid: spec.path: Required value

Possible Solution

Add path: "/" to each of the examples and remove the comment in the Readme that path is optional https://github.com/openfaas-incubator/ingress-operator#schema

Context

Trying to test my last PR by using the example files

Current Behaviour

All target functions are assumed to be within the openfaas-fn namespace, and the spec only takes a function as an input:

apiVersion: openfaas.com/v1
kind: FunctionIngress
metadata:
  name: nodeinfo
  namespace: openfaas
spec:
  domain: "nodeinfo.myfaas.club"
  function: "nodeinfo"
  ingressType: "nginx"
  path: "/v1/profiles/(.*)"

Expected Behaviour

Add a functionNamespace field into the spec:

apiVersion: openfaas.com/v1
kind: FunctionIngress
metadata:
  name: nodeinfo
  namespace: openfaas
spec:
  domain: "nodeinfo.myfaas.club"
  function: "nodeinfo"
+ functionNamespace: "staging-fn"
  ingressType: "nginx"
  path: "/v1/profiles/(.*)"

If we really want to do things properly and make a breaking change, this could be a functionRef:

spec:
  domain: "nodeinfo.myfaas.club"
  functionRef:
     name: "nodeinfo"
   namespace: "staging-fn"

But I think that might be more work than is warranted for now.

Who is this for? What company will benefit from the changes?

Kubiya asked for this in #63

List All Possible Solutions and Workarounds

All FunctionIngress objects and their eventual Ingress objects must be created in the openfaas namespace since they all point at the gateway.

But that said, the namespace is just appended to the path that comes out on the Ingress:

http://gateway.openfaas:8080/function/FUNCTION.NAMESPACE

Expected Behaviour

The badge in the readme should reflect the status of the CI build for this project

Current Behaviour

It shows the status of a Travis build which is unknown

List All Possible Solutions and Workarounds

Update with the GitHub Actions generated badge

Task

Test multiple paths for the same host

i.e. two function ingress entries as per the new feature in #20, but with the same domain and differing paths.

I.e. can we write two FunctionIngress records that would serve two separate back-end functions from the same sub-domain and TLS cert:

api.dx.no/v1/customers
api.dx.no/v1/tickets

apiVersion: openfaas.com/v1alpha2
kind: FunctionIngress
metadata:
  name: api.dx.no
  namespace: openfaas
spec:
  domain: "api.dx.no"
  function: "list-customers"
  ingressType: "nginx"
  path: "/v1/customers/(*.)"
    tls:
      enabled: true
      issuerRef:
         name: "letsencrypt-staging"
         kind: "Issuer"

---
apiVersion: openfaas.com/v1alpha2
kind: FunctionIngress
metadata:
  name: api.dx.no
  namespace: openfaas
spec:
  domain: "api.dx.no"
  function: "list-tickets"
  ingressType: "nginx"
  path: "/v1/tickets/(*.)"
    tls:
      enabled: true
      issuerRef:
         name: "letsencrypt-staging"
         kind: "Issuer"

Expected Behaviour

When adding/updating annotations to the FunctionIngress, these should be added/updated on the underlying Ingress.

Current Behaviour

If the FunctionIngress and the underlying Ingress already exist, changes to the FunctionIngress annotations aren't passed to the underlying Ingress.

Possible Solution

There's probably a validation on whether the ingress already exists that is bypassing the expected behaviour.

Steps to Reproduce (for bugs)

1. Create a FunctionIngress with no custom annotations
2. Apply with kubectl
3. Add a custom annotation, for example nginx.ingress.kubernetes.io/limit-connections: 10
4. Apply with kubectl
5. Annotations aren't in the ingress

Your Environment

• ingress-operator 0.6.5
• v1.18.8 (DigitalOcean)
• Linux

Per the security announcement https://groups.google.com/forum/#!topic/golang-announce/65QixT3tcmg we should update the docker build layers to at least Go 1.11.13

Possible Solution

Because the docker build layers already use golang:1.11 we should only need to rebuild and tag a new release to ensure that it is using the latest 1.11.13 base image

Related

Update Dockerfile and rebuild/release

openfaas/faas#1291
openfaas/templates#170
openfaas/faas-netes#494
openfaas/nats-queue-worker#66
openfaas/of-watchdog#78
https://github.com/openfaas-incubator/faas-idler/issues/32
openfaas/golang-http-template#28
openfaas-incubator/faas-federation#4
openfaas-incubator/vcenter-connector#27
openfaas-incubator/faas-memory#3
openfaas-incubator/faas-rancher#8

Only Rebuild

openfaas/faas-swarm#56
#10
https://github.com/openfaas-incubator/openfaas-operator/issues/87

Expected Behaviour

The function deployed in the steps below should be invokable at http://example.kube/api/v1/test/.

Current Behaviour

The function deployed is not reachable at http://example.kube/api/v1/test/ but rather a 404 (by Nginx) is returned. Visiting http://example.kube/api/v1/test/(.*) also returns a 404 (OpenFaaS). While the function works as intended at http://gw.example.kube/function/test-fn

Are you a GitHub Sponsor (Yes/No?)

Check at: https://github.com/sponsors/openfaas

• Yes
• No

List All Possible Solutions and Workarounds

The only workarounds I can think of right now. Are to either call the gateway path (i.e., not using the ingress-operator) or manually compose and maintain ingresses.

Which Solution Do You Recommend?

I'd prefer it if I got ingress-operators working as intended.

Steps to Reproduce (for bugs)

1. Start with a clean kubernetes cluster.
2. Add 127.0.0.1 example.kube and 127.0.0.1 gw.example.kube to your hosts file.
3. Execute the following shell script

GW_HOST="http://gw.example.kube"
FN_NAME="test-fn"

# Create namespaces
kubectl create namespace ingress-nginx
kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml

# Update chart repos
helm repo add nginx-stable https://helm.nginx.com/stable
helm repo add openfaas https://openfaas.github.io/faas-netes/

helm repo update

# Apply charts
helm install example-ingress nginx-stable/nginx-ingress \
    -n ingress-nginx \
    --set rbac.create=true

helm install openfaas openfaas/openfaas \
    -n openfaas  \
    --values openfaas.yml

# Wait for OpenFaaS to start
sleep 60

# Login to gateway
echo $(kubectl -n openfaas get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode) | faas-cli login -g $GW_HOST -u admin --password-stdin

# Get and host base node14 template
faas-cli template store pull node14
faas-cli new $FN_NAME --lang node14 -g $GW_HOST
faas-cli build -f "$FN_NAME.yml"
faas-cli deploy -f "$FN_NAME.yml"

# Apply FunctionIngress for test-fn
kubectl apply -f test-fi.yml

with the following files:

# openfaas.yml
functionNamespace: openfaas-fn
generateBasicAuth: true
ingressOperator:
  create: true
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
  hosts:
    - host: gw.example.kube
      serviceName: gateway
      servicePort: 8080
      path: /
faasnetes:
  imagePullPolicy: IfNotPresent

# test-fi.yml
apiVersion: openfaas.com/v1alpha2
kind: FunctionIngress
metadata:
  name: test-fn
  namespace: openfaas
spec:
  domain: "example.kube"
  function: "test-fn"
  ingressType: "nginx"
  path: "/api/v1/test/(.*)"

4. Go to http://example.kube/api/v1/test/

Context

I'm trying to create a more REST-like path for my api. As well as, replace existing apis with functions.

Your Environment

• FaaS-CLI version ( Full output from: faas-cli version ):

  ___                   _____           ____
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
      |_|

CLI:
 commit:  72816d486cf76c3089b915dfb0b66b85cf096634
 version: 0.13.13

• Docker version docker version (e.g. Docker 17.0.05 ):
  20.10.7
• What version and distriubtion of Kubernetes are you using? kubectl version
  I'm using Docker Desktop's Kubernetes at version 1.21.2
• Operating System and version (e.g. Linux, Windows, MacOS):
  Windows 10, kubectl reports that the cluster is running on Linux
• Helm version
  3.5.3

v1alpha2 cert-manager API referred to in the README is getting deprecated.

Possible Solution

Update the example configs to use the latest v1 API.

Expected Behaviour

The YAML spec file should contain a field for Path for validation, it's missing.

Current Behaviour

The Path field from FunctionIngress' Spec is missing from the YAML CRD

Possible Solution

Add it, or regenerate the spec.

Steps to Reproduce (for bugs)

There is no bug, it still works, but it's better to have the schema correct.

Reported in #31

Expected Behaviour

Instead of using spec.ingressType to select an Ingress class, we should use the default annotation kubernetes.io/ingress.class, since #7 was implemented.

Current Behaviour

In the version 0.4.0, by the docs, we need to define the Ingress class using the spec.ingressType configuration.

BUT

After merging #7 we actually don't need that. I tested two cases:

Empty annotation and spec.ingressType

Deploying the following FunctionIngress without spec.ingressType and without kubernetes.io/ingress.class annotation defaults the Ingress class to nginx and works. I believe this should be considered a bug and specifying an Ingress class should be mandatory.

apiVersion: openfaas.com/v1alpha2
kind: FunctionIngress
metadata:
  name: nodeinfo
  namespace: openfaas
spec:
  domain: "nodeinfo.deployeveryday.com"
  function: "nodeinfo"
  # ingressType: "nginx"  # disable ingressType

Use kubernetes.io/ingress.class annotation

Deploying the following FunctionIngress without spec.ingressType but using the kubernetes.io/ingress.class works like a charm. I believe this should be the default way of doing things.

apiVersion: openfaas.com/v1alpha2
kind: FunctionIngress
metadata:
  name: nodeinfo
  namespace: openfaas
  annotations:
    # Uses default K8S annotation
    kubernetes.io/ingress.class: "nginx"
spec:
  domain: "nodeinfo.deployeveryday.com"
  function: "nodeinfo"
  # ingressType: "nginx"  # disable ingressType

Possible Solution

1. getClass should yield an error if not class is defined.
2. makeAnnotations should get the Class from annotations and not from function.Spec.IngressType.

Context

Using the default K8S annotation to select an Ingress class requires lower learning curve and it's more a standard across other K8S controllers.