• change the repo name
• changes to go.mod
• changes to all files with Rafay
• CI / CD needs to be changed
• readme's, contributing docs, issue, features templates ..
• governance related docs - to be checked

Expected vs actual behavior

• User should be able to perform web kubectl to imported cluster always

Steps to reproduce the bug

1. Import a cluster
2. After a day or two, hit web kubectl and it would fail to establish a session ( happens sporadically )

Are you using the latest version of the project?

You can check your version by running helm ls|grep '^<deployment-name>' or using pctl, pctl version, and provide the output.

• yes, 0.1.3

• I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.

• I'm using the latest version of the project.

Expected vs actual behavior

Unable to connect to cluster via Lens after a while since initial connection. Relay agent needs to be rebooted to allow connection to target cluster again.

relay-agent logs:
{"level":"info","ts":"2023-08-21T16:11:04.457Z","caller":"tunnel/client.go:562","msg":"Relay Agent.Client.paralus-core-relay-agent::action handshake addr 18.204.85.83:443 "}
{"level":"info","ts":"2023-08-21T16:11:13.757Z","caller":"agent/agent.go:444","msg":"Relay Agent::setting hash of configmap data hash: 6b4af4b720c5f97671316f2d25f3a7802ef690289f7e0bd465b1c407616e36e2 "}
{"level":"info","ts":"2023-08-21T16:16:04.350Z","caller":"cleanup/authz.go:143","msg":"finding authz matching","selector":"authz-expiry<1692634564,paralus-relay=true"}
{"level":"info","ts":"2023-08-21T16:16:04.388Z","caller":"tunnel/client.go:209","msg":"Relay Agent.Client::scaled client timer totalScaledClient : 0 "}

Steps to reproduce the bug

1. Deploy paralus
2. Connect to cluster (do not disconnect)
3. Try to connect after 5

Are you using the latest version of the project?

     NAME      NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION

paralus paralus 5 2023-08-16 17:14:50.752987 -0400 EDT deployed ztka-0.2.5 v0.2.4

You can check your version by running helm ls|grep '^<deployment-name>' or using pctl, pctl version, and provide the output.

What is your environment setup? Please tell us your cloud provider, operating system, and include the output of kubectl version --output=yaml and helm version. Any other information that you have, eg. logs and custom values, is highly appreciated!

helm version
version.BuildInfo{Version:"v3.11.0", GitCommit:"472c5736ab01133de504a826bd9ee12cbe4e7904", GitTreeState:"clean", GoVersion:"go1.18.10"}

kubectl version --output=yaml
clientVersion:
buildDate: "2022-01-25T21:25:17Z"
compiler: gc
gitCommit: 816c97ab8cff8a1c72eccca1026f7820e93e0d25
gitTreeState: clean
gitVersion: v1.23.3
goVersion: go1.17.6
major: "1"
minor: "23"
platform: darwin/amd64
serverVersion:
buildDate: "2023-07-28T16:53:07Z"
compiler: gc
gitCommit: 222fd0c4adcf243ebeaee986cc0d41a39e570d8f
gitTreeState: clean
gitVersion: v1.23.17-eks-2d98532
goVersion: go1.19.6
major: "1"
minor: 23+
platform: linux/amd64

Kubernetes cluster running on AWS

(optional) If you have ideas on why the bug happens or how it can be solved, please provide it here

• I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
• I'm using the latest version of the project.

Expected vs actual behavior

If you run kubectl get secret for a read-only user then it works, whereas kubectl get secrets doesn't.

Steps to reproduce the bug

1. Grant readonly rights to user
2. Run kubectl get pods -o json | jq '.items[].spec.containers[].env[]?.valueFrom.secretKeyRef.name' | grep -v null | uniq command for authorized namespace
3. Secrets should return
4. Run kubectl get secret <secret_name> and it works

Are you using the latest version of the project?

Using version 2.0

What is your environment setup? Please tell us your cloud provider, operating system, and include the output of kubectl version --output=yaml and helm version. Any other information that you have, eg. logs and custom values, is highly appreciated!

Kubernetes 1.24

(optional) If you have ideas on why the bug happens or how it can be solved, please provide it here

The guard against secrets should be a wildcard so that it can protect against secret and sec

• [X ] I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
• I'm using the latest version of the project.

The relay agent golang library contains the following vulnerabilities:

github.com/prometheus/client_golang fixed in version 1.12.2

https://bugzilla.redhat.com/show_bug.cgi?id=2067400

golang.org/x/text fixed in version 0.3.8

https://www.cvedetails.com/cve/CVE-2021-38561/

These libraries need to be updated.

• I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
• I'm using the latest version of the project.

Go test failed for go test ./...:

starting client
Waiting for workersstarting clientCompleted--- FAIL: TestRelayClient (0.00s)
    client_test.go:25: failed to connect  dial unix /tmp/relay-unix-kubectldialin.relay.rafay.dev: connect: no such file or directory
FAIL
FAIL	github.com/RafaySystems/relay/pkg/tunnel	0.028s
?   	github.com/RafaySystems/relay/pkg/utils	[no test files]
FAIL

Briefly describe the feature

As of now, if the user selects a namespace when specifying a project-user-role mapping which does not exist, we create the namespace on the first use. This is done as we rely on the namespace being available to generate SA and other necessary resources, but it would be better to not create the namespace and just return with error/empty if they try to access it.

What problem does this feature solve? Please link any relevant documentation or Issues

Make the behavior of missing namespaces more to what people might expect out of the box.

(optional) What is your current workaround?

-

Briefly describe the feature

• Need for a cross arch / platform docker image to be built

What problem does this feature solve? Please link any relevant documentation or Issues

• Currently it builds only linux/amd64 based images. Solves compatibility issues.

Describe the request

Currently GH actions uses a user's account to build and publish images, change this to use a build specific account

• changes to create namespaces from step objects
• verify namespaced roles and role mapping creation

This is a cross post from this issue

After applying the -boostrap.yaml for the relay-agent onto clusters that I want to import, the agent is not able to connect to Paralus to register clusters. I did some debugging and found that it was not Okta, but the relay application with this problem.

The certificate generated for SSL was created following the Deploy ClusterIssuer and Certificate Objects with cert-manager.

Expected vs actual behavior

• Expect

  • clusters to register with Paralus and the Cluster Connection status to read SUCCESSFUL when viewing the clusters in a project

• Actual

  • relay-agent unable to connect and register cluster with Paralus due to Method not allowed
  • Error shown:
    [POST /v2/sentry/bootstrap/{templateToken}/register][501] Bootstrap_RegisterBootstrapAgent default &{Code:12 Details:[] Message:Method Not Allowed}

• cluster registration stuck pending and Cluster Connection status reads FAILURE

Steps to reproduce the bug

1. Deploy Paralus
2. Enable SSL
3. Try to import another cluster following the instructions in the Paralus Console

Are you using the latest version of the project?

• chart version: ztka-0.2.4
• app version: v0.2.3

What is your environment setup? Please tell us your cloud provider, operating system, and include the output of kubectl version --output=yaml and helm version. Any other information that you have, eg. logs and custom values, is highly appreciated!

• AWS following the eks-quickstart blog post
• helm v3.12.2
• kubectl v1.25.x

(optional) If you have ideas on why the bug happens or how it can be solved, please provide it here

• Before adding the certificate I was able to import other clusters and use the kubectl terminal, perhaps the registration function(s) cannot communicate over HTTPS?

• I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
• I'm using the latest version of the project.

Expected vs actual behavior

When attempting to connect to certain servers, the kubectl client returns this error:

Error from server (InternalError): an error on the server ("read unix @->/tmp/relay-unix-*.core-connector.paralus.iherbpreprod.net: read: connection reset by peer") has prevented the request from succeeding (get pods)

or

Error from server (unable to proxy kubectl service): ERROR: Unauthenticated access not allowed. Please log in to the portal via browser, or set up API key, for access via the secure kubectl proxy. Error:context deadline exceeded

And on the server side I see this issue from the relay-server

{"level":"error","ts":"2023-02-23T02:26:21.061Z","caller":"tunnel/server.go:1677","msg":"Relay Server.Server::failed to compete TLS handshake error: remote error: tls: bad certificate  ","stacktrace":"github.com/paralus/relay/pkg/tunnel.(*Server).processDialinTLSState\n\t/build/pkg/tunnel/server.go:1677\ngithub.com/paralus/relay/pkg/tunnel.(*Server).handleDialinConnection\n\t/build/pkg/tunnel/server.go:1794"}
{"level":"error","ts":"2023-02-23T02:26:21.061Z","caller":"tunnel/server.go:1796","msg":"Relay Server.Server.handleDialinConnection::failed to process TLS state error: ailed to compete TLS handshake  ","stacktrace":"github.com/paralus/relay/pkg/tunnel.(*Server).handleDialinConnection\n\t/build/pkg/tunnel/server.go:1796"}

Restarting the relay agent of the cluster seems to fix it.

Steps to reproduce the bug

Uncertain what causes it.

Are you using the latest version of the project?

You can check your version by running helm ls|grep '^<deployment-name>' or using pctl, pctl version, and provide the output.

VERSION: 0.1.0
BUILD: 0.1.0
BUILD-TIME: 1656329967
ARCH: darwin/amd64

What is your environment setup? Please tell us your cloud provider, operating system, and include the output of kubectl version --output=yaml and helm version. Any other information that you have, eg. logs and custom values, is highly appreciated!

Kubectl

clientVersion:
  buildDate: "2022-08-23T17:44:59Z"
  compiler: gc
  gitCommit: a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2
  gitTreeState: clean
  gitVersion: v1.25.0
  goVersion: go1.19
  major: "1"
  minor: "25"
  platform: darwin/amd64
kustomizeVersion: v4.5.7
serverVersion:
  buildDate: "2022-11-29T18:41:42Z"
  compiler: gc
  gitCommit: 52e500d139bdef42fbc4540c357f0565c7867a81
  gitTreeState: clean
  gitVersion: v1.22.16-eks-ffeb93d
  goVersion: go1.16.15
  major: "1"
  minor: 22+
  platform: linux/amd64

Helm

version.BuildInfo{Version:"v3.10.3", GitCommit:"835b7334cfe2e5e27870ab3ed4135f136eecc704", GitTreeState:"clean", GoVersion:"go1.19.4"}

(optional) If you have ideas on why the bug happens or how it can be solved, please provide it here

The only workaround I've found is to delete the relay agent pod and have it get recreated. But then it happens almost immediately again.

• I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
• I'm using the latest version of the project.

Expected vs actual behavior

If you attempt to upgrade a paralus server from 0.2.0 with relay version 0.1.2 to 0.2.5 paralus with relay 0.1.5 (starting with 0.1.4), you'll get the error:

"level":"info","ts":"2023-08-16T23:25:59.118Z","caller":"agent/agent.go:171","msg":"Relay Agent::relay agent namespace: paralus-system  fingerprint: 4d63c263-0e39-4451-aa97-6882e339fc30  "}
{"level":"info","ts":"2023-08-16T23:25:59.119Z","caller":"agent/agent.go:394","msg":"Relay Agent::config: &{TemplateToken:cifi3glc3m5b406jcbc0 TemplateName: Scheme:https Mode: Addr:console.paralus.iherbtest.net:443 ClientID:cisqaj3ppcveb4n2bcrg ClientIP:10.7.228.141 Name:relay-agent-557f56bb69-x2rcc PrivateKey:[] CSR:[] Certificate:[] CACertificate:[] ServerHost: ServerPort:0 Fingerprint:4d63c263-0e39-4451-aa97-6882e339fc30} "}
[POST /v2/sentry/bootstrap/{templateToken}/register][500] bootstrapRegisterBootstrapAgentInternalServerError  map[code:2 message:fingerprint mismatch for token cisqaj3ppcveb4n2bcrg]
{"level":"error","ts":"2023-08-16T23:25:59.159Z","caller":"agent/agent.go:397","msg":"Relay Agent::failed to register relay agent error: [POST /v2/sentry/bootstrap/{templateToken}/register][500] bootstrapRegisterBootstrapAgentInternalServerError  map[code:2 message:fingerprint mismatch for token cisqaj3ppcveb4n2bcrg]  ","stacktrace":"github.com/paralus/relay/pkg/agent.registerRelayAgent\n\t/build/pkg/agent/agent.go:397\ngithub.com/paralus/relay/pkg/agent.handleRelayNetworks\n\t/build/pkg/agent/agent.go:606"}

The only way to fix this is to go to the database and update the fingerprint value in the sentry_bootstrap_agent table for the specific token row with the uid from the paralus-system namespace for that client (also found within the log)

Steps to reproduce the bug

1. Create a cluster in paralus 0.2.0 or 0.2.1 (so the relay agent is at 0.1.3 or lower)
2. Bootstrap it in and make sure it works
3. Upgrade the paralus to 0.2.3 or higher (so relay agent is 0.1.4 or higher)
4. Try to have the client connect and it will fail with the above issue

(optional) If you have ideas on why the bug happens or how it can be solved, please provide it here

This appears to be happening because an existing cluster does not automatically update the fingerprint if it's already bootstrapped into the database. You have to do a new bootstrapping, or update the value manually

• I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
• I'm using the latest version of the project.

Hi i have setup of paralus aka cloudflare ( plus ssl termination on it ) and then ingress take incoming connection to 80 ports, so i do not generate certificate in this way. Now i try connect external cluster and get error

err: x509: certificate signed by unknown authority

on core-connector. If it's possible skip it or set trusted certs or whatever ?
Thx

TODO: Add description
Got following log from relay-server container:

p.TypeEC PRIVATE KEY{"level":"error","ts":"2022-06-08T06:46:49.001Z","caller":"relay/relay.go:370","msg":"Relay Server::failed to register peering relay error: context deadline exceeded  \n","stacktrace":"github.com/RafayLabs/relay/pkg/relay.registerRelayPeerService\n\t/build/pkg/relay/relay.go:370\ngithub.com/RafayLabs/relay/pkg/relay.relayServerBootStrap\n\t/build/pkg/relay/relay.go:642\ngithub.com/RafayLabs/relay/pkg/relay.RunRelayServer\n\t/build/pkg/relay/relay.go:786"}
{"level":"error","ts":"2022-06-08T06:46:49.001Z","caller":"relay/relay.go:644","msg":"Relay Server::failed to register relay with peer-service-bootstrap service, will retry error: context deadline exceeded  \n","stacktrace":"github.com/RafayLabs/relay/pkg/relay.relayServerBootStrap\n\t/build/pkg/relay/relay.go:644\ngithub.com/RafayLabs/relay/pkg/relay.RunRelayServer\n\t/build/pkg/relay/relay.go:786"}

relay-tail container is restarting repeatedly with following error:

p.TypeEC PRIVATE KEY{"level":"info","ts":"2022-06-08T06:51:18.043Z","caller":"tail/register.go:82","msg":"unable to register","error":"context deadline exceeded"}
{"level":"panic","ts":"2022-06-08T06:51:18.050Z","caller":"tail/run.go:112","msg":"unable to create sentry authorization pool","error":"context deadline exceeded","stacktrace":"github.com/RafayLabs/relay/pkg/tail.runTail\n\t/build/pkg/tail/run.go:112\ngithub.com/RafayLabs/relay/pkg/tail.RunRelayTail\n\t/build/pkg/tail/run.go:212"}
panic: unable to create sentry authorization pool

goroutine 38 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0001aa000, {0xc000379500, 0x1, 0x2})
	/build/vendor/go.uber.org/zap/zapcore/entry.go:232 +0x446
go.uber.org/zap.(*SugaredLogger).log(0xc00011a660, 0x4, {0x1b75ad4, 0xc000593f90}, {0x0, 0xc000593eb0, 0x2}, {0xc000161e88, 0x2, 0x2})
	/build/vendor/go.uber.org/zap/sugar.go:227 +0xee
go.uber.org/zap.(*SugaredLogger).Panicw(...)
	/build/vendor/go.uber.org/zap/sugar.go:204
github.com/RafayLabs/relay/pkg/tail.runTail({0x1dd45f8, 0xc000718f00})
	/build/pkg/tail/run.go:112 +0xe5
github.com/RafayLabs/relay/pkg/tail.RunRelayTail({0x1dd45f8, 0xc000718f00})
	/build/pkg/tail/run.go:212 +0x34
created by main.main
	/build/main.go:122 +0x5d9

Expected vs actual behavior

After installing a relay client into a cluster, I got a TLS error. I had expected the error to be reported in the log as an "error" level. But instead it was "info"

Sample error:

{"level":"info","ts":"2023-02-10T17:07:24.326Z","caller":"tunnel/client.go:460","msg":"Relay Agent.Client.paralus-core-relay-agent::action backoff sleep: 26.373942655s  address: 6f881749-fa78-4a50-b6a7-f41ab57273a0.core-connector.paralus.iherbpreprod.net:443  "}
{"level":"info","ts":"2023-02-10T17:07:28.308Z","caller":"tunnel/client.go:416","msg":"Relay Agent.Client.paralus-core-relay-agent::dial failed network: tcp  addr: 6f881749-fa78-4a50-b6a7-f41ab57273a0.core-connector.paralus.iherbpreprod.net:443  err: x509: certificate has expired or is not yet valid: current time 2023-02-10T17:07:28Z is after 2020-09-18T12:00:00Z  "}

Steps to reproduce the bug

Not sure, given it's a TLS error that is sent by paralus

Are you using the latest version of the project?

You can check your version by running helm ls|grep '^<deployment-name>' or using pctl, pctl version, and provide the output.

VERSION: 0.1.0
BUILD: 0.1.0
BUILD-TIME: 1656329967
ARCH: darwin/amd64

What is your environment setup? Please tell us your cloud provider, operating system, and include the output of kubectl version --output=yaml and helm version. Any other information that you have, eg. logs and custom values, is highly appreciated!

Kubectl

clientVersion:
  buildDate: "2022-08-23T17:44:59Z"
  compiler: gc
  gitCommit: a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2
  gitTreeState: clean
  gitVersion: v1.25.0
  goVersion: go1.19
  major: "1"
  minor: "25"
  platform: darwin/amd64
kustomizeVersion: v4.5.7
serverVersion:
  buildDate: "2022-11-29T18:41:42Z"
  compiler: gc
  gitCommit: 52e500d139bdef42fbc4540c357f0565c7867a81
  gitTreeState: clean
  gitVersion: v1.22.16-eks-ffeb93d
  goVersion: go1.16.15
  major: "1"
  minor: 22+
  platform: linux/amd64

Helm

version.BuildInfo{Version:"v3.10.3", GitCommit:"835b7334cfe2e5e27870ab3ed4135f136eecc704", GitTreeState:"clean", GoVersion:"go1.19.4"}

(optional) If you have ideas on why the bug happens or how it can be solved, please provide it here

Catch certain client errors and report them as error and not info

• [X ] I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
• [ X] I'm using the latest version of the project.