paralus / relay Goto Github PK
View Code? Open in Web Editor NEWRelay server and agent for kubernetes cluster interactions
Home Page: https://www.paralus.io/
License: Apache License 2.0
Relay server and agent for kubernetes cluster interactions
Home Page: https://www.paralus.io/
License: Apache License 2.0
After installing a relay client into a cluster, I got a TLS error. I had expected the error to be reported in the log as an "error" level. But instead it was "info"
Sample error:
{"level":"info","ts":"2023-02-10T17:07:24.326Z","caller":"tunnel/client.go:460","msg":"Relay Agent.Client.paralus-core-relay-agent::action backoff sleep: 26.373942655s address: 6f881749-fa78-4a50-b6a7-f41ab57273a0.core-connector.paralus.iherbpreprod.net:443 "}
{"level":"info","ts":"2023-02-10T17:07:28.308Z","caller":"tunnel/client.go:416","msg":"Relay Agent.Client.paralus-core-relay-agent::dial failed network: tcp addr: 6f881749-fa78-4a50-b6a7-f41ab57273a0.core-connector.paralus.iherbpreprod.net:443 err: x509: certificate has expired or is not yet valid: current time 2023-02-10T17:07:28Z is after 2020-09-18T12:00:00Z "}
Not sure, given it's a TLS error that is sent by paralus
You can check your version by running helm ls|grep '^<deployment-name>'
or using pctl, pctl version
, and provide the output.
VERSION: 0.1.0
BUILD: 0.1.0
BUILD-TIME: 1656329967
ARCH: darwin/amd64
kubectl version --output=yaml
and helm version
. Any other information that you have, eg. logs and custom values, is highly appreciated!Kubectl
clientVersion:
buildDate: "2022-08-23T17:44:59Z"
compiler: gc
gitCommit: a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2
gitTreeState: clean
gitVersion: v1.25.0
goVersion: go1.19
major: "1"
minor: "25"
platform: darwin/amd64
kustomizeVersion: v4.5.7
serverVersion:
buildDate: "2022-11-29T18:41:42Z"
compiler: gc
gitCommit: 52e500d139bdef42fbc4540c357f0565c7867a81
gitTreeState: clean
gitVersion: v1.22.16-eks-ffeb93d
goVersion: go1.16.15
major: "1"
minor: 22+
platform: linux/amd64
Helm
version.BuildInfo{Version:"v3.10.3", GitCommit:"835b7334cfe2e5e27870ab3ed4135f136eecc704", GitTreeState:"clean", GoVersion:"go1.19.4"}
Catch certain client errors and report them as error and not info
The relay agent golang library contains the following vulnerabilities:
github.com/prometheus/client_golang fixed in version 1.12.2
https://bugzilla.redhat.com/show_bug.cgi?id=2067400
golang.org/x/text fixed in version 0.3.8
https://www.cvedetails.com/cve/CVE-2021-38561/
These libraries need to be updated.
This is a cross post from this issue
After applying the -boostrap.yaml for the relay-agent onto clusters that I want to import, the agent is not able to connect to Paralus to register clusters. I did some debugging and found that it was not Okta, but the relay application with this problem.
The certificate generated for SSL was created following the Deploy ClusterIssuer and Certificate Objects with cert-manager.
Expect
SUCCESSFUL
when viewing the clusters in a projectActual
Method not allowed
[POST /v2/sentry/bootstrap/{templateToken}/register][501] Bootstrap_RegisterBootstrapAgent default &{Code:12 Details:[] Message:Method Not Allowed}
cluster registration stuck pending and Cluster Connection status reads FAILURE
kubectl version --output=yaml
and helm version
. Any other information that you have, eg. logs and custom values, is highly appreciated!When attempting to connect to certain servers, the kubectl client returns this error:
Error from server (InternalError): an error on the server ("read unix @->/tmp/relay-unix-*.core-connector.paralus.iherbpreprod.net: read: connection reset by peer") has prevented the request from succeeding (get pods)
or
Error from server (unable to proxy kubectl service): ERROR: Unauthenticated access not allowed. Please log in to the portal via browser, or set up API key, for access via the secure kubectl proxy. Error:context deadline exceeded
And on the server side I see this issue from the relay-server
{"level":"error","ts":"2023-02-23T02:26:21.061Z","caller":"tunnel/server.go:1677","msg":"Relay Server.Server::failed to compete TLS handshake error: remote error: tls: bad certificate ","stacktrace":"github.com/paralus/relay/pkg/tunnel.(*Server).processDialinTLSState\n\t/build/pkg/tunnel/server.go:1677\ngithub.com/paralus/relay/pkg/tunnel.(*Server).handleDialinConnection\n\t/build/pkg/tunnel/server.go:1794"}
{"level":"error","ts":"2023-02-23T02:26:21.061Z","caller":"tunnel/server.go:1796","msg":"Relay Server.Server.handleDialinConnection::failed to process TLS state error: ailed to compete TLS handshake ","stacktrace":"github.com/paralus/relay/pkg/tunnel.(*Server).handleDialinConnection\n\t/build/pkg/tunnel/server.go:1796"}
Restarting the relay agent of the cluster seems to fix it.
Uncertain what causes it.
You can check your version by running helm ls|grep '^<deployment-name>'
or using pctl, pctl version
, and provide the output.
VERSION: 0.1.0
BUILD: 0.1.0
BUILD-TIME: 1656329967
ARCH: darwin/amd64
kubectl version --output=yaml
and helm version
. Any other information that you have, eg. logs and custom values, is highly appreciated!Kubectl
clientVersion:
buildDate: "2022-08-23T17:44:59Z"
compiler: gc
gitCommit: a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2
gitTreeState: clean
gitVersion: v1.25.0
goVersion: go1.19
major: "1"
minor: "25"
platform: darwin/amd64
kustomizeVersion: v4.5.7
serverVersion:
buildDate: "2022-11-29T18:41:42Z"
compiler: gc
gitCommit: 52e500d139bdef42fbc4540c357f0565c7867a81
gitTreeState: clean
gitVersion: v1.22.16-eks-ffeb93d
goVersion: go1.16.15
major: "1"
minor: 22+
platform: linux/amd64
Helm
version.BuildInfo{Version:"v3.10.3", GitCommit:"835b7334cfe2e5e27870ab3ed4135f136eecc704", GitTreeState:"clean", GoVersion:"go1.19.4"}
The only workaround I've found is to delete the relay agent pod and have it get recreated. But then it happens almost immediately again.
If you run kubectl get secret
for a read-only user then it works, whereas kubectl get secrets
doesn't.
kubectl get pods -o json | jq '.items[].spec.containers[].env[]?.valueFrom.secretKeyRef.name' | grep -v null | uniq
command for authorized namespacekubectl get secret <secret_name>
and it worksUsing version 2.0
kubectl version --output=yaml
and helm version
. Any other information that you have, eg. logs and custom values, is highly appreciated!Kubernetes 1.24
The guard against secrets should be a wildcard so that it can protect against secret
and sec
As of now, if the user selects a namespace when specifying a project-user-role mapping which does not exist, we create the namespace on the first use. This is done as we rely on the namespace being available to generate SA and other necessary resources, but it would be better to not create the namespace and just return with error/empty if they try to access it.
Make the behavior of missing namespaces more to what people might expect out of the box.
-
You can check your version by running helm ls|grep '^<deployment-name>'
or using pctl, pctl version
, and provide the output.
yes, 0.1.3
I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
I'm using the latest version of the project.
Hi i have setup of paralus aka cloudflare ( plus ssl termination on it ) and then ingress take incoming connection to 80 ports, so i do not generate certificate in this way. Now i try connect external cluster and get error
err: x509: certificate signed by unknown authority
on core-connector. If it's possible skip it or set trusted certs or whatever ?
Thx
Unable to connect to cluster via Lens after a while since initial connection. Relay agent needs to be rebooted to allow connection to target cluster again.
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
paralus paralus 5 2023-08-16 17:14:50.752987 -0400 EDT deployed ztka-0.2.5 v0.2.4
You can check your version by running helm ls|grep '^<deployment-name>'
or using pctl, pctl version
, and provide the output.
kubectl version --output=yaml
and helm version
. Any other information that you have, eg. logs and custom values, is highly appreciated!helm version
version.BuildInfo{Version:"v3.11.0", GitCommit:"472c5736ab01133de504a826bd9ee12cbe4e7904", GitTreeState:"clean", GoVersion:"go1.18.10"}
kubectl version --output=yaml
clientVersion:
buildDate: "2022-01-25T21:25:17Z"
compiler: gc
gitCommit: 816c97ab8cff8a1c72eccca1026f7820e93e0d25
gitTreeState: clean
gitVersion: v1.23.3
goVersion: go1.17.6
major: "1"
minor: "23"
platform: darwin/amd64
serverVersion:
buildDate: "2023-07-28T16:53:07Z"
compiler: gc
gitCommit: 222fd0c4adcf243ebeaee986cc0d41a39e570d8f
gitTreeState: clean
gitVersion: v1.23.17-eks-2d98532
goVersion: go1.19.6
major: "1"
minor: 23+
platform: linux/amd64
TODO: Add description
Got following log from relay-server container:
p.TypeEC PRIVATE KEY{"level":"error","ts":"2022-06-08T06:46:49.001Z","caller":"relay/relay.go:370","msg":"Relay Server::failed to register peering relay error: context deadline exceeded \n","stacktrace":"github.com/RafayLabs/relay/pkg/relay.registerRelayPeerService\n\t/build/pkg/relay/relay.go:370\ngithub.com/RafayLabs/relay/pkg/relay.relayServerBootStrap\n\t/build/pkg/relay/relay.go:642\ngithub.com/RafayLabs/relay/pkg/relay.RunRelayServer\n\t/build/pkg/relay/relay.go:786"}
{"level":"error","ts":"2022-06-08T06:46:49.001Z","caller":"relay/relay.go:644","msg":"Relay Server::failed to register relay with peer-service-bootstrap service, will retry error: context deadline exceeded \n","stacktrace":"github.com/RafayLabs/relay/pkg/relay.relayServerBootStrap\n\t/build/pkg/relay/relay.go:644\ngithub.com/RafayLabs/relay/pkg/relay.RunRelayServer\n\t/build/pkg/relay/relay.go:786"}
relay-tail container is restarting repeatedly with following error:
p.TypeEC PRIVATE KEY{"level":"info","ts":"2022-06-08T06:51:18.043Z","caller":"tail/register.go:82","msg":"unable to register","error":"context deadline exceeded"}
{"level":"panic","ts":"2022-06-08T06:51:18.050Z","caller":"tail/run.go:112","msg":"unable to create sentry authorization pool","error":"context deadline exceeded","stacktrace":"github.com/RafayLabs/relay/pkg/tail.runTail\n\t/build/pkg/tail/run.go:112\ngithub.com/RafayLabs/relay/pkg/tail.RunRelayTail\n\t/build/pkg/tail/run.go:212"}
panic: unable to create sentry authorization pool
goroutine 38 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0001aa000, {0xc000379500, 0x1, 0x2})
/build/vendor/go.uber.org/zap/zapcore/entry.go:232 +0x446
go.uber.org/zap.(*SugaredLogger).log(0xc00011a660, 0x4, {0x1b75ad4, 0xc000593f90}, {0x0, 0xc000593eb0, 0x2}, {0xc000161e88, 0x2, 0x2})
/build/vendor/go.uber.org/zap/sugar.go:227 +0xee
go.uber.org/zap.(*SugaredLogger).Panicw(...)
/build/vendor/go.uber.org/zap/sugar.go:204
github.com/RafayLabs/relay/pkg/tail.runTail({0x1dd45f8, 0xc000718f00})
/build/pkg/tail/run.go:112 +0xe5
github.com/RafayLabs/relay/pkg/tail.RunRelayTail({0x1dd45f8, 0xc000718f00})
/build/pkg/tail/run.go:212 +0x34
created by main.main
/build/main.go:122 +0x5d9
Currently GH actions uses a user's account to build and publish images, change this to use a build specific account
Go test failed for go test ./...
:
starting client
Waiting for workersstarting clientCompleted--- FAIL: TestRelayClient (0.00s)
client_test.go:25: failed to connect dial unix /tmp/relay-unix-kubectldialin.relay.rafay.dev: connect: no such file or directory
FAIL
FAIL github.com/RafaySystems/relay/pkg/tunnel 0.028s
? github.com/RafaySystems/relay/pkg/utils [no test files]
FAIL
If you attempt to upgrade a paralus server from 0.2.0 with relay version 0.1.2 to 0.2.5 paralus with relay 0.1.5 (starting with 0.1.4), you'll get the error:
"level":"info","ts":"2023-08-16T23:25:59.118Z","caller":"agent/agent.go:171","msg":"Relay Agent::relay agent namespace: paralus-system fingerprint: 4d63c263-0e39-4451-aa97-6882e339fc30 "}
{"level":"info","ts":"2023-08-16T23:25:59.119Z","caller":"agent/agent.go:394","msg":"Relay Agent::config: &{TemplateToken:cifi3glc3m5b406jcbc0 TemplateName: Scheme:https Mode: Addr:console.paralus.iherbtest.net:443 ClientID:cisqaj3ppcveb4n2bcrg ClientIP:10.7.228.141 Name:relay-agent-557f56bb69-x2rcc PrivateKey:[] CSR:[] Certificate:[] CACertificate:[] ServerHost: ServerPort:0 Fingerprint:4d63c263-0e39-4451-aa97-6882e339fc30} "}
[POST /v2/sentry/bootstrap/{templateToken}/register][500] bootstrapRegisterBootstrapAgentInternalServerError map[code:2 message:fingerprint mismatch for token cisqaj3ppcveb4n2bcrg]
{"level":"error","ts":"2023-08-16T23:25:59.159Z","caller":"agent/agent.go:397","msg":"Relay Agent::failed to register relay agent error: [POST /v2/sentry/bootstrap/{templateToken}/register][500] bootstrapRegisterBootstrapAgentInternalServerError map[code:2 message:fingerprint mismatch for token cisqaj3ppcveb4n2bcrg] ","stacktrace":"github.com/paralus/relay/pkg/agent.registerRelayAgent\n\t/build/pkg/agent/agent.go:397\ngithub.com/paralus/relay/pkg/agent.handleRelayNetworks\n\t/build/pkg/agent/agent.go:606"}
The only way to fix this is to go to the database and update the fingerprint value in the sentry_bootstrap_agent
table for the specific token row with the uid from the paralus-system namespace for that client (also found within the log)
This appears to be happening because an existing cluster does not automatically update the fingerprint if it's already bootstrapped into the database. You have to do a new bootstrapping, or update the value manually
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.