kubesphere / tower Goto Github PK

tower有自动更新机制嘛，通过平台ks更新、或者自动更新

tower:
E1118 17:18:08.550844 1 proxy.go:152] Failed to upgrade connectionwrite tcp 10.233.90.20:8080->10.233.97.0:61511: write: connection reset by peer

agent:
agent.go:168] Connection error: websocket: bad handshake (Attempt: 18)

How many member clusters can Kubesphere federation support through tower agent?

run agen

./bin/agent --kubeconfig ~/.kube/config --kubesphere-service http://0.0.0.0:8000 --token 13530761f40fc3161ed59316a6b3ed7b6ba44c5cca613f03d37db3fe9d903e87 --name alpha --v 4 --proxy-server http://192.168.1.31:30880

http://0.0.0.0:8000是本地的一个python文件服务 (python3 -m http.server)

集群pod内访问代理的80端口, proxy panic,

日志:

http: panic serving 10.233.124.47:41500: runtime error: invalid memory address or nil pointer dereference
goroutine 588 [running]:
net/http.(*conn).serve.func1(0xc000238280)
        /usr/local/go/src/net/http/server.go:1795 +0x139
panic(0x13594c0, 0x208c260)
        /usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/apimachinery/pkg/util/proxy.normalizeLocation(0xc000017b88, 0x20c5d60)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/proxy/upgradeaware.go:168 +0x48
k8s.io/apimachinery/pkg/util/proxy.NewUpgradeAwareHandler(...)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/proxy/upgradeaware.go:178
kubesphere.io/tower/pkg/proxy.(*Server).ServeHTTP(0xc000337b00, 0x16d7f80, 0xc00012a700, 0xc000260c00)
        /workspace/pkg/proxy/proxy_server.go:184 +0x28b
net/http.serverHandler.ServeHTTP(0xc0002aa380, 0x16d7f80, 0xc00012a700, 0xc000260c00)
        /usr/local/go/src/net/http/server.go:2831 +0xa4
net/http.(*conn).serve(0xc000238280, 0x16dbc40, 0xc0000a6dc0)
        /usr/local/go/src/net/http/server.go:1919 +0x875
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2957 +0x384

2023/05/16 02:43:18 http: panic serving 172.64.182.254:59840: runtime error: invalid memory address or nil pointer dereference
goroutine 58308 [running]:
net/http.(*conn).serve.func1(0xc000186e60)
/usr/local/go/src/net/http/server.go:1795 +0x139
panic(0x13594c0, 0x208c260)
/usr/local/go/src/runtime/panic.go:679 +0x1b2
bufio.(*Writer).Flush(0xc00290c3c0, 0x16a6d20, 0xc00290c3c0)
/usr/local/go/src/bufio/bufio.go:593 +0x5c
net/http.(*Request).write(0xc001908e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:691 +0x7bb
net/http.(*Request).Write(...)
/usr/local/go/src/net/http/request.go:524
k8s.io/apimachinery/pkg/util/proxy.dial(0xc001908e00, 0x16a84c0, 0xc00027a280, 0xc0003af160, 0xc0003aefe0, 0xc000fd7a60, 0x14efb96)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/proxy/upgradeaware.go:428 +0x165
k8s.io/apimachinery/pkg/util/proxy.(*UpgradeAwareHandler).DialForUpgrade(0xc00335a5a0, 0xc001908e00, 0x890575e8, 0xffffffffffffffff, 0x0, 0x3e0000c002b22fa8)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/proxy/upgradeaware.go:399 +0x11b
k8s.io/apimachinery/pkg/util/proxy.(*UpgradeAwareHandler).tryUpgrade(0xc00335a5a0, 0x16d7f80, 0xc002af42a0, 0xc001908c00, 0x5f9f00)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/proxy/upgradeaware.go:280 +0x15ea
k8s.io/apimachinery/pkg/util/proxy.(*UpgradeAwareHandler).ServeHTTP(0xc00335a5a0, 0x16d7f80, 0xc002af42a0, 0xc001908c00)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/proxy/upgradeaware.go:189 +0x6a
kubesphere.io/tower/pkg/proxy.(*Server).ServeHTTP(0xc0006da120, 0x16d7f80, 0xc002af42a0, 0xc001908c00)
/workspace/pkg/proxy/proxy_server.go:186 +0x355
net/http.serverHandler.ServeHTTP(0xc0000288c0, 0x16d7f80, 0xc002af42a0, 0xc001908c00)
/usr/local/go/src/net/http/server.go:2831 +0xa4
net/http.(*conn).serve(0xc000186e60, 0x16dbc40, 0xc001c9af40)
/usr/local/go/src/net/http/server.go:1919 +0x875
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2957 +0x384
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x63bc15]

goroutine 82318 [running]:
crypto/tls.(*Conn).write(0xc001523180, 0xc0009ba300, 0xf3, 0x100, 0xc000460300, 0xee, 0x180)
/usr/local/go/src/crypto/tls/conn.go:915 +0x135
crypto/tls.(*Conn).writeRecordLocked(0xc001523180, 0x16, 0xc000460300, 0xee, 0x180, 0x0, 0x0, 0xc000460300)
/usr/local/go/src/crypto/tls/conn.go:964 +0x29f
crypto/tls.(*Conn).writeRecord(0xc001523180, 0xc000460316, 0xc000460300, 0xee, 0x180, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:986 +0xe2
crypto/tls.(*Conn).clientHandshake(0xc001523180, 0x0, 0x0)
/usr/local/go/src/crypto/tls/handshake_client.go:167 +0x179
crypto/tls.(*Conn).Handshake(0xc001523180, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1362 +0x12b
net/http.(*persistConn).addTLS.func2(0x0, 0xc001523180, 0x0, 0xc0005dcf00)
/usr/local/go/src/net/http/transport.go:1414 +0x42
created by net/http.(*persistConn).addTLS
/usr/local/go/src/net/http/transport.go:1410 +0x1aa

主集群节点上的tower pod报错如下，网络均没有任何问题，但是经常出现，等待2分钟左右又恢复了，
E0115 01:46:14.038945 1 upgradeaware.go:367] Error proxying data from client to backend: read tcp 10.233.78.199:16614->10.233.78.138:48506: read: connection reset by peer

E0115 01:46:14.040117 1 upgradeaware.go:367] Error proxying data from client to backend: read tcp 10.233.78.199:16614->10.233.78.138:48472: read: connection reset by peer

I found that the project uses a custom ssh-based protocol, would it be interesting to replace it with one that is compatible with OpenSSH?

https://github.com/wzshiming/sshproxy
https://github.com/wzshiming/sshd

Proxy(at least two replications):
Add leader election and kubernetes readiness probe. They both make sure that there are only one proxy running and one endpoint behind the svc. Once one proxy replication is down, another one can start quickly.
Agent(at least two replications):
Add leader election to make sure that there is only one agent running, once one agent replication is down, another one can become the leader quickly.

how can i use postman to test tower-server api ？

➜ tower git:(master) ✗ kubectl -n kubesphere-system get cluster alpha -o jsonpath='{.spec.connection.kubeconfig}' | base64 -d > config_alpha
➜ tower git:(master) ✗ kubectl --kubeconfig config_alpha get node
Unable to connect to the server: dial tcp 10.100.231.111:6443: i/o timeout
➜ tower git:(master) ✗ kubectl --kubeconfig config_alpha get node
Unable to connect to the server: dial tcp 10.100.231.111:6443: i/o timeout
➜ tower git:(master) ✗ kubectl --kubeconfig config_alpha get node
Unable to connect to the server: dial tcp 10.100.231.111:6443: i/o timeout
➜ tower git:(master) ✗ kubectl --kubeconfig config_alpha get node
Unable to connect to the server: dial tcp 10.100.231.111:6443: i/o timeout
➜ tower git:(master) ✗ kubectl --kubeconfig config_alpha get node
Unable to connect to the server: dial tcp 10.100.231.111:6443: i/o timeout
➜ tower git:(master) ✗ kubectl --kubeconfig config_alpha get node
Unable to connect to the server: dial tcp 10.100.231.111:6443: i/o timeout

apiVersion: v1
clusters:

• cluster:
  certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKYzNCb1pYSmxNQjRYRFRJd01ETXlOVEV3TVRjeE5sb1hEVE13TURNeU16RXdNVGN4Tmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYTndhR1Z5WlRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTTV0CjVGVGtCS0tBaHhidUw0VjVsdE9MR0FmSlZoQ3ArRk9RUEtSalJqdHNsZStUa2hkbkJ4ODdWVk1LRFZUeVFVeFgKWlZxNC9TODlYb213TGhpd1VtVjlLTjlEditmc0ZsRDQ0QnoyUDdQdHVmUnYxMkJSSnhHamw1R3JaZktPeVo1cgo3cnhFV0xFcGtEc2hjSnlITDhoSmF1MUtSVHVSOG8zbVp4SVVYUTBuSEtSS3FDL3dwVkh2Y0xDR2tBajY3Uld4CkpFTmwxaXcvMGRvbXcrM0svemlaVW55dE90WjVPRkREOVlGV2RSV052dDZUOGNkRmkwUFZxVENsWjZSbFpYUVQKbFZmelFyckhNSmZNK1EvMmZhbGRHYmpaN2pRV3hpTVhaQjNuUmdMVkw5YVBqVUpNNnlhRDVhZGVZNndGVE4yYgpXMzhKREFjcFNWWkcxbUFPQTlFQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFHUlFBUm9DU0xBVWhpbmhTR2cyUHgveVFldEEKNkVROU8rOUpqWTB3M1pLUjBxa1p4VW1JRUNRNDFMeHp1QXJ2SnR1UURTd3hWY3QwcG9ZRWFWTFZmSlF5Q2w1UApUVGp2QlBrdUJrSXc3NTExRnF0dXJJVnRMWWtSQlNvTGIxeWdCS2JxcGtuRWNMR2tRUTdmUEc1T3Fpb28wUE5nCnhicldRSUNHK0pmUjA3c0t1bU5Zd090Nml4Q2NHY2lQRHZjU282S0JVZ20zMUM4OWFiaG9kSk1pRHNXMzZsL2QKVWx5MHNxUXFLWHd0VE5CVHI2ZE1hU0RLbmRieWJRZytJN2ZRNXhxeEhJQTBEUEgzQVZHQjVudTFMYitFenlBYwowZW1YNVRUSDJJaGFIUjcvMm1EcjFrczI5RFNxTWxFaTRxcnZFeUorSC9BYWoyeFZ1QUlrTjk1ejdGST0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  server: https://10.100.231.111:6443
  name: alpha
  contexts:
• context:
  cluster: alpha
  user: kubernetes-admin
  name: kubernetes-admin@alpha
  current-context: kubernetes-admin@alpha
  kind: Config
  preferences: {}
  users:
• name: kubernetes-admin
  user:
  client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4akNDQWRxZ0F3SUJBZ0lJQTdpSmU1N0F5bmt3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWE53YUdWeVpUQWVGdzB5TURBek1qVXhNREUzTVRaYUZ3MHpNakExTWpnd056QTBNalphTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTFRR2JmMWE2M1l1Z3dTK3UKUDdocHBqWVl3bTQ5UHREWllhMzV1K3RrQVp0MDRKVHlyd2VvRnFWaXNWRmRKVnFvMDdDUnEveXJkdXdSNUFQNgphejdQbjBhK1kxSnVheHhZUVc0V0haSCt2N2h5MXFJY2pyUENUMjdnbTdPSW5LZnpOcFRiUzNqZVIzU3g4cW9qCnVqbzNpOExvb0x5K0dHdDVVUU1QdGVhQWxiMXk2WUR3dXFiWG1EK3JqQ2FGdjN6ZDBOREJsVzIzQjkvYnFPcG0KYStQMEFxdDRsN1J4T2dwcVZXUGFBSUNvVkJja0NtdlZPRnE5Z2swRkdFN2FTNHZDbkl0Tzh0bUZyRmtKQ0RtYwpLUks5dUxQMGhwWHRVWTJTQzg0aWQwQmk1ZXdoYUlpTHE5d1NiM01mMWtqVlJYd0JBdFhBQll2UU40S2tPbG5kCk96cnRHd0lEQVFBQm95Y3dKVEFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLL1VTc3JUWjJtZzYzNldubGVYNTF1NGE0Y1ZPQ1lsSGh1SwpiemlSQmwyNzRrZ3JjSEhJcDhiTzVEajJYQmpIYnQ5S2YzT3E5TVlyZk9TWVFpa25jQzRVS2pmUDNTVSt6WlRMClpMZ2JwazRCUThWMXFSL2pnS21rY0tHbEVKcCtrbWRpVWdnMHI1NUlVK1pIK2VOTi9hUXdZbm1HTjB3a1YrU2kKVHRMMHorRTRxamc3NkhBUy9xMlo0cHpsbVJ2WVhWVGxGL3hkMTRDSCtKQU5yZzRnbWg4aVNXYTFzRlcybWlFOApzbzY3TzBhRjdoeVA5OWhJeDlXYjFMRURIYXFweldHVkYraHdvK3AvY3IxZkE2NlZ4SFJ4TmJUR3NUdzZUWFNpCllLRWxVR0szK0M5cTFxZVV5UzJ4RnA5K0MzMzZuNEtIbHlwamh2N0IxbFN4VFpIWGg2VT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBMVFHYmYxYTYzWXVnd1MrdVA3aHBwallZd200OVB0RFpZYTM1dSt0a0FadDA0SlR5CnJ3ZW9GcVZpc1ZGZEpWcW8wN0NScS95cmR1d1I1QVA2YXo3UG4wYStZMUp1YXh4WVFXNFdIWkgrdjdoeTFxSWMKanJQQ1QyN2dtN09Jbktmek5wVGJTM2plUjNTeDhxb2p1am8zaThMb29MeStHR3Q1VVFNUHRlYUFsYjF5NllEdwp1cWJYbUQrcmpDYUZ2M3pkME5EQmxXMjNCOS9icU9wbWErUDBBcXQ0bDdSeE9ncHFWV1BhQUlDb1ZCY2tDbXZWCk9GcTlnazBGR0U3YVM0dkNuSXRPOHRtRnJGa0pDRG1jS1JLOXVMUDBocFh0VVkyU0M4NGlkMEJpNWV3aGFJaUwKcTl3U2IzTWYxa2pWUlh3QkF0WEFCWXZRTjRLa09sbmRPenJ0R3dJREFRQUJBb0lCQVFDU1NXMVNrZXo2RkZmTgo5WUlWVHdCMzN6VTM2dC96RVJvS0pLS2J0YmpVQmNES01HRlBaZEhCNDhtWS84a1p5MTZoSytXUG1aTGt0SkpBCjU1NkhJdmNTYTFKOEFmUEZhZzBPdXAxRHA2MnVXKzI2akZGU2pCQUozNXN3TTZmQjl2Sm83T2lDdDdyb2xqTlQKOWhkZFlYZ2Vhemk2azRWMlhmRk9ldXpDUDFEV0JLbEZ6NERJZ3E3NjBKV1ZNblJQQlE2SnJNSzFLMXV3cnU3YQpZUCtEQmszUS9TdjErV2FjMkNPM21UOGxVWTlaSU9pcG9MODEzUmhRRzNnMU9ReWFoVk9hOHNSQVV5SXZPSTVXCnRrc1hjVWxqdTBQalMrSzZZMzI0cFVlNGtCYlZCQnl5ZEhCNTNkMzNZc0RQSWFRam42UTlpM0JCSCt6WHB4Vy8KZ2REekJCd3hBb0dCQU5aUzBGekZROFJoM1h6eEZaWU5SaytTeCtQejB3UkxmNUtJUTVmS09nV09weFoycnJ1eQpyRE9uMVYvclpyVko4WlRSc094S0ZDUGNFaDJidm40MWZzRHFTMlFTdHArZ2ZFOHUzZTErRzg3WTJEbVBlTzE3CmE1OUo5TU92RHllaGFtUlFYL1pkZ1JybE5WNkZxdEVObm4vVUpUdzNQbDJtMGdjVzFiRmU5ZndUQW9HQkFQNXQKT0wxR04vdGE3Skw4QWdyN3VETDJESHNobFlHTnVqSFB5aWc3QXlhQ2o0NmorQ2JmZGhsVXBWVHJ6MW5JOGxIWApqdWtOYTVCYXVmZUJTZm1LNS9OQTV1MEFra1gwOVloakNFRm01YWtOTDV3WnNMNkhNOU1CNWh2VkZtZmtvbVk3CnhrSnNJK0JZQ3ZDTlpsSUt6dUhXeHBpS1N3WlVabkFGSnVEVUhOdlpBb0dBQnQvRkd4U2dwYzNWd3NvZ3ZSMVQKNGYxOU44eDhPaWtKd0hOcVo1WHU0bnErMVRPZFN4dmg5ZzR1QTlPVENjR0RaQmlOcWlaK0d1QldIR2tBWlIvbQpIdDNrNlNMSWhMb3JBb2NvU3hPK1pKWUQ2V1MycGpkTkxtQzYzd1BnWW84RTBJN2JaUlYrckpRcGNBeXRlK01KCjhHT25oaFRuaFMxeWt4T2lhdnYydTgwQ2dZQWsxWXN4VzhrZkNyclZUTWErOEZKcWpsU2dQdXgySlI2ZnQ0VjgKN1k0M05RK3podW1XTnlQUjV2OUluQnNZMUk3QWFvUzJHTXRKeXRqQmpHWTVTbkxxTktzNkJLc29sWjQyY0oveApXQVJIeFREdldJak9ES1RrRjVKSTFVQzVLZENaSEc1SGo0QmlLbHpyRm8ySnN5YmdhMGUxTGRCT3lrbEtUblNpCmVCTDFnUUtCZ0h5WHM3amJlQ3ZlamhMMmRtcWF1eC8rcDZMS2VaWnIyckYyWHFvck14WFJSb3FhcXV6UjRZRWIKb2dzaDlsYW9VaFN5cGViZDlpRmdYcWVqdkZLYVA5NnJVTlRTdDlwR2F5Um1IWEFUQXU4UEpjS2hPOUlqQXBrVwpXTThKZWxsa3FRazZ6VmJVVk90d3JYN2xvMVpqSk1XaitVbXRXNy9SdktLWGRvWjFOaDFRCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==

src kubectl get all -n kubesphere-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mc-alpha ClusterIP 10.100.231.111 6443/TCP,80/TCP 47m
➜ src

kubectl get cluster alpha -oyaml -n kubesphere-system
apiVersion: cluster.kubesphere.io/v1alpha1
kind: Cluster
metadata:
creationTimestamp: "2022-05-31T07:00:04Z"
generation: 11
managedFields:

• apiVersion: cluster.kubesphere.io/v1alpha1
  fieldsType: FieldsV1
  fieldsV1:
  f:spec:
  .: {}
  f:connection:
  .: {}
  f:type: {}
  f:joinFederation: {}
  manager: kubectl
  operation: Update
  time: "2022-05-31T07:00:04Z"
• apiVersion: cluster.kubesphere.io/v1alpha1
  fieldsType: FieldsV1
  fieldsV1:
  f:spec:
  f:connection:
  f:kubeconfig: {}
  f:kubernetesAPIEndpoint: {}
  f:kubernetesAPIServerPort: {}
  f:kubesphereAPIEndpoint: {}
  f:kubesphereAPIServerPort: {}
  f:token: {}
  f:status:
  .: {}
  f:conditions: {}
  manager: proxy
  operation: Update
  time: "2022-05-31T07:04:26Z"
  name: alpha
  resourceVersion: "105883911"
  selfLink: /apis/cluster.kubesphere.io/v1alpha1/clusters/alpha
  uid: 48e216ae-9d96-46a1-b344-21873c2820d7
  spec:
  connection:
  kubeconfig: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VONVJFTkRRV0pEWjBGM1NVSkJaMGxDUVVSQlRrSm5hM0ZvYTJsSE9YY3dRa0ZSYzBaQlJFRldUVkpOZDBWUldVUldVVkZFUlhkd2NtUlhTbXdLWXpOQ2IxcFlTbXhOUWpSWVJGUkpkMDFFVFhsT1ZFVjNUVlJqZUU1c2IxaEVWRTEzVFVSTmVVMTZSWGROVkdONFRteHZkMFpVUlZSTlFrVkhRVEZWUlFwQmVFMUxZVE5XYVZwWVRuZGhSMVo1V2xSRFEwRlRTWGRFVVZsS1MyOWFTV2gyWTA1QlVVVkNRbEZCUkdkblJWQkJSRU5EUVZGdlEyZG5SVUpCVFRWMENqVkdWR3RDUzB0QmFIaGlkVXcwVmpWc2RFOU1SMEZtU2xab1EzQXJSazlSVUV0U2FsSnFkSE5zWlN0VWEyaGtia0o0T0RkV1ZrMUxSRlpVZVZGVmVGZ0tXbFp4TkM5VE9EbFliMjEzVEdocGQxVnRWamxMVGpsRWRpdG1jMFpzUkRRMFFub3lVRGRRZEhWbVVuWXhNa0pTU25oSGFtdzFSM0phWmt0UGVWbzFjZ28zY25oRlYweEZjR3RFYzJoalNubElURGhvU21GMU1VdFNWSFZTT0c4emJWcDRTVlZZVVRCdVNFdFNTM0ZETDNkd1ZraDJZMHhEUjJ0QmFqWTNVbGQ0Q2twRlRtd3hhWGN2TUdSdmJYY3JNMHN2ZW1sYVZXNTVkRTkwV2pWUFJrUkVPVmxHVjJSU1YwNTJkRFpVT0dOa1Jta3dVRlp4VkVOc1dqWlNiRnBZVVZRS2JGWm1lbEZ5Y2toTlNtWk5LMUV2TW1aaGJHUkhZbXBhTjJwUlYzaHBUVmhhUWpOdVVtZE1Wa3c1WVZCcVZVcE5ObmxoUkRWaFpHVlpObmRHVkU0eVlncFhNemhLUkVGamNGTldXa2N4YlVGUFFUbEZRMEYzUlVGQllVMXFUVU5GZDBSbldVUldVakJRUVZGSUwwSkJVVVJCWjB0clRVRTRSMEV4VldSRmQwVkNDaTkzVVVaTlFVMUNRV1k0ZDBSUldVcExiMXBKYUhaalRrRlJSVXhDVVVGRVoyZEZRa0ZIVWxGQlVtOURVMHhCVldocGJtaFRSMmN5VUhndmVWRmxkRUVLTmtWUk9VOHJPVXBxV1RCM00xcExVakJ4YTFwNFZXMUpSVU5STkRGTWVIcDFRWEoyU25SMVVVUlRkM2hXWTNRd2NHOVpSV0ZXVEZabVNsRjVRMncxVUFwVVZHcDJRbEJyZFVKclNYYzNOVEV4Um5GMGRYSkpWblJNV1d0U1FsTnZUR0l4ZVdkQ1MySnhjR3R1UldOTVIydFJVVGRtVUVjMVQzRnBiMjh3VUU1bkNuaGljbGRSU1VOSEswcG1VakEzYzB0MWJVNVpkMDkwTm1sNFEyTkhZMmxRUkhaalUyODJTMEpWWjIwek1VTTRPV0ZpYUc5a1NrMXBSSE5YTXpac0wyUUtWV3g1TUhOeFVYRkxXSGQwVkU1Q1ZISTJaRTFoVTBSTGJtUmllV0pSWnl0Sk4yWlJOWGh4ZUVoSlFUQkVVRWd6UVZaSFFqVnVkVEZNWWl0RmVubEJZd293WlcxWU5WUlVTREpKYUdGSVVqY3ZNbTFFY2pGcmN6STVSRk54VFd4RmFUUnhjblpGZVVvclNDOUJZV295ZUZaMVFVbHJUamsxZWpkR1NUMEtMUzB0TFMxRlRrUWdRMFZTVkVsR1NVTkJWRVV0TFMwdExRbz0KICAgIHNlcnZlcjogaHR0cHM6Ly8xMC4xMDAuMjMxLjExMTo2NDQzCiAgbmFtZTogYWxwaGEKY29udGV4dHM6Ci0gY29udGV4dDoKICAgIGNsdXN0ZXI6IGFscGhhCiAgICB1c2VyOiBrdWJlcm5ldGVzLWFkbWluCiAgbmFtZToga3ViZXJuZXRlcy1hZG1pbkBhbHBoYQpjdXJyZW50LWNvbnRleHQ6IGt1YmVybmV0ZXMtYWRtaW5AYWxwaGEKa2luZDogQ29uZmlnCnByZWZlcmVuY2VzOiB7fQp1c2VyczoKLSBuYW1lOiBrdWJlcm5ldGVzLWFkbWluCiAgdXNlcjoKICAgIGNsaWVudC1jZXJ0aWZpY2F0ZS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VNNGFrTkRRV1J4WjBGM1NVSkJaMGxKUlZST1lreGlZMlZ4T0d0M1JGRlpTa3R2V2tsb2RtTk9RVkZGVEVKUlFYZEdWRVZVVFVKRlIwRXhWVVVLUVhoTlMyRXpWbWxhV0U1M1lVZFdlVnBVUVdWR2R6QjVUVVJCZWsxcVZYaE5SRVV6VFZSYVlVWjNNSHBOYWtFeFRXcG5kMDU2VFhsTmVrNWhUVVJSZUFwR2VrRldRbWRPVmtKQmIxUkViazQxWXpOU2JHSlVjSFJaV0U0d1dsaEtlazFTYTNkR2QxbEVWbEZSUkVWNFFuSmtWMHBzWTIwMWJHUkhWbnBNVjBackNtSlhiSFZOU1VsQ1NXcEJUa0puYTNGb2EybEhPWGN3UWtGUlJVWkJRVTlEUVZFNFFVMUpTVUpEWjB0RFFWRkZRVEZFYjJNNFVVeGxSak5VT1c0ck5WVUtZVEprYm5veldVbEpWakZoUTJsMU9WaGxVVVp6SzA1UGNrWTNWR0prUnpoaldHRnVRVUpIYWtsMFFqVmlkaklyUjBkWFdTdHhVbVZQYkRGRVQxTnVPUXBYUm1wMVdXTk9Ra0ZOTTI1VE9GVTJaa3RMWlhST1ZGTTRZVEZJWWs1bWJGWktOR2xHY1M5eWIxRXhVMVpuZFRCQ2FHaFdhMkp0UVV0dVdtdENhbUZUQ2pCd2VWRmxhWEY2UVZOdmNUZE1VMlpMVm1WRWNuWjVjR3RTYzFNNFRHWmxSMGhZWW0xcWMwTkRZa0pYWWpKWldtc3ZaVkp0VFdoRlNYVnJiVEo0YlRZS1kzWTBkV2xGUTNJMmRtZGxiemxQZFhWSFduTmxObU41YVVoaWRVcHVPVFowUkRoU01saHVNR1kxZG14TGJXRXhSVmhGTnpCTlFWaHRkSGd4ZDNjemFRcGxXV0ZIY1hSNmNXVkRPWEIyUkhGa2IwaDViMVphTVZocmNXSXdja2x5T0U0dmRrMTNVbXhYUVd0U1pUQlNZMWhvUVROa2JuQmhRVUZOWXpaSmNFMTRDbWM1Vm00eWQwbEVRVkZCUW05NVkzZEtWRUZQUW1kT1ZraFJPRUpCWmpoRlFrRk5RMEpoUVhkRmQxbEVWbEl3YkVKQmQzZERaMWxKUzNkWlFrSlJWVWdLUVhkSmQwUlJXVXBMYjFwSmFIWmpUa0ZSUlV4Q1VVRkVaMmRGUWtGQ2FWcGpjRFE1VjNoNVJHZHVTRlZPYjB0eWNFNDRMMWR6UXpVNWJTdFlNVzl3TXdwaFFVY3ljR3RZYzNrMVVsa3pjamh4Tlc5Q1MzWXZjV2t2TlROb2EyWXdZV2xXVVdNMVdsWkJVa2hEWVVGVVdsaE1jeTk2UXpjd0wxVkZkaTlCVkVOVENuSkRhMUo2U1cxWFFWWnNTVWxJZWxCc1RYTnJaRGMzSzBseGRqWXpPRzVwYTBWNmVYWm9hVUUwU0hRclYxQXZlSE5wVVdaR2FsZ3dPRVpoYVZNNVVuSUtkRmw1Y25OSlowNTBUSHA1WjFkcWVWb3hSRTVYUkV0S1lYTkdPRUVyYUdkR09YQTVlRzl4UWtKNFJHZDRXSEZ5Y1d4R1ZEUXJiblpXYVRoUGRFMUZiUXBrTlRJeGFVUkhVVkI2VGtkVVZtODJSRlIwUWtjdlJYQm5LeTh6Wnk5TVlYaFFjMXBtZVhWWVYxb3hVVEZaVUM5blkycHdSVUZrUVd0TlRVeHNiRVpEQ2xka056UnphbmROYVdOM1JHNVlRbUpYWkRaVWRrOXBNVWR2UzFnMmJEVTBXbEY1ZFRkMGVXRlJaVmxZU0ZwNWVFaG9SVDBLTFMwdExTMUZUa1FnUTBWU1ZFbEdTVU5CVkVVdExTMHRMUW89CiAgICBjbGllbnQta2V5LWRhdGE6IExTMHRMUzFDUlVkSlRpQlNVMEVnVUZKSlZrRlVSU0JMUlZrdExTMHRMUXBOU1VsRmIyZEpRa0ZCUzBOQlVVVkJNVVJ2WXpoUlRHVkdNMVE1YmlzMVZXRXlaRzU2TTFsSlNWWXhZVU5wZFRsWVpWRkdjeXRPVDNKR04xUmlaRWM0Q21OWVlXNUJRa2RxU1hSQ05XSjJNaXRIUjFkWkszRlNaVTlzTVVSUFUyNDVWMFpxZFZsalRrSkJUVE51VXpoVk5tWkxTMlYwVGxSVE9HRXhTR0pPWm13S1ZrbzBhVVp4TDNKdlVURlRWbWQxTUVKb2FGWnJZbTFCUzI1YWEwSnFZVk13Y0hsUlpXbHhla0ZUYjNFM1RGTm1TMVpsUkhKMmVYQnJVbk5UT0V4bVpRcEhTRmhpYldwelEwTmlRbGRpTWxsYWF5OWxVbTFOYUVWSmRXdHRNbmh0Tm1OMk5IVnBSVU55Tm5ablpXODVUM1YxUjFwelpUWmplV2xJWW5WS2JqazJDblJFT0ZJeVdHNHdaalYyYkV0dFlURkZXRVUzTUUxQldHMTBlREYzZHpOcFpWbGhSM0YwZW5GbFF6bHdka1J4Wkc5SWVXOVdXakZZYTNGaU1ISkpjamdLVGk5MlRYZFNiRmRCYTFKbE1GSmpXR2hCTTJSdWNHRkJRVTFqTmtsd1RYaG5PVlp1TW5kSlJFRlJRVUpCYjBsQ1FVWkNXVk01Y1cxa2ExRXhTVEIxYmdvME9HWm9iSFpwUVhVclZUbDJWR1UyYUZjclJsRjFPWFZpTjNSelJIVnJNV3BRZWpkeFdTdHhTR0pQZUdKNlpXOHdiek5VT0UweE5qWnlNVE42TVhCV0NqTnFZbFJRUkZKQmQyeDNNbXcwYTJzcmRGcDVhbXhxZUdSWllsQlVaMjlVTVdoMGNETnZablZSSzNSME5uRjZkWE14U0ZoYWNGaE1kalI2VkRBeGVGTUtlblJVZWs1MlRUVk9VbTFOZVVWR2FYaENjbGh4WkM5WFoyRXJVQ3QwZEhwQmVHazJTMnB4UjFCbVZVNWpjVFkxTld4dFIxQnBXVVowVkROb2NEWjJSZ3B5U1VOV2VXeEhUMnBTVm1oMFNpODNPV0pyTUVKdWR6QnpkVmhyY0V0S2FGTTVRM1JIY21jd1J6ZHVXVU5yUldJM1IwMU5halJwUXpsdWREVk9NelpFQ21Ka1R6QTNOSFZIVERObVVpdFVURXQwUlZoVmJsbEhTRU1yUkRnM2VUVTJOVlJ5YTFsek5FbDNXSGhTWkdab05uSkRkREJtWlZkVmNFUTJLekphVVc4S0wwdG1WRTVWYTBObldVVkJORGRPVWpoYU9YcHhSMDFLUjBoT1MybHFVbkpDY3pFelVuVlpkVFF2VVVSRlVVcE9VVlJxVnpKdlNqQkNkSEo0Y1dGQll3cHVUMVE0VGtsYWMxTTBURFY2VHpGeGVpOTROMk54YURkeFFsYzVabGxXZEZwblNYQm9ZbVU0ZVdaekwydG1VRFowVkZsWVl6SldjR1JuVTJsbWJqbHRDalZSY1RkSE4ybE1VRXhEU0ZGSVprRndhVVIyY2tsUVdETTNPVlJHVFM5cWRGSktNbXh5VWpaSE1UUXZMMEY2VDJab1REQlRSRlZEWjFsRlFUZHdjRFlLWVRSTE5YUXlPRWdyWVVWRGRrMU5UV05DUlVGV00wdDFUMk5aUldJeWJTdEZOM2xTT1hveFZrUllRM2xaUTJWaEwwczJPSEpGWm1Sa2FWZE9UWGxWYkFwd1psbHNlWGxqTlZaeU5FeHZNVW81WVd4dWFGQk1iblJoU2toVFNIVjRjbXBCYUhFeE5rUnJWbGg1Y1hkcFV6TkJWams0UTBreFRXdHhla3QzVVcxaUNqRm5XbmRsVERSdVIzSnlUM0ptVTNkVVJWUmtjV0V4T0hOV01uRlBhamhxVVdONmJHdGpPRU5uV1VKTlEyeENkRkJ1UTJwYVVqbGxaVkJKUm05WFpUQUthRGt3ZWxsb09UWndUSGcxWVRac1pFSktlRXh6U2tWTE1GRmpkbEJJWTFKMGFUWTNXVGhETm01MWVrcEZUMmhCZFZCTFNUUmtZMW94VG1wS1IwVjJiUXBoZDFwMWVqSk9UMjFKWTNOUFMzZGhOVUYxUlhvdlRXVktTVWxZWlVob2JVOVBWWFphY25Wdk9IUk1XVXB4V0VKV1VFb3hWRVJ3UmxWSFJuZGhPVk5DQ2xoaU9GbHhVVU15VEhjeGVrbE9kVGh2Y0haT2JWRkxRbWRHZUhCMWJsb3ZaVVI2UTNCVU1HSkViblJpZEVOeFNHVjZSbXBsYmxadU5qTnVXRms0V1hvS1NpOXRlR0ZKY1M5R2JXUkdhamRNUlZoMFJDOVBTbWN3TjFJMVpFTmtjbWR1YW5sNk5GaEtiVkYzTW5KMWJWZENhMFZ2TnpOUFlTdGhOM0ZOWVZFNFp3cG9NMnRwYlVzd0wycHpaVWt2ZEZKeFZHaHFiREpVTWtvM1ZFZGpOMmd5VkN0MVdGcERSRFZFUldncmQwRk5VbHBsT1hwdVJUY3JRMlp3WWpKek1HOWtDaXRQUW1wQmIwZEJTVFZpVmpSQlYyVjZWMnM0TVc5bFZXbEhPVVJaTlhSa1puSTFOMGxSTm1KSEwxcHRkVXROTDNjM1NHOXFUMlZzWmt0ck5UQm1ZMVFLYURkYU9WRnFkMEpWVERoWlprTlFjRk5wYmxKVGFFbGFXbWRGUzB0VVdqTkxUakpaZUV4UGFsSkxkVk53YVdaYVRWRjRjM1ZvZERaeGNYRm1hVWN6VHdwTVNGQlVUWG9yYjNGUVdHZDBaMlJFT1RreE4ySjRNMlJFY3paWFZrTlliM0F2ZEhnNVNreFdOVkZUSzNwRmJWY3hjRms5Q2kwdExTMHRSVTVFSUZKVFFTQlFVa2xXUVZSRklFdEZXUzB0TFMwdENnPT0K
  kubernetesAPIEndpoint: https://10.100.231.111:6443
  kubernetesAPIServerPort: 6304
  kubesphereAPIEndpoint: http://10.100.231.111:80
  kubesphereAPIServerPort: 16304
  token: 3383e9fed1d683b21b3c2c76289f3ef9af9677f650898ea3c2663a17963cd06e
  type: proxy
  joinFederation: true
  status:
  conditions:
• lastTransitionTime: "2022-05-31T07:32:33Z"
  lastUpdateTime: "2022-05-31T07:32:33Z"
  message: Agent has connected to proxy successfully.
  status: "True"
  type: AgentAvailable
• lastTransitionTime: "2022-05-31T07:41:07Z"
  lastUpdateTime: "2022-05-31T07:41:07Z"
  message: Cluster has been initialized
  reason: Initialized
  status: "True"
  type: Initialized

When executing kubectl rollout restart -n kubesphere-system deployment cluster-agent, the status of the new pod is CrashLoopBackOff and get logs like A session already allocated for this client. We should retry to connect the server instead of just pending.

If there is a docker mirror cache, printing out the build date and version is good for debugging

Problem:

Tower is panic and agent connect lost.

Toplogy:

Host KS-- Tower(Public Internet) <--------Agent(Internal network)---K8S/KS

Tower Log

Seemed to Recv error payload and Sig Segment Error

always output proxy error

Agent Log