Ensure Devicelab bots are synced with salt server about flutter HOT 7 CLOSED

Okay, we were able to recover access. I accepted keys for windows-4 and windows-17.

from flutter.

Please disregard previous errors, I able fixed issue from my end. Problem has been resolved and fixed. Thanks.

from flutter.

I have added myself, @zanderso and @bdero to the configuration of who can access the server. For future reference, the playbook for this is at: go/flutter-salt#accessing-master-servers

from flutter.

Re-opening this as 3 days after landing CLs to add ssh public keys, they do not appear to have propagated (in any case, when I try to ssh into the server, I get rejected for pubkey authentication).

Assigning to @bdero, cc @kentnguyen99

Per Kent, #146912 is blocked by the server accepting its keys.

from flutter.

Bot flutter-devicelab-windows-4.mtv.corp.google.com could not sync with Salt master. Someone need to login Salt master and then remove old keys and then add new keys so win-4 can re-sync again from Salt master.

I'm getting errors below when run command "salt-call state.apply" on win-4 machine.

flutter@FLUTTER-WIN-4 C:\Users\flutter>salt-call state.apply
[ERROR ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate
Minion failed to authenticate with the master, has the minion key been accepted?

from flutter.

No one can login to the salt master right now, unfortunately. We're working on recovering access.

from flutter.

Hi @bdero Can you delete and accept the keys for flutter-win-4 on Salt master one more time? I'm getting errors below:

flutter@FLUTTER-WIN-4 C:\Users\flutter>salt-call state.apply
[ERROR ] Command 'C:\Windows\System32\cipd.exe' failed with return code: 5
[ERROR ] retcode: 5
[WARNING ] State for file: c:\luci_machine_tokend\etc\pki\flutter-devicelab-win-4.mtv.corp.google.com.crt - Neither 'source' nor 'contents' nor 'conte
nts_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set
to 'False' to avoid reading the file unnecessarily.
[ERROR ] Command 'c:\luci_machine_tokend\opt\luci\machine_tokend\run.bat' failed with return code: 1
[ERROR ] stdout:
flutter@FLUTTER-WIN-4 C:\Users\flutter>c:\luci_machine_tokend\opt\luci\machine_tokend\luci_machine_tokend.exe -backend "luci-token-server.appspot.com"
-pkey-pem "c:\luci_machine_tokend\etc\pki\flutter-devicelab-win-4.mtv.corp.google.com.key" -cert-pem "c:\luci_machine_tokend\etc\pki\flutter-devicela
b-win-4.mtv.corp.google.com.crt" -status-file "c:\luci_machine_tokend\status.json" -token-file "c:\luci_machine_tokend\token.json" -timeout 45s
[ERROR ] stderr: [I2024-05-01T10:26:03.030455-07:00 6524 0 iface.go:167] tsmon is disabled because no endpoint is configured
[D2024-05-01T10:26:03.039716-07:00 6524 0 client.go:450] RPC luci-token-server.appspot.com/tokenserver.minter.TokenMinter.MintMachineToken [deadline 4
4.9907389s] {"host":"luci-token-server.appspot.com", "method":"MintMachineToken", "service":"tokenserver.minter.TokenMinter"}
[E2024-05-01T10:26:03.154716-07:00 6524 0 main.go:250] Failed to generate a new token - token server error BAD_TOKEN_ARGUMENTS - not a valid FQDN "flu
tter-win-4"
[ERROR ] retcode: 1
[ERROR ] {'pid': 6412, 'retcode': 1, 'stdout': '\r\nflutter@FLUTTER-WIN-4 C:\Users\flutter>c:\luci_machine_tokend\opt\luci\machine_tokend\luc
i_machine_tokend.exe -backend "luci-token-server.appspot.com" -pkey-pem "c:\luci_machine_tokend\etc\pki\flutter-devicelab-win-4.mtv.corp.google.co
m.key" -cert-pem "c:\luci_machine_tokend\etc\pki\flutter-devicelab-win-4.mtv.corp.google.com.crt" -status-file "c:\luci_machine_tokend\status.js
on" -token-file "c:\luci_machine_tokend\token.json" -timeout 45s', 'stderr': '[I2024-05-01T10:26:03.030455-07:00 6524 0 iface.go:167] tsmon is disab
led because no endpoint is configured\n[D2024-05-01T10:26:03.039716-07:00 6524 0 client.go:450] RPC luci-token-server.appspot.com/tokenserver.minter.T
okenMinter.MintMachineToken [deadline 44.9907389s] {"host":"luci-token-server.appspot.com", "method":"MintMachineToken", "service":"tokenserver.minter
.TokenMinter"}\n[E2024-05-01T10:26:03.154716-07:00 6524 0 main.go:250] Failed to generate a new token - token server error BAD_TOKEN_ARGUMENTS - not a
valid FQDN "flutter-win-4"'}
[WARNING ] The function "module.run" is using its deprecated version and will expire in version "Phosphorus".

ID: c:\luci_machine_tokend\opt\luci\machine_tokend\run.bat
Function: cmd.run
Result: False
Comment: Command "c:\luci_machine_tokend\opt\luci\machine_tokend\run.bat" run
Started: 10:26:02.986572
Duration: 171.584 ms
Changes:
----------
pid:
6412
retcode:
1
stderr:
[I2024-05-01T10:26:03.030455-07:00 6524 0 iface.go:167] tsmon is disabled because no endpoint is configured
[D2024-05-01T10:26:03.039716-07:00 6524 0 client.go:450] RPC luci-token-server.appspot.com/tokenserver.minter.TokenMinter.MintMachin
eToken [deadline 44.9907389s] {"host":"luci-token-server.appspot.com", "method":"MintMachineToken", "service":"tokenserver.minter.TokenMinter"}
[E2024-05-01T10:26:03.154716-07:00 6524 0 main.go:250] Failed to generate a new token - token server error BAD_TOKEN_ARGUMENTS - not
a valid FQDN "flutter-win-4"
stdout:

              flutter@FLUTTER-WIN-4 C:\Users\flutter>c:\luci_machine_tokend\opt\luci\machine_tokend\luci_machine_tokend.exe -backend "luci-token-s

erver.appspot.com" -pkey-pem "c:\luci_machine_tokend\etc\pki\flutter-devicelab-win-4.mtv.corp.google.com.key" -cert-pem "c:\luci_machine_tokend\etc\pk
i\flutter-devicelab-win-4.mtv.corp.google.com.crt" -status-file "c:\luci_machine_tokend\status.json" -token-file "c:\luci_machine_tokend\token.json" -
timeout 45s

from flutter.