Comments (15)
wadells
commented on June 8, 2024
cc @a-palchikov
from gravity.
a-palchikov
commented on June 8, 2024
Seems like tcp port 61009 is already reserved, can you check what it is reserved for?
$ semanage port -l | grep 61009
from gravity.
ahmed58479
commented on June 8, 2024
@a-palchikov Got past this issue by running on redhat 7.9. Getting a different issue now. Gravity gets stuck waiting for planet container to start.
$ ./gravity install --advertise-addr=172.31.30.40 --token=secret123 --cloud-provider=generic --selinux
Fri Sep 3 23:11:00 UTC Bootstrapping installer for SELinux
Fri Sep 3 23:11:25 UTC Starting enterprise installer
To abort the installation and clean up the system,
press Ctrl+C two times in a row.
If you get disconnected from the terminal, you can reconnect to the installer
agent by issuing 'gravity resume' command.
If the installation fails, use 'gravity plan' to inspect the state and
'gravity resume' to continue the operation.
See https://gravitational.com/gravity/docs/cluster/#managing-an-ongoing-operation for details.
Fri Sep 3 23:11:25 UTC Connecting to installer
Fri Sep 3 23:11:38 UTC Connected to installer
Fri Sep 3 23:11:38 UTC Successfully added "node" node on 172.31.30.40
Fri Sep 3 23:11:38 UTC All agents have connected!
Fri Sep 3 23:11:39 UTC Operation has been created
Fri Sep 3 23:11:40 UTC Executing "/selinux/ip-172-31-30-40.us-west-2.compute.internal" locally
Fri Sep 3 23:11:40 UTC Configure SELinux on node ip-172-31-30-40.us-west-2.compute.internal
Fri Sep 3 23:11:41 UTC Configuring SELinux
Fri Sep 3 23:11:45 UTC Initialize operation on all nodes
Fri Sep 3 23:11:46 UTC Executing "/init/ip-172-31-30-40.us-west-2.compute.internal" locally
Fri Sep 3 23:11:46 UTC Initialize operation on node ip-172-31-30-40.us-west-2.compute.internal
Fri Sep 3 23:11:47 UTC Executing "/checks" locally
Fri Sep 3 23:11:47 UTC Running pre-flight checks
Fri Sep 3 23:11:47 UTC Execute pre-flight checks
Fri Sep 3 23:11:57 UTC Still running pre-flight checks (10 seconds elapsed)
Fri Sep 3 23:12:05 UTC Executing "/configure" locally
Fri Sep 3 23:12:06 UTC Configuring cluster packages
Fri Sep 3 23:12:06 UTC Configure packages for all nodes
Fri Sep 3 23:12:11 UTC Bootstrap all nodes
Fri Sep 3 23:12:12 UTC Executing "/bootstrap/ip-172-31-30-40.us-west-2.compute.internal" locally
Fri Sep 3 23:12:12 UTC Configuring system directories
Fri Sep 3 23:12:12 UTC Configuring application-specific volumes
Fri Sep 3 23:12:12 UTC Bootstrap master node ip-172-31-30-40.us-west-2.compute.internal
Fri Sep 3 23:12:13 UTC Executing "/pull/ip-172-31-30-40.us-west-2.compute.internal" locally
Fri Sep 3 23:12:13 UTC Pull packages on master node ip-172-31-30-40.us-west-2.compute.internal
Fri Sep 3 23:12:14 UTC Pulling applications
Fri Sep 3 23:12:14 UTC Pulling application privaci-appliance:1.55.0
Fri Sep 3 23:12:24 UTC Still pulling application privaci-appliance:1.55.0 (10 seconds elapsed)
Fri Sep 3 23:12:34 UTC Still pulling application privaci-appliance:1.55.0 (20 seconds elapsed)
Fri Sep 3 23:12:44 UTC Still pulling application privaci-appliance:1.55.0 (30 seconds elapsed)
Fri Sep 3 23:12:54 UTC Still pulling application privaci-appliance:1.55.0 (40 seconds elapsed)
Fri Sep 3 23:13:04 UTC Still pulling application privaci-appliance:1.55.0 (50 seconds elapsed)
Fri Sep 3 23:13:14 UTC Still pulling application privaci-appliance:1.55.0 (1 minute elapsed)
Fri Sep 3 23:13:24 UTC Still pulling application privaci-appliance:1.55.0 (1 minute elapsed)
Fri Sep 3 23:13:34 UTC Still pulling application privaci-appliance:1.55.0 (1 minute elapsed)
Fri Sep 3 23:13:44 UTC Still pulling application privaci-appliance:1.55.0 (1 minute elapsed)
Fri Sep 3 23:13:47 UTC Pulling configured packages
Fri Sep 3 23:13:49 UTC Unpacking pulled packages
Fri Sep 3 23:13:50 UTC Install system software on master nodes
Fri Sep 3 23:13:51 UTC Executing "/masters/ip-172-31-30-40.us-west-2.compute.internal/teleport" locally
Fri Sep 3 23:13:51 UTC Installing system service teleport:3.2.17
Fri Sep 3 23:13:51 UTC Install system package teleport:3.2.17 on master node ip-172-31-30-40.us-west-2.compute.internal
Fri Sep 3 23:13:52 UTC Executing "/masters/ip-172-31-30-40.us-west-2.compute.internal/planet" locally
Fri Sep 3 23:13:53 UTC Install system package planet:7.0.60-11709 on master node ip-172-31-30-40.us-west-2.compute.internal
Fri Sep 3 23:13:53 UTC Installing system service planet:7.0.60-11709
Fri Sep 3 23:14:03 UTC Still installing system service planet:7.0.60-11709 (10 seconds elapsed)
Fri Sep 3 23:14:12 UTC Executing "/wait" locally
Fri Sep 3 23:14:12 UTC Wait for Kubernetes to become available
Fri Sep 3 23:14:22 UTC Still executing "/wait" locally (10 seconds elapsed)
Fri Sep 3 23:14:32 UTC Still executing "/wait" locally (20 seconds elapsed)
Fri Sep 3 23:14:36 UTC Executing "/rbac" locally
Fri Sep 3 23:14:36 UTC Creating Kubernetes RBAC resources
Fri Sep 3 23:14:36 UTC Bootstrap Kubernetes roles and PSPs
Fri Sep 3 23:14:38 UTC Executing "/coredns" locally
Fri Sep 3 23:14:39 UTC Configuring CoreDNS
Fri Sep 3 23:14:39 UTC Executing "/system-resources" locally
Fri Sep 3 23:14:39 UTC Create system Kubernetes resources
Fri Sep 3 23:14:40 UTC Configuring system Kubernetes resources
Fri Sep 3 23:14:40 UTC Executing "/user-resources" locally
Fri Sep 3 23:14:40 UTC Create user-supplied Kubernetes resources
Fri Sep 3 23:14:40 UTC Creating user-supplied Kubernetes resources
Fri Sep 3 23:14:41 UTC Export applications layers to Docker registries
Fri Sep 3 23:14:42 UTC Executing "/export/ip-172-31-30-40.us-west-2.compute.internal" locally
Fri Sep 3 23:14:42 UTC Unpacking application rbac-app:7.0.34
Fri Sep 3 23:14:42 UTC Exporting application rbac-app:7.0.34 to local registry
Fri Sep 3 23:14:42 UTC Unpacking application dns-app:7.0.4
Fri Sep 3 23:14:42 UTC Populate Docker registry on master node ip-172-31-30-40.us-west-2.compute.internal
Fri Sep 3 23:14:43 UTC Exporting application dns-app:7.0.4 to local registry
Fri Sep 3 23:14:44 UTC Unpacking application storage-app:0.0.3
Fri Sep 3 23:14:47 UTC Exporting application storage-app:0.0.3 to local registry
Fri Sep 3 23:14:57 UTC Still exporting application storage-app:0.0.3 to local registry (10 seconds elapsed)
Fri Sep 3 23:15:03 UTC Unpacking application bandwagon:6.0.1
Fri Sep 3 23:15:04 UTC Exporting application bandwagon:6.0.1 to local registry
Fri Sep 3 23:15:05 UTC Unpacking application logging-app:7.0.1
Fri Sep 3 23:15:06 UTC Exporting application logging-app:7.0.1 to local registry
Fri Sep 3 23:15:08 UTC Unpacking application monitoring-app:7.0.11
Fri Sep 3 23:15:09 UTC Exporting application monitoring-app:7.0.11 to local registry
Fri Sep 3 23:15:15 UTC Unpacking application tiller-app:7.0.2
Fri Sep 3 23:15:15 UTC Exporting application tiller-app:7.0.2 to local registry
Fri Sep 3 23:15:16 UTC Unpacking application site:7.0.34
Fri Sep 3 23:15:16 UTC Exporting application site:7.0.34 to local registry
Fri Sep 3 23:15:17 UTC Unpacking application privaci-appliance:1.55.0
Fri Sep 3 23:15:27 UTC Still unpacking application privaci-appliance:1.55.0 (10 seconds elapsed)
Fri Sep 3 23:15:37 UTC Still unpacking application privaci-appliance:1.55.0 (20 seconds elapsed)
Fri Sep 3 23:15:47 UTC Still unpacking application privaci-appliance:1.55.0 (30 seconds elapsed)
Fri Sep 3 23:15:51 UTC Exporting application privaci-appliance:1.55.0 to local registry
Fri Sep 3 23:16:01 UTC Still exporting application privaci-appliance:1.55.0 to local registry (10 seconds elapsed)
Fri Sep 3 23:16:11 UTC Still exporting application privaci-appliance:1.55.0 to local registry (20 seconds elapsed)
Fri Sep 3 23:16:21 UTC Still exporting application privaci-appliance:1.55.0 to local registry (30 seconds elapsed)
Fri Sep 3 23:16:31 UTC Still exporting application privaci-appliance:1.55.0 to local registry (40 seconds elapsed)
Fri Sep 3 23:16:41 UTC Still exporting application privaci-appliance:1.55.0 to local registry (50 seconds elapsed)
Fri Sep 3 23:16:51 UTC Still exporting application privaci-appliance:1.55.0 to local registry (1 minute elapsed)
Fri Sep 3 23:17:01 UTC Still exporting application privaci-appliance:1.55.0 to local registry (1 minute elapsed)
Fri Sep 3 23:17:11 UTC Still exporting application privaci-appliance:1.55.0 to local registry (1 minute elapsed)
Fri Sep 3 23:17:15 UTC Executing "/health" locally
Fri Sep 3 23:17:16 UTC Waiting for the planet to start
Fri Sep 3 23:17:16 UTC Wait for cluster to pass health checks
Fri Sep 3 23:17:26 UTC Still waiting for the planet to start (10 seconds elapsed)
Fri Sep 3 23:17:36 UTC Still waiting for the planet to start (20 seconds elapsed)
Fri Sep 3 23:17:46 UTC Still waiting for the planet to start (30 seconds elapsed)
Fri Sep 3 23:17:56 UTC Still waiting for the planet to start (40 seconds elapsed)
Fri Sep 3 23:18:06 UTC Still waiting for the planet to start (50 seconds elapsed)
I see the following error in /var/log/messages
3 23:14:32 ip-172-31-30-40 gravity: activating
Sep 3 23:14:32 ip-172-31-30-40 rootfs/usr/bin/planet[9975]: WARN Failed to query service status. error:[#012ERROR REPORT:#012Original Error: *box.ExitError exit status 3#012Stack Trace:#012#011/gopath/src/github.com/gravitational/planet/lib/box/enter.go:213 github.com/gravitational/planet/lib/box.enter#012#011/gopath/src/github.com/gravitational/planet/lib/box/enter.go:67 github.com/gravitational/planet/lib/box.(*Box).CombinedOutput#012#011/gopath/src/github.com/gravitational/planet/tool/planet/start.go:1050 main.getStatus#012#011/gopath/src/github.com/gravitational/planet/tool/planet/start.go:999 main.monitorUnits#012#011/go/src/runtime/asm_amd64.s:1337 runtime.goexit#012User Message: exit status 3] service:kube-controller-manager planet/start.go:1004
Sep 3 23:14:32 ip-172-31-30-40 gravity: activating
Sep 3 23:14:32 ip-172-31-30-40 rootfs/usr/bin/planet[9975]: WARN Failed to query service status. error:[#012ERROR REPORT:#012Original Error: *box.ExitError exit status 3#012Stack Trace:#012#011/gopath/src/github.com/gravitational/planet/lib/box/enter.go:213 github.com/gravitational/planet/lib/box.enter#012#011/gopath/src/github.com/gravitational/planet/lib/box/enter.go:67 github.com/gravitational/planet/lib/box.(*Box).CombinedOutput#012#011/gopath/src/github.com/gravitational/planet/tool/planet/start.go:1050 main.getStatus#012#011/gopath/src/github.com/gravitational/planet/tool/planet/start.go:999 main.monitorUnits#012#011/go/src/runtime/asm_amd64.s:1337 runtime.goexit#012User Message: exit status 3] service:kube-scheduler planet/start.go:1004
Sep 3 23:14:32 ip-172-31-30-40 systemd-journald[50]: Failed to open runtime journal: No such file or directory
Sep 3 23:14:32 ip-172-31-30-40 systemd-journald[50]: Failed to open runtime journal: No such file or directory
Sep 3 23:14:32 ip-172-31-30-40 gravity: active
Sep 3 23:14:32 ip-172-31-30-40 systemd-journald[50]: Failed to open runtime journal: No such file or directory
Sep 3 23:14:32 ip-172-31-30-40 systemd-journald[50]: Failed to open runtime journal: No such file or directory
Sep 3 23:14:32 ip-172-31-30-40 gravity: activating
Sep 3 23:14:32 ip-172-31-30-40 rootfs/usr/bin/planet[9975]: WARN Failed to query service status. error:[#012ERROR REPORT:#012Original Error: *box.ExitError exit status 3#012Stack Trace:#012#011/gopath/src/github.com/gravitational/planet/lib/box/enter.go:213 github.com/gravitational/planet/lib/box.enter#012#011/gopath/src/github.com/gravitational/planet/lib/box/enter.go:67 github.com/gravitational/planet/lib/box.(*Box).CombinedOutput#012#011/gopath/src/github.com/gravitational/planet/tool/planet/start.go:1050 main.getStatus#012#011/gopath/src/github.com/gravitational/planet/tool/planet/start.go:999 main.monitorUnits#012#011/go/src/runtime/asm_amd64.s:1337 runtime.goexit#012User Message: exit status 3] service:kube-kubelet planet/start.go:1004
from gravity.
ahmed58479
commented on June 8, 2024
Sep 3 23:14:10 ip-172-31-30-40 rootfs/usr/bin/planet[9933]: WARN Failed to run. error:[
ERROR REPORT:
Original Error: *libcontainer.genericError container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"mod keyring permissions: permission denied\""
Stack Trace:
/gopath/src/github.com/gravitational/planet/lib/box/srv.go:207 github.com/gravitational/planet/lib/box.Start
/gopath/src/github.com/gravitational/planet/tool/planet/start.go:293 main.start
/gopath/src/github.com/gravitational/planet/tool/planet/start.go:68 main.startAndWait
/gopath/src/github.com/gravitational/planet/tool/planet/main.go:466 main.run
/gopath/src/github.com/gravitational/planet/tool/planet/main.go:64 main.main
/go/src/runtime/proc.go:200 runtime.main
/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"mod keyring permissions: permission denied\""] planet/main.go:696
from gravity.
a-palchikov
commented on June 8, 2024
Can you share the audit log for the specific failure? It is definitely distribution version-specific and the gravity's policy has not been updated in awhile.
from gravity.
ahmed58479
commented on June 8, 2024
@a-palchikov audit logs.
type=SERVICE_STOP msg=audit(1631033116.341:4668269): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=gravity__gravitational.io__planet__7.0.60-11709 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_STOP msg=audit(1631033118.168:4668273): pid=10001 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:gravity_container_init_t:s0 msg='unit=coredns comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_STOP msg=audit(1631033118.661:4668277): pid=10001 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:gravity_container_init_t:s0 msg='unit=registry comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_STOP msg=audit(1631033121.180:4668318): pid=32223 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:gravity_container_init_t:s0 msg='unit=coredns comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_STOP msg=audit(1631033121.420:4668319): pid=32223 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:gravity_container_init_t:s0 msg='unit=registry comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_STOP msg=audit(1631033121.845:4668325): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=gravity__gravitational.io__planet__7.0.60-11709 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_STOP msg=audit(1631033123.666:4668329): pid=10001 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:gravity_container_init_t:s0 msg='unit=coredns comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_STOP msg=audit(1631033124.161:4668351): pid=10001 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:gravity_container_init_t:s0 msg='unit=registry comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_STOP msg=audit(1631033126.921:4668376): pid=32223 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:gravity_container_init_t:s0 msg='unit=coredns comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_STOP msg=audit(1631033126.928:4668379): pid=32223 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:gravity_container_init_t:s0 msg='unit=registry comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_STOP msg=audit(1631033127.225:4668381): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=gravity__gravitational.io__planet__7.0.60-11709 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
from gravity.
ahmed58479
commented on June 8, 2024
[root@ip-172-31-30-40 gravity_system_user]# systemctl status gravity__gravitational.io__planet__7.0.60-11709.service
● gravity__gravitational.io__planet__7.0.60-11709.service - Auto-generated service for the gravitational.io/planet:7.0.60-11709 package
Loaded: loaded (/etc/systemd/system/gravity__gravitational.io__planet__7.0.60-11709.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-09-07 19:16:36 UTC; 205ms ago
Main PID: 27762 (planet)
Tasks: 2
Memory: 8.0M
CGroup: /system.slice/gravity__gravitational.io__planet__7.0.60-11709.service
Sep 07 19:16:36 ip-172-31-30-40.us-west-2.compute.internal systemd[1]: Started Auto-generated service for the gravitational.io/planet:7.0.60-11709 package.
Sep 07 19:16:36 ip-172-31-30-40.us-west-2.compute.internal gravity-cli[27762]: [RUNNING]: /usr/bin/gravity package command "start" "gravitational.io/planet:7.0.60-11709" "ferventbooth1035/planet-con...0.60-11709"
Sep 07 19:16:36 ip-172-31-30-40.us-west-2.compute.internal rootfs/usr/bin/planet[27762]: WARN signal: killed libcontainer/container_linux.go:347
Sep 07 19:16:36 ip-172-31-30-40.us-west-2.compute.internal gravity[27762]: [ERROR]: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"mod keyri...n denied\""
Sep 07 19:16:36 ip-172-31-30-40.us-west-2.compute.internal systemd[1]: gravity__gravitational.io__planet__7.0.60-11709.service: main process exited, code=exited, status=255/n/a
Hint: Some lines were ellipsized, use -l to show in full.
from gravity.
ahmed58479
commented on June 8, 2024
@a-palchikov i do see the following error on audit2allow.
[root@ip-172-31-30-40 mnt]# audit2allow -a -w
type=AVC msg=audit(1631040505.687:4745807): avc: denied { execute } for pid=17447 comm="gravity" name="planet" dev="nvme0n1p2" ino=1212154316 scontext=sysadm_u:sysadm_r:gravity_installer_t:s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=file permissive=0
Was caused by:
Unknown - would be allowed by active policy
Possible mismatch between this policy and the one under which the audit message was generated.
Possible mismatch between current in-memory boolean settings vs. permanent ones.
type=AVC msg=audit(1631041794.230:4758859): avc: denied { execute_no_trans } for pid=25829 comm="gravity" path="/mnt/gravity/local/packages/unpacked/gravitational.io/planet/7.0.60-11709/rootfs/usr/bin/planet" dev="nvme0n1p2" ino=490733665 scontext=sysadm_u:sysadm_r:gravity_installer_t:s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=file permissive=0
Was caused by:
Unknown - would be allowed by active policy
Possible mismatch between this policy and the one under which the audit message was generated.
Possible mismatch between current in-memory boolean settings vs. permanent ones.
[root@ip-172-31-30-40 mnt]# audit2allow -a
from gravity.
a-palchikov
commented on June 8, 2024
It looks like the planet rootfs directory was not labeled properly as these transitions show. The planet binary needs to be labeled gravity_container_runtime_exec_t and not container_file_t.
I'll have to repro this to see if it's anything obvious.
from gravity.
ahmed58479
commented on June 8, 2024
@a-palchikov Is there a workaround for this? I tried changing the label using the app.yaml file as mentioned here. https://goteleport.com/gravity/docs/selinux/
Added the following
- name: app-data
path: /mnt
targetPath: /mnt
seLinuxLabel: "system_u:object_r:gravity_container_runtime_exec_t:s0"
Also tried running
chcon -t gravity_container_runtime_exec_t -R mnt/
Planet process still fails to start
Getting the following error now.
Sep 14 18:04:23 ip-172-31-30-40 gravity-cli[9333]: [RUNNING]: /usr/bin/gravity package command "start" "gravitational.io/planet:7.0.60-11709" "reverentvillani7368/planet-config-172313040reverentvillani7368:7.0.60-11709"
Sep 14 18:04:23 ip-172-31-30-40 gravity-cli[9333]: [FAILURE]: /usr/bin/gravity package command "start" "gravitational.io/planet:7.0.60-11709" "reverentvillani7368/planet-config-172313040reverentvillani7368:7.0.60-11709": [ERROR]: mkdir /mnt/gravity/local/packages/unpacked/gravitational.io/planet: permission denied
Sep 14 18:04:23 ip-172-31-30-40 gravity: #033[31m[ERROR]: mkdir /mnt/gravity/local/packages/unpacked/gravitational.io/planet: permission denied
Sep 14 18:04:23 ip-172-31-30-40 gravity: #033[0m
Sep 14 18:04:23 ip-172-31-30-40 systemd: gravity__gravitational.io__planet__7.0.60-11709.service: main process exited, code=exited, status=255/n/a
Sep 14 18:04:23 ip-172-31-30-40 systemd: Unit gravity__gravitational.io__planet__7.0.60-11709.service entered failed state.
Sep 14 18:04:23 ip-172-31-30-40 systemd: gravity__gravitational.io__planet__7.0.60-11709.service failed.
Sep 14 18:04:23 ip-172-31-30-40 setroubleshoot: failed to retrieve rpm info for /usr/bin/gravity
Sep 14 18:04:23 ip-172-31-30-40 setroubleshoot: SELinux is preventing /usr/bin/gravity from create access on the directory /mnt/gravity/local/packages/unpacked/gravitational.io/planet. For complete SELinux messages run: sealert -l 6d9984cf-2712-407c-ab67-8ae25fbdfd8f
Sep 14 18:04:23 ip-172-31-30-40 python: SELinux is preventing /usr/bin/gravity from create access on the directory /mnt/gravity/local/packages/unpacked/gravitational.io/planet.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that gravity should be allowed create access on the planet directory by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'gravity' --raw | audit2allow -M my-gravity#012# semodule -i my-gravity.pp#012
from gravity.
ahmed58479
commented on June 8, 2024
Getting the following error on teleport starting
Sep 14 18:32:04 ip-172-31-30-40 systemd: gravity__gravitational.io__teleport__3.2.17.service holdoff time over, scheduling restart.
Sep 14 18:32:04 ip-172-31-30-40 systemd: Stopped Auto-generated service for the gravitational.io/teleport:3.2.17 package.
Sep 14 18:32:04 ip-172-31-30-40 systemd: Started Auto-generated service for the gravitational.io/teleport:3.2.17 package.
Sep 14 18:32:05 ip-172-31-30-40 gravity-cli[28607]: [RUNNING]: /usr/bin/gravity package command "start" "gravitational.io/teleport:3.2.17" "stoicturing2182/teleport-node-config-172313040stoicturing2182:3.2.17"
Sep 14 18:32:05 ip-172-31-30-40 gravity: INFO [PROC:1] Joining the cluster with a secure token. service/connect.go:294
Sep 14 18:32:05 ip-172-31-30-40 gravity: WARN [AUTH] Joining cluster without validating the identity of the Auth Server. This may open you up to a Man-In-The-Middle (MITM) attack if an attacker can gain privileged network access. To remedy this, use the CA pin value provided when join token was generated to validate the identity of the Auth Server. auth/register.go:139
Sep 14 18:32:05 ip-172-31-30-40 gravity: WARN [PROC:1] Teleport process has exited with error: missing parameter HostID service:register.node service/supervisor.go:247
Sep 14 18:32:05 ip-172-31-30-40 gravity: ERRO [PROC:1] Critical service register.node has exited with error missing parameter HostID, aborting. service/signals.go:148
Sep 14 18:32:05 ip-172-31-30-40 gravity: error: missing parameter HostID
Sep 14 18:32:05 ip-172-31-30-40 systemd: gravity__gravitational.io__teleport__3.2.17.service: main process exited, code=exited, status=1/FAILURE
Sep 14 18:32:05 ip-172-31-30-40 systemd: Unit gravity__gravitational.io__teleport__3.2.17.service entered failed state.
Sep 14 18:32:05 ip-172-31-30-40 systemd: gravity__gravitational.io__teleport__3.2.17.service failed.
from gravity.
ahmed58479
commented on June 8, 2024
@a-palchikov After relabeling the binaries i was able to get planet to start. The kube-kubelet service is now failing with the following error.
Flag --kube-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cgroup-root has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --tls-min-version has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --feature-gates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --kube-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cgroup-root has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --tls-min-version has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --feature-gates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
W0915 23:26:11.338492 26857 plugins_providers.go:44] Unexpected CSI Migration Feature Flags combination detected: enabling "CSIMigrationAWSComplete" requires "CSIMigrationAWS" to be enabled. CSI Migration may not take effect
W0915 23:26:11.338540 26857 plugins_providers.go:44] Unexpected CSI Migration Feature Flags combination detected: enabling "CSIMigrationGCEComplete" requires "CSIMigrationGCE" to be enabled. CSI Migration may not take effect
W0915 23:26:11.338547 26857 plugins_providers.go:44] Unexpected CSI Migration Feature Flags combination detected: enabling "CSIMigrationOpenStack" requires CSIMigration to be enabled. CSI Migration may not take effect
W0915 23:26:11.338555 26857 plugins_providers.go:44] Unexpected CSI Migration Feature Flags combination detected: enabling "CSIMigrationAzureDisk" requires CSIMigration to be enabled. CSI Migration may not take effect
W0915 23:26:11.338561 26857 plugins_providers.go:44] Unexpected CSI Migration Feature Flags combination detected: enabling "CSIMigrationAzureFile" requires CSIMigration to be enabled. CSI Migration may not take effect
I0915 23:26:11.338667 26857 server.go:416] Version: v1.17.9
I0915 23:26:11.338871 26857 plugins.go:100] No cloud provider specified.
I0915 23:26:11.343445 26857 dynamic_cafile_content.go:166] Starting client-ca-bundle::/var/state/root.cert
I0915 23:26:11.400196 26857 server.go:642] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
I0915 23:26:11.400573 26857 container_manager_linux.go:265] container manager verified user specified cgroup-root exists: []
I0915 23:26:11.400584 26857 container_manager_linux.go:270] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
I0915 23:26:11.400712 26857 topology_manager.go:120] [topologymanager] Creating topology manager with none policy
I0915 23:26:11.400720 26857 container_manager_linux.go:300] [topologymanager] Initializing Topology Manager with none policy
I0915 23:26:11.400725 26857 container_manager_linux.go:305] Creating device plugin manager: true
I0915 23:26:11.400765 26857 state_mem.go:36] [cpumanager] initializing new in-memory state store
I0915 23:26:11.401031 26857 state_mem.go:84] [cpumanager] updated default cpuset: ""
I0915 23:26:11.401037 26857 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
I0915 23:26:11.401093 26857 kubelet.go:311] Watching apiserver
I0915 23:26:11.402457 26857 client.go:75] Connecting to docker on unix:///var/run/docker.sock
I0915 23:26:11.402476 26857 client.go:104] Start docker client with request timeout=2m0s
W0915 23:26:11.411749 26857 docker_service.go:563] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
I0915 23:26:11.411772 26857 docker_service.go:240] Hairpin mode set to "hairpin-veth"
I0915 23:26:11.419823 26857 docker_service.go:255] Docker cri networking managed by cni
I0915 23:26:11.427276 26857 docker_service.go:260] Docker Info: &{ID:F2MN:CM4O:7O52:PBHK:KLJP:DBKC:6BTW:TZIT:S3SW:GC7M:32AL:2VOC Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:2 Driver:overlay2 DriverStatus:[[Backing Filesystem xfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:21 OomKillDisable:true NGoroutines:40 SystemTime:2021-09-15T23:26:11.420398753Z LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:3.10.0-1160.15.2.el7.x86_64 OperatingSystem:Debian GNU/Linux 9 (stretch) (containerized) OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc00023c380 NCPU:8 MemTotal:33079832576 GenericResources:[] DockerRootDir:/ext/docker HTTPProxy: HTTPSProxy: NoProxy:0.0.0.0/0,.local Name:ip-172-31-30-40.us-west-2.compute.internal Labels:[] ExperimentalBuild:false ServerVersion:20.10.7 ClusterStore: ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[]} io.containerd.runtime.v1.linux:{Path:runc Args:[]} runc:{Path:runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:d71fcd7d8303cbf684402823e425e9dd2e99285d Expected:d71fcd7d8303cbf684402823e425e9dd2e99285d} RuncCommit:{ID:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 Expected:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=default name=selinux] ProductLicense:Community Engine Warnings:[]}
I0915 23:26:11.427355 26857 docker_service.go:273] Setting cgroupDriver to cgroupfs
I0915 23:26:11.435587 26857 remote_runtime.go:59] parsed scheme: ""
I0915 23:26:11.435599 26857 remote_runtime.go:59] scheme "" not registered, fallback to default scheme
I0915 23:26:11.435644 26857 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 <nil>}] <nil>}
I0915 23:26:11.435658 26857 clientconn.go:577] ClientConn switching balancer to "pick_first"
I0915 23:26:11.435695 26857 remote_image.go:50] parsed scheme: ""
I0915 23:26:11.435700 26857 remote_image.go:50] scheme "" not registered, fallback to default scheme
I0915 23:26:11.435711 26857 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 <nil>}] <nil>}
I0915 23:26:11.435716 26857 clientconn.go:577] ClientConn switching balancer to "pick_first"
E0915 23:26:11.438511 26857 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
I0915 23:26:11.446181 26857 kuberuntime_manager.go:211] Container runtime docker initialized, version: 20.10.7, apiVersion: 1.41.0
I0915 23:26:11.453242 26857 server.go:1114] Started kubelet
I0915 23:26:11.453282 26857 server.go:144] Starting to listen on 0.0.0.0:10250
E0915 23:26:11.453295 26857 kubelet.go:1303] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
I0915 23:26:11.454290 26857 server.go:384] Adding debug handlers to kubelet server.
I0915 23:26:11.454809 26857 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
I0915 23:26:11.455024 26857 volume_manager.go:265] Starting Kubelet Volume Manager
I0915 23:26:11.455448 26857 desired_state_of_world_populator.go:138] Desired state populator starts to run
I0915 23:26:11.486908 26857 status_manager.go:157] Starting to sync pod status with apiserver
I0915 23:26:11.486934 26857 kubelet.go:1821] Starting kubelet main sync loop.
E0915 23:26:11.486978 26857 kubelet.go:1845] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
I0915 23:26:11.555070 26857 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach
W0915 23:26:11.555090 26857 kubelet_node_status.go:311] the node label kubernetes.io/arch=amd64 will overwrite default setting amd64
W0915 23:26:11.555096 26857 kubelet_node_status.go:311] the node label kubernetes.io/os=linux will overwrite default setting linux
W0915 23:26:11.555100 26857 kubelet_node_status.go:311] the node label kubernetes.io/hostname=172.31.30.40 will overwrite default setting 172.31.30.40
I0915 23:26:11.581145 26857 kubelet_node_status.go:70] Attempting to register node 172.31.30.40
E0915 23:26:11.587076 26857 kubelet.go:1845] skipping pod synchronization - container runtime status check may not have completed yet
I0915 23:26:11.587384 26857 kubelet_node_status.go:112] Node 172.31.30.40 was previously registered
I0915 23:26:11.587429 26857 kubelet_node_status.go:73] Successfully registered node 172.31.30.40
E0915 23:26:11.787177 26857 kubelet.go:1845] skipping pod synchronization - container runtime status check may not have completed yet
I0915 23:26:12.029623 26857 cpu_manager.go:173] [cpumanager] starting with none policy
I0915 23:26:12.029650 26857 cpu_manager.go:174] [cpumanager] reconciling every 10s
I0915 23:26:12.029666 26857 policy_none.go:43] [cpumanager] none policy: Start
F0915 23:26:12.030769 26857 kubelet.go:1381] Failed to start ContainerManager open /proc/sys/kernel/panic: permission denied
Seems like it does not have access to /proc directory. Do we need to any any special permissions for this? I do not seem any audit log for this.
from gravity.
a-palchikov
commented on June 8, 2024
If the whole rootfs directory is not labelled properly, then various binaries will fail at random places. I will try to reproduce this once I get a chance and will let you know if it's anything obvious or something that requires changes to the policy.
from gravity.
ahmed58479
commented on June 8, 2024
@a-palchikov What are the correct labels for the rootfs directory. I got the following lables.
[root@ip-172-31-30-40 rootfs]# ls -lZ
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 bin
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 boot
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 dev
drwxrwxr-x. root root system_u:object_r:gravity_container_file_t:s0 etc
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 ext
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 home
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 lib
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 lib64
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 media
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 mnt
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 opt
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 proc
drwx------. root root system_u:object_r:gravity_container_file_t:s0 root
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 run
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 sbin
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 srv
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 sys
drwxrwxrwt. root root system_u:object_r:gravity_container_file_t:s0 tmp
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 usr
drwxr-xr-x. root root system_u:object_r:gravity_container_file_t:s0 var
Started a discussion about this here.
from gravity.
ahmed58479
commented on June 8, 2024
@a-palchikov Were you able to try this?
from gravity.
Related Issues (20)
- Prometheus Telegraf issue
- Gravity builds fail: "gcr.io/kubernetes-helm/tiller:v2.15.2 not found" HOT 3
- Gravity release automation fails during publish task HOT 1
- Feature: AWS integrations track autoscaling groups in multiple regions
- gravity plan --tail excessive CPU usage
- taint phase retry loop exits on temporary connectivity failures
- Gravity UPGRADE FAILED, hook failed while upgrading application from one version to another
- Create RFD for tracking SQS queues across multiple regions for autoscaling
- Master node unable to start gravity-site Pod
- Tracking: golang CVE CVE-2021-44716 / CVE-2021-44717 HOT 2
- Tracking: Update container images for kube-rbac-proxy and kube-state-metrics HOT 1
- Robotest failure to find image suse-cloud/sles-15-sp2-v20210604 HOT 2
- Gravity incompatible with systemd 242+
- How to Ship a single container when a customer installed with the provided gravity tarball HOT 1
- 7.0.36 unable to mount volumes on Google Cloud HOT 4
- Upgrade from 6.1.33 to 7.0.36 fails on missing etcd.bak file when using custom install directories HOT 1
- Cluster experiencing intermittent degraded state due to etcd-healthz failure HOT 4
- [7.0] default to serf -encrypt
- etcd tests fail due to expired certificate HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gravity.