Comments (6)
jp83
commented on September 16, 2024
I guess there's a similar issue with rook csi, this issue/comment seems relevant
I have:
[rancher@k8s-node-2 ~]$ ls -1 /opt/rke/var/lib/kubelet
cpu_manager_state
plugin-containers
plugins
plugins_registry
pod-resources
pods
But this doesn't resolve the iscsi related directories, maybe this is the other issue you were referring to previously.
from charts.
travisghansen
commented on September 16, 2024
OK, interesting stuff. Are you running with rke or using rancher to deploy k8s?
Does rancherOS not come with iscsi tools installed by default?
from charts.
travisghansen
commented on September 16, 2024
Also if you could send over the data from docker inspect kubelet on one of your nodes. That will give me the full picture of how the thing is launch in RancherOS.
from charts.
jp83
commented on September 16, 2024
Um, I used rancher to deploy, but I think it uses a lot of RKE, but I haven't converted/exported my cluster it to a yaml file or anything.
I had to install iscsi to use OpenEBS, basically followed these instructions
openebs/openebs#1146 (comment)
from charts.
jp83
commented on September 16, 2024
One caveat is that I was on older rancher version (still 2+) and k8s 1.13, and updated a few weeks ago and had a few problems, mostly dns. I intend to start fresh again once I'm comfortable with this new underlying storage.
[rancher@k8s-node-2 ~]$ docker inspect kubelet
[
{
"Id": "da4ad41662aa8981f1ea5a138e6e26371ac240c72edb6c6249c0f3f6321bfd6e",
"Created": "2020-02-25T14:14:05.472255563Z",
"Path": "/opt/rke-tools/entrypoint.sh",
"Args": [
"kubelet",
"--client-ca-file=/etc/kubernetes/ssl/kube-ca.pem",
"--cloud-provider=",
"--fail-swap-on=false",
"--volume-plugin-dir=/var/lib/kubelet/volumeplugins",
"--authentication-token-webhook=true",
"--cluster-dns=10.43.0.10",
"--kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml",
"--network-plugin=cni",
"--cni-conf-dir=/etc/cni/net.d",
"--read-only-port=0",
"--authorization-mode=Webhook",
"--resolv-conf=/etc/resolv.conf",
"--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
"--hostname-override=k8s-node-2",
"--pod-infra-container-image=rancher/pause:3.1",
"--anonymous-auth=false",
"--cgroups-per-qos=True",
"--address=0.0.0.0",
"--event-qps=0",
"--cni-bin-dir=/opt/cni/bin",
"--cluster-domain=cluster.local",
"--root-dir=/opt/rke/var/lib/kubelet",
"--make-iptables-util-chains=true",
"--streaming-connection-idle-timeout=30m",
"--v=2"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2252,
"ExitCode": 0,
"Error": "",
"StartedAt": "2020-02-29T19:58:08.689659683Z",
"FinishedAt": "2020-02-29T18:58:36.107019579Z"
},
"Image": "sha256:cdf19c45b54666841b869dd4211237a2b25758f28f2f363f5a8d19b4db275dcb",
"ResolvConfPath": "/etc/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/da4ad41662aa8981f1ea5a138e6e26371ac240c72edb6c6249c0f3f6321bfd6e/hostname",
"HostsPath": "/var/lib/docker/containers/da4ad41662aa8981f1ea5a138e6e26371ac240c72edb6c6249c0f3f6321bfd6e/hosts",
"LogPath": "/var/lib/docker/containers/da4ad41662aa8981f1ea5a138e6e26371ac240c72edb6c6249c0f3f6321bfd6e/da4ad41662aa8981f1ea5a138e6e26371ac240c72edb6c6249c0f3f6321bfd6e-json.log",
"Name": "/kubelet",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/etc/kubernetes:/etc/kubernetes:z",
"/etc/cni:/etc/cni:rw,z",
"/opt/cni:/opt/cni:rw,z",
"/opt/rke/var/lib/cni:/var/lib/cni:z",
"/var/lib/calico:/var/lib/calico:z",
"/etc/resolv.conf:/etc/resolv.conf",
"/sys:/sys:rprivate",
"/var/lib/docker:/var/lib/docker:rw,rslave,z",
"/opt/rke/var/lib/kubelet:/opt/rke/var/lib/kubelet:shared,z",
"/var/lib/rancher:/var/lib/rancher:shared,z",
"/var/run:/var/run:rw,rprivate",
"/run:/run:rprivate",
"/opt/rke/etc/ceph:/etc/ceph",
"/dev:/host/dev:rprivate",
"/var/log/containers:/var/log/containers:z",
"/var/log/pods:/var/log/pods:z",
"/usr:/host/usr:ro",
"/etc:/host/etc:ro",
"/var/lib/kubelet/volumeplugins:/var/lib/kubelet/volumeplugins:shared,z"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {
"max-file": "2",
"max-size": "25m"
}
},
"NetworkMode": "host",
"PortBindings": {},
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": [
"service-sidekick"
],
"CapAdd": null,
"CapDrop": null,
"Capabilities": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "shareable",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "host",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"label=disable"
],
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/84e96aa2138937c8b41d1fbab798ba42e61872f55edf7e4fc403e4cd97ffe417-init/diff:/var/lib/docker/overlay2/db0aefad5d29f05608bf2b52b1586929923bf8983ef03e719a41658c7ba8d8e3/diff:/var/lib/docker/overlay2/aa3fac404d7843a2c259bbfb5adda5bc91f4b48ec7104d6fe6d719b7e3911b3a/diff:/var/lib/docker/overlay2/bfddd63b2beeb2ac41c85ec4084cc56fe348e7d25b3eff3ecc60d6f008ca1401/diff:/var/lib/docker/overlay2/3ab650be37172d94da0d958ee2916d498376f5b654a5b8f25a9a9228a2f4909a/diff:/var/lib/docker/overlay2/b17b21e31698d1f2b7a440fde6f6cdd8327b4199cf32c43fdb993609d1dcfe3c/diff:/var/lib/docker/overlay2/1dbae6017447c251c3a151ba9e1d44519abdcfd39396ffe77e82911063426b4d/diff:/var/lib/docker/overlay2/e22b392641b881b2e54c7326165b9b131f94fd1b52041fe91c46819fc541e608/diff:/var/lib/docker/overlay2/4f953174368b6dc884e3f3e35f3f74cb7bd5ac1cb6dff5a1e3b29022ffff4d9e/diff:/var/lib/docker/overlay2/b58aef9bd78434f08ffe3c854b0dd2a107ae95ad34027f3d715db919205e8dfd/diff:/var/lib/docker/overlay2/a4b573f773d173ca061ad84127905d849bff9d51651ff59509685ee995bf7276/diff",
"MergedDir": "/var/lib/docker/overlay2/84e96aa2138937c8b41d1fbab798ba42e61872f55edf7e4fc403e4cd97ffe417/merged",
"UpperDir": "/var/lib/docker/overlay2/84e96aa2138937c8b41d1fbab798ba42e61872f55edf7e4fc403e4cd97ffe417/diff",
"WorkDir": "/var/lib/docker/overlay2/84e96aa2138937c8b41d1fbab798ba42e61872f55edf7e4fc403e4cd97ffe417/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/var/lib/calico",
"Destination": "/var/lib/calico",
"Mode": "z",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/lib/rancher",
"Destination": "/var/lib/rancher",
"Mode": "shared,z",
"RW": true,
"Propagation": "shared"
},
{
"Type": "bind",
"Source": "/var/run",
"Destination": "/var/run",
"Mode": "rw,rprivate",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/etc/kubernetes",
"Destination": "/etc/kubernetes",
"Mode": "z",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/etc/resolv.conf",
"Destination": "/etc/resolv.conf",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/dev",
"Destination": "/host/dev",
"Mode": "rprivate",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/opt/cni",
"Destination": "/opt/cni",
"Mode": "rw,z",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "ac6941434ae742add3d7fc5bf135020671baa78dbe6c701b3d5f87f6fe6ec806",
"Source": "/var/lib/docker/volumes/ac6941434ae742add3d7fc5bf135020671baa78dbe6c701b3d5f87f6fe6ec806/_data",
"Destination": "/opt/rke-tools",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/usr",
"Destination": "/host/usr",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/opt/rke/var/lib/kubelet",
"Destination": "/opt/rke/var/lib/kubelet",
"Mode": "shared,z",
"RW": true,
"Propagation": "shared"
},
{
"Type": "bind",
"Source": "/sys",
"Destination": "/sys",
"Mode": "rprivate",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/log/containers",
"Destination": "/var/log/containers",
"Mode": "z",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/etc/cni",
"Destination": "/etc/cni",
"Mode": "rw,z",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/etc",
"Destination": "/host/etc",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/opt/rke/var/lib/cni",
"Destination": "/var/lib/cni",
"Mode": "z",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/lib/kubelet/volumeplugins",
"Destination": "/var/lib/kubelet/volumeplugins",
"Mode": "shared,z",
"RW": true,
"Propagation": "shared"
},
{
"Type": "bind",
"Source": "/opt/rke/etc/ceph",
"Destination": "/etc/ceph",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/run",
"Destination": "/run",
"Mode": "rprivate",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/lib/docker",
"Destination": "/var/lib/docker",
"Mode": "rw,rslave,z",
"RW": true,
"Propagation": "rslave"
},
{
"Type": "bind",
"Source": "/var/log/pods",
"Destination": "/var/log/pods",
"Mode": "z",
"RW": true,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "k8s-node-2.int.jonathanphelps.com",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "rancher/hyperkube:v1.15.10-rancher1",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/opt/rke-tools/entrypoint.sh",
"kubelet",
"--client-ca-file=/etc/kubernetes/ssl/kube-ca.pem",
"--cloud-provider=",
"--fail-swap-on=false",
"--volume-plugin-dir=/var/lib/kubelet/volumeplugins",
"--authentication-token-webhook=true",
"--cluster-dns=10.43.0.10",
"--kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml",
"--network-plugin=cni",
"--cni-conf-dir=/etc/cni/net.d",
"--read-only-port=0",
"--authorization-mode=Webhook",
"--resolv-conf=/etc/resolv.conf",
"--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
"--hostname-override=k8s-node-2",
"--pod-infra-container-image=rancher/pause:3.1",
"--anonymous-auth=false",
"--cgroups-per-qos=True",
"--address=0.0.0.0",
"--event-qps=0",
"--cni-bin-dir=/opt/cni/bin",
"--cluster-domain=cluster.local",
"--root-dir=/opt/rke/var/lib/kubelet",
"--make-iptables-util-chains=true",
"--streaming-connection-idle-timeout=30m",
"--v=2"
],
"OnBuild": null,
"Labels": {
"io.rancher.rke.container.name": "kubelet",
"org.label-schema.build-date": "2020-02-13T19:18:10Z",
"org.label-schema.schema-version": "1.0",
"org.label-schema.vcs-ref": "902bbaa24c1d91a5ee867670fb5977d925132362",
"org.label-schema.vcs-url": "https://github.com/rancher/hyperkube.git"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "57782bcc8e0cbb4610ebdb644a1324f90fa45d3971b01022c4e9ef336db86cfa",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/default",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"host": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "9a06d398b53b02d07584ef533bc260e7d45e6ec0c6791ff4b46e377d57e3e7d2",
"EndpointID": "f11ba92f7d7c198781abfd9c0fef514248044c81c5a560b1005829d280dada11",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"DriverOpts": null
}
}
}
}
]
from charts.
travisghansen
commented on September 16, 2024
Yeah, I see they aren't doing a blanket /opt/rke/var/lib/kubelet mount. I'm running pure rke against CentOS 7 nodes ATM and this is what the mounts look like for me.
"HostConfig": {
"Binds": [
"/etc/kubernetes:/etc/kubernetes:z",
"/etc/cni:/etc/cni:rw,z",
"/opt/cni:/opt/cni:rw,z",
"/var/lib/cni:/var/lib/cni:z",
"/var/lib/calico:/var/lib/calico:z",
"/etc/resolv.conf:/etc/resolv.conf",
"/sys:/sys:rprivate",
"/var/lib/docker:/var/lib/docker:rw,rslave,z",
"/var/lib/kubelet:/var/lib/kubelet:shared,z",
"/var/lib/rancher:/var/lib/rancher:shared,z",
"/var/run:/var/run:rw,rprivate",
"/run:/run:rprivate",
"/etc/ceph:/etc/ceph",
"/dev:/host/dev:rprivate",
"/var/log/containers:/var/log/containers:z",
"/var/log/pods:/var/log/pods:z",
"/usr:/host/usr:ro",
"/etc:/host/etc:ro",
"/vault:/vault",
"/usr/local/sbin/iscsiadm:/usr/local/sbin/iscsiadm",
"/:/host-iscsi"
],
I suspect if you're on a newer version those issues may go away, otherwise you'll have to call out each dir individually.
You'll also likely need to work around this here: rancher/rke#1846 in whatever form that takes with RancherOS + Rancher (not sure if you can customize stuff with rancher directly like that).
I realize it's a lot of moving parts here so appreciate the patience getting it going. I'd love to have steps detailed out using rancher + rancherOS though from a documentation standpoint.
The general issues are the same as what you mentioned in the openebs link (they aren't unique to this project). It's just the nature of how csi + containerized kubelet + iscsi all have to interact. I know the openebs folks and we've actually discussed collaborating on a few things (they like some of the work involved with democratic-csi that can be re-used by other projects). In any case, I'm confident you can get it going, but will probably take a few tweaks to get all the stars aligned properly.
from charts.
Related Issues (20)
- Resource limits HOT 6
- unable to upgrade to 0.12.0 HOT 2
- Way to Move TrueNAS Secrets out of freenas-iscsi.yaml (values.yaml)? HOT 2
- Breaking changes: CustomResourceDefinition & field is immutable HOT 1
- Initial setup assist, installed truenas core last week HOT 12
- Regardless of chart version, latest democratic-csi image is always pulled and deployed HOT 3
- Mirror this Helm repo to ArtifactHub HOT 3
- Ability to set pods priority HOT 1
- lustre support from helm chart HOT 2
- k8s.gcr.io Image Registry Will Be Frozen HOT 2
- Proposal: Add helm-docs support on charts HOT 6
- Questions about `datasetPermissions*` and `shareMaproot*` fields HOT 8
- appVersion set to 1.0 for all helm releases HOT 3
- Weird behavior on directory creation for generic NFS mounts HOT 1
- Can't find smb-client.yaml file in the examples HOT 1
- Helm error if defaultClass is set to true HOT 1
- Question about the grpc-proxy HOT 2
- how can I replicate data in 3 different truenas ?
- Split image and tag into different values in the helm chart
- CA_BUNDLE preventing snapshot controller from being installed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from charts.