Comments (11)
this is the problem
librados: client.admin authentication error (95) Operation not supported
[errno 95] error connecting to the cluster
At the time the user is to be deleted, the volume should have already be deleted, confirming the credential and cluster are working.
@mkimuram what ceph cluster you are using? could you please get the ceph mon logs and see if there is any message about deleting user?
@gman0 have you seen this error before?
from ceph-csi.
@rootfs no. Indeed, ceph logs could help.
from ceph-csi.
what ceph cluster you are using?
I'm using ceph docker image used in e2e testing.
(Created by using https://github.com/kubernetes/kubernetes/tree/master/test/images/volumes-tester/rbd .)
could you please get the ceph mon logs and see if there is any message about deleting user?
Logs are extracted below way. Please check it.
ceph_logs.txt
# kubectl exec -it ceph-server -n e2e-tests-csi-mock-plugin-6hphk -- find /var/log/ceph/ -name "*log"
/var/log/ceph/ceph-osd.1.log
/var/log/ceph/ceph-mon.a.log
/var/log/ceph/ceph-client.admin.log
/var/log/ceph/ceph.log
/var/log/ceph/ceph-osd.0.log
/var/log/ceph/ceph.audit.log
/var/log/ceph/ceph-mds.cephfs.log
# kubectl exec -it ceph-server -n e2e-tests-csi-mock-plugin-6hphk -- find /var/log/ceph/ -name "*log" -print -exec cat {} \; > /tmp/ceph_logs.txt
from ceph-csi.
there is no auth rm
in ceph-audit.log
, last msgs are:
2018-07-19 18:20:48.820491 mon.0 172.17.0.3:6789/0 78 : audit [INF] from='client.? 172.17.0.3:0/386396941' entity='client.admin' cmd=[{"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"}]: dispatch
2018-07-19 18:20:51.959515 mon.0 172.17.0.3:6789/0 81 : audit [INF] from='client.? 172.17.0.3:0/386396941' entity='client.admin' cmd='[{"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"}]': finished
from ceph-csi.
@mkimuram can you post the relevant e2e test console messages too? I wonder if the ceph cluster was (in the process of) teardown before the PV was deleted in the background.
from ceph-csi.
@mkimuram this is what message i would look for to ensure the PV is deleted before the cluster is teared down.
https://github.com/mkimuram/kubernetes/blob/b461cb7acad0a0912fda198b8df46cd5ae452f0a/test/e2e/storage/volume_provisioning.go#L166
from ceph-csi.
I run e2e test again with saving console log(e2e-console.txt) and wait until "Waiting up to 20m0s for PersistentVolume pvc-xxx to get deleted" message shows up.
After that, I got the logs (all_logs.txt) with the below script.
However, I still can't find auth rm
.
This time, I also include logs for csi-external-provisioner and cephfs-driver for the relevant execution.
I hope this will be of your help.
# cat getlog.sh
#! /bin/bash -x
date
kubectl logs ceph-server -n $namespace
kubectl logs csi-cephfs-controller-0 -n $namespace -c csi-external-provisioner
kubectl logs $cephfsnode -n $namespace -c cephfs-driver
kubectl exec -it ceph-server -n $namespace -- find /var/log/ceph/ -name "*log"
kubectl exec -it ceph-server -n $namespace -- find /var/log/ceph/ -name "*log" -print -exec cat {} \;
# export namespace=e2e-tests-csi-mock-plugin-9sbdt
# export cephfsnode=csi-cephfs-node-jkxxx
# ./getlog.sh > all_logs.txt 2>&1
from ceph-csi.
right, the timeline matches: in CSI logs, delete volume happened before cluster was teared down.
from ceph-csi.
Sharing logs with debug messages:
all_logs5.txt
e2e-console5.txt
from ceph-csi.
This is what happened.
The e2e test just creates/deletes volume, it doesn't publish the volume, thus the volume is never mounted during the test.
However, the cephfs controller service doesn't create the user; the node service creates it here. So by the time volume is to be deleted by controller service, the user doesn't exist.
from ceph-csi.
@rootfs very nice find, thanks! I'll move
this to NodeStageVolume/NodeUnstageVolume.
from ceph-csi.
Related Issues (20)
- Concurrency issue when pvc is mounted to multiple pods HOT 23
- CentOS Stream 8 is EOL HOT 4
- Add support for ModifyVolume HOT 5
- allow podSecurityContexts to be set in `nodeplugin` and `provisioner` of chart `ceph-csi-cephfs` HOT 2
- Add http health endpoint for ceph-csi-cephfs and ceph-csi-rbd HOT 4
- cephfs-csi Pod has always been in containercreating HOT 3
- CephFS keyring requires nonsensicaly enormous and insecure privileges to work HOT 6
- Update modprobe in csi-rbdplugin to support zstd compressed rbd and nbd kernel module HOT 7
- Unable to create CephFS subvolume dynamically (`no available topology found`) HOT 2
- cephFS: Remove the 400 snapshot limitation from the doc HOT 2
- Allow to change mounter option from an existing PV HOT 9
- cephfs csi error,mds mds status Start request repeated too quickly. Failed with result 'signal'. HOT 3
- New csiplugin-configmap.yaml setting to override PVs's volumeAttributes.mounter HOT 2
- 'luksOpen'` encrypted volumes HOT 1
- rbd remap on network failure HOT 8
- Remove podSecurityPolicy from the helm documentation
- Ensure ceph-lock is always released when staging encrypted cephfs volume HOT 4
- Improve performance in encrypted cephfs provisioning HOT 3
- Update the capabilities needed for client.csi-cephfs-node in case of encryption HOT 3
- Ceph-csi enable deployments to multiple namespaces per cluster HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ceph-csi.