mv: can't preserve ownership: Permission denied about mysql-backup HOT 18 CLOSED

Thank you very much! Now it's working fine.

from mysql-backup.

Hmm, can't preserve ownership? That sounds like an NFS issue.

You mounted the NFS share as root, and chowned the backup folder as root to 1005:1005. It sounds like it is trying to do a move of the file somewhere.

The backup seems to be created

In the NFS folder? Maybe share your config for running mysql-backup (without any secrets, of course)?

from mysql-backup.

Thank you for your fast response! I also assume it's a kind of NFS issue.

Exactly. The NFS share is mounted as root and then I chowned the backup folder as root to 1005:1005. I checked inside the containter 1005:1005 is correctly mapped to appuser:appuser. Yes, the backup is created inside the backup-folder (on the NFS share). Except for the error message everything seems to be working fine.

My assumption is that there is a problem when moving the backup file accross different files systems (https://github.com/deitch/mysql-backup/blob/master/entrypoint#L227): in this case from the local temp folder to the mounted NFS share. I had similar problems in the past in an other contexts. I'm not an expert in this, but maybe it's better to copy and delete the backup file instead of moving it.

The configuration of mysql-backup is pretty simple:
DB_USER | root
DB_SERVER | mydb-syserver
DB_PASS | my-password
DB_DUMP_TARGET | /mnt/mysqlbackup/backup
DB_DUMP_FREQ | 1440

Where /mnt/mysqlbackup is the mounted NFS share.

from mysql-backup.

maybe it's better to copy and delete the backup file instead of moving it

In theory, that should do the same thing. In practice, it might not.

Did you run it in debug mode to spew out logs, make 100% sure at what point it is complaining? Just set DB_DUMP_DEBUG=true. I cannot think of anywhere other than the mv, but if I had a dime for every time I was convinced at what stage something was breaking and I was completely wrong...

from mysql-backup.

Sure, this is the debug output:

+ file_env DB_SERVER
+ local var=DB_SERVER
+ local fileVar=DB_SERVER_FILE
+ local def=
+ '[' mydb-db ']'
+ '[' '' ']'
+ local val=
+ '[' mydb-db ']'
+ val=mydb-db
+ export DB_SERVER=mydb-db
+ DB_SERVER=mydb-db
+ unset DB_SERVER_FILE
+ file_env DB_PORT
+ local var=DB_PORT
+ local fileVar=DB_PORT_FILE
+ local def=
+ '[' '' ']'
+ local val=
+ '[' '' ']'
+ '[' '' ']'
+ export DB_PORT=
+ DB_PORT=
+ unset DB_PORT_FILE
+ file_env DB_USER
+ local var=DB_USER
+ local fileVar=DB_USER_FILE
+ local def=
+ '[' root ']'
+ '[' '' ']'
+ local val=
+ '[' root ']'
+ val=root
+ export DB_USER=root
+ DB_USER=root
+ unset DB_USER_FILE
+ file_env DB_PASS
+ local var=DB_PASS
+ local fileVar=DB_PASS_FILE
+ local def=
+ '[' mydb-pass ']'
+ '[' '' ']'
+ local val=
+ '[' mydb-pass ']'
+ val=mydb-pass
+ export DB_PASS=mydb-pass
+ DB_PASS=mydb-pass
+ unset DB_PASS_FILE
+ file_env DB_NAMES
+ local var=DB_NAMES
+ local fileVar=DB_NAMES_FILE
+ local def=
+ '[' '' ']'
+ local val=
+ '[' '' ']'
+ '[' '' ']'
+ export DB_NAMES=
+ DB_NAMES=
+ unset DB_NAMES_FILE
+ file_env DB_DUMP_FREQ 1440
+ local var=DB_DUMP_FREQ
+ local fileVar=DB_DUMP_FREQ_FILE
+ local def=1440
+ '[' 1440 ']'
+ '[' '' ']'
+ local val=1440
+ '[' 1440 ']'
+ val=1440
+ export DB_DUMP_FREQ=1440
+ DB_DUMP_FREQ=1440
+ unset DB_DUMP_FREQ_FILE
+ file_env DB_DUMP_BEGIN +0
+ local var=DB_DUMP_BEGIN
+ local fileVar=DB_DUMP_BEGIN_FILE
+ local def=+0
+ '[' '' ']'
+ local val=+0
+ '[' '' ']'
+ '[' '' ']'
+ export DB_DUMP_BEGIN=+0
+ DB_DUMP_BEGIN=+0
+ unset DB_DUMP_BEGIN_FILE
+ file_env DB_DUMP_DEBUG
+ local var=DB_DUMP_DEBUG
+ local fileVar=DB_DUMP_DEBUG_FILE
+ local def=
+ '[' true ']'
+ '[' '' ']'
+ local val=
+ '[' true ']'
+ val=true
+ export DB_DUMP_DEBUG=true
+ DB_DUMP_DEBUG=true
+ unset DB_DUMP_DEBUG_FILE
+ file_env DB_DUMP_TARGET /backup
+ local var=DB_DUMP_TARGET
+ local fileVar=DB_DUMP_TARGET_FILE
+ local def=/backup
+ '[' /mnt/mysqlbackup/backup ']'
+ '[' '' ']'
+ local val=/backup
+ '[' /mnt/mysqlbackup/backup ']'
+ val=/mnt/mysqlbackup/backup
+ export DB_DUMP_TARGET=/mnt/mysqlbackup/backup
+ DB_DUMP_TARGET=/mnt/mysqlbackup/backup
+ unset DB_DUMP_TARGET_FILE
+ file_env DB_RESTORE_TARGET
+ local var=DB_RESTORE_TARGET
+ local fileVar=DB_RESTORE_TARGET_FILE
+ local def=
+ '[' '' ']'
+ local val=
+ '[' '' ']'
+ '[' '' ']'
+ export DB_RESTORE_TARGET=
+ DB_RESTORE_TARGET=
+ unset DB_RESTORE_TARGET_FILE
+ file_env AWS_ENDPOINT_URL
+ local var=AWS_ENDPOINT_URL
+ local fileVar=AWS_ENDPOINT_URL_FILE
+ local def=
+ '[' '' ']'
+ local val=
+ '[' '' ']'
+ '[' '' ']'
+ export AWS_ENDPOINT_URL=
+ AWS_ENDPOINT_URL=
+ unset AWS_ENDPOINT_URL_FILE
+ file_env AWS_ENDPOINT_OPT
+ local var=AWS_ENDPOINT_OPT
+ local fileVar=AWS_ENDPOINT_OPT_FILE
+ local def=
+ '[' '' ']'
+ local val=
+ '[' '' ']'
+ '[' '' ']'
+ export AWS_ENDPOINT_OPT=
+ AWS_ENDPOINT_OPT=
+ unset AWS_ENDPOINT_OPT_FILE
+ file_env AWS_ACCESS_KEY_ID
+ local var=AWS_ACCESS_KEY_ID
+ local fileVar=AWS_ACCESS_KEY_ID_FILE
+ local def=
+ '[' '' ']'
+ local val=
+ '[' '' ']'
+ '[' '' ']'
+ export AWS_ACCESS_KEY_ID=
+ AWS_ACCESS_KEY_ID=
+ unset AWS_ACCESS_KEY_ID_FILE
+ file_env AWS_SECRET_ACCESS_KEY
+ local var=AWS_SECRET_ACCESS_KEY
+ local fileVar=AWS_SECRET_ACCESS_KEY_FILE
+ local def=
+ '[' '' ']'
+ local val=
+ '[' '' ']'
+ '[' '' ']'
+ export AWS_SECRET_ACCESS_KEY=
+ AWS_SECRET_ACCESS_KEY=
+ unset AWS_SECRET_ACCESS_KEY_FILE
+ file_env AWS_DEFAULT_REGION
+ local var=AWS_DEFAULT_REGION
+ local fileVar=AWS_DEFAULT_REGION_FILE
+ local def=
+ '[' '' ']'
+ local val=
+ '[' '' ']'
+ '[' '' ']'
+ export AWS_DEFAULT_REGION=
+ AWS_DEFAULT_REGION=
+ unset AWS_DEFAULT_REGION_FILE
+ file_env SMB_USER
+ local var=SMB_USER
+ local fileVar=SMB_USER_FILE
+ local def=
+ '[' '' ']'
+ local val=
+ '[' '' ']'
+ '[' '' ']'
+ export SMB_USER=
+ SMB_USER=
+ unset SMB_USER_FILE
+ file_env SMB_PASS
+ local var=SMB_PASS
+ local fileVar=SMB_PASS_FILE
+ local def=
+ '[' '' ']'
+ local val=
+ '[' '' ']'
+ '[' '' ']'
+ export SMB_PASS=
+ SMB_PASS=
+ unset SMB_PASS_FILE
+ [[ -n true ]]
+ set -x
+ '[' -n root ']'
+ DBUSER=-uroot
+ '[' -n mydb-pass ']'
+ DBPASS=-pmydb-pass
+ DUMPVARS=
++ env
++ awk -F_ '/^MYSQLDUMP_/ {print $2}'
+ '[' -z mydb-db ']'
+ '[' -z '' ']'
+ echo 'DB_PORT not provided, defaulting to 3306'
DB_PORT not provided, defaulting to 3306
+ DB_PORT=3306
+ TMPDIR=/tmp/backups
+ TMPRESTORE=/tmp/restorefile
+ declare -A uri
+ [[ -n '' ]]
+ uri_parser /mnt/mysqlbackup/backup
+ uri=()
+ full=/mnt/mysqlbackup/backup
+ full=/mnt/mysqlbackup/backup
+ full=/mnt/mysqlbackup/backup
+ [[ / == \/ ]]
+ full=file://localhost/mnt/mysqlbackup/backup
+ [[ file://l == \f\i\l\e\:\/\/\/ ]]
+ pattern='^(([a-z0-9]{2,5})://)?((([^:\/]+)(:([^@\/]*))?@)?([^:\/?]+)(:([0-9]+))?)(\/[^?]*)?(\?[^#]*)?(#.*)?$'
+ [[ file://localhost/mnt/mysqlbackup/backup =~ ^(([a-z0-9]{2,5})://)?((([^:\/]+)(:([^@\/]*))?@)?([^:\/?]+)(:([0-9]+))?)(\/[^?]*)?(\?[^#]*)?(#.*)?$ ]]
+ full=file://localhost/mnt/mysqlbackup/backup
+ uri[uri]=file://localhost/mnt/mysqlbackup/backup
+ uri[schema]=file
+ uri[address]=localhost
+ uri[user]=
+ uri[password]=
+ uri[host]=localhost
+ uri[port]=
+ uri[path]=/mnt/mysqlbackup/backup
+ uri[query]=
+ uri[fragment]=
+ [[ file == \s\m\b ]]
+ [[ -n '' ]]
+ return 0
++ date
+ echo Starting at Thu Dec 27 14:34:15 UTC 2018
Starting at Thu Dec 27 14:34:15 UTC 2018
++ date +%s
+ current_time=1545921255
+ freq_time=86400
++ date +%Y%m%d
+ today=20181227
+ [[ +0 =~ ^\+(.*)$ ]]
+ waittime=0
+ target_time=1545921255
+ '[' -z '' ']'
+ sleep 0
+ true
+ mkdir -p /tmp/backups
+ '[' -d /scripts.d/pre-backup/ ']'
++ date -u +%Y%m%d%H%M%S
+ now=20181227143415
+ SOURCE=db_backup_20181227143415.gz
+ TARGET=db_backup_20181227143415.gz
+ [[ -n '' ]]
+ DB_LIST=-A
+ mysqldump -h mydb-db -P 3306 + gzip
-uroot -pmydb-pass -A
+ '[' -d /scripts.d/post-backup/ ']'
+ '[' -f /scripts.d/source.sh ']'
+ '[' -f /scripts.d/target.sh ']'
+ case "${uri[schema]}" in
+ mkdir -p /mnt/mysqlbackup/backup
+ mv /tmp/backups/db_backup_20181227143415.gz /mnt/mysqlbackup/backup/db_backup_20181227143415.gz
mv: can't preserve ownership of '/mnt/mysqlbackup/backup/db_backup_20181227143415.gz': Permission denied
+ '[' -z '' ']'
++ date +%s
+ current_time=1545921395
+ backup_time=140
+ freq_time_count=0
+ freq_time_count_to_add=1
+ extra_time=86400
+ target_time=1546007655
+ waittime=86260
+ sleep 86260

from mysql-backup.

Yeah, as we suspected, right at that mv.

It has been a few (ok, more than a few) years since I did NFS admin (yes, I really did that once), but that looks suspiciously to me like insufficient rights to create a file as uid.gid 1005.1005 on the NFS mount. It might be both how it is mounted as a client, or how the NFS export is configured. What is the mount command you are using? Is is it in /etc/fstab or a manual mount command? And what options are you passing? We can start there, then look at the NFS server.

from mysql-backup.

I assume there are the rights to create the file (otherwise it won't be available on the share).

I'm exporting the NFS share from a Windows Server 2016 and it's mounted on Debian 9.6. My /etc/fstab contains: my-server-ip:/share-name /my-mount-point nfs auto,rw 0 0 . A subdirectory of /my-mount-point is then mounted by Kubernetes inside /mnt/mysqlbackup/ of the container.

from mysql-backup.

Try making a file manually on the mount and chown-ing it to 1005?

from mysql-backup.

I tried the following things:

1. Creating a file as root and then chown-ed to 1005:1005. Which was working.
2. Creating a file inside the container on the mount. Worked and the file is owned by the appuser.
3. Creating a file inside the container in /tmp/backup. Then trying to mv the file to the share (also inside the container as appuser):

bash-4.4$ pwd
/tmp/backups
bash-4.4$ touch foo.test
bash-4.4$ mv foo.test /mnt/mysqlbackup/backup/
mv: can't preserve ownership of '/mnt/mysqlbackup/backup/foo.test': Permission denied
bash-4.4$ ls -la /mnt/mysqlbackup/backup/
total 18234
drwxr-xr-x 2 appuser appuser 4096 Dec 28 09:49 .
drwxr-xr-x 2 root root 64 Dec 23 12:26 ..
-rw-r--r-- 1 appuser appuser 0 Dec 28 09:47 foo.test

from mysql-backup.

Oh that is even stranger. So it complains about can't preserve ownership but moves the file anyway, and keeps the ownership? Can you run ls -ln /mnt/mysqlbackup/backup/ to check the actual uid/gid?

from mysql-backup.

Yes, exactly. The files are moved anyway. Like you can see in the output event the backups of the last days have been created:

bash-4.4$ ls -ln /mnt/mysqlbackup/backup/
total 20553
-rw-r--r-- 1 1005 1005 2312095 Dec 23 12:24 db_backup_20181223122435.gz
-rw-r--r-- 1 1005 1005 2312001 Dec 23 12:34 db_backup_20181223123419.gz
-rw-r--r-- 1 1005 1005 2309769 Dec 23 12:58 db_backup_20181223125827.gz
-rw-r--r-- 1 1005 1005 2336176 Dec 24 13:00 db_backup_20181224130046.gz
-rw-r--r-- 1 1005 1005 2336582 Dec 25 12:56 db_backup_20181225125609.gz
-rw-r--r-- 1 1005 1005 2336786 Dec 26 12:58 db_backup_20181226125828.gz
-rw-r--r-- 1 1005 1005 2351828 Dec 27 13:00 db_backup_20181227130044.gz
-rw-r--r-- 1 1005 1005 2351932 Dec 27 14:36 db_backup_20181227143415.gz
-rw-r--r-- 1 1005 1005 2379495 Dec 28 14:32 db_backup_20181228143159.gz
-rw-r--r-- 1 1005 1005 0 Dec 28 09:47 foo.test
-rw-r--r-- 1 1005 1005 4 Dec 28 09:47 foo.txt
-rw-r--r-- 1 1005 1005 4 Dec 28 09:46 text.txt

from mysql-backup.

OK, digging a little deeper (i.e. looking at if busybox implements it in any unique way), it does it fairly normally:

1. copy the file
2. set any properties (uid, gid, permissions, times)
3. remove the old file

It is reaching the error in the second step, specifically the chown. As is clear from the source code, this is not a fatal error, and the mv continues, but without the proper ownership.

FWIW, the particular line is here . This is version 1.28.4 of busybox, which is what is used in alpine:3.8, the basis of the current iteration of mysql-backup.

In this case, it is getting the proper ownership, but cannot change them as user 1005 cannot really chown anything.

What strikes me as odd now is that calling chown() when I do not have permissions should fail only if it isn't already owned by that uid.gid. The source to chown() in musl is here but it isn't very exciting, just makes a syscall().

bash-4.4$ echo bar > /tmp/foo
bash-4.4$ ls -ln /tmp/foo
-rw-r--r--    1 1005     1005             4 Dec 30 09:02 /tmp/foo
bash-4.4$ id
uid=1005(appuser) gid=1005(appuser)
bash-4.4$ chown 1005.1005 /tmp/foo
bash-4.4$ echo $?
0
bash-4.4$ chown 1006.1006 /tmp/foo
chown: /tmp/foo: Operation not permitted
bash-4.4$ echo $?
1

Once again, I suspect it has something to do with nfs. Not sure if it is the client implementation or server-side.

Going to try and replicate. What is the underlying host running (distro and kernel version?). I don't think it matters, but good to know...

from mysql-backup.

Let's keep it simpler. Just create the file in the container on the mount and try chown-ing it?

bash-4.4$ echo test > test
bash-4.4$ cat test
test
bash-4.4$ chown 1005.1005 test
bash-4.4$ echo $?
0

As you can see, this has no issues (above tried NFS mount from Linux server). For whatever reason, the syscall is returning an error even when going from self to self.

If the above test fails, we should check how you have the NFS export configured on the server. Also, did you use any special options when mounting it on the client?

from mysql-backup.

@Nephelo can you check the latest image? Moved to cp -a anyways.

from mysql-backup.

Thank you very much for your deep analysis. Sorry for my late response.
The underlying host is running Debian 4.9.130.

I'm trying the following in the container on the mounted share:

bash-4.4$ cat test
test
bash-4.4$ chown 1005.1005 test
chown: test: Permission denied
bash-4.4$ echo $?
1
bash-4.4$

I'm mounting the folder with the following line my-server-ip:/share-name /my-mount-point nfs auto,rw 0 0 in /etc/fstab.

I tried the new image, but there's still the same issue:
cp: can't preserve ownership of '/mnt/mysqlbackup/backup/db_backup_20190109175142.gz': Permission denied.

I've access to an other cluster with a similar setup, but the share is served from a Linux host instead of a Windows host. I'll try it tomorrow.

from mysql-backup.

I checked it on an other cluster which has a similar setup except the NFS-share is provided by a Linux host. There it's working without any problems.

So I assume the problem is the Windows NFS server and it has nothing to do with your implementation. Since the backup files are created and moved to the share, the error message is no problem for me.

from mysql-backup.

It is the cp -a that is causing it. If we just did cp, it would be fine (but wouldn't preserve ownership). We can put in an option to cp without -a and it should work fine.

from mysql-backup.

See
#72

from mysql-backup.