woxxy / mysql-backup-to-amazon-s3 Goto Github PK
View Code? Open in Web Editor NEWBackup daily/weekly/monhtly all your MySQL databases to Amazon S3 via SH and s3cmd
License: MIT License
Backup daily/weekly/monhtly all your MySQL databases to Amazon S3 via SH and s3cmd
License: MIT License
Just curious if anyone else has hit this and how they are getting around it?
I configured all and i think it's all fine but i got an error when the system try to upload to s3:
WARNING: Upload failed: /day/mysqldbBackup.12.18.2013.tar.gz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
I got this error several time until i got:
ERROR: Upload of '/root/mysqldbBackup.12.18.2013.tar.gz' failed too many times. Skipping that file.
Do i have to change something on bucket configuration?
Thanks in advantage,
Dimitri
I get this error when I try using s3cmd
mimoo@server:~$ s3cmd mb s3://my-database-backups
ERROR: Bucket 'my-database-backups' already exists
ERROR: S3 error: 409 (BucketAlreadyExists): The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
And the program gets me this error:
ERROR: S3 error: 400 (InvalidRequest): The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
Old backup removed.
Moving the backup from past week to another folder...
ERROR: S3 error: 400 (InvalidRequest): The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
Past backup moved.
After a little troubleshooting I found out why my daily and weekly backups were running at the same time. Apparently if you set a "day of month" and "day of week" it will execute the command when either condition was met.
I found this in the crontab man page:
Note: The day of a command's execution can be specified by two fields -- day of month, and day of week. If both fields are restricted (ie, are not *), the command will be run when either field matches the current time.
For example, "30 4 1,15 * 5'' would cause a command to be run at 4:30 am on the 1st and 15th of each month, plus every Friday.
At the line of moving the backup from day to previous day I get the below errors. Result is that the old backup is moved to /previous_day but it also still shows up in /day. Apparently the delete part of the mv command fails.
Moving the backup from past day to another folder...
WARNING: Retrying failed request: /sql/previous_day/databases.03.26.2013.tar.gz (timed out)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /sql/previous_day/databases.03.26.2013.tar.gz (timed out)
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: /sql/previous_day/databases.03.26.2013.tar.gz (timed out)
WARNING: Waiting 9 sec...
WARNING: Retrying failed request: /sql/previous_day/databases.03.26.2013.tar.gz (timed out)
WARNING: Waiting 12 sec...
WARNING: Retrying failed request: /sql/previous_day/databases.03.26.2013.tar.gz (timed out)
WARNING: Waiting 15 sec...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please report the following lines to:
[email protected]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Problem: S3RequestError: Request failed for: /sql/previous_day/databases.03.26.2013.tar.gz
S3cmd: 1.0.0
Traceback (most recent call last):
File "/usr/bin/s3cmd", line 2006, in
main()
File "/usr/bin/s3cmd", line 1950, in main
cmd_func(args)
File "/usr/bin/s3cmd", line 618, in cmd_mv
subcmd_cp_mv(args, s3.object_move, "move", "File %(src)s moved to %(dst)s")
File "/usr/bin/s3cmd", line 607, in subcmd_cp_mv
response = process_fce(src_uri, dst_uri, extra_headers)
File "/usr/share/s3cmd/S3/S3.py", line 315, in object_move
response_copy = self.object_copy(src_uri, dst_uri, extra_headers)
File "/usr/share/s3cmd/S3/S3.py", line 311, in object_copy
response = self.send_request(request)
File "/usr/share/s3cmd/S3/S3.py", line 487, in send_request
return self.send_request(request, body, retries - 1)
File "/usr/share/s3cmd/S3/S3.py", line 487, in send_request
return self.send_request(request, body, retries - 1)
File "/usr/share/s3cmd/S3/S3.py", line 487, in send_request
return self.send_request(request, body, retries - 1)
File "/usr/share/s3cmd/S3/S3.py", line 487, in send_request
return self.send_request(request, body, retries - 1)
File "/usr/share/s3cmd/S3/S3.py", line 487, in send_request
return self.send_request(request, body, retries - 1)
File "/usr/share/s3cmd/S3/S3.py", line 489, in send_request
raise S3RequestError("Request failed for: %s" % resource['uri'])
S3RequestError: Request failed for: /sql/previous_day/databases.03.26.2013.tar.gz
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please report the above lines to:
[email protected]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Past backup moved.
Is there a way of deleting those files after backup is completed?
some example from 1 file:
d470328cfe947e583c786ac96870cbd backup [31/Mar/2013:01:19:25 +0000] 10.36.175.32 3272ee08a76771012BADB7E10721E6 REST.PUT.OBJECT 2013-03-31-01-19-25-B645228B9AC6A250 "PUT TTP/1.1" 200 - - 385 28 9 "-" "Jakarta Commons-HttpClient/3.0" -
It just works much better than mysqldump in every way, especially with very large DBs.
Hello;
How can i use this to upload multiple DB on a server?
I saw that currently it uses mysqldump to login and create the mysql dump. I would need the option to also define the mysql host.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.