Comments (17)
This would be a great addition indeed.
One question upfront: is it possible to spin up a Azure Blob Storage compatible tool in a Docker container so it can be used in tests (and development) for this image? I would like to avoid to a. create an Azure account just to be able to support this feature and b. run tests against remote infrastructure. Ideally, this could work like with S3 where the tests run a MinIO server or WebDAV where an instance of bytemark/webdav
is running.
from docker-volume-backup.
This would be a great addition indeed.
One question upfront: is it possible to spin up a Azure Blob Storage compatible tool in a Docker container so it can be used in tests (and development) for this image? I would like to avoid to a. create an Azure account just to be able to support this feature and b. run tests against remote infrastructure. Ideally, this could work like with S3 where the tests run a MinIO server or WebDAV where an instance of
bytemark/webdav
is running.
Absolutely and totally understandable! It can be done with Azurite: https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azurite
See also this chapter to configure it with the SDK: https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azurite?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&tabs=visual-studio#authorization-for-tools-and-sdks
For the Docker container, see: https://hub.docker.com/_/microsoft-azure-storage-azurite
If you use CI/CD pipelines, here is a great example of automating all of that: https://docs.microsoft.com/en-us/azure/storage/blobs/use-azurite-to-run-automated-tests
from docker-volume-backup.
Sounds good. If you or someone else wants to pick this up right away, I am happy to help with getting PRs merged (it should be relatively straight forward). Otherwise I'll keep it on the roadmap so we can get this supported sooner or later. Thanks for the suggestion.
from docker-volume-backup.
so I can't make any commitments
No need to worry here.
from docker-volume-backup.
@DevSecNinja I created a PR that adds support for Azure Blob Storage in #171 - it seems to work well when working against Azurite, but as I have zero experience in the Azure Ecosystem, I am wondering if I am doing something "wrong" here (e.g. non-standard way of passing credentials and such). If you're interested, I would be very happy if you could give this PR a quick sanity check and let me know if you think it makes sense like this. Thank you š©
from docker-volume-backup.
@DevSecNinja Do you think you'll find the time to give this another test soon? In case you're too busy, that's no problem (it really is no problem, please do not feel obliged to do anything), I'd just like to wrap up that PR at some point. Considering the test setup against Azurite works, I'd be fine with merging this and seeing if people encounter issues in the wild, but if you still would like to test it, it'd be much appreciated as well.
Thanks @m90, understood! I think it's best to merge it and I'll test it afterwards as well. A new semester for my study program just started and I'll try to find some time for it!
from docker-volume-backup.
Sounds good. If you or someone else wants to pick this up right away, I am happy to help with getting PRs merged (it should be relatively straight forward). Otherwise I'll keep it on the roadmap so we can get this supported sooner or later. Thanks for the suggestion.
Awesome! I must say that I'm also a part time CS grad student besides work, so I can't make any commitments. But it's high on my list to pick up when I have some spare time. Thanks!
from docker-volume-backup.
In case anyone starts working on this, it'd be wise to check in on the progress of #94 which might make implementing this feature a bit easier (also completely different).
from docker-volume-backup.
Really cool, thanks so much @m90! PR looks good, but I'm not a Go expert! :) Made a small comment on the PR.
With regards to authentication, I always like to use my managed identities. I have them on my Azure VMs and on my lab VMs via Azure Arc. A managed identity basically allows you to authenticate to resources without a password (and is therefore often times safer). With an RBAC role, you can just give read/write access to the managed identity of the VM and the application can easily authenticate via azidentity.NewDefaultAzureCredential(nil)
: https://learn.microsoft.com/en-us/azure/developer/go/azure-sdk-authentication?tabs=powershell
If you're interested in implementing this, you could think of running azidentity.NewDefaultAzureCredential(nil)
if no Azure Storage account key was provided. In that case, the Go client will try the options defined in the Learn article I shared earlier, including managed identity auth.
from docker-volume-backup.
If you're interested in implementing this, you could think of running azidentity.NewDefaultAzureCredential(nil) if no Azure Storage account key was provided.
This sounds great and is exactly why I was asking. I'll implement it like this.
from docker-volume-backup.
If you're interested in implementing this, you could think of running azidentity.NewDefaultAzureCredential(nil) if no Azure Storage account key was provided.
This sounds great and is exactly why I was asking. I'll implement it like this.
Awesome, thanks! If you could push out a prerelease Docker image, Iām happy to give it a spin in my lab before releasing it?
from docker-volume-backup.
There now is v2.24.0-pre.1
on Docker Hub.
If you find the time to give this a test drive, it'd be much appreciated. Updated documentation is here: https://github.com/offen/docker-volume-backup/blob/azure-storage/README.md
from docker-volume-backup.
There now is
v2.24.0-pre.1
on Docker Hub.If you find the time to give this a test drive, it'd be much appreciated. Updated documentation is here: https://github.com/offen/docker-volume-backup/blob/azure-storage/README.md
That's awesome, thanks for the Christmas present! š I'll test the following soon:
- Storage Account via public endpoint (via
AZURE_STORAGE_ACCOUNT_NAME
) - Storage Account via private endpoint (via
AZURE_STORAGE_ENDPOINT
) - Storage Account with Access Key
- Storage Account with Managed Identity
Let me know if I'm missing a test case!
Storage Account via private endpoint (via AZURE_STORAGE_ENDPOINT
)
- Noticed in testing with
AZURE_STORAGE_ENDPOINT
that if the storage account cannot be accessed for whatever reason, the email log (with info notification level) didn't tell me that the blob upload failed. Is that as expected? AZURE_STORAGE_ENDPOINT
seems to require a slash at the end. That's not very clear from the docs and I would suggest to either throw an error in the log if the string doesn't end with a slash or just add the slash yourself in the code.AZURE_STORAGE_ENDPOINT
doesn't seem to work with public (xyz..blob.core.windows.net) and private (privatelink.blob.core.windows.net) endpoints. Not sure why but the log doesn't tell me anything:
Attaching to backup
backup | Installing cron.d entry with expression @daily.
backup | crond: crond (busybox 1.35.0) started, log level 8
backup | Starting cron in foreground.
time="2022-12-25T15:08:18Z" level=info msg="Created backup of `/backup` at `/tmp/backup-2022-12-25T15-05-14.tar.gz`."
time="2022-12-25T15:08:46Z" level=info msg="Encrypted backup using given passphrase, saving as `/tmp/backup-2022-12-25T15-05-14.tar.gz.gpg`."
time="2022-12-25T15:09:02Z" level=info msg="Removed tar file `/tmp/backup-2022-12-25T15-05-14.tar.gz`."
time="2022-12-25T15:09:02Z" level=info msg="Removed GPG file `/tmp/backup-2022-12-25T15-05-14.tar.gz.gpg`."
time="2022-12-25T15:09:04Z" level=info msg="Finished running backup tasks."
from docker-volume-backup.
Noticed in testing with AZURE_STORAGE_ENDPOINT that if the storage account cannot be accessed for whatever reason, the email log (with info notification level) didn't tell me that the blob upload failed. Is that as expected?
Hmm, this is kind of mysterious. If something fails, there should be an error and a non-zero exit. No errors are swallowed in the code and I've seen it fail a lot in development, too. I need to dig into the code of the SDK to see if there are any conditions that could cause a failing upload while still not returning an error.
When I deliberately break the test case by passing a bad account name, I will get the following log output:
ā azure git:(azure-storage) ā TEST_VERSION=v2.24.0-pre.1 docker-compose up -d
Recreating azure_backup_1 ...
azure_storage_1 is up-to-date
Recreating azure_backup_1 ... done
Starting azure_az_cli_1 ... done
ā azure git:(azure-storage) ā docker-compose exec backup backup
time="2022-12-26T08:09:39Z" level=info msg="Stopping 1 container(s) labeled `docker-volume-backup.stop-during-backup=true` out of 4 running container(s)."
time="2022-12-26T08:09:39Z" level=info msg="Created backup of `/backup` at `/tmp/test.tar.gz`."
time="2022-12-26T08:09:40Z" level=info msg="Restarted 1 container(s) and the matching service(s)."
time="2022-12-26T08:09:40Z" level=error msg="Fatal error running backup: copyArchive: error copying archive: (*azureBlobStorage).Copy: error uploading file /tmp/test.tar.gz: PUT http://storage:10000/devstoreaccount1xxx/test-container/path/to/backup/test.tar.gz\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Invalid storage account.\nERROR CODE: InvalidOperation\n--------------------------------------------------------------------------------\n<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n <Code>InvalidOperation</Code>\n <Message>Invalid storage account.\nRequestId:cb1979da-5926-4c55-bde8-4d82451cab96\nTime:2022-12-26T08:09:40.565Z</Message>\n</Error>\n--------------------------------------------------------------------------------\n"
time="2022-12-26T08:09:40Z" level=info msg="Removed tar file `/tmp/test.tar.gz`."
so I am not sure why you are seeing a successful exit right now.
AZURE_STORAGE_ENDPOINT seems to require a slash at the end. That's not very clear from the docs and I would suggest to either throw an error in the log if the string doesn't end with a slash or just add the slash yourself in the code.
Auto-appending a slash if needed probably makes sense.
AZURE_STORAGE_ENDPOINT doesn't seem to work with public (xyz..blob.core.windows.net) and private (privatelink.blob.core.windows.net) endpoints. Not sure why but the log doesn't tell me anything:
The endpoint needs to include a protocol, e.g. https://xyz.blob.core.windows.net/
. Did you pass the values like this?
from docker-volume-backup.
Apologies for my late reply, wanted to dive into it before replying.
Hmm strange, I'm not seeing those logs. Could it be that for AZURE_STORAGE_ENDPOINT
, also AZURE_STORAGE_ACCOUNT_NAME
needs to be set? I assumed that when I set AZURE_STORAGE_ENDPOINT
I don't need to set AZURE_STORAGE_ACCOUNT_NAME
but that might be wrong.
The endpoint needs to include a protocol, e.g. https://xyz.blob.core.windows.net/. Did you pass the values like this?
Yeah, I set it like this: AZURE_STORAGE_ENDPOINT=https://MyAccountName.privatelink.blob.core.windows.net/
and I've also tested with AZURE_STORAGE_ENDPOINT="https://MyAccountName.blob.core.windows.net/"
I'm getting an interesting error now that I'm using the AZURE_STORAGE_ENDPOINT
in combination with the AZURE_STORAGE_ACCOUNT_NAME
:
time="2022-12-30T16:25:13Z" level=error msg="Fatal error running backup: copyArchive: error copying archive: (*azureBlobStorage).Copy: error uploading file /tmp/backup-2022-12-30T16-21-39.tar.gz.gpg: parse \"\\\"https://MyAccountName.blob.core.windows.net\\\"/stct-mycontainername/backup-2022-12-30T16-21-39.tar.gz.gpg\": first path segment in URL cannot contain colon"
from docker-volume-backup.
Could it be that for AZURE_STORAGE_ENDPOINT, also AZURE_STORAGE_ACCOUNT_NAME needs to be set?
Yes, for the version you are testing against, AZURE_STORAGE_ACCOUNT_NAME
always needs to be set if you want to target Azure Blob Storage.
I'm getting an interesting error now that I'm using the AZURE_STORAGE_ENDPOINT in combination with the AZURE_STORAGE_ACCOUNT_NAME
The double quoted string in the error message makes me wonder if you're running into the bane of this issue tracker: docker/compose#2854
Assuming you are using Docker compose, could you try defining your environment variables in object notation instead, i.e. AZURE_STORAGE_ENDPOINT: https://MyAccountName.privatelink.blob.core.windows.net/
In case this is not causing the error: do you set a value for AZURE_STORAGE_PATH
?
from docker-volume-backup.
@DevSecNinja Do you think you'll find the time to give this another test soon? In case you're too busy, that's no problem (it really is no problem, please do not feel obliged to do anything), I'd just like to wrap up that PR at some point. Considering the test setup against Azurite works, I'd be fine with merging this and seeing if people encounter issues in the wild, but if you still would like to test it, it'd be much appreciated as well.
from docker-volume-backup.
Related Issues (20)
- Hard to understand error message during exec HOT 3
- archive-pre is triggered on each container backup instead of just container with exec-label HOT 3
- Health Check HOT 6
- Notifications Delay HOT 1
- Replace Log line feeds with HTML <br> tag HOT 3
- Backup the entire content without directory HOT 4
- Backup to subfolders HOT 2
- Resolve date in AWS_S3_PATH HOT 1
- Can't get stop during backup working HOT 2
- Remove local backup after upload to S3 HOT 3
- Invalid memory address crash when running a backup HOT 16
- Fail to configure cron schedule HOT 3
- Docker swarm error HOT 5
- "swarm exec" support HOT 1
- "input/output error" when copying file to NFS mount HOT 4
- Azure storage: Support access tier HOT 2
- schedule the docker-volume-backup? HOT 1
- [Feature] Automated backups of all named volumes HOT 6
- [FR] upgrade `shoutrrr` to v0.8.0 to support custom fields in calls to Generic Webhooks HOT 3
- [Feature] Allow BACKUP_COMPRESSION to be disabled HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-volume-backup.