AWS S3 Frankfurt region not working about django-storages HOT 28 CLOSED

I had the same issue and fixed it by using S3Boto3Storage instead of S3BotoStorage and two not documented parameters in my settings.py.

requirements.txt:

• boto3==1.4.1
• django-storages==1.5.1

settings.py:

• STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
• AWS_ACCESS_KEY_ID = ''
• AWS_SECRET_ACCESS_KEY = ''
• AWS_STORAGE_BUCKET_NAME = ''
• AWS_S3_REGION_NAME = 'eu-central-1' (not documented)
• AWS_S3_SIGNATURE_VERSION = 's3v4' (not documented)
• AWS_QUERYSTRING_AUTH = False
• AWS_S3_FILE_OVERWRITE = True

To be honest, I didn't set the STATICFILES_STORAGE setting, because my application uses the S3Boto3Storage class for specific tasks, not for all static files. But I think it should work this way.

I stumbled upon the not documented options by going through the source code of django-storages.

Hope this will be a solution for others too.

from django-storages.

s3 boto user can find a temporary workaround here: boto/boto#2741

EDIT: here is a sample workaround

import os
from storages.backends.s3boto import S3BotoStorage

os.environ['S3_USE_SIGV4'] = 'True'

class S3Storage(S3BotoStorage):
    @property
    def connection(self):
        if self._connection is None:
            self._connection = self.connection_class(
                self.access_key, self.secret_key,
                calling_format=self.calling_format, host='s3.eu-central-1.amazonaws.com')
        return self._connection

from django-storages.

In my case for Frankfurt region it was enough to only add these two lines to settings (no need to override S3BotoStorage)
S3_USE_SIGV4 = True
AWS_S3_HOST = 's3.eu-central-1.amazonaws.com'

Tested with
django-storages==1.5.2
boto==2.46.1

from django-storages.

@miracle2k Make sure you're providing the modified Storage Class in STATICFILES_STORAGE and not the boto default one.

This works for me:

import os
from storages.backends.s3boto import S3BotoStorage

os.environ['S3_USE_SIGV4'] = 'True'

class S3Storage(S3BotoStorage):
    @property
    def connection(self):
        if self._connection is None:
            self._connection = self.connection_class(
                self.access_key, self.secret_key,
                calling_format=self.calling_format, host='s3.eu-central-1.amazonaws.com')
        return self._connection

STATICFILES_STORAGE = 'MyProject.settings.S3Storage'

from django-storages.

Funny this has started to throw 403:Forbidden all of a sudden. Only when requesting from localhost though. Might have something to do with http / https?

#EDIT#
This had to do with my local timesettings. Synced to ntp and everything worked as expected. I blame friday afternoon. And apologies for the stupid comment.

from django-storages.

@somacci solution by @FPurchess works. Try it.

from django-storages.

So I solved my problem my switching to:

boto3==1.4.4
django-storages==1.5.2

Not sure what the problem was, but I am using eu-central-1. Maybe that is the issue, but anyway, seems like its fixed.

from django-storages.

Adding
AWS_S3_REGION_NAME = 'eu-central-1'
fixed it for me. I guess you just need to set your region, and you good to go.
Full code:

AWS_SECRET_ACCESS_KEY = ""
AWS_STORAGE_BUCKET_NAME = ""
AWS_S3_REGION_NAME = 'eu-central-1'

AWS_S3_FILE_OVERWRITE = False
AWS_DEFAULT_ACL = None

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

from django-storages.

I can confirm that the above workaround works for Frankfurt.

Big thanks @FPurchess !

from django-storages.

I can confirm this, too! Thank you!!

from django-storages.

You can just add this to the settings:

os.environ['S3_USE_SIGV4'] = 'True'
AWS_S3_HOST = 's3.eu-central-1.amazonaws.com'

(Note: I'm using the bitbucket repo)

from django-storages.

@BodhiSukha thanks for that, going to close now. If you notice any regression in moving from the BitBucket repo to this one please open an issue.

from django-storages.

@BodhiSukha AWS_S3_HOST = 's3.eu-central-1.amazonaws.com' does not work for me. Any idea why? Hacking class as @FPurchess works fine.

boto==2.36.0
django-storages==1.1.8

from django-storages.

@satyrius sorry just saw you comment.

I'm using it in 2 different setups

boto==2.34.0 
django-storages==1.1.8

and

boto==2.36.0 
django-storages==1.1.8

works fine on both of them. Unfortunately I'm not sure about your problem. Only other thing different in my settings is the calling format. In python 2.7.9 environment I use this extra setting

AWS_S3_CALLING_FORMAT = boto.s3.connection.OrdinaryCallingFormat()

from django-storages.

When I try using this workaround, I get HostRequiredError: BotoClientError: When using SigV4, you must specify a 'host' parameter..

django-storages==1.1.8
boto==2.38.0

from django-storages.

I have checked the issues about Signature 4 with boto for Frankfurt region. I have had same problem with Seoul Frankfurt. I know that both Seoul and Frankfurt regions only support Signture 4 only.
So, is this closed or not. If it is closed, how to solve it. Please let me know.

from django-storages.

Thanks @Leistungsabfall!
The 400 Bad Request exception persisted until I updated to boto==2.38.0 (I was running 2.22).

from django-storages.

Could we re-open this issue please ? It's not solved, we just have a work-around.
Suport for s3v4 auth is really important for all who are deploying outside the U.S.

from django-storages.

@sdeleeuw 's work around works beautifully.

from django-storages.

I experienced an issue with using the London region (eu-west-2) and @tramwaj29's solution fixed it for me, substituting the region name in the value of AWS_S3_HOST.

from django-storages.

I'm going to reopen this until I can get it added to the docs.

from django-storages.

I used the method by @sdeleeuw, and at least it does not throw any errors anymore. However, when I upload a File, it does not create the file in S3. I create an IAM user and use the credentials from the user. Also, I added the user as a principal

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "PublicReadForGetBucketObjects",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::bucket/media/*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789:user/myuser"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::bucket/media",
                "arn:aws:s3:::bucket/media/*"
            ]
        }
    ]
}

And CORS:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>HEAD</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>Authorization</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

Any ideas?

from django-storages.

@oesah I think that s3boto3 handles this automagically but the same doesn't play out for s3boto. I have opened #335 which includes documentation for the fixes discovered in this thread and changes the default signature version so no one will be bit by this in the future. Thanks for your patience and your discoveries everyone. If you see anything wrong in #335 please say something. I'm going to merge it in the morning and immediately cut 1.6 afterwards.

from django-storages.

I'm getting :

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>
        InvalidRequest
    </Code>
    <Message>
        The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
    </Message>
    <RequestId>548992053424FAF0</RequestId> . 
    <HostId>
        bE/0RI6Wn5J41Qkf/kKRnA8A+uLuy9N1C6wyGsptJeqqOqmyaV+JCbXbj8hzHI7Bp1PtFOo897k= . 
    </HostId>
</Error>

with this code:

AWS_S3_HOST = 's3.eu-west-2.amazonaws.com'
s3 = boto.connect_s3(settings.S3_CALLS_KEY, settings.S3_CALLS_SECRET, host=AWS_S3_HOST)
bucket = s3.get_bucket(settings.S3_CALLS_BUCKET)

Can anyone see the issue here?

I'm trying for london, I got the s3 host from here

EDIT : I updated boto but it went to a different python environment, updated the correct boto and now my code above works.

from django-storages.

This also appears to happen in the 'Canada (Central)' region. Was getting a Cross-Origin Read Blocking (CORB) blocked cross-origin (error was caused because the response was just error XML and content type xml instead of the expected image content type) in the browser until I changed region. Going to implement workaround by @sdeleeuw

Edit: Workaround worked great for Canada region.

from django-storages.

Just my 2 cents here, I might be missing something but it seems like boto reads S3_USE_SIGV4 straight from the env vars and thus setting it in settings.py has no effect whatsoever. See here:
https://github.com/boto/boto/blob/03b2268348ea81d80e3e5ddea0970f4968561010/boto/auth.py#L1067

Can someone confirm this? If this is true docs need to be updated.

from django-storages.

Setting the region name to one of the following also does the job because of this

SIGV4_DETECT = [
    '.cn-',
    # In eu-central and ap-northeast-2 we support both host styles for S3
    '.eu-central',
    '-eu-central',
    '.ap-northeast-2',
    '-ap-northeast-2',
    '.ap-south-1',
    '-ap-south-1',
    '.us-east-2',
    '-us-east-2',
    '-ca-central',
    '.ca-central',
    '.eu-west-2',
    '-eu-west-2',
]```

from django-storages.

So I solved my problem my switching to:

boto3==1.4.4
django-storages==1.5.2

Not sure what the problem was, but I am using eu-central-1. Maybe that is the issue, but anyway, seems like its fixed.

Saved my day. Also using eu-central-1

from django-storages.