etianen / django-s3-storage Goto Github PK
View Code? Open in Web Editor NEWDjango Amazon S3 file storage.
License: BSD 3-Clause "New" or "Revised" License
django-s3-storage's People
Contributors
Stargazers
Watchers
Forkers
achiku constantineg1 devendralaulkar un33k 1e0ng alexkahn tony-chenl jeffcjohnson dylanmccall areahq attakei jjlorenzo moraga paylasso georgewhewell redking1 shauns valohai jschneier benedictking matthiask sysradium kencochrane adaptiware flimm flo-dhalluin mgalgs darkmattervale asagami twistopayments myang32 adamyala funkydunc pchristos enriquepablo mitchelljkotler michael-k hramezani bhargavaganti michielvermeir ckirby fdemmer emesik multiplicom chrisjakuta philimol pogro micmalti michaelanckaert heckad thelebster bellyfat ipmb richardarpanet quantbulls riggedcoinflip rooseveltiny dduong42 mcallistersean ryanbagwell hzdg djaizz kevinmarsh samulihakkinen somyaranjan26 almost bramverhelst cedriccook yananliu1115 casualuser uxio0 alexanderpg96 gorslawa silnychyi routhinator tabdon roriz jnns codenature dukepan2005 xhonino brahimmade shariforz arpitjain799 or 1928d machtfit rollingstonedjango-s3-storage's Issues
Setting to disable GZIP compression?
Amazon CloudFront recently added (opt-in) support for auto-magic transparent GZIP compression.
This means it is no longer needed to pre-compress the S3 objects for gzipped serving over HTTP if you use CloudFront and opted in to this.
So it would be cool if we'd have a setting to disable the GZIP compression in django-s3-storage (maybe with a use-case note in the readme in case people missed the CF announcement).
New point relase?
Would it be possible to get a new point release up on pypi now those 2 issues got merged?
403 error with version 1.6.3 of boto3
django version: 2.0.2
django-s3-storage version: 0.12.4
boto3 version: 1.6.3
Uploading a file then accessing the url results in a 403 error with the message:
The request signature we calculated does not match the signature you provided. Check your key and signing method.
Last version of boto3 I used that made django-s3-storage work was 1.5.2. This time version 1.6.3 autoinstalled as a dep and it threw the above error.
As always, thank you for such a wonderful module.
In AWS Lambda, InvalidAccessKeyId error unless AWS_SESSION_TOKEN is given
Using django-s3-storage in Lambda (for example with zappa) will throw InvalidAccessKey error because it doesn't give AWS_SESSION_TOKEN which is required for temporary security credentials given by Lambda. e.g.
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.
How to fix this:
- Support
AWS_SESSION_TOKENin Djangosettings.py(and also add it to README/docs) - Pass that token to boto3
Exists method returning a false positive
The exists method returns a false positive when a file with a similar name exists within the bucket.
I did comment on the comment on the commit in question but have copied is here for completeness.
This causes false positives for keys that have the same prefix.
"css/main.css.map" existing on s3 for "css/main.css" would cause this to return true.
Do we need to check for directory existence in this method? the docs state this is just for files?
Looking further this looks like it was changed in relation to #16 the Django docs does state
Returns True if a file referenced by the given name already exists in the storage system, or False if the name is available for a new file.
But is is used with a directory in the collectstatic --clear command: https://github.com/django/django/blob/master/django/contrib/staticfiles/management/commands/collectstatic.py#L106
Question: Migration from OG django-storages
Anyone have advice on doing a migration from the OG django-storages to this storage backend?
I serve a ton of images and would like know if this would be a good fit.
I'm going to give it a go anyway and would be happy to document the process.
File name should use forward slash even in Windows
I'm using ImageField and django-imagekit with django-s3-storage and I noticed that file name of a ImageField/FileField is using Windows path separators, e.g.
"photo_id": "family-child\\26-abu-bakar.jpg",
"photo_url": "https://s3.amazonaws.com/media.keluargasamara.com/prd/family-child/26-abu-bakar.jpg",
"thumbnail_id": "CACHE\\images\\family-child\\26-abu-bakar\\352a50472bc938d5e792c202401046be.jpg",
"thumbnail_url": "https://s3.amazonaws.com/media.keluargasamara.com/prd/CACHE/images/family-child/26-abu-bakar/352a50472bc938d5e792c202401046be.jpg",
Note that I recently migrated from django-storages due to this bug. django-storages properly used forward slash, but it made django-imagekit unusable (For now).
I presume that this bug doesn't occur on Linux OS but the behavior should be consistent on Windows, because these paths are S3 paths, they're not Windows paths.
Collectstatic fail with some files
When I try to collectstatic it fails, it seems like is trying to sync non existent files.
The folder fonts not even exist there '''plugins/bower_components/jquery-wizard-master/libs/fonts/'''
Post-processing 'plugins/bower_components/jquery-wizard-master/libs/bootstrap/bootstrap.min.css' failed!
Traceback (most recent call last):
File "./manage.py", line 15, in <module>
execute_from_command_line(sys.argv)
File "/home/ubuntu/apps/env/lib/python3.5/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
utility.execute()
File "/home/ubuntu/apps/env/lib/python3.5/site-packages/django/core/management/__init__.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/ubuntu/apps/env/lib/python3.5/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/ubuntu/apps/env/lib/python3.5/site-packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/home/ubuntu/apps/env/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 189, in handle
collected = self.collect()
File "/home/ubuntu/apps/env/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 135, in collect
raise processed
File "/home/ubuntu/apps/env/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 296, in _post_process
content = pattern.sub(converter, content)
File "/home/ubuntu/apps/env/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 197, in converter
force=True, hashed_files=hashed_files,
File "/home/ubuntu/apps/env/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 134, in _url
hashed_name = hashed_name_func(*args)
File "/home/ubuntu/apps/env/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 345, in _stored_name
cache_name = self.clean_name(self.hashed_name(name))
File "/home/ubuntu/apps/env/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 94, in hashed_name
raise ValueError("The file '%s' could not be found with %r." % (filename, self))
ValueError: The file 'plugins/bower_components/jquery-wizard-master/libs/fonts/glyphicons-halflings-regular.eot' could not be found with <django_s3_storage.storage.ManifestStaticS3Storage object at 0x7f2e58f163c8>.
An error occurred (InvalidArgument) when calling the ListObjectsV2 operation
When I upload a file, I get the below error.
I have also try django-storages, when I use boto2, everthing is ok. but when I user boto3, I got the same error? Are there some bugs?
Traceback:
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
132. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
58. return view_func(*args, **kwargs)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/rest_framework/viewsets.py" in view
83. return self.dispatch(request, *args, **kwargs)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
483. response = self.handle_exception(exc)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/rest_framework/views.py" in handle_exception
443. self.raise_uncaught_exception(exc)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
480. response = handler(request, *args, **kwargs)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/rest_framework/mixins.py" in create
21. self.perform_create(serializer)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/rest_framework/mixins.py" in perform_create
26. serializer.save()
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/rest_framework/serializers.py" in save
214. self.instance = self.create(validated_data)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/rest_framework/serializers.py" in create
906. instance = ModelClass.objects.create(**validated_data)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/db/models/manager.py" in manager_method
127. return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/db/models/query.py" in create
348. obj.save(force_insert=True, using=self.db)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/db/models/base.py" in save
734. force_update=force_update, update_fields=update_fields)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/db/models/base.py" in save_base
762. updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/db/models/base.py" in _save_table
846. result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/db/models/base.py" in _do_insert
885. using=using, raw=raw)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/db/models/manager.py" in manager_method
127. return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/db/models/query.py" in _insert
920. return query.get_compiler(using=using).execute_sql(return_id)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
973. for sql, params in self.as_sql():
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in as_sql
931. for obj in self.query.objs
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/db/models/fields/files.py" in pre_save
314. file.save(file.name, file, save=False)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/db/models/fields/files.py" in save
93. self.name = self.storage.save(name, content, max_length=self.field.max_length)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/core/files/storage.py" in save
53. name = self.get_available_name(name, max_length=max_length)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django/core/files/storage.py" in get_available_name
89. while self.exists(name) or (max_length and len(name) > max_length):
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/django_s3_storage/storage.py" in exists
273. Prefix=self._get_key_name(name),
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/botocore/client.py" in _api_call
253. return self._make_api_call(operation_name, kwargs)
File "/home/yzxn1709/git-lab/top_cpt/env/local/lib/python2.7/site-packages/botocore/client.py" in _make_api_call
557. raise error_class(parsed_response, operation_name)
Exception Type: ClientError at /v1/api/package
Exception Value: An error occurred (InvalidArgument) when calling the ListObjectsV2 operation: Unknown
Problems with staticfiles.json while using ManifestStaticS3Storage
Hello @etianen
I'm having trouble setting up static files using django_s3_storage.storage.ManifestStaticS3Storage. While accessing static files via static tag or using collectstatic, I'm getting Exception:
OSError: S3Storage error at u'staticfiles.json': An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
Everything works while using django_s3_storage.storage.StaticS3Storage.
Files not updated on s3
I am trying to push a change to some css files that I have and they aren't overwriting the existing files on s3 even though they had changed. I saw the other issue #43 and thought that this might be related to that "fix". So I'm trying to use 0.11.0. I will let you know how that goes.
This is on Django 1.10.5 and Python 3.4 (on elastic beanstalk).
0.9.11 ManifestStaticS3Storage incompatible with django-grappelli 2.9.1 / django 1.10
I'm getting the following error:
Post-processing 'grappelli/tinymce/jscripts/tiny_mce/themes/simple/skins/o2k7/ui.css' failed!
[...]
ValueError: The file 'grappelli/tinymce/jscripts/tiny_mce/themes/simple/skins/o2k7/../../img/icons.gif' could not be found with <django_s3_storage.storage.ManifestStaticS3Storage object>.
Here's some relevant shell code:
>>> storage.exists('grappelli/tinymce/jscripts/tiny_mce/themes/simple/skins/o2k7/../../img/icons.gif')
False
>>> storage.exists('grappelli/tinymce/jscripts/tiny_mce/themes/simple/img/icons.gif')
True
FWIW this broke when I upgraded django and grappelli, not when I upgraded django-s3-storage. But it seems like a bug in the storage somehow maybe? I don't see anything relevant in the 1.10 release notes.
Automatically remove S3 Key on model delete
Hi,
How can I setup django-s3-storage to "remove S3 Key on model delete"?
If it's not possible, is there any plan?
Feature suggestion: support for reduced redundancy
This would involve:
- adding a new setting
- adding a new keyword argument for S3Storage.init and a new attribute for S3Storage
- passing
reduced_redundancy=self.aws_s3_reduced_redundancyin the call toset_contents_from_file
Add support for "immutable" Cache-Control header
Hey @etianen, something that Whitenoise supports is adding the "immutable" value to the Cache-Control header for static files with hashed filenames. It looks like you already added support for caching hashed files longer, so why not add this wonderful feature?
See here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control and here: http://bitsup.blogspot.de/2016/05/cache-control-immutable.html
I imagine it would be something like a new option: AWS_S3_IMMUTABLE_CACHED_STATIC = True.
Would you be open to me submitting a PR?
Unable pickle File objects with s3 stprage for django celery tasks
EncodeError: Can't pickle <class 'botocore.client.S3'>: attribute lookup botocore.client.S3 failed
django s3 storages does not recognise the bucket name
I am using django-s3-storage==0.11.2 and boto3==1.4.4. These are in the settings.py:
STATIC_URL = '/static/'
STATICFILES_DIRS = [
os.path.join(BASE_DIR, "static"),
]
STATIC_ROOT = os.path.join(BASE_DIR, 'static_cdn')
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media_cdn')
AWS_S3_BUCKET_NAME = "my-bucket-name"
AWS_ACCESS_KEY_ID = 'test_id_x'
AWS_SECRET_ACCESS_KEY = 'test_id_x+test_id_x'
DEFAULT_FILE_STORAGE = "django_s3_storage.storage.S3Storage"
STATICFILES_STORAGE = "django_s3_storage.storage.StaticS3Storage"
AWS_S3_ADDRESSING_STYLE = "auto"
AWS_S3_BUCKET_AUTH_STATIC = False
AWS_S3_MAX_AGE_SECONDS_STATIC = 60 * 60 * 24 * 365 # 1 year.
AWS_S3_BUCKET_AUTH = False
AWS_S3_MAX_AGE_SECONDS = 60 * 60 * 24 * 365 # 1 year.
I have also ran these command:
manage.py s3_sync_meta django.core.files.storage.default_storage
But when I run collectstatic or this command
manage.py s3_sync_meta django.contrib.staticfiles.storage.staticfiles_storage
I get this error:
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "": Bucket name must match the regex "^[a-zA-Z0-9.-_]{1,255}$"
I have already created the bucket, and the bucket name is correct, because this works and does not gives any error:
`s3.meta.client.head_bucket(Bucket='my-bucket-name')`
I don't know what am I missing here? Could you help me out please.
Signature mismatch error
Hi! I get
`boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method.when try tocollectstatic`, though credentials are fine - works perfectly with this library https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html
Not compatible with django = 1.10.1
No storage images in bucket with django =1.10.1, I test in local and prod (apache2.4)
Not show any errors.
Every things is ok, but the image not store in bucket, But when I change django to 1.10 is ok
When I create a file in my bucket Django==1.10.1, every things is ok.
from django.core.files.storage import default_storage
from django.core.files.base import ContentFile
from django.core.cache import cache
default_storage.exists('storage_test')
False
file = default_storage.open('storage_test', 'w')
file.write('storage contents')
file.close()
default_storage.exists('storage_test')
True
(my first issues)
Incompatible with riak-cs
https://github.com/basho/riak_cs
Is supposed to be an s3 compatible store. When using it I get weird errors :
File "/home/flo/code/python/ablums/.venv/lib/python3.5/site-packages/django/db/models/fields/files.py", line 94, in save
self.name = self.storage.save(name, content, max_length=self.field.max_length)
File "/home/flo/code/python/ablums/.venv/lib/python3.5/site-packages/django/core/files/storage.py", line 53, in save
name = self.get_available_name(name, max_length=max_length)
File "/home/flo/code/python/django-s3-storage/django_s3_storage/storage.py", line 385, in get_available_name
return super(S3Storage, self).get_available_name(name, max_length)
File "/home/flo/code/python/ablums/.venv/lib/python3.5/site-packages/django/core/files/storage.py", line 77, in get_available_name
while self.exists(name) or (max_length and len(name) > max_length):
File "/home/flo/code/python/django-s3-storage/django_s3_storage/storage.py", line 295, in exists
return self.exists(name + "/")
File "/home/flo/code/python/django-s3-storage/django_s3_storage/storage.py", line 289, in exists
return bool(results["KeyCount"])
KeyError: 'KeyCount'
The list_objects_v2 does not indeed returns a KeyCount parameters, when I manually call the api.
I have a working PR, that does the trick by checking that a "Content" key exists instead of checking "KeyCount" but not sure if this belongs here, or upstream (boto3).
Using s3_sync_meta causes TypeError exception
Anytime, I try to run s3_sync_meta command, that raises:
TypeError: sync_meta_iter() missing 1 required positional argument: 'self'
So, I end up with this small fix, which simply instantiate storage object before calling sync_meta_iter().
I replaced in s3_sync_meta.py:
# Sync the meta.
for path in storage.sync_meta_iter():with:
for path in storage().sync_meta_iter():And now, I'm able to run s3_sync_meta as expected.
NOTE: This happen on python 3.4.3, so maybe this is related.
s3_sync_meta issue
when trying to sync the meta data after switching from django-storages-redux i get:
django-admin s3_sync_meta django.core.files.storage.default_storage
Syncing meta for django.core.files.storage.default_storage
Traceback (most recent call last):
File "/app/.heroku/python/bin/django-admin", line 11, in <module>
sys.exit(execute_from_command_line())
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/app/.heroku/python/lib/python2.7/site-packages/django_s3_storage/management/commands/s3_sync_meta.py", line 22, in handle
for path in storage.sync_meta_iter():
File "/app/.heroku/python/lib/python2.7/site-packages/django_s3_storage/storage.py", line 368, in sync_meta_iter
for path in sync_meta_impl(""):
File "/app/.heroku/python/lib/python2.7/site-packages/django_s3_storage/storage.py", line 354, in sync_meta_impl
metadata = key.metadata.copy()
AttributeError: 'NoneType' object has no attribute 'metadata'
However, I suspect this is user error; specifically, I'm not sure what is meant by path.to.your.storage in the documentation. I'm trying to run this on my heroku instance, and I'm not at all clear what the 'path' is in this context...
Thanks.
collectstatic -c fails since django 1.9
it seems that since django 1.9 specificaly this PR django/django@87d7824 it breaks the "collectstatic -c" since it tries to validate that the root directory exists (which is empty in out case) because the bucket.get_key fails if the key name is empty
In the case of s3-storages, maybe we can check if the bucket exists or just return true if its empty? (I think I would go for the true if empty)
Here is the Traceback:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/app/.heroku/python/lib/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 176, in handle
collected = self.collect()
File "/app/.heroku/python/lib/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 89, in collect
self.clear_dir('')
File "/app/.heroku/python/lib/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 210, in clear_dir
if not self.storage.exists(path):
File "/app/.heroku/python/lib/python3.4/site-packages/django_s3_storage/storage.py", line 262, in exists
return self._get_key(name).exists()
File "/app/.heroku/python/lib/python3.4/site-packages/django_s3_storage/storage.py", line 194, in _get_key
return self.bucket.get_key(self._get_key_name(name), validate=validate)
File "/app/.heroku/python/lib/python3.4/site-packages/boto/s3/bucket.py", line 183, in get_key
return self.new_key(key_name)
File "/app/.heroku/python/lib/python3.4/site-packages/boto/s3/bucket.py", line 622, in new_key
raise ValueError('Empty key names are not allowed')
ValueError: Empty key names are not allowed
Files being overwritten.
I have a test model with a FileField. I find that if I create two objects of this model, and both objects have the same file name, the second object I create, overwrites the file on S3.
For example, the filename of the first object is test.pdf.
If I create a second object with the same file name test.pdf (but different contents), the test.pdf on S3 gets overwritten, and the first model also now refers to the new test.pdf.
Is this normal behaviour?
Thanks.
Files are always re-uploaded to S3 storage
Bug with manifest storage and relative URLs
Consider Django's base.css: https://github.com/django/django/blob/1.10.5/django/contrib/admin/static/admin/css/base.css#L356
It uses a relative reference to another resource:
background: url(../img/sorting-icons.svg) 0 0 no-repeat;
When it comes to post-processing, we end up calling .exists('../img/sorting-icons.svg') method in this library. _get_key_name returns shaun-test/admin/css/../img/sorting-icons.svg, not shaun-test/admin/img/sorting-icons.svg as expected. This looks to be a regression caused by 8936080
I'll try and get a test case/fix together.
Forcing the cache-control header to bytes causes SignatureDoesNotMatch exception
I encountered a SignatureDoesNotMatch exception while trying to upload files.
I'm using the eu-central-1 region. Apparently this is a recipe for disaster because it uses Amazon's v4 signature algorithm.
I wrote the following minimal reproduction script to emulates what django-s3-storage does:
import os
from boto import s3
s3_connection = s3.connect_to_region(
'eu-central-1',
aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],
aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'],
)
bucket = s3_connection.get_bucket(
os.environ['AWS_S3_BUCKET_NAME'],
validate=False,
)
key = bucket.get_key(
'test.txt',
validate=False,
)
key.set_contents_from_file(
open('test.txt'),
policy='private',
headers={'Content-Type': 'text/plain', 'Cache-Control': b'private, max-age=3600'},
)and I got this exception, which is the same as the one I'm getting when I'm trying to upload a file:
Traceback (most recent call last):
File "test.py", line 24, in <module>
headers={'Content-Type': 'text/plain', 'Cache-Control': b'private, max-age=3600'},
File "/Users/myk/.virtualenvs/fractalideas_com/lib/python3.4/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
chunked_transfer=chunked_transfer, size=size)
File "/Users/myk/.virtualenvs/fractalideas_com/lib/python3.4/site-packages/boto/s3/key.py", line 750, in send_file
chunked_transfer=chunked_transfer, size=size)
File "/Users/myk/.virtualenvs/fractalideas_com/lib/python3.4/site-packages/boto/s3/key.py", line 951, in _send_file_internal
query_args=query_args
File "/Users/myk/.virtualenvs/fractalideas_com/lib/python3.4/site-packages/boto/s3/connection.py", line 664, in make_request
retry_handler=retry_handler
File "/Users/myk/.virtualenvs/fractalideas_com/lib/python3.4/site-packages/boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "/Users/myk/.virtualenvs/fractalideas_com/lib/python3.4/site-packages/boto/connection.py", line 940, in _mexe
request.body, request.headers)
File "/Users/myk/.virtualenvs/fractalideas_com/lib/python3.4/site-packages/boto/s3/key.py", line 884, in sender
response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAJF6PMZVKYXRGWSYQ</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20151124T073315Z
20151124/eu-central-1/s3/aws4_request
705754741114d0b3b9751c7320182dbd4df2bb74b31c4fb997b1411646fcf4b3</StringToSign><SignatureProvided>d2bc85477ef151b21859c28dde1715511852a0aefd4e9bd33ab14c923b2d9faa</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 35 31 31 32 34 54 30 37 33 33 31 35 5a 0a 32 30 31 35 31 31 32 34 2f 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 37 30 35 37 35 34 37 34 31 31 31 34 64 30 62 33 62 39 37 35 31 63 37 33 32 30 31 38 32 64 62 64 34 64 66 32 62 62 37 34 62 33 31 63 34 66 62 39 39 37 62 31 34 31 31 36 34 36 66 63 66 34 62 33</StringToSignBytes><CanonicalRequest>PUT
/test.txt
cache-control:private, max-age=3600
content-length:5
content-md5:2Oj8otwPiW/Xy0ywAxuiSQ==
content-type:text/plain
expect:100-Continue
host:fractalideas-com-dev.s3.eu-central-1.amazonaws.com:443
user-agent:Boto/2.38.0 Python/3.4.3 Darwin/15.0.0
x-amz-acl:private
x-amz-content-sha256:f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2
x-amz-date:20151124T073315Z
cache-control;content-length;content-md5;content-type;expect;host;user-agent;x-amz-acl;x-amz-content-sha256;x-amz-date
f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2</CanonicalRequest><CanonicalRequestBytes>50 55 54 0a 2f 74 65 73 74 2e 74 78 74 0a 0a 63 61 63 68 65 2d 63 6f 6e 74 72 6f 6c 3a 70 72 69 76 61 74 65 2c 20 6d 61 78 2d 61 67 65 3d 33 36 30 30 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 35 0a 63 6f 6e 74 65 6e 74 2d 6d 64 35 3a 32 4f 6a 38 6f 74 77 50 69 57 2f 58 79 30 79 77 41 78 75 69 53 51 3d 3d 0a 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 74 65 78 74 2f 70 6c 61 69 6e 0a 65 78 70 65 63 74 3a 31 30 30 2d 43 6f 6e 74 69 6e 75 65 0a 68 6f 73 74 3a 66 72 61 63 74 61 6c 69 64 65 61 73 2d 63 6f 6d 2d 64 65 76 2e 73 33 2e 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 3a 34 34 33 0a 75 73 65 72 2d 61 67 65 6e 74 3a 42 6f 74 6f 2f 32 2e 33 38 2e 30 20 50 79 74 68 6f 6e 2f 33 2e 34 2e 33 20 44 61 72 77 69 6e 2f 31 35 2e 30 2e 30 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 72 69 76 61 74 65 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 66 32 63 61 31 62 62 36 63 37 65 39 30 37 64 30 36 64 61 66 65 34 36 38 37 65 35 37 39 66 63 65 37 36 62 33 37 65 34 65 39 33 62 37 36 30 35 30 32 32 64 61 35 32 65 36 63 63 63 32 36 66 64 32 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 35 31 31 32 34 54 30 37 33 33 31 35 5a 0a 0a 63 61 63 68 65 2d 63 6f 6e 74 72 6f 6c 3b 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3b 63 6f 6e 74 65 6e 74 2d 6d 64 35 3b 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3b 65 78 70 65 63 74 3b 68 6f 73 74 3b 75 73 65 72 2d 61 67 65 6e 74 3b 78 2d 61 6d 7a 2d 61 63 6c 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 66 32 63 61 31 62 62 36 63 37 65 39 30 37 64 30 36 64 61 66 65 34 36 38 37 65 35 37 39 66 63 65 37 36 62 33 37 65 34 65 39 33 62 37 36 30 35 30 32 32 64 61 35 32 65 36 63 63 63 32 36 66 64 32</CanonicalRequestBytes><RequestId>F178C70B4C87A673</RequestId><HostId>YGQWiH5L4DUm1888IJ4bFcqNMhIVza7Y2XiBOqESwecq/aiuYGVUlxnjHff13eb7nhQN73JomKk=</HostId></Error>
The b prefix on the cache-control value looked odd. I tried removing it and no longer got the exception.
This led me to this code in _get_cache_control:
return force_bytes("{privacy}, max-age={max_age}".format(
privacy = privacy,
max_age = self.aws_s3_max_age_seconds,
)) # Have to use bytes else the space will be percent-encoded. Odd, eh?
I changed it to:
return "{privacy}, max-age={max_age}".format(
privacy = privacy,
max_age = self.aws_s3_max_age_seconds,
)
And uploads worked again.
I'm not sure in which circumstances the space gets %-encoded and how bad it is but it is probably not as bad as not being able to upload files at all :-)
As a workaround, perhaps you could simply remove the space i.e. "{privacy},max-age={max_age}"; I believe that's a valid value as well.
Upload files to "." directory
Hi,
Not sure if this is an issue or me setting things up wrong...most likely the latter. I really don't know where else to ask for support, I've tried IRC #django but did not really get anywhere.
I have it setup with private file upload and static storage and it works as expected, except for the private upload bit. It upload everything to a directory in the bucket called "." so it would be; bucketname/./filethatwasuploaded
I am not sure why or how to get it to upload just to the bucket, I've tried putting in a prefix, but then i get bucketname/prefix/./filethatwasuploaded
I am using:
Django==1.9.6
boto==2.40.0
django-s3-storage==0.9.8
The settings in the settings file is almost identical to the ones in the readme, except for the keys.
Any help would be very much appreciated and thanks for a great package.
Boto3 client error in multi threaded environment.
I am using uwsgi behind django in production with threads enabled. I occasionally get
KeyError
'endpoint_resolver'
from this line django_s3_storage/storage.py in _setup at line 140
if self.settings.AWS_S3_ENDPOINT_URL:
connection_kwargs["endpoint_url"] = self.settings.AWS_S3_ENDPOINT_URL
self.s3_connection = boto3.client("s3", config=Config(
s3={"addressing_style": self.settings.AWS_S3_ADDRESSING_STYLE},
signature_version=self.settings.AWS_S3_SIGNATURE_VERSION,
), **connection_kwargs)
def _setting_changed_received(self, setting, **kwargs):
if setting.startswith("AWS_"):
# HACK: No supported way to close the HTTP session from boto3... :S
self.s3_connection._endpoint.http_session.close()
upon searching I got this from boto3 documentation. Is it possible to do the same? like django-storages does here.
Feature request: Add a dir prefix option to directory
Would be great if I I could add a directory prefix setting to uploaded files so I can use the same bucket for different projects.
Example:
my-s3-bucket-name/project1/media
my-s3-bucket-name/project2/media
Here, project1 and project2 would be added to e.g. django_s3_storage.storage.dir_prefix setting.
"The file cannot be reopened" when used with Django imagekit
I'm using django-s3-storage along with django-imagekit, but there seems to be a problem with the way django-s3-storage opens files. It's easy to reproduce, and I have a full django app and repro instructions for the issue, here: https://github.com/mgalgs/iktest
I was a bit thrown off for a while because django-s3-storage isn't anywhere in the stack trace of the erroneous situation, but @scottmx81 figured out that the test app works fine with django-storages, or by monkey-patching pilkit.open_image like so:
from pilkit.lib import Image
from pilkit import utils
def open_image(target):
try:
target.seek(0)
return Image.open(target)
except ValueError:
return Image.open(target.storage.open(target.name))
utils.open_image = open_imageWith django-storages no monkey-patch is required...
/cc @scottmx81
Upload file without storing whem on disk first
Hello! I'm want to use a s3 storage, can you please tell me - can this module load files directly to s3 storage without storing whem localy first in some /tmp dir?
I have to work with big files and i dont want to store them localy before sending to s3 cloud.
thanks in advance for reply.
Use boto3
It's now the recommended library, and boto has some bugs. #30
Support for ckeditor file-upload
Any plans on adding support for file-uploads to S3 with CKEditor?
Each collectstatic generates a new available name for certain specific files
We have the weirdest issue where every collectstatic copies a specific file to S3 under a new random name (looks like it is generated by the storage API's get_available_name()). This is of course very problematic as it break URL's
In the current project it does this for a .woff file from fontawesome (the one from the django-suit V2 preview).
In another projects this happens for a few different files.
Example, note only this particular .woff file gets duplicated. The weird thing is that the original file name (without the random postfix) is missing.
I feel this might have something to do with eventual consistency? It looks like storage gets syncing, somehow thinks the files changed (incorrectly), then tries to replace it with a delete and create, but hitting a race condition between the delete and create in django's storage mechanism?
This is deployed from EC2 on the official Ubuntu 16 AMI with python 3.6, django 1.11, django-s3-storage 0.11.1. This particular .woff file comes from a "pip install -e git+git://.." directly from django-suit's github.
Support windows
Currently, if you run collectstatic on a windows environment paths get uploaded with windows-style path separators.
Example:
https://s3-ap-southeast-2.amazonaws.com/nextgame-images/admin%5Ccss%5Cbase.css
It would be great if this could automatically convert the paths to the correct unix-style path separators
listdir() doesn't work at bucket root
storage.listdir(path) normalizes path as a bucket key, adds a slash, and then passes this prefix to the boto3 API here:
This doesn't seem to work for the bucket root.
Any of '', '/' or '.' gets normalized to '.' and passing Prefix="./" to list_objects_v2 doesn't work, as shown below.
# expected result with Prefix=""
>>> paginator = self.s3_connection.get_paginator("list_objects_v2")
>>> pages = paginator.paginate(Bucket=self.settings.AWS_S3_BUCKET_NAME, Delimiter="/", Prefix="")
>>> list(pages)
[{'ResponseMetadata': {'RequestId': 'EAC9F38060385E98', 'HostId': 'DjpPUviEQNUvnoBeje+TgUQDwR0QtMWJXR+7nfNxjimsj3I2dBYZ9fTvbwZR1ufG9ZN9FN/LOVs=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'DjpPUviEQNUvnoBeje+TgUQDwR0QtMWJXR+7nfNxjimsj3I2dBYZ9fTvbwZR1ufG9ZN9FN/LOVs=', 'x-amz-request-id': 'EAC9F38060385E98', 'date': 'Sun, 28 May 2017 08:36:24 GMT', 'x-amz-bucket-region': 'eu-west-1', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Name': 'myks-org--photos', 'Prefix': '', 'Delimiter': '/', 'MaxKeys': 1000, 'CommonPrefixes': [{'Prefix': '2002/'}, {'Prefix': '2003/'}, {'Prefix': '2004/'}, {'Prefix': '2005/'}, {'Prefix': '2006/'}, {'Prefix': '2007/'}, {'Prefix': '2008/'}, {'Prefix': '2009/'}, {'Prefix': '2010/'}, {'Prefix': '2011/'}, {'Prefix': '2012/'}, {'Prefix': '2013/'}, {'Prefix': '2014/'}, {'Prefix': '2015/'}, {'Prefix': '2016/'}, {'Prefix': '2017/'}], 'KeyCount': 16}]
# other Prefix values don't work
>>> pages = paginator.paginate(Bucket=self.settings.AWS_S3_BUCKET_NAME, Delimiter="/", Prefix=".")
>>> list(pages)
[{'ResponseMetadata': {'RequestId': '8A7908AE11249562', 'HostId': 'YlaPCrxFiS95+BZ1bhFFLv6Efx9iYvfrM4hm/DvSUmLR5TpACQX9tReixvcLP2NnCkw8yM3lU7k=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'YlaPCrxFiS95+BZ1bhFFLv6Efx9iYvfrM4hm/DvSUmLR5TpACQX9tReixvcLP2NnCkw8yM3lU7k=', 'x-amz-request-id': '8A7908AE11249562', 'date': 'Sun, 28 May 2017 08:36:33 GMT', 'x-amz-bucket-region': 'eu-west-1', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Name': 'myks-org--photos', 'Prefix': '.', 'Delimiter': '/', 'MaxKeys': 1000, 'KeyCount': 0}]
>>> pages = paginator.paginate(Bucket=self.settings.AWS_S3_BUCKET_NAME, Delimiter="/", Prefix="/")
>>> list(pages)
[{'ResponseMetadata': {'RequestId': '364AEA1FB40BB0AE', 'HostId': 'j4Ysb++IZlDgXPbrYw7Jq0DjcQAANECKwBXSzUZT3UMzcKGd8phHcyqZd6rMZxX76/AT5HzE/ew=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'j4Ysb++IZlDgXPbrYw7Jq0DjcQAANECKwBXSzUZT3UMzcKGd8phHcyqZd6rMZxX76/AT5HzE/ew=', 'x-amz-request-id': '364AEA1FB40BB0AE', 'date': 'Sun, 28 May 2017 08:36:40 GMT', 'x-amz-bucket-region': 'eu-west-1', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Name': 'myks-org--photos', 'Prefix': '/', 'Delimiter': '/', 'MaxKeys': 1000, 'KeyCount': 0}]
>>> pages = paginator.paginate(Bucket=self.settings.AWS_S3_BUCKET_NAME, Delimiter="/", Prefix="./")
>>> list(pages)
[{'ResponseMetadata': {'RequestId': '401D677C8330DAEB', 'HostId': 'E2qWXeK/pXA9TTfH2i4cDs+R3zm+Op7GNSRoeTxYLup/V5yDDF2akgRYxdKq9MzX6L2HX1aH2S4=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'E2qWXeK/pXA9TTfH2i4cDs+R3zm+Op7GNSRoeTxYLup/V5yDDF2akgRYxdKq9MzX6L2HX1aH2S4=', 'x-amz-request-id': '401D677C8330DAEB', 'date': 'Sun, 28 May 2017 08:40:26 GMT', 'x-amz-bucket-region': 'eu-west-1', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Name': 'myks-org--photos', 'Prefix': './', 'Delimiter': '/', 'MaxKeys': 1000, 'KeyCount': 0}]
I think we need a special case like this:
path = self._get_key_name(path)
path = "" if path == "." else path + "/"There's a variety of ways to express this logic.
Library not compressing uploaded files
Hi,
I tried to use this library for uploading django files to AWS S3 especially because it allows to compress files, but it doesn't work. Even when I tried to manually define AWS_S3_GZIP = True.
Please help
Replace boto with boto3
Is there any plan to replace boto with boto3 ? The developers at boto prefer say that it is stable enough and projects should start integrating boto3.
Error in readme
```
Could you push a release to PyPI?
Hello,
I've been applying the patch from #26 to projects where I need it. I'd love to use an off-the-shelf version of django-s3-storage instead. When you have time, would it be possible to make a release?
Thanks!
Why should I use django-s3-storage instead of django-storages-redux?
I would find it useful to address this question in the README.
Perhaps you just wrote django-s3-storage because it was fun.
Perhaps you wanted a library that did one thing and did it well. django-storages clearly exihibits kitchen sink syndrome.
Perhaps you wanted a modern and well-documented library. django-storages was abandoned and revived as django-storages-redux; it shows.
Perhaps you needed features that django-storages-redux didn't provide.
I'm interested in trying alternatives to the current de facto solution. It would be an easier decision if I knew why the alternative exists!
Thanks!
Support for the AWS KMS encryption type
This package currently supports AES256 encryption. AWS KMS should be supported as well.
Currently this package uses the boolean AWS_S3_ENCRYPT_KEY to mediate whether to use no encryption or to use AES256 encryption.
A backward compatible change would add another variable (e.g., AWS_S3_ENCRYPT_KEY_ID) to specify the key type iff AWS_S3_ENCRYPT_KEY is true.
A somewhat breaking change would be to use AWS_S3_ENCRYPT_KEY such that if it's empty or None then encryption is disabled, otherwise the encryption type == the value of AWS_S3_ENCRYPT_KEY. This should still act correctly if encryption isn't being used, but if it is, then the AES256 case would need to be explicitly stated (or assumed to be the case if the value is a boolean value of True).
Edit: in addition to ServerSideEncryption, optional params Key and SSEKMSKeyId would need to be handled as well.
Your thoughts?
sorl-thumbnail is incompitable with django-s3-storage
I am trying to run this command, which is part of sorl-thumbnail:
$ python manage.py thumbnail clear_delete_all
Clear the Key Value Store ... [Done]
Starting new HTTPS connection (1): s3-eu-west-1.amazonaws.com
Delete all thumbnail files in THUMBNAIL_PREFIX ... Traceback (most recent call last):
File "manage.py", line 20, in <module>
execute_from_command_line(sys.argv)
File "python/lib/python3.6/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
utility.execute()
File "python/lib/python3.6/site-packages/django/core/management/__init__.py", line 355, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "python/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "python/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "python/lib/python3.6/site-packages/sorl/thumbnail/management/commands/thumbnail.py", line 74, in handle
delete_all_thumbnails()
File "python/lib/python3.6/site-packages/sorl/thumbnail/images.py", line 222, in delete_all_thumbnails
path = os.path.join(storage.location, settings.THUMBNAIL_PREFIX)
File "python/lib/python3.6/site-packages/django/utils/functional.py", line 239, in inner
return func(self._wrapped, *args)
AttributeError: 'S3Storage' object has no attribute 'location'
It seems that sorl-thumbnail expects there to be a location attribute on S3Storage.
Document required IAM policies
I'm having a bit of trouble figuring out what the most restrictive IAM policy for S3 storage is. I'd expect a simple read/write policy like this to work (but it doesn't)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::my-bucket"]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": ["arn:aws:s3:::my-bucket/*"]
}
]
}I've googled around a bit, and I see the following a lot. It seems to work, but it feels a bit to broad and cargo-culted, especially the "s3:Object" wildcards (it includes Torrents, Version en ACL stuff).
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListBucketMultipartUploads",
"s3:ListBucketVersions"
],
"Resource": [
"arn:aws:s3:::my-bucket"
]
},
{
"Effect": "Allow",
"Action": [
"s3:*Object*",
"s3:ListMultipartUploadParts",
"s3:AbortMultipartUpload"
],
"Resource": [
"arn:aws:s3:::my-bucket/*"
]
}
]
}S3 custom domain and collectstatic
I have a cdn.domain.com that points to a s3 bucket with the same name. In my settings for this app I've added:
AWS_S3_CALLING_FORMAT_STATIC = 'boto.s3.connection.VHostCallingFormat'
AWS_S3_HOST_STATIC = 'cdn.domain.com'
and URLs that are generated by {% static %} now work properly. I still have one problem though - when I run collectstatic command I get:
<Error>
<Code>InvalidRequest</Code>
<Message>
The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
</Message>
<RequestId>FCA19FC7046E571A</RequestId>
<HostId>...</HostId>
</Error>
If I remove AWS_S3_CALLING_FORMAT_STATIC and AWS_S3_HOST_STATIC settings then collectstatic runs fine and I can see files being uploaded to S3. Am I missing some other setting here?
Changes to caching
I'd like to suggest some changes to the way django-s3-storage handles the caching of static files.
The first issue is that the default expiration time—1 year—is too long. That default applies not just to the versioned ManifestStaticS3Storage files but to the regular StaticS3Storage files as well. By comparison, django-storages doesn't set the header at all, by default, and whitenoise uses either 0 or 60 seconds, depending on DEBUG.
It's important to set a conservative default here because once a user's browser has cached a file there's no way for the developer to force it to refresh. So someone who starts with the default value and then decides to change it later will find that it's too late for some users.
The second issue is that ManifestStaticS3Storage doesn't distinguish between the versioned (e.g. myfile.123456.css) and the unversioned (myfile.css) files when it comes to caching. The way this should work is that the versioned files get cached forever, while the unversioned files get cached for a relatively short time. whitenoise gets this right, using the filename to figure out if it's a versioned file or not and setting the expiration time accordingly.
So my suggestions are:
- Lower the default expiration time (perhaps to one hour, to match the file storage default, or lower).
- With
ManifestStaticS3Storage, set the expiration time differently depending on whether or not it's a versioned file. If it is, set it to cache forever. If it isn't, use theAWS_S3_MAX_AGE_SECONDS_STATICsetting.
IOError: S3Storage error on Heroku
With version 0.11.0, Python 2.7.13
I ran the collectstatic command manually, from a Heroku bash shell with --clear. I then tried to do an automated build (normal Heroku deploy) and got this error.
-----> Python app detected
$ pip install -r requirements.txt
$ python manage.py collectstatic --noinput
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py", line 294, in run_from_argv
self.execute(*args, **cmd_options)
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py", line 345, in execute
output = self.handle(*args, **options)
File "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 193, in handle
collected = self.collect()
File "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 124, in collect
handler(path, prefixed_path, storage)
File "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 349, in copy_file
if not self.delete_file(path, prefixed_path, source_storage):
File "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 258, in delete_file
target_last_modified = self.storage.get_modified_time(prefixed_path)
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/files/storage.py", line 230, in get_modified_time
dt = self.modified_time(name)
File "/app/.heroku/python/lib/python2.7/site-packages/django_s3_storage/storage.py", line 322, in modified_time
return make_naive(self.meta(name)["LastModified"])
File "/app/.heroku/python/lib/python2.7/site-packages/django_s3_storage/storage.py", line 32, in _do_wrap_errors
raise IOError("S3Storage error at {!r}: {}".format(name, force_text(ex)))
IOError: S3Storage error at u'rest_framework/fonts/glyphicons-halflings-regular.woff': An error occurred (404) when calling the HeadObject operation: Not Found
! Error while running '$ python manage.py collectstatic --noinput'.
See traceback above for details.
You may need to update application code to resolve this error.
Or, you can disable collectstatic for this application:
$ heroku config:set DISABLE_COLLECTSTATIC=1
https://devcenter.heroku.com/articles/django-assets
Downgrading to 0.9.11 resolved the issue.
Manually assigning existing S3 file to ImageField
If I uploaded a file to s3 without django-s3-storage how easy is it to assign that file to a model without downloading it of s3 and uploading again?
Should I instantiate an ImageField and assign some django_s3_storage.storage.S3File object? If so then which fields are mandatory?
Thanks.
In some instances, files are not getting copied over to s3 correctly.
Depending on what you have for file names, some of the files are not getting copied over correctly.
If you have the following two files:
- assets/bootstrap/css/bootstrap.min.css.map
- assets/bootstrap/css/bootstrap.min.css
It will copy over the first one bootstrap.min.css.map but it will not copy over the second one bootstrap.min.css. This is because of the way we determine if a file exists or not.
Look at the exists method found here: https://github.com/etianen/django-s3-storage/blob/master/django_s3_storage/storage.py#L269
def exists(self, name):
# We also need to check for directory existence, so we'll list matching
# keys and return success if any match.
results = self.s3_connection.list_objects_v2(
Bucket=self.settings.AWS_S3_BUCKET_NAME,
MaxKeys=1,
Prefix=self._get_key_name(name),
)
return bool(results["KeyCount"])
What this method does is it will do a listing of the bucket using the prefix of the given name. When it wants to see if assets/bootstrap/css/bootstrap.min.css exists it will list the contents of the bucket, using the prefix of assets/bootstrap/css/bootstrap.min.css and return at most 1 key.
This method will return True even if assets/bootstrap/css/bootstrap.min.css isn't there if the assets/bootstrap/css/bootstrap.min.css.map file is there because the prefix of assets/bootstrap/css/bootstrap.min.css matches the file assets/bootstrap/css/bootstrap.min.css.map. So you end up with a false positive. If the files doesn't exists, it will use the get_available_name() method from here: https://github.com/django/django/blob/master/django/core/files/storage.py#L60 to generate a random file name and store that.
What happens is that the file is copied over, but the name was changed, and thus is unavailable when you try and fetch from s3 directly.
The fix is to do something like this.
def exists(self, name):
try:
self.s3_connection.head_object(
Bucket=self.settings.AWS_S3_BUCKET_NAME,
Key=self._get_key_name(name))
return True
except ClientError as e:
if e.response['Error']['Code'] == "404":
return False
else:
raise e
This code makes a head call for the object, and if it returns with no error, you know it exists, if it returns an error, check to make sure it is a 404, and not another error. If a 404, then we know the file doesn't exists.
Similar code can be found here: https://github.com/jschneier/django-storages/blob/master/storages/backends/s3boto3.py#L452-L467 and https://stackoverflow.com/questions/33842944/check-if-a-key-exists-in-a-bucket-in-s3-using-boto3
I can submit a PR for the fix, but I had one question around the comment in the exists method.
# We also need to check for directory existence, so we'll list matching
# keys and return success if any match.
It says that we need to also check for directory existence. My code from above doesn't work for directories, so we will need a solution for that use case. Do you know if that comment is still relevant?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.