Directory does not exist about django-filebrowser HOT 43 CLOSED

besides the breaking tests, are you able to browse your media-directory with the filebrowser?

from django-filebrowser.

No. Unfortunately as long as I have filebrowser enabled my entire application throws that same exception.

from django-filebrowser.

@vam:
do you have any idea why this line:
https://github.com/sehmaschine/django-filebrowser/blob/master/filebrowser/sites.py#L117
throws an error with s3boto? it relies on self.storage exists which - in that case - ends with a path to .../static/filebrowser/ which again is really strange.

from django-filebrowser.

After clearing my pyc's I'm getting a different error. Still not working but the directory is now showing the proper directory in the traceback. I did login to S3 and create an "uploads" directory. Not sure why it's saying there isn't one.

$ ./manage.py test filebrowser
Traceback (most recent call last):
File "./manage.py", line 14, in
execute_manager(settings)
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/core/management/init.py", line 438, in execute_manager
utility.execute()
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/core/management/init.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/core/management/base.py", line 191, in run_from_argv
self.execute(_args, *_options.dict)
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/core/management/base.py", line 220, in execute
output = self.handle(_args, *_options)
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/south/management/commands/test.py", line 8, in handle
super(Command, self).handle(_args, *_kwargs)
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/core/management/commands/test.py", line 37, in handle
failures = test_runner.run_tests(test_labels)
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/test/simple.py", line 358, in run_tests
suite = self.build_suite(test_labels, extra_tests)
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/test/simple.py", line 248, in build_suite
suite.addTest(build_suite(app))
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/test/simple.py", line 77, in build_suite
test_module = get_tests(app_module)
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/test/simple.py", line 35, in get_tests
test_module = import('.'.join(app_path + [TEST_MODULE]), {}, {}, TEST_MODULE)
File "/Users/greg/.virtualenvs/newyork.com/src/django-filebrowser/filebrowser/tests/init.py", line 2, in
from filebrowser.tests.base import FileObjectPathTests, FileObjectVersionTests, FileObjectUnicodeTests
File "/Users/greg/.virtualenvs/newyork.com/src/django-filebrowser/filebrowser/tests/base.py", line 17, in
from filebrowser.sites import site
File "/Users/greg/.virtualenvs/newyork.com/src/django-filebrowser/filebrowser/sites.py", line 524, in
site = FileBrowserSite(name='filebrowser')
File "/Users/greg/.virtualenvs/newyork.com/src/django-filebrowser/filebrowser/sites.py", line 109, in init
self.directory = DIRECTORY
File "/Users/greg/.virtualenvs/newyork.com/src/django-filebrowser/filebrowser/sites.py", line 118, in _directory_set
raise ImproperlyConfigured("Directory '%(dir)s' for the site %(app_name)s.%(name)s does not exist." % {'dir': val, 'app_name': self.app_name, 'name': self.name})
django.core.exceptions.ImproperlyConfigured: Directory 'uploads/' for the site filebrowser.filebrowser does not exist.

from django-filebrowser.

On line 117 of sites.py it's evaluating self.storage.exists(val).
What exactly is it looking for here?

self.storage.exists returns bound method S3BotoStorage.exists of <storages.backends.s3boto.S3BotoStorage object at 0x1038c9c10
self.storage.exists(val) is false

from django-filebrowser.

try to set DIRECTORY to "" if "uploads/" doesn´t exist on your server.

exists for s3boto is defined here:
http://code.welldev.org/django-storages/src/11bef0c2a410/storages/backends/s3boto.py#cl-204

you may wanna add some print-statements (e.g. print name) to the exists-function in order to exactly find out what´s going on.

from django-filebrowser.

Reading the s3boto code now...

I changed the directory to "" and got further:

Traceback:
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response

1.                     response = callback(request, _callback_args, *_callback_kwargs)
   

   File "/Users/greg/.virtualenvs/newyork.com/src/django-filebrowser/filebrowser/decorators.py" in decorator

2.     if get_path('', site=site) == None:
   

   File "/Users/greg/.virtualenvs/newyork.com/src/django-filebrowser/filebrowser/functions.py" in get_path

3. if path.startswith('.') or os.path.isabs(path) or not site.storage.isdir(os.path.join(site.directory, path)):
   

   File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/utils/functional.py" in getattr

4.     return getattr(self._wrapped, name)
   

Exception Type: AttributeError at /admin/filebrowser/browse/
Exception Value: 'S3BotoStorage' object has no attribute 'isdir'

from django-filebrowser.

we´re on the right track now (thanks again for looking into that).

@vam: guess that´s because the filebrowser storage-mixin defines some additional properties (see http://readthedocs.org/docs/django-filebrowser/en/latest/file_storages.html#storagemixin-class). do you have any idea for a workaround?

from django-filebrowser.

So it seems the first issue was non-existing directory on your s3 server (the constructor of FileBrowserSite needs a face-lift - dirctory should be a keyword argument)

The second issue: When using anything else than FileSystemStorage, it's necessary extend the storage class and implement some of the required methods (StorageMixin class in filebrowser/storage.py defines those methods and can be used as a mixin class) See the use-case at https://github.com/sehmaschine/django-filebrowser/blob/master/filebrowser/sites.py#L33

from django-filebrowser.

I don't see another workaround than implementing those methods for s3boto

from django-filebrowser.

so ... that would be (in urls.py):

from filebrowser.sites import site
from filebrowser.storage import FileSystemStorageMixin
from django.core.files.storage import default_storage

if FileSystemStorageMixin not in default.__bases__:
    default.__bases__ += (FileSystemStorageMixin,)

site.storage = default_storage

right?

from django-filebrowser.

Tried that Patrick. name 'default' is not defined.

from django-filebrowser.

sorry, it´s default_storage, not "default". again:

from filebrowser.sites import site
from filebrowser.storage import FileSystemStorageMixin
from django.core.files.storage import default_storage

if FileSystemStorageMixin not in default_storage.__bases__:
    default_storage.__bases__ += (FileSystemStorageMixin,)

site.storage = default_storage

from django-filebrowser.

Two steps forward, one step back? lol
A new one...

Traceback:
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response

1.                 response = middleware_method(request)
   

   File "/Users/greg/.virtualenvs/newyork.com/src/django-debug-toolbar/debug_toolbar/middleware.py" in process_request

2.             original_urlconf = **import**(getattr(request, 'urlconf', settings.ROOT_URLCONF), {}, {}, ['*'])
   

   File "/Users/greg/dev/newyork.com/newyork/../newyork/urls.py" in
3. if FileSystemStorageMixin not in default_storage.bases:
   File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/utils/functional.py" in getattr

4.     return getattr(self._wrapped, name)
   

Exception Type: AttributeError at /admin/filebrowser/browse/
Exception Value: 'S3BotoStorage' object has no attribute 'bases'

from django-filebrowser.

to @vam again:
is there another way to extend a class? see http://stackoverflow.com/questions/3193158/bases-doesnt-work-whats-next

seems that bases only works on objects ...

from django-filebrowser.

@patrick something similar, however, we have no mixin class for s3boto implemented. it's not possible to use FileSystemStorageMixin with s3. I'm currently looking into http://code.welldev.org/django-storages/src/11bef0c2a410/storages/backends/s3boto.py to see how difficult would be to provide those extra 6 methods

from django-filebrowser.

@vam I've already given Rich Leland a heads up (he's a maintainer of storages). I'm sure he'd be happy to merge those in if you come up with something.

from django-filebrowser.

alright, I think we´re getting closer. we should at least try to solve this, since django-storages seem to be the standard when it comes to alternative storage-engines with django.

from django-filebrowser.

the way to mix-in the class to an object would be to use: default_storage.__class__.__bases__ However, as long we're trying to extend s3storage class with FileSystemMixinClass, we're getting nowhere. What you can try greg, is this:

from filebrowser.sites import site
from filebrowser.storage import StorageMixin
from django.core.files.storage import default_storage

if StorageMixin not in default_storage.__class__.__bases__:
    default_storage.__class__.__bases__ += (StorageMixin,)

site.storage = default_storage

That would make sense and you will get 'NotImplementedError' instead of 'AttributeError', the question is, can we call it a progress? ;)

from django-filebrowser.

@vam yes, it does throw the 'NotImplementedError' but like you said doesn't get any closer:

Traceback:
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response

1.                     response = callback(request, _callback_args, *_callback_kwargs)
   

   File "/Users/greg/.virtualenvs/newyork.com/src/django-filebrowser/filebrowser/decorators.py" in decorator

2.     if get_path('', site=site) == None:
   

   File "/Users/greg/.virtualenvs/newyork.com/src/django-filebrowser/filebrowser/functions.py" in get_path

3. if path.startswith('.') or os.path.isabs(path) or not site.storage.isdir(os.path.join(site.directory, path)):
   

   File "/Users/greg/.virtualenvs/newyork.com/src/django-filebrowser/filebrowser/storage.py" in isdir

4.     raise NotImplementedError()
   

Exception Type: NotImplementedError at /admin/filebrowser/browse/
Exception Value:

from django-filebrowser.

@greg I'm trying to implement the s3mixin class right now, i see a quick and easy way, which unfortunately has efficiency issues. hope to get back to you soon.

from django-filebrowser.

@vam thanks!

On Tue, Nov 1, 2011 at 1:43 PM, Vaclav Mikolasek <
[email protected]>wrote:

@greg I'm trying to implement the s3mixin class right now, i see a quick
and easy way, which unfortunately has efficiency issues. hope to get back
to you soon.

Reply to this email directly or view it on GitHub:

#40 (comment)

from django-filebrowser.

The followin class should solve some of the problem.. please note i did not test it. Note also that some things make still not be working - like creation of a directory. Pull the latest FB version (i've committed some changes that you'll need as well)

IMPORTANT: don't test it on production data.

from filebrowser.storage import StorageMixin

class S3BotoStorageMixin(StorageMixin):

    def isfile(self, name):
        return self.exists(name)

    def isdir(self, name):
        # That's some inefficient implementation...
        # If there are some files having 'name' as their prefix, then 
        # the name is considered to be a directory
        name = self._normalize_name(self._clean_name(name))
        dirlist = self.bucket.list(self._encode_name(name))

        # Check whether the iterator is empty
        for item in dirlist:
            return True
        return False

    def move(self, old_file_name, new_file_name, allow_overwrite=False):

        if self.exists(new_file_name):
            if allow_overwrite:
                self.delete(new_file_name)
            else:
                raise "The destination file '%s' exists and allow_overwrite is False" % new_file_name

        old_key_name = self._encode_name(self._normalize_name(self._clean_name(old_file_name)))
        new_key_name = self._encode_name(self._normalize_name(self._clean_name(new_file_name)))

        k = self.bucket.copy_key(new_key_name, self.bucket.name, old_key_name)

        if not k:
            raise "Couldn't copy '%s' to '%s'" % (old_file_name, new_file_name)

        self.delete(old_file_name)

    def makedirs(self, name):
        pass

    def rmtree(self, name):
        name = self._normalize_name(self._clean_name(name))
        dirlist = self.bucket.list(self._encode_name(name))
        for item in dirlist:
            item.delete()

you need to mix in the class to your storage object, assuming it's called 's3storage':

if S3BotoStorageMixin not in s3storage.__class__.__bases__:
    s3storage.__class__.__bases__ += (S3BotoStorageMixin,)

site.storage = s3storage

thanks a lot for looking into that!

from django-filebrowser.

@vam
This seems to be working fairly well. When I upload an image though it creates a directory using the name of the file I uploaded instead. During the upload process the progress bar flashes to 100% and then never leaves the page. Switching back to the parent directory I see the folder it just created named "imagename.jpg"

from django-filebrowser.

Actually @vam, a correction.
It's reading all images as folders. The upload I did actually did upload the image. Filebrowser thinks it's a directory. I'm assuming this is a problem with isfile or isdir?

The uploader never does take me back to the parent directory after upload or show me the file I uploaded. I don't know which is supposed to happen.

Due to this behavior, using the insert image action from tinymce toolbar obviously isn't working since it traverses it as a directory.

from django-filebrowser.

the isdir() method is the culprit, i suppose. Could you please try this version:

def isdir(self, name):
        # That's some inefficient implementation...
        # If there are some files having 'name' as their prefix, then 
        # the name is considered to be a directory
        if self.isfile(name):
            return False

        name = self._normalize_name(self._clean_name(name))
        dirlist = self.bucket.list(self._encode_name(name))

        # Check whether the iterator is empty
        for item in dirlist:
            return True
        return False

from django-filebrowser.

That throws this exception @vam

Traceback:
File "/Users/greg/.virtualenvs/newyork.com/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response

1.                     response = callback(request, _callback_args, *_callback_kwargs)
   

   File "/Users/greg/.virtualenvs/newyork.com/src/django-filebrowser/filebrowser/decorators.py" in decorator

2.         raise ImproperlyConfigured, _("Error finding Upload-Folder (MEDIA_ROOT + DIRECTORY). Maybe it does not exist?")
   

Exception Type: ImproperlyConfigured at /admin/filebrowser/browse/
Exception Value: Error finding Upload-Folder (MEDIA_ROOT + DIRECTORY). Maybe it does not exist?

from django-filebrowser.

I see.. does this fixes it:

def isdir(self, name):
        # That's some inefficient implementation...
        # If there are some files having 'name' as their prefix, then 
        # the name is considered to be a directory
        if not name: # Empty name is a directory
            return True

        if self.isfile(name):
            return False

        name = self._normalize_name(self._clean_name(name))
        dirlist = self.bucket.list(self._encode_name(name))

        # Check whether the iterator is empty
        for item in dirlist:
            return True
        return False

from django-filebrowser.

Yes. That fixed it @vam!! Thanks!

Now I can browse the dirs, delete images and upload. The select from tinymce now works as well.

One final question @vam. When uploading, should it be redirecting after upload? The progress bar just flashes to 100% then stays on the same view. The upload however is successful.

from django-filebrowser.

Glad it worked! Yeah, the new ajax uploader does not redirect you - it stays at the upload view. That's correct.

The S3BotoStorageMixin class is not really complete -- creation of a directory will not work for sure. I suppose that tests will fail on that.. But for now, i'd leave it at that. Let me know if some further issues poped up...

from django-filebrowser.

Wasn't seeing this on my local instance while testing but on the staging server at rackspace it's throwing a 500 when I try and look inside a directory in filebrowser. Is this a s3boto issue or filebrowser?

Traceback (most recent call last):

File "/sites/.virtualenvs/newyork/lib/python2.6/site-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, _callback_args, *_callback_kwargs)

File "/sites/.virtualenvs/newyork/src/django-filebrowser/filebrowser/decorators.py", line 29, in decorator
return function(request, _args, *_kwargs)

File "/sites/.virtualenvs/newyork/lib/python2.6/site-packages/django/contrib/admin/views/decorators.py", line 19, in _checklogin
return view_func(request, _args, *_kwargs)

File "/sites/.virtualenvs/newyork/lib/python2.6/site-packages/django/views/decorators/cache.py", line 79, in _wrapped_view_func
response = view_func(request, _args, *_kwargs)

File "/sites/.virtualenvs/newyork/src/django-filebrowser/filebrowser/sites.py", line 231, in browse
listing = filelisting.files_listing_filtered()

File "/sites/.virtualenvs/newyork/src/django-filebrowser/filebrowser/base.py", line 127, in files_listing_filtered
listing = filter(self.filter_func, self.files_listing_total())

File "/sites/.virtualenvs/newyork/src/django-filebrowser/filebrowser/base.py", line 104, in files_listing_total
files = sort_by_attr(files, self.sorting_by)

File "/sites/.virtualenvs/newyork/src/django-filebrowser/filebrowser/functions.py", line 120, in sort_by_attr
intermed = map(None, map(getattr, seq, (attr,)*len(seq)), xrange(len(seq)), seq)

File "/sites/.virtualenvs/newyork/src/django-filebrowser/filebrowser/base.py", line 232, in _date
self._date_stored = time.mktime(self.site.storage.modified_time(self.path).timetuple())

File "/sites/.virtualenvs/newyork/src/django-storages/storages/backends/s3boto.py", line 241, in modified_time
raise NotImplementedError()

NotImplementedError

from django-filebrowser.

This seems to be s3boto issue. Check you have the latest version of that
backend.

On Nov 3, 2011 8:17 PM, "Greg Newman" <
[email protected]>
wrote:

Wasn't seeing this on my local instance while testing but on the staging
server at rackspace it's throwing a 500 when I try and look inside a
directory in filebrowser. Is this a s3boto issue or filebrowser?

Traceback (most recent call last):

File
"/sites/.virtualenvs/newyork/lib/python2.6/site-packages/django/core/handlers/base.py",
line 111, in get_response
response = callback(request, _callback_args, *_callback_kwargs)

File
"/sites/.virtualenvs/newyork/src/django-filebrowser/filebrowser/decorators.py",
line 29, in decorator
return function(request, _args, *_kwargs)

File
"/sites/.virtualenvs/newyork/lib/python2.6/site-packages/django/contrib/admin/views/decorators.py",
line 19, in _checklogin
return view_func(request, _args, *_kwargs)

File
"/sites/.virtualenvs/newyork/lib/python2.6/site-packages/django/views/decorators/cache.py",
line 79, in _wrapped_view_func
response = view_func(request, _args, *_kwargs)

File
"/sites/.virtualenvs/newyork/src/django-filebrowser/filebrowser/sites.py",
line 231, in browse
listing = filelisting.files_listing_filtered()

File
"/sites/.virtualenvs/newyork/src/django-filebrowser/filebrowser/base.py",
line 127, in files_listing_filtered
listing = filter(self.filter_func, self.files_listing_total())

File
"/sites/.virtualenvs/newyork/src/django-filebrowser/filebrowser/base.py",
line 104, in files_listing_total
files = sort_by_attr(files, self.sorting_by)

File
"/sites/.virtualenvs/newyork/src/django-filebrowser/filebrowser/functions.py",
line 120, in sort_by_attr
intermed = map(None, map(getattr, seq, (attr,)*len(seq)),
xrange(len(seq)), seq)

File
"/sites/.virtualenvs/newyork/src/django-filebrowser/filebrowser/base.py",
line 232, in _date
self._date_stored =
time.mktime(self.site.storage.modified_time(self.path).timetuple())

File
"/sites/.virtualenvs/newyork/src/django-storages/storages/backends/s3boto.py",
line 241, in modified_time
raise NotImplementedError()

NotImplementedError

---

Reply to this email directly or view it on GitHub:

#40 (comment)

from django-filebrowser.

How is it going greg, is this issue ready to be closed?

from django-filebrowser.

Hey @vam
Everything is working but it's super slow to retrieve the directories and files from S3.
I don't think it's related to this ticket so feel free to close it.
Thanks again for your help!

from django-filebrowser.

You're welcome.

Would be curious to see what makes it so slow. Can't test it myself, so if you eventually figure that out, please let me know.

from django-filebrowser.

Is there a walkthru of this somewhere for integrating storages, s3 & filebrowser? I'm following the comments here but running into errors.

from django-filebrowser.

not really. there's a bit of some info here: http://readthedocs.org/docs/django-filebrowser/en/latest/file_storages.html

if that does not help, open a new issue..

from django-filebrowser.

Just a note:

Even if you provide a mixing for S3Storage, filebrowser won't work well (if at all) with S3 since things like file sizes and images dimensions trigger a read for each file. That means at least 1 (list) + N (get) operations for a directory with N files in it. Some kind of caching is necessary in the case of remote file storages, and those things that trigger unnecessary reads (file sizes, image sizes) have to be removed.

That's a good task for the core-refactor branch, by the way. The current main branch is only really usable with a local file storage.

from django-filebrowser.

bummer! any suggestions for an alternative in the meantime? FTP?

from django-filebrowser.

Just going to use nfs + cloudfront for now. let me know if there's anything i can do to help.

from django-filebrowser.

Is this issue addressed somehow? or is filebrowser still unusable with S3?

from django-filebrowser.

we´re not working on this issue. however, if someone comes up with a patch, we´re happy to implement it.

from django-filebrowser.

To get partially working:

• Manual create uploads directory on S3
• Set MEDIA_ROOT = ""
• Subclass class something like this:

MediaS3BotoStorage = lambda: S3MediaStorage(location='media')

class S3MediaStorage(S3BotoStorage):
    """
    Subclasses :class:`storages.backends.s3boto.S3BotoStorage` and
    sets base location for files to ``/media``.
    """
    def __init__(self, *args, **kwargs):
        kwargs["location"] = "media"
        kwargs['bucket'] = settings.AWS_STORAGE_BUCKET_NAME
        kwargs['custom_domain'] = domain(settings.MEDIA_URL)
        super(S3MediaStorage, self).__init__(*args, **kwargs)

    def isfile(self, name):
        return self.exists(name)

    def isdir(self, name):
        # That's some inefficient implementation...
        # If there are some files having 'name' as their prefix, then
        # the name is considered to be a directory
        if not name: # Empty name is a directory
            return True

        if self.isfile(name):
            return False

        name = self._normalize_name(self._clean_name(name))
        dirlist = self.bucket.list(self._encode_name(name))

        # Check whether the iterator is empty
        for item in dirlist:
            return True
        return False

    def move(self, old_file_name, new_file_name, allow_overwrite=False):

        if self.exists(new_file_name):
            if allow_overwrite:
                self.delete(new_file_name)
            else:
                raise "The destination file '%s' exists and allow_overwrite is False" % new_file_name

        old_key_name = self._encode_name(self._normalize_name(self._clean_name(old_file_name)))
        new_key_name = self._encode_name(self._normalize_name(self._clean_name(new_file_name)))

        k = self.bucket.copy_key(new_key_name, self.bucket.name, old_key_name)

        if not k:
            raise "Couldn't copy '%s' to '%s'" % (old_file_name, new_file_name)

        self.delete(old_file_name)

    def makedirs(self, name):
        pass

    def rmtree(self, name):
        name = self._normalize_name(self._clean_name(name))
        dirlist = self.bucket.list(self._encode_name(name))
        for item in dirlist:
            item.delete()

I can confirm:

• directories are not created.
• Thumbnails are not created.

from django-filebrowser.