django-nonrel / djangotoolbox Goto Github PK

The constructor for django.db.backends.BaseDatabaseOperations has changed and now expects the connection as a parameter. Consider modifying djangotoolbox.db.base.__init__ to pass the connection via the super method instead of storing it (which is what the new constructor does anyway)

It seems that when you save an object with an EmbeddedModelField, if that embedded object has any foreign keys they are not saved as ObjectIDs, but rather just strings. This was reported https://bitbucket.org/Wilfred/djangotoolbox/issue/6/embeddedmodelfield-and-foreignkey, but as far as I'm aware is not resolved.

First question: is this still an issue or did I miss a fix?

Second question: if still an issue, thoughts re: how to fix it? My first thought there was that there wouldn't be an issue if ObjectId's weren't cast to strings in the ORM on the way out of the database. I imagine changing this would make a lot of code unhappy that's assuming strings, but is there a deeper reason for this conversion? If that won't work for whatever reason, I'll give some thought to other ways it might be handled.

Would appreciate any project-leader wisdom here, thanks!

sorry but I'm not using google app engine, just using django-nonrel with djangotoolbox and mongoengine, and I just saw the site you sent and it has no information about what I'm looking for, can you give me a hand on this?
I have this collection in mongo:

{
"_id" : ObjectId("517f00dbe138234abbf790bb"),
"campaigns" : [
{
"payout" : 0.7,
"landing_page" : [ ],
"rotate_urls" : null,
"_module" : "setup.models",
"_model" : "Campaigns",
"cloaking" : true,
"campaign_name" : "DOS",
"rotate_urls_status" : false,
"affiliate_url" : "http://www.DOS.com/[[subid]]"
}
],
"category_name" : "category2"
}

how can I append a embed document into landing_page using django-nonrel, mongoengine without djangodbindexer support?? I have tried this but I always receive errors:
landingpage = LandingPages(nickname=lpnickname,url=lpurl)
b = Categories.objects.get(id=affnetworkid,campaigns__campaign_name=campaignname)
b.landing_page.append(landingpage)
b.save()

The mongoengine documentation says that doble underscore is supported, but, the fact is that django-nonrel or the mongoengine is trying to execute a join which is not what I'm looking for, how can I use dot notation on this??
how can I just add more documents to the landing_page list type inside campaigns???
thanks
note: I have 8 days trying to achieve this.

I'd like to use version 1.4 as a dependency, but the version on PyPI is only 0.9.2.

If it can't be updated on PyPI for some reason, please at least update the setup.py and add the distribution to github.

Thanks!

I get the following error everytime I try to run the server after upgrading to django to version 1.11.

..../models.py", line 5, in
from djangotoolbox.fields import ListField
File "/anaconda/lib/python2.7/site-packages/djangotoolbox/fields.py", line 4, in
from django.utils.importlib import import_module
ImportError: No module named importlib

after a little research, I found out "django.utils.importlib is a compatibility library for when Python 2.6 was still supported. It has been obsolete since Django 1.7, which dropped support for Python 2.6, and is removed in 1.9 per the deprecation cycle." source

but when I tried running the server afterwards, I got this error:

from djangotoolbox.fields import ListField
File "/anaconda/lib/python2.7/site-packages/djangotoolbox/fields.py", line 6, in
from django.db.models.fields.subclassing import Creator
ImportError: No module named subclassing

can anyone help please?

This is complicated to explain, but worth having documented somewhere for the next person.

I have a reasonably complex project built on nonrel/GAE that uses the admin screens quite a bit. There are some constraints in what's possible since the built-in admin logic assumes an underlying relational database. But it's definitely possible to get something quite functional working.

However, there's a problem with executing an 'admin action' (https://docs.djangoproject.com/en/1.3/ref/contrib/admin/actions/) against selected entitities in a change list that's the result of a search. Searches can be made to work with the appropriate dbindexer setup, but there's a weird bit of logic in the vanilla-django code that creates a queryset of selected entities from the search results that causes problems.

The first set of problems comes from djangotoolbox.db.basecompiler.EMULATED_OPS, where the emulated operations don't handle operations against dbindexer list columns. You can get round that like this (or on one line probably):

def _list_starts_with(l, s):
return reduce(lambda result, item: result or item.startswith(s), l)

...
'startswith': lambda x, y: _list_starts_with(x, y) if isinstance(x, (list,tuple)) else x.startswith(y),
...

But then you get an error later on when the action tries to do query set.update. It's complicated and I don't have a fix so there's no point discussing.

Anyway, the upshot of all that is that you probably can't do "admin actions" using nonrel if you enable searching a changelist, unless you'rre prepared to do some changes to vanilla django somewhere along the way... I don't suppose it should be very complicated since the selected entities could be specified with a list of IDs, but that isn't what's going on: that list of IDs is somehow overlaid on a query set that is related to the search you've done. Dunno why. Seems a bit weird to me. Anyway. For info.

Steve

I've been having a ton of trouble getting a ListField to work, even after following the very useful example at https://gist.github.com/jonashaag/1200165

I finally traced it down to adding "null=True" to my list of attributes in the models.py file. If I remove that entry I can see the form fine, but if I add it back in the "form.as_p" call returns nothing in the template.

Looking in the debugger the form object looks different too. The text representation is the python default object ID when the form is broken. When it would display I see the HTML code that would get output for the items in my form.

Now I'm not sure this will show up in the example above, but the code change I'd make would be...

categories = CategoryField()

becomes...

categories = CategoryField(null=True)

I had added that to work around existing documents that didn't have my new ListField value. I assumed the model verification would fail somewhere if there were existing documents without a value in a field that can't be null or blank.

My app is really long and I'm hoping this is enough to reproduce the issue.

In the NonRelQuery _matches_filters method, the code at line 274 which reads:

                    if isinstance(value, (datetime.datetime, datetime.date,
                                          datetime.time)):

is a typo, and should instead read:

                    if isinstance(entity_value, (datetime.datetime, datetime.date,
                                          datetime.time)):

This code has absolutely no API documentation and it's not that easy to understand what needs to be done in order to implement a non relational backend.

Hello,

I use a lot "db_column" in my model classes, mainly to shorten field names in mongodb.
But by reinstalling django_nonrel on my environnement, I noticed that the db_column doesn't work anymore for the EmbeddedModelField.
It worked very well before and I think that the last changes on djangotoolbox broke it.

For example, with the following classes:

class MyClass (models.Model):
comments = ListField(EmbeddedModelField('Comment'))

class Comment (models.Model):
text = models.TextField(db_column='t')
date = models.DateTimeField(db_column='d')

Everytime I query MyClass, I get the following error:
'" t' is an invalid keyword argument for this function"
If I remove the db_column in the fields, it works again, but with "text" as the field name ...

Can you make a fix ?
Thanks a lot

Django 1.7 introduces custom lookups, which breaks the Constraint private API hacks found in djangotoolbox.db.

It seems like the only issues are in basecompiler.NonRelQuery._decode_child(). I found all of the values using the new API except annotations. See Pull Request #56.

I have been trying to get a blogging system running on GAE. I am not familiar with GAE or BigTable (although I am comfortable using Django) so please bare with me.

I followed the instructions here (and have my app running):

http://djangoappengine.readthedocs.org/en/latest/

However, when I try to use the 'ArchiveIndexView' class based generic view the app throws an exception (Nonetype object has no attribute 'model'). Having dug about a bit I notice the following:

• the 'get_selected_fields' method in 'basecompiler.py' is called twice per http request;
• the first time it is called, query.select_fields == [];
• the second time it is called, query.select_fields == [None]

This seems to be the route of the problem. I was wondering if this is a known issue, or if I have missed something along the way.

The View:

class ListingView(ArchiveIndexView):

    model = Post

    date_field = "created_date"

    date_type = 'year'

    allow_future = False

The Model:

class Post(models.Model):

    # Post title
    title = models.CharField(max_length=250)

    # Url safe title for the post
    url_safe_title = models.SlugField(max_length=250)

    # Post content 
    content = models.TextField()

    # Post created date 
    created_date = models.DateTimeField(auto_now_add=True)

    # Post updated date
    updated_date = models.DateTimeField(auto_now=True)

    # Store a reference to the user
    author = models.ForeignKey(User)

    # Method for getting an absolute url for the resource
    def get_absolute_url(self, *args, **kwargs):
        return reverse('admin-post-update', args=[self.id,])

    # Method for getting a delete url for the resource
    def get_delete_url(self, *args, **kwargs):
        return ""

    # Method for getting a listing url for our resources
    @staticmethod
    def get_listing_url():
        return reverse('admin-post-list')

    # Convenience method for generating
    # a friendly unicode representation
    # of the instance 
    def __unicode__(self):
        return self.title

Also, please find the application stacktrace below:

ERROR 2014-01-27 12:20:04,084 base.py:212] Internal Server Error: /blog/
Traceback (most recent call last):
File "/Users/danielstringer/Dev/AppEngine/blog/django/core/handlers/base.py", line 115, in get_response
response = callback(request, _callback_args, *_callback_kwargs)
File "/Users/danielstringer/Dev/AppEngine/blog/django/views/generic/base.py", line 68, in view
return self.dispatch(request, _args, *_kwargs)
File "/Users/danielstringer/Dev/AppEngine/blog/django/views/generic/base.py", line 86, in dispatch
return handler(request, _args, *_kwargs)
File "/Users/danielstringer/Dev/AppEngine/blog/django/views/generic/dates.py", line 333, in get
self.date_list, self.object_list, extra_context = self.get_dated_items()
File "/Users/danielstringer/Dev/AppEngine/blog/django/views/generic/dates.py", line 412, in get_dated_items
date_list = self.get_date_list(qs, ordering='DESC')
File "/Users/danielstringer/Dev/AppEngine/blog/django/views/generic/dates.py", line 391, in get_date_list
if date_list is not None and not date_list and not allow_empty:
File "/Users/danielstringer/Dev/AppEngine/blog/django/db/models/query.py", line 157, in nonzero
return type(self).bool(self)
File "/Users/danielstringer/Dev/AppEngine/blog/django/db/models/query.py", line 151, in bool
next(iter(self))
File "/Users/danielstringer/Dev/AppEngine/blog/django/db/models/query.py", line 139, in _result_iter
self._fill_cache()
File "/Users/danielstringer/Dev/AppEngine/blog/django/db/models/query.py", line 941, in _fill_cache
self._result_cache.append(next(self._iter))
File "/Users/danielstringer/Dev/AppEngine/blog/djangotoolbox/db/basecompiler.py", line 375, in results_iter
fields = self.get_fields()
File "/Users/danielstringer/Dev/AppEngine/blog/djangotoolbox/db/basecompiler.py", line 519, in get_fields
if field.model._meta != query_model._meta:
AttributeError: 'NoneType' object has no attribute 'model'
INFO 2014-01-27 12:20:04,314 dev_appserver.py:3090] "GET /blog/ HTTP/1.1" 500 -

I would be more than happy to help diagnose and fix this issue (if the issue is in the djangotoolbox codebase). I am interested in your thoughts.

Many thanks,
Dan

The docs for djangotoolbox are stale, resulting in this exact issue:

readthedocs/readthedocs.org#4082

To be clear: this is an issue with this package, not with readthedocs itself. Should be fixed by rebuilding and republishing.

This happens with the last version of Python.

Now clicking on a specific user in "Users" in the admin, django throws this error:

 NoReverseMatch at /admin/auth/user/52ade053f9bed911e71dfa5e/ Reverse for  'view_on_site' with arguments '(u'52aca520f9bed9267bd2e23a',      u'52ade053f9bed911e71dfa5e')' and keyword arguments '{}' not found.

Here is the full traceback:

 Environment:


Request Method: GET
Request URL: http://localhost:8000/admin/auth/user/52ade053f9bed911e71dfa5e/

Django Version: 1.5.5
Python Version: 2.7.6
Installed Applications:
('django_mongodb_engine',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.admin',
 'djangotoolbox',
 'django_mongodb_engine',
 'dbarch')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware')


Template error:
In template /home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/contrib/admin/templates/admin/change_form.html, error at line 33
   Reverse for 'view_on_site' with arguments '(u'52aca520f9bed9267bd2e23a', u'52ade053f9bed911e71dfa5e')' and keyword arguments '{}' not found.
   23 : </div>


   24 : {% endblock %}


   25 : {% endif %}


   26 : 


   27 : {% block content %}<div id="content-main">


   28 : {% block object-tools %}


   29 : {% if change %}{% if not is_popup %}


   30 :   <ul class="object-tools">


   31 :     {% block object-tools-items %}


   32 :     <li><a href="{% url opts|admin_urlname:'history' original.pk|admin_urlquote %}" class="historylink">{% trans "History" %}</a></li>


   33 :     {% if has_absolute_url %}<li><a href=" {% url 'admin:view_on_site' content_type_id original.pk%} " class="viewsitelink">{% trans "View on site" %}</a></li>{% endif%}


   34 :     {% endblock %}


   35 :   </ul>


   36 : {% endif %}{% endif %}


   37 : {% endblock %}


   38 : <form {% if has_file_field %}enctype="multipart/form-data" {% endif %}action="{{ form_url }}" method="post" id="{{ opts.module_name }}_form">{% csrf_token %}{% block form_top %}{% endblock %}


   39 : <div>


   40 : {% if is_popup %}<input type="hidden" name="_popup" value="1" />{% endif %}


   41 : {% if save_on_top %}{% block submit_buttons_top %}{% submit_row %}{% endblock %}{% endif %}


   42 : {% if errors %}


   43 :     <p class="errornote">


Traceback:
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  140.                     response = response.render()
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/response.py" in render
  105.             self.content = self.rendered_content
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/response.py" in rendered_content
  82.         content = template.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in render
  140.             return self._render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in _render
  134.         return self.nodelist.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in render
  830.                 bit = self.render_node(node, context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  124.         return compiled_parent._render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in _render
  134.         return self.nodelist.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in render
  830.                 bit = self.render_node(node, context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  124.         return compiled_parent._render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in _render
  134.         return self.nodelist.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in render
  830.                 bit = self.render_node(node, context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  63.             result = block.nodelist.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in render
  830.                 bit = self.render_node(node, context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  63.             result = block.nodelist.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in render
  830.                 bit = self.render_node(node, context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/defaulttags.py" in render
  285.                 return nodelist.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in render
  830.                 bit = self.render_node(node, context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/defaulttags.py" in render
  285.                 return nodelist.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in render
  830.                 bit = self.render_node(node, context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  63.             result = block.nodelist.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in render
  830.                 bit = self.render_node(node, context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/defaulttags.py" in render
  285.                 return nodelist.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/base.py" in render
  830.                 bit = self.render_node(node, context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/home/viyat/Desktop/zmongo/env/lib/python2.7/site-packages/django/template/defaulttags.py" in render
  426.                         raise e

Exception Type: NoReverseMatch at /admin/auth/user/52ade053f9bed911e71dfa5e/
Exception Value: Reverse for 'view_on_site' with arguments '(u'52aca520f9bed9267bd2e23a', u'52ade053f9bed911e71dfa5e')' and keyword arguments '{}' not found.

djangotoolbox doesn't work on django 1.8.

After the work that was done by @kavdev in #56 it seems like most of the errors are generated from Django moving around their db modules and packages significantly in this release.

Can we get #56 merged in so we can work on porting to Django 1.8?

It looks like this specific test can fail if it's run really quickly. Observe it succeed under python 2.5 and 2.6. Couldn't reproduce on my local machine.

Do you guys think it might be cause by something else?

Dudes, I'm sick and tired of trying to look for something and not be able to find it, I'm using django-nonrel, djangotoolbox, mongoengine and is REALLY IMPOSSIBLE to find solutions without a site with all documentation in html format, why don't you just buy a different domain name and place all this project documentation in html format somewhere??? is it something really hard???

I'm using django-nonrel 1.6-beta with djangotoolbox. Some of my admin pages needed to use DISTINCT but djangotoolbox prevented that in /db/basecompiler.py line 451:

diff --git a/djangotoolbox/db/basecompiler.py b/djangotoolbox/db/basecompiler.py
index 71d9ec8..47fa95b 100644
--- a/djangotoolbox/db/basecompiler.py
+++ b/djangotoolbox/db/basecompiler.py
@@ -451,7 +451,7 @@ class NonrelCompiler(SQLCompiler):
             raise EmptyResultSet()
         if (len([a for a in self.query.alias_map if
                  self.query.alias_refcount[a]]) > 1 or
-            self.query.distinct or self.query.extra or self.query.having):
+            self.query.extra or self.query.having):
             raise DatabaseError("This query is not supported by the database.")

     def get_count(self, check_exists=False):

Can somebody explain why using distinct raises this DatabaseError?

I'll also try to push this to a branch.

Thank you.

To reproduce, gather together the latest django-nonrel, djangoappengine, dbindexer, djangotoolbox and the test app. Try to run manage.py syncdb. Here's the error:

Creating tables ...
Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_manager(settings)
  File "/home/lukeb/Potato/Framework/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/home/lukeb/Potato/Framework/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/lukeb/Potato/Framework/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/lukeb/Potato/Framework/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/home/lukeb/Potato/Framework/django/core/management/base.py", line 351, in handle
    return self.handle_noargs(**options)
  File "/home/lukeb/Potato/Framework/django/core/management/commands/syncdb.py", line 109, in handle_noargs
    emit_post_sync_signal(created_models, verbosity, interactive, db)
  File "/home/lukeb/Potato/Framework/django/core/management/sql.py", line 190, in emit_post_sync_signal
    interactive=interactive, db=db)
  File "/home/lukeb/Potato/Framework/django/dispatch/dispatcher.py", line 172, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/lukeb/Potato/Framework/django/contrib/contenttypes/management.py", line 19, in update_contenttypes
    model=opts.object_name.lower())
  File "/home/lukeb/Potato/Framework/django/db/models/manager.py", line 132, in get
    return self.get_query_set().get(*args, **kwargs)
  File "/home/lukeb/Potato/Framework/django/db/models/query.py", line 346, in get
    num = len(clone)
  File "/home/lukeb/Potato/Framework/django/db/models/query.py", line 82, in __len__
    self._result_cache = list(self.iterator())
  File "/home/lukeb/Potato/Framework/django/db/models/query.py", line 275, in iterator
    for row in compiler.results_iter():
  File "/home/lukeb/Potato/Framework/djangotoolbox/db/basecompiler.py", line 335, in results_iter
    results = self.build_query(fields).fetch(
  File "/home/lukeb/Potato/Framework/djangotoolbox/db/basecompiler.py", line 430, in build_query
    query.add_filters(self.query.where)
  File "/home/lukeb/Potato/Framework/djangotoolbox/db/basecompiler.py", line 142, in add_filters
    self.add_filters(child)
  File "/home/lukeb/Potato/Framework/djangotoolbox/db/basecompiler.py", line 145, in add_filters
    self.add_filter(field, lookup_type, self._negated, value)
  File "/home/lukeb/Potato/Framework/djangoappengine/db/compiler.py", line 61, in _func
    return func(*args, **kwargs)
TypeError: add_filter() takes exactly 6 arguments (5 given)

From a quick prod around it looks like the method definition of add_filter differs between djangoappengine.db.compiler and djangotoolbox.db.basecompiler. I'm not sure where this is supposed to be fixed though.

in djangotoolbox/fields.py:238 RemovedInDjango110Warning: SubfieldBase has been deprecated. Use Field.from_db_value instead.
class EmbeddedModelField(models.Field):

Hi, The server gets the following error after updating to the latest code in the master branch. Any ideas on how to fix it? Thanks

File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 1766, in __setitem__
    datastore_types.ValidateProperty(' ', value)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore_types.py", line 1547, in ValidateProperty
    'Unsupported type for property %s: %s' % (name, v.__class__))
DatabaseError: Unsupported type for property  : <class 'django.utils.safestring.SafeText'>

.reverse() toggles standard_ordering. Should be abstracted somehow so that backends don't need to care about this attribute. Also related to django-nonrel/mongodb-engine#106

Create one unique set of test models instead of defining new models for all tests (only where it makes sense)

This doesn't appear to affect ListField, which does serialize into correct json format. Also, this is most likely a problem with djangotoolbox, not django-nonrel, but I'm not sure where to submit issues for that project.

from django.db import models
from djangotoolbox import fields

class Post(models.Model):
    title = models.CharField(max_length=256)
    user = models.ForeignKey(User)
    content = models.TextField()
    tags = fields.SetField(models.CharField(max_length=128, blank=True), null=True, blank=True)

Calling dumpdata on a Post object results in json like this:

{"pk": "4efb8096aae5710ad4000000",
 "model": "blog.post",
 "fields": {"tags": "set([u'home', u'features'])",
            "content":"foo",
            "user": "4ded8376aae5713406000010",
            "title": "My Title",
            "slug": "my-title"}
}

Since the json format doesn't have a set notation, it probably should just serialize to the same format as the ListField.

Environment:

Django==1.8.18
djangotoolbox==1.8.0
Python 2.7.6

When performing a request as http://[2406:da00:ff00::36f3:801a]:80, the process_request method will throw a ValueError while unpacking the host to determine domain/port combination.

def process_request(self, request):
        # Ignore port if it's 80 or 443
        if ':' in request.get_host():
            domain, port = request.get_host().split(':')
            if int(port) not in (80, 443):
                domain = request.get_host()
        else:
            domain = request.get_host().split(':')[0]

I apologize if this is the wrong place to post this, but I thought I would mention it in case others were having the same trouble I was. Also, so I can get feedback if I am missing something or otherwise committing a heinous sin.

My project using django-nonrel and djangotoolbox 1.8.0 with a MongoDB 2.6 data store raises an exception "This database doesn't support filtering on non-primary key ForeignKey fields." when I try to (surprise) filter on non-primary FK fields.

It appears to be a true statement, I cannot find any documentation regarding MongoDB supporting relations between collections on anything other than a primary key. However, just because it isn't supported doesn't mean it doesn't work. In fact it works exactly as I expected it to. Removing the conditional that raises this exception in lines 233-234 of basecompiler.py seems to result in proper filtering behavior from MongoDB. This saved me from writing custom forms for all my admin models.

Disclaimer: I have only tested this in the context of my current application, more testing is recommended. Likely the statement in question is there for a reason, a better solution than removing it entirely is likely necessary.

I have attached a patch file detailing the changes I made.
djangotoolbox-basecompiler-1.8.mongo.patch.txt

Breaks compatibility, in commit 57a0d7c .

Taken from this mailing list discussion: http://groups.google.com/group/django-non-relational/browse_thread/thread/c006d67444bb28f7

Given the models:

from django.db import models

class Person(models.Model):
    name = models.CharField(max_length=200)

class Car(models.Model):
    name = models.CharField(max_length=200)
    owner = models.ForeignKey(Person)

class Wheel(models.Model):
    part_of = models.ForeignKey(Car)

Subqueries return no results, even when there should be

In [1]: from test_app.models import Person, Car, Wheel

In [2]: p = Person.objects.create(name='bob')

In [3]: c1 = Car.objects.create(name='mini', owner=p)

In [4]: c2 = Car.objects.create(name='landrover', owner=p)

In [5]: w = Wheel.objects.create(part_of=c2)

In [6]: cars = Car.objects.filter(owner=p)

In [7]: cars
Out[7]: [<Car: Car object>, <Car: Car object>]

In [8]: Wheel.objects.filter(part_of__in=cars) # cars is a QuerySet
Out[8]: []

In [9]: Wheel.objects.filter(part_of__in=list(cars)) # force QuerySet to list
Out[9]: [<Wheel: Wheel object>]

For some cases it'd be useful to define a max_length for ListField and SetField. It'd work as a maximum number of items for such fields.

django-nonrel/mongodb-engine#92

class Foo(models.Model):
    bar = ListField(EmbeddedModelField('Buzz'))

class Buzz(models.Model):
    class Meta:
        abstract = True

doesn't work because Buzz is abstract

I use django.contrib.auth.views.login
but it raises error NotImplementedError at /login/
The database backend only supports count() queries.

Hi --

I've been thinking that, until someone gets around to putting a nice interface on DictFields and ListFields, it would be nice to be able to edit them as raw JSON in the admin -- just show a Textarea with the contents of the field.

So to that end, I put together the following SerializedObjectWidget:

from django.core.serializers import serialize, deserialize, json
from django.db.models.query import QuerySet
from django.db import models
from django.forms.widgets import Textarea
from django.utils import simplejson

class DjangoJSONEncoder(json.DjangoJSONEncoder):
    """
        Version of DjangoJSONEncoder that knows how to encode embedded QuerySets or Models.
    """
    def default(self, obj):
        if isinstance(obj, QuerySet):
            return serialize('python', obj)
        if isinstance(obj, models.Model):
            return serialize('python', [obj])[0]
        return super(DjangoJSONEncoder, self).default(obj)

def deserialize_model(obj):
    if 'model' in obj and 'pk' in obj:
        obj = list(deserialize('python', [obj]))[0].object
    return obj

class SerializedObjectWidget(Textarea):
    def render(self, name, value, attrs=None):
        value = simplejson.dumps(value, indent=2, cls=DjangoJSONEncoder)
        return super(SerializedObjectWidget, self).render(name, value, attrs)

    def value_from_datadict(self, data, files, name):
        val = data.get(name, None)
        if not val:
            return None
        return simplejson.loads(val, object_hook=deserialize_model)

All this does is take an arbitrary Python object (which can include model instances), encode it as JSON, and put it in a Textarea to display. Then on save it takes the results and decodes them back into a Python object including model instances. So it's pretty agnostic about what you use it for, but in particular it should work fine for DictFields and ListFields. Here's how I'm using it in my project (in models.py):

from .widgets import SerializedObjectWidget
from djangotoolbox.fields import ListField as DTListField, DictField as DTDictField, AbstractIterableField

class ListField(DTListField):
    def formfield(self, **kwargs):
        defaults = {'form_class': Field, 'widget': SerializedObjectWidget}
        defaults.update(kwargs)
        return super(AbstractIterableField, self).formfield(**defaults)

class DictField(DTDictField, Field):
    def formfield(self, **kwargs):
        defaults = {'form_class': Field, 'widget': SerializedObjectWidget}
        defaults.update(kwargs)
        return super(AbstractIterableField, self).formfield(**defaults)

The result in the admin looks like this: http://imgur.com/qExPZ

So, not something you'd want just any user to have access to, but usable if you know what you're doing.

This is working for me so far, but I haven't used it much and there might be all kinds of edge cases and security holes. I just wanted to throw it out there as an option to fold into DictField and ListField until something better comes along.

What do you think?

Thanks,
Jack

======================================================================
FAIL: test_pre_save_untyped (djangotoolbox.tests.EmbeddedModelFieldTest)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "<http://freedomfry.local:8080/job/Google%20Art%20Project/ws/djangotoolbox/tests.py",> line 269, in test_pre_save_untyped
   self.test_pre_save(field='simple_untyped')
 File "<http://freedomfry.local:8080/job/Google%20Art%20Project/ws/djangotoolbox/tests.py",> line 265, in test_pre_save
   self.assertEqualDatetime(getattr(instance, field).auto_now_add, auto_now_add)
 File "<http://freedomfry.local:8080/job/Google%20Art%20Project/ws/djangotoolbox/tests.py",> line 235, in assertEqualDatetime
   self.assertEqual(d1.replace(microsecond=0), d2.replace(microsecond=0))
AssertionError: datetime.datetime(2011, 12, 15, 6, 39, 55) != datetime.datetime(2011, 12, 15, 6, 39, 54)

The system the test ran on was rather slow it seems.

First time I'm trying to use MongoDB with Django.
Error in your module.. boom!

/usr/lib/python3.4/site-packages/djangotoolbox/db/base.py in _value_for_db_collection(self, value, field, field_kind, db_type, lookup)
    481                 # "list" may be used for SetField.
    482                 #print("Type: "+str(db_type))
--> 483                 if db_type in 'list':
    484                     return list(value)
    485                 elif db_type == 'set':

TypeError: 'in <string>' requires string as left operand, not NoneType

When trying to create ListField() in the model as per example.
3-4 hours of debugging step by step brought me to this solution:

class NonrelDatabaseWrapper(BaseDatabaseWrapper):
   data_types={'ListField':'list'}

Next step is to solve this crap:

/usr/lib/python3.4/site-packages/django/db/models/fields/__init__.py in get_prep_value(self, value)
    983         if value is None:
    984             return None
--> 985         return int(value)
    986 
    987     def contribute_to_class(self, cls, name, **kwargs):

TypeError: int() argument must be a string, a bytes-like object or a number, not 'ObjectId'

When saving the actual model.

commit bd292d6 excludes self.query.standard_ordering which handles reverse(). here's a test case. i'm going to work on a fix.

def test_reverse(self):
        class DBColumn(models.Model):
            a = models.IntegerField(db_column='b')
        model1 = DBColumn.objects.create(a=1)
        model2 = DBColumn.objects.create(a=2)
        self.assertEqual(list(DBColumn.objects.all().order_by('a').reverse()), [model2, model1])
        self.assertEqual(list(DBColumn.objects.all().order_by('-a').reverse()), [model1, model2])

All the library I installed comes from pip install django-mongodb-engine only

And I follow the tutorial, still got below error once login admin site, and click User link

I've already got INSTALLED_APP including djangotoolbox and django_mongodb_engine as the troubleshooting suggested

[10/Dec/2013 18:20:04] "GET /admin/ HTTP/1.1" 200 3249
Internal Server Error: /admin/auth/user/
Traceback (most recent call last):
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\django\core\handlers\ba
se.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\django\contrib\admin\op
tions.py", line 372, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\django\utils\decorators
.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\django\views\decorators
\cache.py", line 89, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\django\contrib\admin\si
tes.py", line 202, in inner
    return view(request, *args, **kwargs)
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\django\utils\decorators
.py", line 25, in _wrapper
    return bound_func(*args, **kwargs)
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\django\utils\decorators
.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\django\utils\decorators
.py", line 21, in bound_func
    return func(self, *args2, **kwargs2)
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\django\contrib\admin\op
tions.py", line 1285, in changelist_view
    'selection_note': _('0 of %(cnt)s selected') % {'cnt': len(cl.result_list)},

  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\django\db\models\query.
py", line 106, in __len__
    self._result_cache = list(self.iterator())
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\django\db\models\query.
py", line 317, in iterator
    for row in compiler.results_iter():
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\djangotoolbox\db\baseco
mpiler.py", line 375, in results_iter
    results = self.build_query(fields).fetch(
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\djangotoolbox\db\baseco
mpiler.py", line 477, in build_query
    self.check_query()
  File "D:\Dev\python\Envs\vizpropspy2\lib\site-packages\djangotoolbox\db\baseco
mpiler.py", line 455, in check_query
    raise DatabaseError("This query is not supported by the database.")

Type conversions refactor split into more logical parts, plus a complete PEPification and style normalization and some additional cleanup. The actual changes are described in the original issue -- all is still relevant including the warning at the end. The new branches are here:

https://github.com/django-nonrel/djangotoolbox/compare/feature/type-conversions-refactor-2
https://github.com/django-nonrel/djangoappengine/compare/feature/type-conversions-refactor-2
django-nonrel/mongodb-engine@develop...feature/type-conversions-refactor-2

Hopefully you'll find reviewing them commit-wise doable, unlike before. The additional nonrel branch is optional, it depends on the refactor, but not vice versa [update: not any longer, after renaming nonrel_db_type to db_type it is necessary].

First commit on each branch contains all the pedantic stuff; there are no actual changes in these commits -- other than reflowing, capitalizing & dotting, blank adding there is some method reordering, variable renaming, additional comments, one unused internal method dropped and some small blocks of code rewritten for better readability. If anything in these first commits raises any doubts I'm going to separate it.

The code style chosen is not really any better than what it was, but it is consistent across toolbox and both engines (and dbindexer is to follow). I've based it on PEP8, PEP257, Django & Google code styles (generally trying to stick with the first one unless there was some good reason not to). There are some some pros of cleaning up and normalizing things this way:

• following commits should be more readable, as the temptation to include stylistic changes should be smaller;
• you may run the pep8 verifier on . to check that your code is fine and dandy (use --exclude="docs", and --exclude="multiple_database" for Mongo; ignore W602);
• and also you may comfortably place a traceback and an editor side-by-side, even on a small monitor, and maybe you'll find yourself scrolling a bit too much and reading another class than you though you were a bit less often ;-)

The additional cleanup includes:

• separating parts that are not relevant to the issue (I've committed the set serialization test and EmbeddedModelField.update fixes to the develop branch, tests for the natural usage of ListField with ForeignKeys are on a separate branch);
• some further simplifying of nonrel fields code, namely this commit: 5fce56f;
• and of GAE code: django-nonrel/djangoappengine@8f81ef7;
• I've renamed DatabaseCreation.nonrel_db_type to just db_type after realizing that the parent method has been added by nonrel, moreover back-ends can now actually work without the related_db_type changes. Probably I'll propose letting back-ends override Field.db_type in this way to Django (replacing the #17336 ticket).
• Removing or explaining some TODOs, rewritting a few comments.

**/lib/python2.7/site-packages/djangotoolbox/db/base.py", line 28, in init
super(NonrelDatabaseOperations, self).init()
TypeError: init() takes exactly 2 arguments (1 given)

the following error occurs at the installation process:

Extracting djangotoolbox-1.6.2-py3.4.egg to /home/vyscond/tartarus/infosae-rest/.venv/lib/python3.4/site-packages
  File "/home/vyscond/tartarus/infosae-rest/.venv/lib/python3.4/site-packages/djangotoolbox-1.6.2-py3.4.egg/djangotoolbox/test.py", line 45
    print "DB state not valid:"
                              ^
SyntaxError: Missing parentheses in call to 'print'

I thinks it's good replace this with a compatible between 2 and 3 python versions? or this project is intent to be only available to version 2?

New home for django-nonrel/mongodb-engine#103

Nearly works with Mongo (somewhat similar to issue 47 there), might be expressive in some cases.

def test_list_with_foreignkeys(self):
    class Model(models.Model):
        pass
    class ReferenceList(models.Model):
        keys = ListField(models.ForeignKey(Model))
    model1 = Model.objects.create()
    model2 = Model.objects.create()
    ReferenceList.objects.create(keys=[model1, model2])
    self.assertEqual(
        Model.objects.get(pk=ReferenceList.objects.all()[0].keys[0]),
        model1)