gtaylor / django-eve-db Goto Github PK

Hey,
Nice project, was just in the process of inspectdb'ing a mysql dump, but this might save me the trouble.

Can you please provide a working url for the rubicon 1.0.4 datadump?
Thanks

http://pastebin.com/JpNi4iqR

Liklely a deleted or modified column. We'll also take a look at it but wanted to open an issue.

Which license is this licensed under? README simply states "the BSD license," but the LICENSE file has a copy of the GPL so there's no way to know which BSD license is intended.

Can you include a download or a link to a download for the latest SQLite data dump that works with your tool?

Thanks for the code!

django-eve-db/eve_db/ccp_importer/util.py:

from django.db.backends.sqlite3.base import Database as sqlite3

Summary

Importing the table eve_db_crpnpccorporation throws an django.db.transaction.TransactionManagementError (for detailed stacktrace see Appendix). The source of the problem seems to be the transaction management, which is set to manually on the method (eve_db/ccp_importer/importers/importer_classes.py:SQLImporter#prep_and_run_importer) causing the exception.

The exception message indicates that there is a pending commit or rollback at the end of the method:

django.db.transaction.TransactionManagementError: Transaction managed block ended with pending COMMIT/ROLLBACK

Possible fix

Find out at what point a rollback or commit gets lost and fix it.

How to reproduce

1. Create virtual environment mkvirtualenv djangoevedbtest
2. Install requirements pip install django south psycopg2
3. Create test database createdb -h localhost djangoevedbtest
4. Create test user createuser -h localhost djangoevedbtest
5. Grant privileges to user
   • psql -h localhost
   • grant all privileges on database djangoevedbtest to djangoevedbtest;
   • \quit
6. Create a Django project django-admin.py startproject djangoevedbtest
7. Set up database connection in settings.py
8. Add south and eve_db to INSTALLED_APPS in settings.py
9. Sync database python manage.py syncdb (do not create a superuser)
10. Migrate eve_db python manage.py migrate eve_db
11. Download and unpack latest static database dump from http://www.fuzzwork.co.uk/dump/retribution-1.1-84566/eve.db.bz2
12. Start importing the dump python manage.py eve_import_ccp_dump /path/to/dump

Appendix

File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/Jens/.virtualenvs/django-eve-db-test/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/Users/Jens/.virtualenvs/django-eve-db-test/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/Jens/.virtualenvs/django-eve-db-test/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/Jens/.virtualenvs/django-eve-db-test/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/Users/Jens/Projects/Django/djangoevedbtest/eve_db/management/commands/eve_import_ccp_dump.py", line 116, in handle
    util.run_importers(util.IMPORT_LIST, sqlite_file)
  File "/Users/Jens/Projects/Django/djangoevedbtest/eve_db/ccp_importer/util.py", line 145, in run_importers
    importer.prep_and_run_importer(conn)
  File "/Users/Jens/.virtualenvs/django-eve-db-test/lib/python2.7/site-packages/django/db/transaction.py", line 224, in inner
    return func(*args, **kwargs)
  File "/Users/Jens/.virtualenvs/django-eve-db-test/lib/python2.7/site-packages/django/db/transaction.py", line 218, in __exit__
    self.exiting(exc_value, self.using)
  File "/Users/Jens/.virtualenvs/django-eve-db-test/lib/python2.7/site-packages/django/db/transaction.py", line 303, in exiting
    leave_transaction_management(using=using)
  File "/Users/Jens/.virtualenvs/django-eve-db-test/lib/python2.7/site-packages/django/db/transaction.py", line 67, in leave_transaction_management
    connection.leave_transaction_management()
  File "/Users/Jens/.virtualenvs/django-eve-db-test/lib/python2.7/site-packages/django/db/backends/__init__.py", line 130, in leave_transaction_management
    raise TransactionManagementError("Transaction managed block ended with "
django.db.transaction.TransactionManagementError: Transaction managed block ended with pending COMMIT/ROLLBACK

i'm using sqlite dump from this thread: http://www.eveonline.com/ingameboard.asp?a=topic&threadID=1376938

traceback tail:

eve_db/ccp_importer/importers/map.py in import_row(self, row)
    227         imp_obj.importance = row['importance']
    228 
--> 229         if row['url2d']:
    230             imp_obj.url_2d = row['url2d']

dgmAttributeTypes:

File "django-eve-db/eve_db/ccp_importer/importers/inventory.py", line 172, in import_row
    graphic_id = row['graphicID']

dgmEffects:

django-eve-db/eve_db/ccp_importer/importers/inventory.py(213)import_row()
--> 213         if row['graphicID']:
    214             imp_obj.graphic = EveGraphic.objects.get(id=row['graphicID'])

staStationTypes:

django-eve-db/eve_db/ccp_importer/importers/station.py(123)import_row()
--> 123         if row['dockingBayGraphicID']:
    124             imp_obj.docking_bay_graphic = EveGraphic.objects.get(id=row['dockingBayGraphicID'])
    125             
--> 126         if row['hangarGraphicID']:
    127             imp_obj.hangar_graphic = EveGraphic.objects.get(id=row['hangarGraphicID'])

Maybe a quick scan for ids or object counts or something like that and an option to --reset stuff.

Out of curiosity, given that it appears other apps within this project make use of postgresql (or, presumably, mysql), is there any specific reason to use sqlite for eve-db rather than using the mysql/pgsql conversions so that everything can be handled within a single database system?

-aDe

I ran into the following error loading the latest import:

django.db.utils.IntegrityError: insert or update on table "eve_db_invmarketgroup" violates foreign key constraint "eve_db_invmarket_parent_id_1f53341d_fk_eve_db_invmarketgroup_id"
DETAIL: Key (parent_id)=(1361) is not present in table "eve_db_invmarketgroup".

I noticed that a similar problem was corrected on the thread of Issue #13 by @zweizeichen and was wondering if anyone might be able to give some advice on how to correct the data dump? I am working with the latest sqlite dump from:

https://www.fuzzwork.co.uk/dump/