aroscoe / music-inbox Goto Github PK

Probably from bad data or something...

DEBUG:album_diff:Galactic ...
Ya-Ka-May ...
Exception in thread Thread-3:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 522, in __bootstrap_inner
self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 477, in run
self.__target(_self.__args, *_self.__kwargs)
File "/Library/Python/2.6/site-packages/django/dispatch/dispatcher.py", line 166, in send
response = receiver(signal=self, sender=sender, *_named)
File "/Users/michael/code/music-inbox/apps/library/utils/importer.py", line 26, in itunes
signals.import_done.send(sender=self, library=library)
File "/Library/Python/2.6/site-packages/django/dispatch/dispatcher.py", line 166, in send
response = receiver(signal=self, sender=sender, *_named)
File "/Users/michael/code/music-inbox/apps/library/utils/mb.py", line 34, in album_diff
mb_artist_entry.fetch_albums()
File "/Users/michael/code/music-inbox/apps/library/models.py", line 87, in fetch_albums
date, asin = self.get_release_date(release.id)
File "/Users/michael/code/music-inbox/apps/library/models.py", line 101, in get_release_date
release = release_group.releases[0] # TODO iterate over all
IndexError: list index out of range

currently each library upload spawns a new thread to import the albums. This ends up hitting the musicbrainz ws. There is rate limiting (i.e., time.sleep(1) ). However, these calls are not synchronized across threads, which means concurrent library importing will end up hitting musicbrainz ws more than it should, resulting in HTTP 502 error codes.

Possible solutions include a single thread doing all imports or synchronization around all musicbrainz calls (not even sure if python can do that).

Heavier but more fun solutions include using a message queue with each import thread as a producer with a single consumer on the other end.

When validating a file with less than 4 lines it will not invalidate the form. This sucky code needs cleaned up anyway.

even with rate limiting, sometimes musicbrainz is sending back HTTP 502, but only randomly. It seems to happen during library import after awhile. Musicbrainz requires rate limiting of 1 req / sec / IP. I believe all of our musicbrainz calls have the necessary sleep, but I could be wrong. I have even tried sleep()'ing for 2 seconds, twice what they require. I believe at this point we probably need to wrap all musicbrainz calls in try... except in an infinite loop and do exponential backoff for timeouts, or some such