Crash without log when running backgroud. about gphotos-sync HOT 14 CLOSED

from gphotos-sync.

Yes it's the second time it occurs. Without log.
Each time it start again from last db update (all picasa sync).
Actually I try to abort run when I think about it because it forces a db update.

from gphotos-sync.

from gphotos-sync.

No it's not failing on the same problem.

In fact last time it has continue, and it has started downloading for the first time. (it start with picasa)

Command was launched without log redirect nor background mode, so I had to kill it to go to lunch and restart it with background.

That was 1 hour ago and it's still running but no output .

....
gphotos.picasa: Downloading /media/toshiba/picasa/2017/08/20170824_131826-ANIMATION.gif ...
gphotos.picasa: Downloading /media/toshiba/picasa/2017/08/20170824_105454-ANIMATION.gif ...
gphotos.picasa: Downloading /media/toshiba/picasa/2017/08/20170824_102847.mp4 ...
gphotos.picasa: Downloading /media/toshiba/picasa/2017/08/20170824_093413-COLLAGE.jpg ...
gphotos.picasa: Downloading /media/toshiba/picasa/2017/08/20170823_163016-COLLAGE.jpg ...
gphotos.picasa: Downloading /media/toshiba/picasa/2017/08/20170823_162945-COLLAGE.jpg ...
gphotos.picasa: Downloading /media/toshiba/picasa/2017/08/20170822_141821.mp4 ...
gphotos.picasa: Downloading /media/toshiba/picasa/2017/08/20170812_134932-ANIMATION.gif ...
gphotos.picasa: Downloading /media/toshiba/picasa/2017/07/20170729_103800-ANIMATION.gif ...

gphotos.picasa: Downloading /media/toshiba/picasa/2017/07/20170712_190921.mp4 ...
gphotos.picasa: Downloading /media/toshiba/picasa/2017/07/20170711_194259-ANIMATION.gif ...
^Cgphotos: User cancelled download
gphotos: Done.
gphotos.data: Saving Database ...
gphotos.data: Database Saved.

then :

(gphotos-sync) pi@raspberrypi:~/pyenvs/gphotos-sync $ gphotos-sync --db-path=var/ /media/toshiba/ > var/gphotos.log &
[1] 7976
(gphotos-sync) pi@raspberrypi:~/pyenvs/gphotos-sync $ tail -f var/gphotos.log
gphotos.drive: Indexing Drive Folders ...
gphotos.drive: Resolving paths ...
gphotos.drive: Drive Folders scanned.
gphotos.drive: Indexing Drive Files ...
gphotos.data: Saving Database ...
gphotos.data: Database Saved.
gphotos.picasa: Indexing Albums ...
gphotos.picasa: Album count 23
gphotos.picasa: Album: Auto Backup, photos: 41434, updated: 2017-11-26 09:12:24, published: 2017-11-26 09:03:29

Stuck on this line for 1 hour.

CPU is at 100% on raspberry.

from gphotos-sync.

from gphotos-sync.

I will try what you ask later. Since last post, I went to lunch and back. There are 2 new lines in log file :

gphotos.picasa: Updated /media/toshiba/picasa/2017/03/00013.MTS
gphotos.utils: token refresh: ya29.GlwSBdggnJIqQz-aZEJYxTVOIz8Vsv4uXtNsI0A5rvolRNlXyU0hurXpNggqcPm3BxR3pWfrhaYe798TW1ZhIZHvhqvUxEjlRS6SHAmSOGtewt9n5OHlcG88cs4UEg

I'm still waiting, cpu at 100%.

from gphotos-sync.

Just did it, nothing in etc/

I restart it in background to check.

from gphotos-sync.

Do it again, still nothing.

Restart in foreground mode, I will let it work an hour to check if it's moving forward.

from gphotos-sync.

OK I'll need to fix the stack trace dump, apologies.

It seems like it is still running, just very slow. Is memory OK? I'm not sure how to interpret htop output and the Mem bar looks full.

from gphotos-sync.

Memory was ok. Only 132M really used over 434.

Maybe you should add more logs in each steps, and a verbose mode to display them (that will help us trace where it hangs)

from gphotos-sync.

I just managed to get gphotos-sync to hang.
I was testing background operation and accidentally ran two. One aborted with DB locked, I cancelled the other and then restarted. It then always locked up on indexing folders, very early.

I deleted the DB from the filesystem and all was well.

So it seems it is possible to corrupt the DB in a subtle way that causes a hang. It is possible that this is what you are seeing...

from gphotos-sync.

Further update. I have noticed that the picasa indexing phase is processor bound and hits 100% on one processor for most of its execution. This might be a bit rough on the Pi.
The reason is that the Drive matching is running for each item discovered, each match may make several DB queries.

It would probably make sense to split the indexing of items in picasaWeb and the matching.

Matching would be processor intensive but should not take too long since it does not need to wait for responses from Google.

from gphotos-sync.

The evidence now suggests that this was due to a corrupt database. It seems that this may be caused by running two instances against the same DB (although db locking should handle this). I will add a singleton check to avoid this issue.

from gphotos-sync.

locking so that only one instance can run against a given db location has been added in cbe040d

from gphotos-sync.