Giter Club home page Giter Club logo

Comments (32)

gilesknap avatar gilesknap commented on May 16, 2024

Please check you are redirecting stderr.
The finally clause in the top level try block logs 'Done', so was not called and would trap kill so this was a hard exit.

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

Pierre, is this a duplicate of #15? I thought that #15 related to locking up with 100% processor but maybe you meant it exited as above?

from gphotos-sync.

prigal avatar prigal commented on May 16, 2024

I don't think so because on #15 problem did not kill the process, it was stucked at 100%.

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

OK, good that is what I thought, and I think we have concluded that the 100% processor issue was due to a corrupt database.

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

Also please can you check your system logs for something at the time of termination?

from gphotos-sync.

prigal avatar prigal commented on May 16, 2024

Terminal is hanged on this :

2017-11-30 11:28:51,937 gphotos.drive: downloading /media/toshiba/drive/Google Photos/2016/20160614_123920.jpg ...
2017-11-30 11:28:56,261 gphotos.drive: downloading /media/toshiba/drive/Google Photos/2016/20160521_103028.mp4 ...
2017-11-30 11:30:48,463 gphotos.utils: RETRYING due to: MemoryError()
2017-11-30 11:30:48,588 gphotos.utils: Call was: <bound method GoogleDriveFile.GetContentFile of GoogleDriveFile({u'mimeType': u'video/mp4', u'appDataContents': False, u'thumbnailLink': u'https://lh3.googleusercontent.com/44MnKwCXnOVHTlE9MgrAFnlaxUOoVsQUxWdGUtwJOJiM8FVa0XaFqwf4KjOWoqhmOywAuadIR9I4=s220', u'labels': {u'restricted': False, u'starred': False, u'viewed': False, u'hidden': False, u'trashed': False}, u'explicitlyTrashed': False, u'etag': u'"rOX0fJzgyZU36BYOEcGKIYxPCtI/MTQ2NjcyNTYxNTAwMA"', u'lastModifyingUserName': u'P&S Rigalous', u'writersCanShare': True, u'owners': [{u'picture': {u'url': u'https://lh5.googleusercontent.com/-bHbJW3OjwK4/AAAAAAAAAAI/AAAAAAAAAI0/e_YY6rZWqCU/s64/photo.jpg'}, u'kind': u'drive#user', u'displayName': u'P&S Rigalous', u'permissionId': u'15840638401364870311', u'isAuthenticatedUser': True, u'emailAddress': u'[email protected]'}], 'id': u'1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA', u'videoMediaMetadata': {u'width': 1920, u'durationMillis': u'119479', u'height': 1080}, u'lastModifyingUser': {u'picture': {u'url': u'https://lh5.googleusercontent.com/-bHbJW3OjwK4/AAAAAAAAAAI/AAAAAAAAAI0/e_YY6rZWqCU/s64/photo.jpg'}, u'kind': u'drive#user', u'displayName': u'P&S Rigalous', u'permissionId': u'15840638401364870311', u'isAuthenticatedUser': True, u'emailAddress': u'[email protected]'}, u'title': u'20160521_103028.mp4', u'ownerNames': [u'P&S Rigalous'], u'capabilities': {u'canCopy': True, u'canEdit': True}, u'version': u'102889', u'parents': [{u'isRoot': False, u'kind': u'drive#parentReference', u'id': u'1Cqjhz0YsLoBFgVMhnIn9uWNeNQ', u'selfLink': u'https://www.googleapis.com/drive/v2/files/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/parents/1Cqjhz0YsLoBFgVMhnIn9uWNeNQ', u'parentLink': u'https://www.googleapis.com/drive/v2/files/1Cqjhz0YsLoBFgVMhnIn9uWNeNQ'}], u'shared': False, u'originalFilename': u'20160521_103028.mp4', u'description': u'', u'webContentLink': u'https://drive.google.com/uc?id=1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA&export=download', u'editable': True, u'embedLink': u'https://drive.google.com/file/d/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/preview?usp=drivesdk', u'markedViewedByMeDate': u'1970-01-01T00:00:00.000Z', u'quotaBytesUsed': u'0', u'modifiedDate': u'2016-06-23T23:46:55.000Z', u'createdDate': u'2016-05-21T08:32:29.000Z', u'md5Checksum': u'bbfe9a5cdef25b05b83f6778e1fe2d4f', u'iconLink': u'https://drive-thirdparty.googleusercontent.com/16/type/video/mp4', u'kind': u'drive#file', u'alternateLink': u'https://drive.google.com/file/d/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/view?usp=drivesdk', u'copyable': True, u'modifiedByMeDate': u'2016-06-23T23:46:55.000Z', u'downloadUrl': u'https://doc-0c-54-docs.googleusercontent.com/docs/securesc/ik2846oc7lsicnetn647n4i8r8p8so39/ba2b1d67v063sqtmfoif7stl60uee5dk/1512036000000/15840638401364870311/15840638401364870311/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA?e=download&gd=true', u'userPermission': {u'kind': u'drive#permission', u'etag': u'"rOX0fJzgyZU36BYOEcGKIYxPCtI/UpLg_Q20J2zGB56s-9FtcP0AYww"', u'role': u'owner', u'type': u'user', u'id': u'me', u'selfLink': u'https://www.googleapis.com/drive/v2/files/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/permissions/me'}, u'spaces': [u'photos', u'drive'], u'fileExtension': u'mp4', u'headRevisionId': u'14ll_8ySol2J9sO2zyBItUTyeZLGefmR-RSmCJ3lxn5VmfkooHw', u'selfLink': u'https://www.googleapis.com/drive/v2/files/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA', u'fileSize': u'257903955'})> (('/media/toshiba/.temp-photo',), {})
2017-11-30 11:31:42,617 gphotos.utils: RETRYING due to: MemoryError()
2017-11-30 11:31:42,739 gphotos.utils: Call was: <bound method GoogleDriveFile.GetContentFile of GoogleDriveFile({u'mimeType': u'video/mp4', u'appDataContents': False, u'thumbnailLink': u'https://lh3.googleusercontent.com/44MnKwCXnOVHTlE9MgrAFnlaxUOoVsQUxWdGUtwJOJiM8FVa0XaFqwf4KjOWoqhmOywAuadIR9I4=s220', u'labels': {u'restricted': False, u'starred': False, u'viewed': False, u'hidden': False, u'trashed': False}, u'explicitlyTrashed': False, u'etag': u'"rOX0fJzgyZU36BYOEcGKIYxPCtI/MTQ2NjcyNTYxNTAwMA"', u'lastModifyingUserName': u'P&S Rigalous', u'writersCanShare': True, u'owners': [{u'picture': {u'url': u'https://lh5.googleusercontent.com/-bHbJW3OjwK4/AAAAAAAAAAI/AAAAAAAAAI0/e_YY6rZWqCU/s64/photo.jpg'}, u'kind': u'drive#user', u'displayName': u'P&S Rigalous', u'permissionId': u'15840638401364870311', u'isAuthenticatedUser': True, u'emailAddress': u'[email protected]'}], 'id': u'1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA', u'videoMediaMetadata': {u'width': 1920, u'durationMillis': u'119479', u'height': 1080}, u'lastModifyingUser': {u'picture': {u'url': u'https://lh5.googleusercontent.com/-bHbJW3OjwK4/AAAAAAAAAAI/AAAAAAAAAI0/e_YY6rZWqCU/s64/photo.jpg'}, u'kind': u'drive#user', u'displayName': u'P&S Rigalous', u'permissionId': u'15840638401364870311', u'isAuthenticatedUser': True, u'emailAddress': u'[email protected]'}, u'title': u'20160521_103028.mp4', u'ownerNames': [u'P&S Rigalous'], u'capabilities': {u'canCopy': True, u'canEdit': True}, u'version': u'102889', u'parents': [{u'isRoot': False, u'kind': u'drive#parentReference', u'id': u'1Cqjhz0YsLoBFgVMhnIn9uWNeNQ', u'selfLink': u'https://www.googleapis.com/drive/v2/files/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/parents/1Cqjhz0YsLoBFgVMhnIn9uWNeNQ', u'parentLink': u'https://www.googleapis.com/drive/v2/files/1Cqjhz0YsLoBFgVMhnIn9uWNeNQ'}], u'shared': False, u'originalFilename': u'20160521_103028.mp4', u'description': u'', u'webContentLink': u'https://drive.google.com/uc?id=1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA&export=download', u'editable': True, u'embedLink': u'https://drive.google.com/file/d/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/preview?usp=drivesdk', u'markedViewedByMeDate': u'1970-01-01T00:00:00.000Z', u'quotaBytesUsed': u'0', u'modifiedDate': u'2016-06-23T23:46:55.000Z', u'createdDate': u'2016-05-21T08:32:29.000Z', u'md5Checksum': u'bbfe9a5cdef25b05b83f6778e1fe2d4f', u'iconLink': u'https://drive-thirdparty.googleusercontent.com/16/type/video/mp4', u'kind': u'drive#file', u'alternateLink': u'https://drive.google.com/file/d/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/view?usp=drivesdk', u'copyable': True, u'modifiedByMeDate': u'2016-06-23T23:46:55.000Z', u'downloadUrl': u'https://doc-0c-54-docs.googleusercontent.com/docs/securesc/ik2846oc7lsicnetn647n4i8r8p8so39/ba2b1d67v063sqtmfoif7stl60uee5dk/1512036000000/15840638401364870311/15840638401364870311/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA?e=download&gd=true', u'userPermission': {u'kind': u'drive#permission', u'etag': u'"rOX0fJzgyZU36BYOEcGKIYxPCtI/UpLg_Q20J2zGB56s-9FtcP0AYww"', u'role': u'owner', u'type': u'user', u'id': u'me', u'selfLink': u'https://www.googleapis.com/drive/v2/files/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/permissions/me'}, u'spaces': [u'photos', u'drive'], u'fileExtension': u'mp4', u'headRevisionId': u'14ll_8ySol2J9sO2zyBItUTyeZLGefmR-RSmCJ3lxn5VmfkooHw', u'selfLink': u'https://www.googleapis.com/drive/v2/files/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA', u'fileSize': u'257903955'})> (('/media/toshiba/.temp-photo',), {})

My raspberry is no more accessible via ssh, I'll reboot it tonight.

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

if possible please could you send the .gphotos.terminated file from the above crash. Thanks.
Ooops: you cant because it is still running, so if it is still hanging then kill and send me the .gphotos.terminated. Thanks.

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

and the file size of the last mp4 listed

from gphotos-sync.

prigal avatar prigal commented on May 16, 2024

I'm still at work and my raspberry is unreachable. I will reboot it and send you .terminated file if present when I'll be at home.

from gphotos-sync.

prigal avatar prigal commented on May 16, 2024

Just come back to my home, raspberry was still unreachable from local network, had to reboot it. No .terminated file and nothing more in log.

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

can you tell me how big /media/toshiba/drive/Google Photos/2016/20160521_103028.mp4 (not the local file but the version on google photos), please?

I'm pretty sure that you Pi was out of memory and swapping so much it was unusable.

I've looked at the Pydrive code and indeed it downloads the entire file into memory and then saves it.
I will look at changing this behaviour (this is google code but is open source) but it will probably be a few days.

In the meantime I guess you could try out an entire run by using --skip-video. I hope at this should allow your entire library minus videos to come down since I think we have cracked all the other showstoppers.

I'll let you know when I have a new PyDrive to try out.

from gphotos-sync.

prigal avatar prigal commented on May 16, 2024

20160521_103028.mp4 -> 240Mo

Yesterday before going to bed, I launch script again (with error redirection to stdout)

This morning script was stopped without any error, last line was :

2017-11-30 23:55:33,879 gphotos.picasa: unmatched /media/toshiba/picasa/2016/03/20160328_111303.mp4 will be downloaded

No .terminated file again.

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

I think Linux is sending a sigterm to the process because it has used up all memory, even swap. We can't trap that and terminate without cleaning up (I suggest you delete the DB file after this).

I suggest you try the --skip-video option for the moment and I will have a go at modifying google's gdrive to download big files incrementally (this will take a few days at least)

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

I just realized the last report is for an mp4 downloading via picasa, the first was from drive. So I need to fix the download code in two places!

from gphotos-sync.

prigal avatar prigal commented on May 16, 2024

Last report is not about a download, it's during indexing stage.

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

Oh yeah, sorry. After the reboot did you delete the DB file? You almost certainly had a corrupt DB if you had to reboot.

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

When you get an exit with no error, please can you look in the system logs to see if anything suspicious happened at the time? I can only guess that the process is getting a hard kill, sigterm which it cannot trap, if this is true then the OS must be sending it for some reason and I would expect a log entry.

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

I think I finally have an excuse to buy a raspberry Pi :-). Your use case for gphotos-sync and Pi makes a lot of sense, but it is hard for me to debug from a distance since I am not getting any of the errors you are reporting.

from gphotos-sync.

prigal avatar prigal commented on May 16, 2024

Just had problem again (program killed without any output)
Nothing in any system logs...

I will launch again in foreground since PI is still reachable.

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

Here is a thought. You will be saturating the WiFi when downloading big files as you are trying to read and write simultaneously at your broadband download rate.

If this caused your SSH session to timeout then you would see any child processes die immediately.

If you agree this is s possibility the take a look https://www.bjornjohansen.no/ssh-timeout

from gphotos-sync.

prigal avatar prigal commented on May 16, 2024

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

gphotos-sync is up and running on my Pi Zero W. Using the same setup as you (except my NAS is an NFS share onto my desktop).

I will report back tomorrow.

Note I used the following script to allow disconnecting the ssh session without terminating the child gphotos-sync process:-

#! /bin/bash
set -x
cd /home/pi/gphotos-sync
LOG=~/logs/$(date +"%F_%H.%M.%S").log
./gphotos-sync --all-drive --db-path /home/pi/db $@ /mnt/gklinux/Data/Raspberry/giles-full/ &>${LOG} &
disown -ah
sleep 1 # make sure process has had time to start and output to logfile
tail -f ${LOG}

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

Update: using the above script the pi ran indexing with no issues. Picasa indexing was slow at about 1 image per second.

As expected it got out of memory error when trying to download 300Mb video. It exited cleanly and reported the error with stack trace.

I will endeavour to patch gdrive to fix this. Gdata for Picasa may be more challenging and I may look at replacing it with REST.

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

Update: OK I'm looking at fixing PyDrive to allow large downloads and I've got into totally replacing its transport with the requests library. This is going to take some time but is very interesting.

In the meantime, if you get your Pi up again there is an easy mitigation against the large file issue. You can increase your swap file size to a max of 2GB. If none of your files exceeds 2GB then you are good to go. I tried this on mine. If you do have files > 2GB let me know and I'll add a max-size param that lets you skip them for the moment.

from gphotos-sync.

prigal avatar prigal commented on May 16, 2024

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

@prigal if you are still interested in this project you should find that the latest release will be OK on the PI.

  • It now does file copies via stream so does not need masses of memory for large files.
  • There is now full debug logging to file, so if there are issues we can more easily debug

I will be trying out on my Pi Zero soon and will close this bug if all is well.

from gphotos-sync.

prigal avatar prigal commented on May 16, 2024

@gilesknap Thanks for update. I will try this again one day but I have to find a new disk first, my NAS has gone over :/

from gphotos-sync.

gilesknap avatar gilesknap commented on May 16, 2024

The latest version has been tested on my pi zero and all looks good.
Improvements that allow it to work are:

  • Downloads are streamed so files larger than memory can be downloaded
  • Automatic debug logging allows unexpected exits to be diagnosed

from gphotos-sync.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.