Comments (32)
Please check you are redirecting stderr.
The finally clause in the top level try block logs 'Done', so was not called and would trap kill so this was a hard exit.
from gphotos-sync.
Pierre, is this a duplicate of #15? I thought that #15 related to locking up with 100% processor but maybe you meant it exited as above?
from gphotos-sync.
I don't think so because on #15 problem did not kill the process, it was stucked at 100%.
from gphotos-sync.
OK, good that is what I thought, and I think we have concluded that the 100% processor issue was due to a corrupt database.
from gphotos-sync.
Also please can you check your system logs for something at the time of termination?
from gphotos-sync.
Terminal is hanged on this :
2017-11-30 11:28:51,937 gphotos.drive: downloading /media/toshiba/drive/Google Photos/2016/20160614_123920.jpg ...
2017-11-30 11:28:56,261 gphotos.drive: downloading /media/toshiba/drive/Google Photos/2016/20160521_103028.mp4 ...
2017-11-30 11:30:48,463 gphotos.utils: RETRYING due to: MemoryError()
2017-11-30 11:30:48,588 gphotos.utils: Call was: <bound method GoogleDriveFile.GetContentFile of GoogleDriveFile({u'mimeType': u'video/mp4', u'appDataContents': False, u'thumbnailLink': u'https://lh3.googleusercontent.com/44MnKwCXnOVHTlE9MgrAFnlaxUOoVsQUxWdGUtwJOJiM8FVa0XaFqwf4KjOWoqhmOywAuadIR9I4=s220', u'labels': {u'restricted': False, u'starred': False, u'viewed': False, u'hidden': False, u'trashed': False}, u'explicitlyTrashed': False, u'etag': u'"rOX0fJzgyZU36BYOEcGKIYxPCtI/MTQ2NjcyNTYxNTAwMA"', u'lastModifyingUserName': u'P&S Rigalous', u'writersCanShare': True, u'owners': [{u'picture': {u'url': u'https://lh5.googleusercontent.com/-bHbJW3OjwK4/AAAAAAAAAAI/AAAAAAAAAI0/e_YY6rZWqCU/s64/photo.jpg'}, u'kind': u'drive#user', u'displayName': u'P&S Rigalous', u'permissionId': u'15840638401364870311', u'isAuthenticatedUser': True, u'emailAddress': u'[email protected]'}], 'id': u'1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA', u'videoMediaMetadata': {u'width': 1920, u'durationMillis': u'119479', u'height': 1080}, u'lastModifyingUser': {u'picture': {u'url': u'https://lh5.googleusercontent.com/-bHbJW3OjwK4/AAAAAAAAAAI/AAAAAAAAAI0/e_YY6rZWqCU/s64/photo.jpg'}, u'kind': u'drive#user', u'displayName': u'P&S Rigalous', u'permissionId': u'15840638401364870311', u'isAuthenticatedUser': True, u'emailAddress': u'[email protected]'}, u'title': u'20160521_103028.mp4', u'ownerNames': [u'P&S Rigalous'], u'capabilities': {u'canCopy': True, u'canEdit': True}, u'version': u'102889', u'parents': [{u'isRoot': False, u'kind': u'drive#parentReference', u'id': u'1Cqjhz0YsLoBFgVMhnIn9uWNeNQ', u'selfLink': u'https://www.googleapis.com/drive/v2/files/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/parents/1Cqjhz0YsLoBFgVMhnIn9uWNeNQ', u'parentLink': u'https://www.googleapis.com/drive/v2/files/1Cqjhz0YsLoBFgVMhnIn9uWNeNQ'}], u'shared': False, u'originalFilename': u'20160521_103028.mp4', u'description': u'', u'webContentLink': u'https://drive.google.com/uc?id=1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA&export=download', u'editable': True, u'embedLink': u'https://drive.google.com/file/d/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/preview?usp=drivesdk', u'markedViewedByMeDate': u'1970-01-01T00:00:00.000Z', u'quotaBytesUsed': u'0', u'modifiedDate': u'2016-06-23T23:46:55.000Z', u'createdDate': u'2016-05-21T08:32:29.000Z', u'md5Checksum': u'bbfe9a5cdef25b05b83f6778e1fe2d4f', u'iconLink': u'https://drive-thirdparty.googleusercontent.com/16/type/video/mp4', u'kind': u'drive#file', u'alternateLink': u'https://drive.google.com/file/d/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/view?usp=drivesdk', u'copyable': True, u'modifiedByMeDate': u'2016-06-23T23:46:55.000Z', u'downloadUrl': u'https://doc-0c-54-docs.googleusercontent.com/docs/securesc/ik2846oc7lsicnetn647n4i8r8p8so39/ba2b1d67v063sqtmfoif7stl60uee5dk/1512036000000/15840638401364870311/15840638401364870311/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA?e=download&gd=true', u'userPermission': {u'kind': u'drive#permission', u'etag': u'"rOX0fJzgyZU36BYOEcGKIYxPCtI/UpLg_Q20J2zGB56s-9FtcP0AYww"', u'role': u'owner', u'type': u'user', u'id': u'me', u'selfLink': u'https://www.googleapis.com/drive/v2/files/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/permissions/me'}, u'spaces': [u'photos', u'drive'], u'fileExtension': u'mp4', u'headRevisionId': u'14ll_8ySol2J9sO2zyBItUTyeZLGefmR-RSmCJ3lxn5VmfkooHw', u'selfLink': u'https://www.googleapis.com/drive/v2/files/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA', u'fileSize': u'257903955'})> (('/media/toshiba/.temp-photo',), {})
2017-11-30 11:31:42,617 gphotos.utils: RETRYING due to: MemoryError()
2017-11-30 11:31:42,739 gphotos.utils: Call was: <bound method GoogleDriveFile.GetContentFile of GoogleDriveFile({u'mimeType': u'video/mp4', u'appDataContents': False, u'thumbnailLink': u'https://lh3.googleusercontent.com/44MnKwCXnOVHTlE9MgrAFnlaxUOoVsQUxWdGUtwJOJiM8FVa0XaFqwf4KjOWoqhmOywAuadIR9I4=s220', u'labels': {u'restricted': False, u'starred': False, u'viewed': False, u'hidden': False, u'trashed': False}, u'explicitlyTrashed': False, u'etag': u'"rOX0fJzgyZU36BYOEcGKIYxPCtI/MTQ2NjcyNTYxNTAwMA"', u'lastModifyingUserName': u'P&S Rigalous', u'writersCanShare': True, u'owners': [{u'picture': {u'url': u'https://lh5.googleusercontent.com/-bHbJW3OjwK4/AAAAAAAAAAI/AAAAAAAAAI0/e_YY6rZWqCU/s64/photo.jpg'}, u'kind': u'drive#user', u'displayName': u'P&S Rigalous', u'permissionId': u'15840638401364870311', u'isAuthenticatedUser': True, u'emailAddress': u'[email protected]'}], 'id': u'1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA', u'videoMediaMetadata': {u'width': 1920, u'durationMillis': u'119479', u'height': 1080}, u'lastModifyingUser': {u'picture': {u'url': u'https://lh5.googleusercontent.com/-bHbJW3OjwK4/AAAAAAAAAAI/AAAAAAAAAI0/e_YY6rZWqCU/s64/photo.jpg'}, u'kind': u'drive#user', u'displayName': u'P&S Rigalous', u'permissionId': u'15840638401364870311', u'isAuthenticatedUser': True, u'emailAddress': u'[email protected]'}, u'title': u'20160521_103028.mp4', u'ownerNames': [u'P&S Rigalous'], u'capabilities': {u'canCopy': True, u'canEdit': True}, u'version': u'102889', u'parents': [{u'isRoot': False, u'kind': u'drive#parentReference', u'id': u'1Cqjhz0YsLoBFgVMhnIn9uWNeNQ', u'selfLink': u'https://www.googleapis.com/drive/v2/files/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/parents/1Cqjhz0YsLoBFgVMhnIn9uWNeNQ', u'parentLink': u'https://www.googleapis.com/drive/v2/files/1Cqjhz0YsLoBFgVMhnIn9uWNeNQ'}], u'shared': False, u'originalFilename': u'20160521_103028.mp4', u'description': u'', u'webContentLink': u'https://drive.google.com/uc?id=1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA&export=download', u'editable': True, u'embedLink': u'https://drive.google.com/file/d/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/preview?usp=drivesdk', u'markedViewedByMeDate': u'1970-01-01T00:00:00.000Z', u'quotaBytesUsed': u'0', u'modifiedDate': u'2016-06-23T23:46:55.000Z', u'createdDate': u'2016-05-21T08:32:29.000Z', u'md5Checksum': u'bbfe9a5cdef25b05b83f6778e1fe2d4f', u'iconLink': u'https://drive-thirdparty.googleusercontent.com/16/type/video/mp4', u'kind': u'drive#file', u'alternateLink': u'https://drive.google.com/file/d/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/view?usp=drivesdk', u'copyable': True, u'modifiedByMeDate': u'2016-06-23T23:46:55.000Z', u'downloadUrl': u'https://doc-0c-54-docs.googleusercontent.com/docs/securesc/ik2846oc7lsicnetn647n4i8r8p8so39/ba2b1d67v063sqtmfoif7stl60uee5dk/1512036000000/15840638401364870311/15840638401364870311/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA?e=download&gd=true', u'userPermission': {u'kind': u'drive#permission', u'etag': u'"rOX0fJzgyZU36BYOEcGKIYxPCtI/UpLg_Q20J2zGB56s-9FtcP0AYww"', u'role': u'owner', u'type': u'user', u'id': u'me', u'selfLink': u'https://www.googleapis.com/drive/v2/files/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA/permissions/me'}, u'spaces': [u'photos', u'drive'], u'fileExtension': u'mp4', u'headRevisionId': u'14ll_8ySol2J9sO2zyBItUTyeZLGefmR-RSmCJ3lxn5VmfkooHw', u'selfLink': u'https://www.googleapis.com/drive/v2/files/1zjEx3qpV9sEZYXINLGWudb6o6CZfFvYsmA', u'fileSize': u'257903955'})> (('/media/toshiba/.temp-photo',), {})
My raspberry is no more accessible via ssh, I'll reboot it tonight.
from gphotos-sync.
from gphotos-sync.
from gphotos-sync.
if possible please could you send the .gphotos.terminated file from the above crash. Thanks.
Ooops: you cant because it is still running, so if it is still hanging then kill and send me the .gphotos.terminated. Thanks.
from gphotos-sync.
and the file size of the last mp4 listed
from gphotos-sync.
I'm still at work and my raspberry is unreachable. I will reboot it and send you .terminated file if present when I'll be at home.
from gphotos-sync.
Just come back to my home, raspberry was still unreachable from local network, had to reboot it. No .terminated file and nothing more in log.
from gphotos-sync.
can you tell me how big /media/toshiba/drive/Google Photos/2016/20160521_103028.mp4 (not the local file but the version on google photos), please?
I'm pretty sure that you Pi was out of memory and swapping so much it was unusable.
I've looked at the Pydrive code and indeed it downloads the entire file into memory and then saves it.
I will look at changing this behaviour (this is google code but is open source) but it will probably be a few days.
In the meantime I guess you could try out an entire run by using --skip-video. I hope at this should allow your entire library minus videos to come down since I think we have cracked all the other showstoppers.
I'll let you know when I have a new PyDrive to try out.
from gphotos-sync.
20160521_103028.mp4 -> 240Mo
Yesterday before going to bed, I launch script again (with error redirection to stdout)
This morning script was stopped without any error, last line was :
2017-11-30 23:55:33,879 gphotos.picasa: unmatched /media/toshiba/picasa/2016/03/20160328_111303.mp4 will be downloaded
No .terminated file again.
from gphotos-sync.
I think Linux is sending a sigterm to the process because it has used up all memory, even swap. We can't trap that and terminate without cleaning up (I suggest you delete the DB file after this).
I suggest you try the --skip-video option for the moment and I will have a go at modifying google's gdrive to download big files incrementally (this will take a few days at least)
from gphotos-sync.
I just realized the last report is for an mp4 downloading via picasa, the first was from drive. So I need to fix the download code in two places!
from gphotos-sync.
Last report is not about a download, it's during indexing stage.
from gphotos-sync.
Oh yeah, sorry. After the reboot did you delete the DB file? You almost certainly had a corrupt DB if you had to reboot.
from gphotos-sync.
When you get an exit with no error, please can you look in the system logs to see if anything suspicious happened at the time? I can only guess that the process is getting a hard kill, sigterm which it cannot trap, if this is true then the OS must be sending it for some reason and I would expect a log entry.
from gphotos-sync.
I think I finally have an excuse to buy a raspberry Pi :-). Your use case for gphotos-sync and Pi makes a lot of sense, but it is hard for me to debug from a distance since I am not getting any of the errors you are reporting.
from gphotos-sync.
Just had problem again (program killed without any output)
Nothing in any system logs...
I will launch again in foreground since PI is still reachable.
from gphotos-sync.
from gphotos-sync.
Here is a thought. You will be saturating the WiFi when downloading big files as you are trying to read and write simultaneously at your broadband download rate.
If this caused your SSH session to timeout then you would see any child processes die immediately.
If you agree this is s possibility the take a look https://www.bjornjohansen.no/ssh-timeout
from gphotos-sync.
from gphotos-sync.
from gphotos-sync.
gphotos-sync is up and running on my Pi Zero W. Using the same setup as you (except my NAS is an NFS share onto my desktop).
I will report back tomorrow.
Note I used the following script to allow disconnecting the ssh session without terminating the child gphotos-sync process:-
#! /bin/bash
set -x
cd /home/pi/gphotos-sync
LOG=~/logs/$(date +"%F_%H.%M.%S").log
./gphotos-sync --all-drive --db-path /home/pi/db $@ /mnt/gklinux/Data/Raspberry/giles-full/ &>${LOG} &
disown -ah
sleep 1 # make sure process has had time to start and output to logfile
tail -f ${LOG}
from gphotos-sync.
Update: using the above script the pi ran indexing with no issues. Picasa indexing was slow at about 1 image per second.
As expected it got out of memory error when trying to download 300Mb video. It exited cleanly and reported the error with stack trace.
I will endeavour to patch gdrive to fix this. Gdata for Picasa may be more challenging and I may look at replacing it with REST.
from gphotos-sync.
Update: OK I'm looking at fixing PyDrive to allow large downloads and I've got into totally replacing its transport with the requests library. This is going to take some time but is very interesting.
In the meantime, if you get your Pi up again there is an easy mitigation against the large file issue. You can increase your swap file size to a max of 2GB. If none of your files exceeds 2GB then you are good to go. I tried this on mine. If you do have files > 2GB let me know and I'll add a max-size param that lets you skip them for the moment.
from gphotos-sync.
from gphotos-sync.
@prigal if you are still interested in this project you should find that the latest release will be OK on the PI.
- It now does file copies via stream so does not need masses of memory for large files.
- There is now full debug logging to file, so if there are issues we can more easily debug
I will be trying out on my Pi Zero soon and will close this bug if all is well.
from gphotos-sync.
@gilesknap Thanks for update. I will try this again one day but I have to find a new disk first, my NAS has gone over :/
from gphotos-sync.
The latest version has been tested on my pi zero and all looks good.
Improvements that allow it to work are:
- Downloads are streamed so files larger than memory can be downloaded
- Automatic debug logging allows unexpected exits to be diagnosed
from gphotos-sync.
Related Issues (20)
- CloudFormation multiple resource file error? HOT 2
- FREEBSD - pstuil dependency too high HOT 2
- Shared photos are removed and then re-added in version 2.14.2 HOT 4
- Incremental Indexing HOT 7
- Comparison to rclone with google-photos backend HOT 1
- Error When Connecting - Running Cloud Flare and NGINX HOT 2
- NFS Share [Errno 28] No space left on device HOT 7
- Error when running gphotos-sync HOT 3
- Container Restarting HOT 5
- Please delete this thread - Sorry. HOT 1
- No new photos HOT 15
- Why are my photos being deleted HOT 5
- Invalid Syntax HOT 4
- Syntax error on syncing folders called '3 years later' HOT 4
- Not working after upgrade to Python 3.12 & gphotos-sync 3.2 HOT 11
- Many 404 client error while downloading HOT 7
- Creating Duplicate Photos Once Finished HOT 10
- Escape special characters HOT 3
- why an image do not have exif data. HOT 6
- Is there any way to make the timezone correct? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gphotos-sync.