This may not be an issue with gphotos-cdp per-se, but I think it should be documented.

If you have photos with a percent sign in the file name (not the description), such as (from my library):

Downtown C%F3rdoba.JPG
C%F3rdoba Capital.JPG

These will fail to download in Chrome. This causes gphotos-cdp to fail.

If running a headless session, the program will exit after a time out:

2021/01/01 17:48:57 ERROR: unhandled page event *page.EventDownloadProgress
2021/01/01 17:48:58 ERROR: unhandled page event *page.EventDownloadProgress
2021/01/01 17:48:58 ERROR: unhandled page event *page.EventDownloadProgress
2021/01/01 17:51:30 downloading in "/GooglePhotos" took too long to start
exit status 1

You can observe the download failure by running the program non-headless. You can also see that Chrome cannot download these files by running Chrome normally and attempting to download them. Firefox can download them fine.

Workaround is just to use the --start flag to skip the photo, and download it manually in another browser.

Since this appears to be a Chrome issue, I don't think a fix in gphotos-cdp would make sense.

Tested on FreeBSD 12.2-RELEASE, Chrome 87.0.4280.88.

Thanks for the great project. I'm trying this tool first time, but got this failure. Should I do anything?

$ ./gphotos-cdp -v
2020/11/15 18:26:32 Session Dir: /var/folders/p1/bx0fgbnj0ljbygmb7hr32zj00000gn/T/gphotos-cdp897911744
------ <nil>
2020/11/15 18:26:33 pre-navigate
2020/11/15 18:26:35 Not yet authenticated, at: https://www.google.com/photos/about/
2020/11/15 18:26:36 Not yet authenticated, at: https://www.google.com/photos/about/

After the workaround described in #1 I was able to sign in and start downloading photos.

The first 20-30 photos downloaded as expected, but then I wanted to test resume. So I ctrl+c'ed the process and restarted it (I'm invoking with ./gphotos-cdp -dev=true -v=true). It then downloaded one photo and exited:

$ ./gphotos-cdp  -dev=true -v=true
2020/01/25 17:48:21 Session Dir: /tmp/gphotos-cdp
2020/01/25 17:48:21 pre-navigate
2020/01/25 17:48:23 post-navigate
2020/01/25 17:48:25 Event: {Type:keyDown Modifiers:Shift Timestamp:<nil> Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0}
2020/01/25 17:48:25 Event: {Type:keyUp Modifiers:Shift Timestamp:<nil> Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0}
2020/01/25 17:48:25 Marking https://photos.google.com/photo/XYZredacted as done
OK

Running gphotos-cdp subsequent times results in the same behavior: one photo downloads with the output above and then the process exits. If I add -n=100 when executing, the behavior is the same.

If I start from scratch and this time close Chrome by clicking the X in the window decoration, I notice the process does not exit, but no output is produced. Ctrl+c'ing from here and re-rerunning results in the same behavior.

Has there been any community or private discussion of the Google Photos Library API which I presume didn't exist when this project was last significantly developed? Has anyone taken a look at this? I'm trying to gauge if I'm interested enough (and if there's enough community interest) to contribute to this.

Following a successful sync, if the most recent item (the one in .lastdone) is deleted from Google Photos, the next sync hangs indefinitely.

Uploading more items (=creating a new most recent item in the feed) won't fix the problem. Only restoring the deleted item from the trash enables successful completion.

Removing .lastdone triggers full sync.

In a navigation flow where navToEnd and navToLast are invoked (i.e. no --start or .lastdone present), navToLast goes into an infinite loop.

This seems to be due to a recent change in the web app's behavior.

• navToLast starts by entering the Detail/Photo page by sending a \n
• This assumes that an element on the Main/Album page is selected/focused.

What has changed is that the action of scrolling (navToEnd) no longer preserves a selected item, so that the \n in navToLast has no effect, and goes into an infinite loop.

Would like to know if I am the only one experiencing this.

Hi there,

Seems like I have issues building from master, certainly due to third party dependencies not matching.
Is there a migration to go modules planned or any package dependency system?

Thanks!

I don't think you can solve this problem, but shared photos when downloaded do not have any (useful) EXIF metadata such as creation/modified timestamp, geo. In the web UI they do (sOcIaL) but not in the downloaded item.

The navToEnd function on slower internet connections does not achieve the expected because the unloaded images gray boxes makes the process think it has reached the end as there are identical screen shots. Updating line 362 as follows resolves this issue without impacting the process too much.

time.Sleep(10 * tick)

This could be done as a post-processing step externally, but then you either have to do it on all files every time or try to track new items. It doesn't seem outside of the responsibility of the tool to consider making this a built-in feature.

hi guys,

iam pretty new in unraid and need some help to configure this great tool on my unraid server.

can please somebody help me there?

Thanks for a very useful tool.

Sorry if this kind of advice is elsewhere on the site already.

More of a warning to anyone that runs into the problem I did regarding the .lastdone file. (Put a copy in another directory before a restart.)

A tl;dr summary, when you are about to fire up a new run of the tool, copy the .lastdone from the downloads directory to a safe place in case you want to put it back before a subsequent restart.

The .lastdone file comes in very handy for sessions that are going to be long and that could be interrupted and therefore require a manual restart. This happened to me yesterday and again today.

The initial session can take a very long time to work its way from the last photo or video to the first (in my case about ten year's worth of images). Once it finds the first, it begins downloading and writing/rewriting a .lastdone file in the chosen download directory with the URL of each successive successful download.

Having a .lastdone saves time in two ways, it lets the tool avoid starting from the end and having to scroll its way backwards to the beginning, and it lets the tool avoid redownloading files from the beginning that it had already downloaded successfully.

Any number of things can cause the tool to have to exit. In my case, the one minute timeout for a download was exceeded (caused by a 240MB video). I found the Chrome window still open and on my Mac, closed it using the red button, had a look at the .lastdone file (only because I had looked through the code earlier) and then started the command again with the same arguments (-v -dev -dldir="$HOME/dldir"). But because the same Chrome process was still running, the tool wasn't allowed to start a new test version of Chrome and it aborted immediately with an appropriate log message. (On a mac, closing the window isn't the same as quitting the process, I was reminded.) But the bad part is the .lastdone file had been removed. There is a cleanup of all non-directory entries in the download directory that seems to get triggered even in the case when the chrome process is not going to be started successfully. Restarting the tool showed it starting from scratch.

I had the contexts of the last .lastdone file still in a terminal so I could manually create it to let the tool start from the problem video again. Since then, I copy the .lastdone to a temp directory before any restart.

I'm getting the following error message after downloading the last item: "timeout waiting for left navigation". As a workaround, I get the url of the first item and compare it with the "location" variable. If they are the same "navN" breaks. Is this expected behavior?

Somehow, my .lastdone file got deleted in between runs of the tool (no external program deleted it). So new run now has to find the end again which takes hour.

I see there's a TODO for this which could prevent this:

https://github.com/perkeep/gphotos-cdp/blob/master/main.go#L417

Steps to repro:

1. go run main.go -dev -v
2. chrome opens up
3. navigate to the login, and enter email address
4. observe an error message returned by Google servers

In another attempt I launched chrome without automation but with the data dir set to the same location as gphoto-cdp uses:

1. launch: /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=/var/folders/tt/5skfg0yd4q1bwhr_fh_0t39h0000gn/T/gphotos-cdp
2. Complete authentication successfully!
3. go run main.go -dev -v
4. observe that the same user data dir is used
5. observe that the browser is not authenticated
6. Somehow Google server detects the browser is automated and logs the browser out. OR Chrome itself logs the user out when running automated.

And my apologies for yesterday's issue.

Moving from JakeWharton/docker-gphotos-sync#38:

$ docker run --name gphotos-cdp-sync --rm -v /gphotos-sync:/tmp/gphotos-cdp -v /dl/cdp-gphotos-sync:/download jakewharton/gphotos-sync:0.3.1 /app/sync.sh

...snip
2020/11/15 21:26:24 Running /app/fix_time.sh on /download/AF1QipBizz_jZXshGM1QuBcdxkMmiP-_EDMi7eLjKxHY/DSC00874.JPG
/download/AF1QipBizz_jZXshGM1QuBcdxkMmiP-_EDMi7eLjKxHY/DSC00874.JPG
2020/11/15 21:26:25 Event: {Type:keyDown Modifiers:Shift Timestamp:<nil> Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0}
2020/11/15 21:26:25 Event: {Type:keyUp Modifiers:Shift Timestamp:<nil> Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0}
2020/11/15 21:26:25 ERROR: unhandled page event *page.EventDownloadWillBegin
2020/11/15 21:26:26 more than one file (2) in download dir "/download"
[cmd] /app/sync.sh exited 1
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

This seems to be happening after a few photos are downloaded.

And the files:

-rwxr-xr-x 1 user grp       0 Nov 15 13:26 DSC00853.JPG*
-rwxr-xr-x 1 user grp 3009501 Nov 15 13:26 DSC00853.JPG.crdownload*
-rwxr-xr-x 1 user grp      76 Nov 15 13:26 .lastdone*

Reading the code, the cdp code will freak out if it sees more than 1 file in the directory (as the error says).

I put a watch on my download dir, and here's what I can tell is happening:

1. Chrome starts the download. File name is 100_1291.JPG.crdownload.
2. Chrome finishes the download. File is copied in to 100_1291.JPG.

For that brief period, both files exist. If you get unlucky, the if check can happen during that period, causing the container to die.

I think this can be resolved by one of:

1. ignore .crdownload files.
2. if >1 files, continue after waiting a beat (and let the disk state reconcile

Running in a Docker container you really need Chrome to launch with --no-sandbox. It's not strictly required, as you can elevate the permissions of the Docker container so that Chrome is happy, but it would be nice to just be able to specify this flag somehow so it worked without additional user effort. And since we trust Google Photos (at least except to provide an API...) the sandbox provides no value (especially in an otherwise empty Docker container).

I figure instead of explicitly adding some kind of no-sandbox flag in the tool, you could add a some kind of mechanism for passing additional flags like --chrome-flags "--foo --bar --no-sandbox".

Or you could consider always specifying --no-sandbox.

Hi,

I have rewritten this into Nodejs, Its faster and has fewer bugs. Please check ou: https://github.com/vikas5914/google-photos-downloader

Photos in Google Photos are often too many to download all of them.
It would be great to provide an option to download only the favorite photos.

I am finding that the method to find the oldest photo is not quite working as expected. It doesn't truly go to the oldest place (~2002) for me for some reason.

So what I'm trying is delete the .lastdone file and run the tool repeatedly, however, that causes all existing images to be downloaded again even if they exist on disk.

I can understand why that would be useful by default (sync changes to existing photos), but it would be nice to have a flag that only downloads new files and ignores the ones that exist on disk.

I am using WSL2 to run gphotos-cdp and have noticed that the performance noticeably drops the more files are already downloaded in the download folder. I have measured it to take roughly 1s slower for each download for each 1000 files which have already been downloaded.

I have traced it to a readDir call in the download function, which seems to take longer and longer to scan the whole directory when there are already thousands of directories in the download folder.

As a quick fix I have modified the moveDownload function to move the files/folders into a subdirectory called 'results' (see newDir := line):

func (s *Session) moveDownload(ctx context.Context, dlFile, location string) (string, error) {
	log.Printf("Move Download start")
	parts := strings.Split(location, "/")
	if len(parts) < 5 {
		return "", fmt.Errorf("not enough slash separated parts in location %v: %d", location, len(parts))
	}
	newDir := filepath.Join(s.dlDir, "results", parts[4])
	if err := os.MkdirAll(newDir, 0700); err != nil {

I'm wrapping this tool up in a Docker container for easy consumption by the masses (or, at least, by a few people). Inside the container I want people to mount /config as the Chrome directory which is sort of standard. Right now I have no choice but to use -dev to get a stable path, but that goes to /tmp/gphotos-cdp. I can do a symlink, but it would be nice to be able to pass a path similar to -dldir.

Alternatively, if the arg parsing library does not support optional values for arguments, you could consider replacing -dev with -chromedir or -configdir or anything which always takes a path. Development would then change to specifying -chromedir /tmp/gphotos-cdp (or whatever you wanted).

Or you can do nothing and I can still make things work. Just thought I'd propose it.

Hello,

I'm starting to have issues while downloading my library. I can't figure if it's because I have too many folders in my library and the program is hanging or because of something else.

I'm not a Go developer but I have an env up and running and I'm able to build and add some logs to the program.

This is a trace :

2023/12/10 10:51:19 Session Dir: /tmp/gphotos-cdp
2023/12/10 10:51:20 pre-navigate
2023/12/10 10:51:22 post-navigate
2023/12/10 10:51:23 Page loaded, most recent item in the feed is: AF1QipNDfV7M14cFI66EfUHLzzMEqskDttntw7SQZpRq
2023/12/10 10:51:24 Event: {Type:keyDown Modifiers:Shift Timestamp:<nil> Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0 Commands:[]}
2023/12/10 10:51:24 Event: {Type:keyUp Modifiers:Shift Timestamp:<nil> Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0 Commands:[]}
2023/12/10 10:51:28 Marking https://photos.google.com/photo/AF1QipOaVU56JChJFU-3RafrUU9q5cA_nEMYqIPpW3yQ as done
2023/12/10 10:51:29 Event: {Type:keyDown Modifiers:Shift Timestamp:<nil> Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0 Commands:[]}
2023/12/10 10:51:29 Event: {Type:keyUp Modifiers:Shift Timestamp:<nil> Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0 Commands:[]}
2023/12/10 10:56:29 downloading in "/media/prigal/TOSHIBA2T/gphoto" took too long to start

It looks like it's not able to enter this loop

for _, v := range entries {
	if v.IsDir() {
		continue
	}
	if v.Name() == ".lastdone" {
		continue
	}
	if v.Name() == ".lastdone.bak" {
		continue
	}
	fileEntries = append(fileEntries, v)
}

I tried to allocate more time to deadline but with no more luck

deadline := time.Now().Add(time.Minute * 5)

Any idea of what could go wrong ?

NB : I have already downloaded 238GB over ~2TB and I have 30408 folders in the destination folder.

I'd swear this had worked before, but now, when I attempt to run this, the automated Chrome window refuses to allow me to log in. The details say that automated tooling cannot be used with Chrome to log into my account. Is there some way around this that I'm missing?

I can see that the last commit to this project was 4 years ago, which makes me suspicious.

There is also #37 from May 2023 which I ran into when I tried. I saw the exact same "Couldn't sign you in" screen. That bug is still open, contains some suggestions that where unclear and they apparently didn't work for everyone.

So is the expectation that this project still works in 2024, and exactly how should one invoke it?

If so, please document how to get it running in light of #37.

If not, please update the README to indicate this is abandoned, e.g. by simply adding this at the top of the README.

**WARNING**: This project is abandoned and is known to no longer work. See #37.

That is fair. It saves us mortals from spending useless time trying to get it to work.

I did this on Debian:

$ apt install golang
$ go run main.go

It would be nice if this was documented (once it works), and if that is not it, then document exactly what should work for first-time users.

I am using this via Jake Wharton's docker containerized deployment
https://github.com/JakeWharton/docker-gphotos-sync/blob/trunk/root/app/sync.sh

however even when I run it directly, it seems to finish but it hasn't fetched all of the photos.
I have tried catting a photo URL into .latest, but this does not seem to help.
it pulls one photo and stops.
Here is the output I see:
gphotos-cdp -v -dev -headless -dldir /download

2022/02/03 17:36:49 Session Dir: /tmp/gphotos-cdp
2022/02/03 17:36:50 pre-navigate
2022/02/03 17:36:52 post-navigate
2022/02/03 17:36:53 Page loaded, most recent item in the feed is: AF1QipNc61YU-maOFCUNdEV7MTgiqA4GZpmmDVb-66KR
2022/02/03 17:36:55 Event: {Type:keyDown Modifiers:Shift Timestamp: Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0}
2022/02/03 17:36:55 Event: {Type:keyUp Modifiers:Shift Timestamp: Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0}
2022/02/03 17:36:55 ERROR: unhandled page event *page.EventDownloadWillBegin
2022/02/03 17:36:59 Marking https://photos.google.com/photo/AF1QipNc61YU-maOFCUNdEV7MTgiqA4GZpmmDVb-66KR as done
OK

what have I missed?

Thanks

It feels like Ctrl+End is much more efficient as it just jumps to the end at once rather goes through your entire library.
Is there any specific reason for using PgDowns and comparing screenthots ?

I'm getting an error during login to the google account, stating that the app is not secured.
I'm hoping updating chromedp might help with that. And we'll have to do it anyway to keep up with chrome changes.
However, we should not update to the latest version yet because:
chromedp/chromedp#627
And indeed I did hit that problem with 81.0.4044.138 here.

Hello all,

This is not really an issue, more like a discussion/forum.

Thanks to this project, @msfjarvis fork and this article https://msfjarvis.dev/posts/backing-up-your-content-from-google-photos/ I manage to make everything work to start backuping my google photos (I now need to setup a nas with 2TB to start downloading everything !)

This is the list of file I'm getting when using this project :

Is there any solution to have them ordered by year and month for example or it's not a good idea and I should use a tool to preview them instead of wanting to have them sorted on my filesystem ? I found projects like photoprism or photostructure, what is your solution ?

Thank you !

I know it supposed to navigate to the oldest photo in my collection first then it will start downloading the photos, but it always fails and stops at some random portrait photos. If I rerun the program it will start downloading photos from the photo it crashed before instead of the oldest photo.

If the platform I'm running the program with can be the problem then I'm running it with https://github.com/JakeWharton/docker-gphotos-sync on a Synology NAS, but I also tried it on a Mac version of Docker and it's still not working properly.

This is the message I get after authenticating with a google account (from the chromedp opened window):

Sign in with a supported browser
To help protect your account, Google doesn’t let you sign in from some browsers. Google might > stop sign-ins from browsers that:

• Don’t support JavaScript or have Javascript turned off.
• Have unsecure or unsupported extensions added.
• Use automation testing frameworks.
• Are embedded in a different application.

Is it possible to bypass this behavior?

I am on MacOS, with an up-to-date Chrome (Version 78.0.3904.108)

Hi! Not sure what to expect from the tool, once I've made the initial backup, if I then modify photos (from the web interface or mobile app etc), will it sync them on subsequent runs? Or does it only download originals?
Basically I'm wondering if the tool has access to the elusive "last modified time". I was initially tempted to use rclone as a backup solution, but was put off by the limitations.

Could cpd potentially be used as a 2-ways sync tool? My internet access is not super fast, and viewing pictures in a browser isn't ideal... I still miss Picasa, and having pictures right there in the local filesystem :'( !