simonedegiacomi / gphotosuploader Goto Github PK

Unofficial Google Photos uploader and Go library

License: GNU General Public License v3.0

Go 93.54% HTML 0.07% JavaScript 4.04% Shell 2.35%

unofficial photo-upload google-photos uploader

gphotosuploader's Introduction

G Photos Uploader - Deprecated

IMPORTANT NOTICE: this project has been deprecated, since Google now released an official google photos API.

New projects built on top of the Google photos API are:
https://github.com/nmrshll/gphotos-uploader-cli for a command-line uploader
https://github.com/nmrshll/google-photos-api-client-go for a go client library

The new projects currently count against yout storage quota, once the client side compression is implemented this project will be deprecated.

Why? What is this?

Google Photos does not have a desktop uploader for Linux, ~~neither an API to upload a photo programmatically.~~ (now there is an Official Google Photos API).

G Photos Uploader lets you upload photos from Linux (and, in theory, any OS for which you can compile a Go program) specifying the file name or watching a directory for changes. Furthermore, the project can also be used as a library that you can include in other Go programs.

Disclaimer

G Photos Uploader is an unofficial tool, I (and any possible contributor) don't guarantee any result. Any security or other kind of issues are at your own risk.

Requirements

To use the tool you need to install Go and Git (used by go get to download the dependencies). If you will use the authentication wizard you will also need a WebDriver.

Install

go get github.com/simonedegiacomi/gphotosuploader

How can I use it?

Standalone tool

To launch the tool you have two options:

Add the $GOPATH/bin folder to your path: doing this you can start the program just typing gphotosuploader (if you use this method and don't specify the auth.json and uploaded.txt file paths, these files will be created in the current working directory);
Enter the project folder and use go run main.go;

To use G Photos Uploader as a standalone tool you need to be authenticated. Authentication is implemented with a JSON file that contains your cookies and user ID.

Authentication

Every time you run the tool, it will check for the auth file. If the file is not found or the cookies seem to be expired, the tool will ask you if you want to run a wizard to get new cookies.

Authentication wizard

The authentication wizard uses the WebDrivers protocol, which is usually used to perform automation tests, that allows G Photos Uploader to control a browser and read the cookies from it. To use the WebDrivers Protocol you need to install a web driver (e.g. chromedriver):

On Ubuntu

sudo apt-get install chromium-chromedriver

# Create a link to launch the driver just typing 'chromedriver'
sudo ln -s /usr/lib/chromium-browser/chromedriver /usr/bin/chromedriver

# Then launch
chromedriver

On macOS Install Homebrew and then:

brew install chromedriver

# Then launch
chromedriver

On Windows:
- Download latest Chrome Web Driver from Google;
- Copy chromedriver.exe in the path (C:\WINDOWS for example);
- Then launch it with a command prompt or Win key + R then chromedriver.exe;

Note: If you are running G Photos Uploader on a headless machine, you can run chromedriver on a separate machine as such:

chromedriver --whitelisted-ips="HEADLESS_MACHINE_IP"

When the Driver starts it will print the address at which it is listening. Once you enter the name of the browser (refer to browserName here) and the address of the web driver in the tool, a new browser window will appear with the Google Photos Login page. Then you can login with your account just like you always do. When you're logged in the tool will read the cookies from the browser, save them into the auth file and close the browser window.
You can now stop the web driver server.

Authentication using a Chrome extension

You can also get the authentication file using a Chrome extension. You can read more about it here.

Upload a photo or watch a directory

Once you have the auth file, you're ready to go. For example, to upload a file named image.png:

gphotosuploader --upload ./image.png

Or to watch a directory:

gphotosuploader --watch path/to/photos --maxConcurrent 4

You can even upload all the photos of a directory and then start to watch another one:

gphotosuploader --upload /path/to/old/photos --upload /downloads/cat.png --watch path/to/new/photos

If you also want to add your photos to a specific existing album you can use the 'album' argument:

gphotosuploader --album albumId --upload ./image.png

Where the albumId is the string that you see in the url when you open the album in the Google Photos Web App (something like: https://photos.google.com/u/2/album/album_id)

If you also want create a new album to add your photos, you can use the 'albumName' argument:

gphotosuploader --albumName foo --upload ./image.png

The tool creates a file (default name: uploaded.txt) which is a list of uploaded files, which will not be re-uploaded. You can specify your own file using the uploadedList argument. To see all the available arguments, use --help.

Library

You can read a simple example here or get the documentation here.

Development

if you want to continue the development of this tool/library, execute first the following script:

githooks/create-links

This will links the hooks used to handle the version of the tool.

Used libreries

fsnotify: To watch for file system events;
Selenium: To authenticate using a browser;

Creators:

gphotosuploader's People

Contributors

Stargazers

Watchers

Forkers

ojacques nmrshll ammgws andrea689 siongui sasajib muyouming shaileshaanand marcelaferoli gaphi hswmartin adrianocrestani gqshell tinyx3k

gphotosuploader's Issues

Show created album ID in log

I need to get back the created album ID.

The solution I have implemented is to display the albumID in the log.
Here is the the patch: diff.txt

Can you update github with it?
I have not succeeded in doing it myself (I am new in github.com).

Thank you!

Error: unexpected JSON response structure

Hi, I get the following error when I try to upload an image:

2018/08/27 21:11:25 Auth file loaded, checking validity ...
2018/08/27 21:11:26 Auth file seems to be valid
2018/08/27 21:11:26 Getting a new At token ...
2018/08/27 21:11:26 At token taken
2018/08/27 21:11:27 [WARNING] The file has been uploaded, but the image url in the reply was not found. The image may not appear.
2018/08/27 21:11:27 Upload error: Error with '/home/user/image.jpg': unexpected JSON response structure

Nothing uploaded

Thanks for sharing your beatiful idea.

However, gphotouploader doesn't seem to upload any files.

Image for testing

$ ls /path/to/directory
11R.jpg

Tests

-upload directory name

./github.com/simonedegiacomi/gphotosuploader/bin/gphotosuploader -upload /path/to/directory
2018/07/13 07:40:28 Auth file loaded, checking validity ...
2018/07/13 07:40:30 Auth file seems to be valid
2018/07/13 07:40:30 Getting a new At token ...
2018/07/13 07:40:31 At token taken
2018/07/13 07:40:31 Done (0 files uploaded, 0 files ignored, 0 errors)

-upload absolute path of one image

./github.com/simonedegiacomi/gphotosuploader/bin/gphotosuploader -upload /path/to/directory/11R.jpg
2018/07/13 07:40:52 Auth file loaded, checking validity ...
2018/07/13 07:40:54 Auth file seems to be valid
2018/07/13 07:40:54 Getting a new At token ...
2018/07/13 07:40:55 At token taken
2018/07/13 07:40:55 Done (0 files uploaded, 0 files ignored, 0 errors)

--upload directory name

./github.com/simonedegiacomi/gphotosuploader/bin/gphotosuploader --upload /path/to/directory
2018/07/13 07:41:15 Auth file loaded, checking validity ...
2018/07/13 07:41:17 Auth file seems to be valid
2018/07/13 07:41:17 Getting a new At token ...
2018/07/13 07:41:18 At token taken
2018/07/13 07:41:18 Done (0 files uploaded, 0 files ignored, 0 errors)

--upload absolute path of one image

./github.com/simonedegiacomi/gphotosuploader/bin/gphotosuploader --upload /path/to/directory/11R.jpg
2018/07/13 07:41:23 Auth file loaded, checking validity ...
2018/07/13 07:41:25 Auth file seems to be valid
2018/07/13 07:41:25 Getting a new At token ...
2018/07/13 07:41:26 At token taken
2018/07/13 07:41:26 Done (0 files uploaded, 0 files ignored, 0 errors)

Test environment

go version
go version go1.6.2 linux/amd64

uname -r
4.13.0-43-generic

os
Ubuntu 16.04

I've tested with main.go but it had same result like above.

Error: can't get an upload url

I'm having the same issue as in #29, but I can't understand why I should be "blacklisted"? I have not uploaded photos (other from the Google Photos app on my iPhone) for ages.

I do use rclone to backup files, but that works fine. As far as I can tell everything in Googles API dashboard also looks fine.

I've tested this both on macOS and Linux Ubuntu.

[username:~/go/bin]$ ./gphotosuploader -upload "/path/to/dir/2002/" -maxConcurrent 4
2018/08/28 23:06:42 Auth file loaded, checking validity ...
2018/08/28 23:06:43 Auth file seems to be valid
2018/08/28 23:06:43 Getting a new At token ...
2018/08/28 23:06:43 At token taken
2018/08/28 23:06:43 Not uploading '/path/to/dir/2002/.DS_Store', it's already been uploaded or it's not a image/video!
2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_013_12a_002.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_016_09a_004.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_015_10a_003.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_012_13a_001.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_017_08a_005.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_020_05a_007.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_021_04a_008.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_018_07a_006.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_022_03a_009.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_024_01a_011.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_023_02a_010.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_025_0a_012.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_026_00a_013.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of directory with spaces/20021230_027_xa_014.jpg': can't get an upload url

2018/08/28 23:06:43 Upload error: Error with '/path/to/dir/2002/2002 Name of another directory with spaces/20021230_002_001.jpg': can't get an upload url

2018/08/28 23:06:44 Upload error: Error with '/path/to/dir/2002/2002 Name of another directory with spaces/20021230_003_002.jpg': can't get an upload url

2018/08/28 23:06:44 Upload error: Error with '/path/to/dir/2002/2002 Name of another directory with spaces/20021230_004_003.jpg': can't get an upload url

2018/08/28 23:06:44 Upload error: Error with '/path/to/dir/2002/2002 Name of another directory with spaces/20021230_006_004.jpg': can't get an upload url

2018/08/28 23:06:44 Upload error: Error with '/path/to/dir/2002/2002 Name of another directory with spaces/20021230_007_005.jpg': can't get an upload url

2018/08/28 23:06:44 Upload error: Error with '/path/to/dir/2002/2002 Name of another directory with spaces/20021230_008_006.jpg': can't get an upload url

2018/08/28 23:06:44 Done (0 files uploaded, 1 files ignored, 20 errors)

--albumName with --maxConcurrent >1

With the following command line, several albums are created with the same name.
gphotosuploader --albumName "New album name" --upload toTransfer/ --maxConcurrent 4

This is probably due to the parallel threads all creating a new Album (and several albumId) without any mutex for this album creation operation.

The command-line workaround, for now, is to set maxConcurrent to 1 when creating a new album and set it to 4 with an existing albumId.

TryUpload() should be named Upload()

It returns an error if it fails anyway.

Of course, there's already an upload() method, but maybe we could do some refactoring and put the 3 upload steps (get uploadID, upload, enable) in a sub-package named uploadSteps ?

That way the top level method that does these 3 upload steps can be called Upload() too.

panic: interface conversion: interface {} is nil, not map[string]interface {}

Not sure what caused this and if this is any help, but here is the last bits of the output.

This is running on a Macbook Pro Retina with macOS 10.13.3.
Images are located on an external drive.

2018/04/06 02:57:55 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0715.JPG' completed
2018/04/06 02:57:58 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0716.JPG' completed
2018/04/06 02:59:51 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0718.JPG' completed
2018/04/06 03:00:00 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0717.JPG' completed
2018/04/06 03:01:34 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0719.JPG' completed
2018/04/06 03:01:40 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0720.JPG' completed
2018/04/06 03:03:21 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0722.JPG' completed
2018/04/06 03:03:46 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0721.JPG' completed
2018/04/06 03:04:45 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0724.JPG' completed
2018/04/06 03:05:57 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0723.JPG' completed
2018/04/06 03:06:23 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0725.JPG' completed
2018/04/06 03:06:28 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0726.JPG' completed
2018/04/06 03:07:11 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0727.JPG' completed
2018/04/06 03:08:40 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0728.JPG' completed
2018/04/06 03:09:14 Upload of '/Volumes/path/to/dir/Photos Library.photoslibrary/Masters/2016/08/03/20160803-164101/IMG_0730.JPG' completed
panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 33 [running]:
github.com/simonedegiacomi/gphotosuploader/api.EnableImageResponse.getEnabledImageId(0xc420064640, 0x3, 0x4, 0x128a320, 0xc420144098)
	/Users/<removed>/go/src/github.com/simonedegiacomi/gphotosuploader/api/models.go:108 +0x27f
github.com/simonedegiacomi/gphotosuploader/api.(*Upload).enablePhoto(0xc420b94500, 0xc42069e120, 0x0, 0x0, 0x0)
	/Users/<removed>/go/src/github.com/simonedegiacomi/gphotosuploader/api/uploadSteps.go:238 +0xe3c
github.com/simonedegiacomi/gphotosuploader/api.(*Upload).Upload(0xc420b94500, 0xc420085410, 0xc42004acf0, 0xc42004ae50)
	/Users/<removed>/go/src/github.com/simonedegiacomi/gphotosuploader/api/upload.go:125 +0x151
github.com/simonedegiacomi/gphotosuploader/utils.(*ConcurrentUploader).uploadFile(0xc4201bc150, 0xc420614600, 0x79)
	/Users/<removed>/Downloads/gphotoupload/go/src/github.com/simonedegiacomi/gphotosuploader/utils/uploader.go:144 +0x16c
created by github.com/simonedegiacomi/gphotosuploader/utils.(*ConcurrentUploader).EnqueueUpload
	/Users/<removed>/Downloads/gphotoupload/go/src/github.com/simonedegiacomi/gphotosuploader/utils/uploader.go:107 +0x137
exit status 2

Upload error can't get an upload url.

Hi, I have just discovered this tool and wanted to test it out but unfortunately an error message faced me:

2018/08/27 19:15:48 Auth file loaded, checking validity ...
2018/08/27 19:15:59 Auth file seems to be valid
2018/08/27 19:15:59 Getting a new At token ...
2018/08/27 19:15:59 At token taken
2018/08/27 19:15:59 Upload error: Error with '/home/user/pictures/1.jpg': can't get an upload url

I hope one can help me out. Thanks.

What is address of the WebDriver

Sorry for dumb question and for my english, what means phrase "Insert the address of the WebDriver" where i can get it? could you give me example of that addres, i use headless machine on ubuntu

Selenium WebDrivers Dependancy

It wasn't 100% clear in Readme -- So,

The Selenium WebDrivers is only needed for authentication to create/update that JSON file, right?

I.e., if I'm able to authenticate using the Chrome browser extension, I don't need it for the normal uploading, right?

thx

Key path not found

I get a lot of messages saying:
Key path not found
2018/07/21 20:47:43 [WARNING] Image uploaded but url not found

Any idea where this could come from?

does this tool can delete photos from Google photo ?

Thanks for this tool,but I will delete some photos from Google photo ,can this tool do it?

installation problem

Hi, thank for tool!

but I can't run it after installation on ubuntu server 16.04 LTS ( go version go1.10.1 )

every time after "go run main.go ... (with any arguments)" I get this errors:

# command-line-arguments
./main.go:46:50: undefined: version
./main.go:46:59: undefined: versionDate

maybe I missed something

[Question] Do uploaded photos count toward your storage quota?

As per the title. When uploading through the official channels (photos.google.com, Backup and Sync, Google Photos app) you can have unlimited storage by allowing Google to resize your photos/videos. Have you been able to reverse engineer that part of the upload API as well?

Google has announced a Google Photos API

More info:

Maybe that could help this project.

Or possibly merge with rclone (https://forum.rclone.org/t/google-photos-api/5593/10)?

does it upload subdirectories on watched directory atomatically ?

or we need to add them individually ?

impossible to retrieve uploaded image ID/URL from the library

So far it's impossible to access the ID of an image right after upload, it's only used internally by the library, but not returned to "the outside" (the code that uses the library)
It could be useful for several reasons:

for storing it in some database and accessing the pictures later
for downloading the picture
(in my case) for performing checks on the uploaded image to see the upload went fine and the uploaded data is not corrupted

A solution would be to include it in some struct returned by the tryUpload method.

I'm working on that, tell me what you think about it.
My plan is to return a struct that contains only the ID, since the URL can be deduced from the ID I'll add a method to re-generate it from the ID.

Problem uploading ARW files (Sony RAW) [works in gphotos web gui]

Hi, thanks for creating this!

I'm having problems uploading Sony RAW format. It is supported ref. this help page and it works in Google Photos web gui.

2018/04/03 23:01:59 Auth file loaded, checking validity ...
2018/04/03 23:02:05 Auth file seems to be valid
2018/04/03 23:02:05 Getting a new At token ...
2018/04/03 23:02:05 At token taken
2018/04/03 23:02:05 Not uploading '/path/to/file/DSC00181.ARW', it's already been uploaded or it's not a image/video!
2018/04/03 23:02:05 Done (0 files uploaded, 1 files ignored, 0 errors)

This is uploaded from a Apple Photos master directory on an HFS+ formated external drive. There is also a jpg of each arw-file in the same directory.

Let me know if I can provide any more info.

Auto create album based on folder name and/or date

It would be nice to auto-create an album and put all the uploaded images into it (if an album id is not specified via the command line parameter). The album name could be based on the local folder name and/or the date of the folder/imagese.

What is needed to run the program?

Hi,

I did not find in the readme the information what is needed to be installed as bare minimum on the Linux machine in order to get and run your script.

Can you please specify that? Can you please update the readme as well?

I use Debian. Do I need to install golang only or maybe some other packages as well?

Regards,
Piotr

Path of already uploaded files (in uploaded.txt) is not absolute

The path of already uploaded files recorded in uploaded.txt is not absolute. Thus, the same photo referenced with two different paths (e.g., ~/Foto/foo.jpg and Foto/foo.jpg), will be uploaded twice.

chromedriver doesn't ask for browser

after running chromedriver i got stuck in:

Starting ChromeDriver 2.36 (0) on port 9515
Only local connections are allowed.

two different gphoto accounts

Thanks for tool!
Is it possible to synchronize two different folders with two different gphoto accounts in the same time?

OAuth failed on first start

I use gphotos-uploader-cli init to generate config file.
Then I change the account to my gmail account in config file.
Run gphotos-uploader-cli, it take me to google OAuth page, but the page throw an error:

400. That’s an error.

Error: redirect_uri_mismatch

The redirect URI in the request, http://localhost:38373, does not match the ones authorized for the OAuth client.

So I am unable to get token, is it any way to do OAuth manually?

Adding photos to specific (precreated) album

Dears, I wonder how to implement subj.

I am absolutely lost looking for docs for these api url

NewUploadUrl = "https://photos.google.com/_/upload/uploadmedia/rupio/interactive?authuser=2"
EnablePhotoUrl = "https://photos.google.com/_/PhotosUi/mutate"

Especially the last one!
How did you get them? (:

Timestamp is wrong

Timestamp in the upload options is wrong, it should be multiplied by 1000.

At the moment in the library I'm just doing this to fix it :

options, err := api.NewUploadOptionsFromFile(file)
if err != nil {
	return err
}
options.Timestamp = options.Timestamp * 1000

But I guess this should be in the library directly.

Correct me if I'm wrong or if this works well in some cases but not for all files.

By the way, how can I contribute ? Just fork and create pull requests ? I quite like your project so far, and I can see a few ways I would like to change it.