pySmartDl, geckodriver, not uploading shapefile zip? about geeup HOT 17 CLOSED

FYI:

If I comment from metadata_loader import load_metadata_from_csv in batch_uploader.py (just to test), I get an error in sel-latest-linux.py:

Traceback (most recent call last):
File "sel-latest-linux.py", line 4, in
from pySmartDL import SmartDL
ImportError: No module named pySmartDL

Which is a bit strange, since this module is installed and can be loaded.

Python 3.6.9 (default, Jul  3 2019, 07:38:46) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pySmartDL import SmartDL
>>> 

Especially also since the module is listed in the list of dependencies above ...

from geeup.

Fixed in v0.3.0, upgrade, uninstall and reinstall pySmartDL for Python3 and Python2

from geeup.

Fixed in v0.3.0, upgrade, uninstall and reinstall pySmartDL for Python3 and Python2

I removed 0.2.9 and installed v0.3.0. Yet on geeup init

I still get an error related to pySmartDL:

Traceback (most recent call last):
File "sel-latest-linux.py", line 4, in
from pySmartDL import SmartDL
ImportError: No module named pySmartDL

Upgrade unfortunately did not fix this?

from geeup.

Can you send the output for the following runs

pip show geeup

pip show pySmartDL

pip3 show pySmartDL

I have a feeling that the installation pathways might be different for pySmarDL and geeup

from geeup.

Good point, here is the output of these commands:

12:18:21-demuzmp4@rub:~$ pip3 show geeup
Name: geeup
Version: 0.3.0
Summary: Simple Client for Earth Engine Uploads with Selenium Support
Home-page: https://github.com/samapriya/geeup
Author: Samapriya Roy
Author-email: [email protected]
License: Apache 2.0
Location: /home/demuzmp4/.local/lib/python3.6/site-packages
Requires: selenium, pytest, pandas, google-cloud-storage, requests, retrying, lxml, pathlib, pySmartDL, requests-toolbelt, oauth2client, earthengine-api, psutil, beautifulsoup4

12:19:04-demuzmp4@rub:~$ pip3 show pySmartDL
Name: pySmartDL
Version: 1.3.2
Summary: A Smart Download Manager for Python
Home-page: http://pypi.python.org/pypi/pySmartDL/
Author: Itay Brandes
Author-email: [email protected]
License: Public Domain
Location: /home/demuzmp4/.local/lib/python3.6/site-packages
Requires:

12:19:20-demuzmp4@rub:~$ pip show pySmartDL
bash: pip: command not found

So both geeup and pySmartDL do have the same installation path.

In addition, I checked my python library path, and /home/demuzmp4/.local/lib/python3.6/site-packages is listed in there:

Python 3.6.9 (default, Jul 3 2019, 07:38:46)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import sys
print (sys.path)
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/demuzmp4/.local/lib/python3.6/site-packages', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']

from geeup.

In addition, these are the errors I get when using geeup and python3 geeup:

12:22:37-demuzmp4@rub:~$ python3 /home/demuzmp4/.local/lib/python3.6/site-packages/geeup/geeup.py init
Traceback (most recent call last):
File "/home/demuzmp4/.local/lib/python3.6/site-packages/geeup/geeup.py", line 24, in
from .batch_uploader import upload
ModuleNotFoundError: No module named 'main.batch_uploader'; 'main' is not a package

12:22:57-demuzmp4@rub:~$ geeup init
Traceback (most recent call last):
File "sel-latest-linux.py", line 4, in
from pySmartDL import SmartDL
ImportError: No module named pySmartDL

from geeup.

Hi @matthiasdemuzere
I have tried to replicate this issue many times on both my windows and linux machines and I do not seem to have this issue. I just updated the relative import issue, so what you can try as a last resort is maybe download the repo, browse to the geeup folder inside the main repo folder, make sure it has the geeup.py file and try

python3 geeup.py -h

from geeup.

I followed your suggestion, and launch geeup.py as you suggest. That allows me to see the help menu.
Yet with the pip install that works as well.

The problem is related to init. That is where I get the pySmartDL error. Also when launching geeup.py from the repo:

17:43:49-demuzmp4@rub:geeup$ python3 geeup.py init
Traceback (most recent call last):
File "sel-latest-linux.py", line 4, in
from pySmartDL import SmartDL
ImportError: No module named pySmartDL

If I do not do init, and try to upload a tif like that, I get the error below. Which makes sense, as init does install the selenium drivers for firefox ....

Password:
Traceback (most recent call last):
File "/home/demuzmp4/.local/lib/python3.6/site-packages/selenium/webdriver/common/service.py", line 76, in start
stdin=PIPE)
File "/usr/lib/python3.6/subprocess.py", line 729, in init
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/demuzmp4/.local/lib/python3.6/site-packages/geeup/geckodriver': '/home/demuzmp4/.local/lib/python3.6/site-packages/geeup/geckodriver'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/demuzmp4/.local/bin/geeup", line 11, in
sys.exit(main())
File "/home/demuzmp4/.local/lib/python3.6/site-packages/geeup/geeup.py", line 170, in main
args.func(args)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/geeup/geeup.py", line 92, in upload_from_parser
nodata_value=args.nodata)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/geeup/batch_uploader.py", line 96, in upload
google_session = __get_google_auth_session(user, password)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/geeup/batch_uploader.py", line 243, in __get_google_auth_session
driver = Firefox(executable_path=os.path.join(lp,"geckodriver"),firefox_options=options)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/selenium/webdriver/firefox/webdriver.py", line 164, in init
self.service.start()
File "/home/demuzmp4/.local/lib/python3.6/site-packages/selenium/webdriver/common/service.py", line 83, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'geckodriver' executable needs to be in PATH.

So perhaps way around this is to update these drivers outside of geeup, so that init is not required? Or won't that work?

from geeup.

For the time being, this should work, go to https://github.com/mozilla/geckodriver/releases/tag/v0.25.0

download and unzip the geckodriver based on your system and place it in the same folder as the geeup.py file. This is exactly what init is downloading

from geeup.

Great. This indeed does the trick for both the pip install geeup and the repo download approach.
No more need to do init indeed, I can start uploading a raster as EE asset.

I do however get an error with that as well, but I believe its unrelated to this issue, so I will open a new one ...

from geeup.

Hey,

I am back to geeup, and still experience the issues as above?
After installation, I try init, yet then I get the pySmartDl error:

Traceback (most recent call last):
File "sel-latest-linux.py", line 4, in
from pySmartDL import SmartDL
ImportError: No module named pySmartDL

geeup and PySmartDL are stored in the same folder:

Name: geeup
Version: 0.3.3
Summary: Simple Client for Earth Engine Uploads with Selenium Support
Home-page: https://github.com/samapriya/geeup
Author: Samapriya Roy
Author-email: [email protected]
License: Apache 2.0
Location: /home/demuzmp4/.local/lib/python3.6/site-packages
Requires: requests-toolbelt, google-cloud-storage, requests, pySmartDL, lxml, psutil, pytest, selenium, retrying, earthengine-api, beautifulsoup4, pathlib, oauth2client, pandas

Name: pySmartDL
Version: 1.3.3
Summary: A Smart Download Manager for Python
Home-page: http://pypi.python.org/pypi/pySmartDL/
Author: Itay Brandes
Author-email: [email protected]
License: Public Domain
Location: /home/demuzmp4/.local/lib/python3.6/site-packages
Requires:

I tried the geckodrive download as instructed above (this time version 0.26).

Whatever I do, the upload of my shapefile does not work:

geeup zipshape --input /home/demuzmp4/Desktop/zip --output users/kode

Files in my input folder:

test.dbf
test.shp
test.prj
test.shx

I also tried seltabup:

geeup seltabup --source /home/demuzmp4/Desktop/zip/tmp --dest projects/WUDAPT/lczEE_TAshp/tmp

which creates a tmp folder, yet gives me this error:

Password:
New folder projects/WUDAPT/lczEE_TAshp/tmp created
Traceback (most recent call last):
File "/home/demuzmp4/.local/bin/geeup", line 11, in
sys.exit(main())
File "/home/demuzmp4/.local/lib/python3.6/site-packages/geeup/geeup.py", line 200, in main
args.func(args)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/geeup/geeup.py", line 127, in seltabup_from_parser
destination=args.dest)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/geeup/sel_tuploader.py", line 68, in seltabup
username.send_keys(uname)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 478, in send_keys
{'text': "".join(keys_to_typing(value)),
File "/home/demuzmp4/.local/lib/python3.6/site-packages/selenium/webdriver/common/utils.py", line 150, in keys_to_typing
for i in range(len(val)):
TypeError: object of type 'NoneType' has no len()

FYI:
a) you can get the shapefile from here.
b) I also tried with python3 python3 /home/demuzmp4/.local/lib/python3.6/site-packages/geeup/geeup.py -h,yet then I get the following error:

Traceback (most recent call last):
File "/home/demuzmp4/.local/lib/python3.6/site-packages/geeup/geeup.py", line 54, in
from .batch_uploader import upload
ModuleNotFoundError: No module named 'main.batch_uploader'; 'main' is not a package

Any idea how to fix this??

from geeup.

Hi @matthiasdemuzere
Try installing v0.3.4 and let me know if the issue persists. You will use zipshape to zip the shapefiles and seltabup for table uploads

pip install geeup

or

pip install geeup --upgrade

from geeup.

Hi.

I just update my version.

So do I understand you well that zipshape zips shapefiles to a zip archive, on my local machine? I just tried, yet I do not get this archive?

I just created the zip archive using a system command. The zip archive (stored in folder zip/) contains a shp, shx, dbf and prj file (IvoryCoast_Abidjan_v1.zip).

Command to upload:
geeup seltabup --source /home/demuzmp4/Desktop/gee_up_test/zip --dest projects/WUDAPT/lczEE_TAshp/test

Error:

New folder projects/WUDAPT/lczEE_TAshp/test created
object of type 'NoneType' has no len()
Failed to setup & use selenium

All fields in the shapefile have information. So I am not sure where this 'NoneType' comes from?

FYI: uploading directly via EE browser > new Table Upload works.

Any ideas?

from geeup.

I am hoping you are providing your username as well in the command

geeup seltabup --source "/home/demuzmp4/Desktop/gee_up_test/zip" --dest "projects/WUDAPT/lczEE_TAshp/test" -u "[email protected]"

You might want to use the geeup selsetup as well. Google has a way of handling accounts which do not have third party apps enabled or if your account has 2 factor authentication enabled.

In any case, it works at my end unfortunately so I cannot reproduce your issue

from geeup.

Ha, I did not define my username in the end. Now I did, yet I still get an error:

New folder projects/WUDAPT/lczEE_TAshp/test created
invalid syntax (, line 2)

If I use geeup selsetup, I get the following error:

Traceback (most recent call last):
File "sel_setup.py", line 7, in
from selenium import webdriver
ImportError: No module named selenium

I then installed selenium, and tried again, with the same error.

If I execute the full path: python3 /home/demuzmp4/.local/lib/python3.6/site-packages/geeup/geeup.py selsetup, I get the following error:

Traceback (most recent call last):
File "/home/demuzmp4/.local/lib/python3.6/site-packages/geeup/geeup.py", line 60, in
from .batch_uploader import upload
ModuleNotFoundError: No module named 'main.batch_uploader'; 'main' is not a package

Outputs of the pip3 show commands for geeup / selenium:

Name: geeup
Version: 0.3.6
Summary: Simple Client for Earth Engine Uploads with Selenium Support
Home-page: https://github.com/samapriya/geeup
Author: Samapriya Roy
Author-email: [email protected]
License: Apache 2.0
Location: /home/demuzmp4/.local/lib/python3.6/site-packages
Requires: pandas, requests-toolbelt, selenium, pytest, requests, earthengine-api, beautifulsoup4, google-cloud-storage, psutil, pySmartDL, pathlib, oauth2client, retrying, lxml

Name: selenium
Version: 3.141.0
Summary: Python bindings for Selenium
Home-page: https://github.com/SeleniumHQ/selenium/
Author: UNKNOWN
Author-email: UNKNOWN
License: Apache 2.0
Location: /home/demuzmp4/.local/lib/python3.6/site-packages
Requires: urllib3

from geeup.

^ I am having the same issue. After providing my username, I get

invalid syntax (<unknown>, line 2)

from geeup.

It is unfortunate but I am unable to replicate this issue, I just upload a zipped shapefile into two different google accounts. This takes into account that both have less secure apps enabled on the google account, these are not service accounts but personal user accounts.

from geeup.