Giter Club home page Giter Club logo

pygce's Introduction

PyGCE | Export, save, and analyze your Garmin Connect data

Open Source Love Contributions welcome

Install

Different ways, all equals

  • $ pip3 install . --upgrade --force-reinstall from the source
  • $ pip3 install pygce via pip

Just run python3 setup.py install --force to upgrade from local installation

Chromedriver

In order to run the scraper you need a chromedriver (i.e. Chrome made for developing purposes, fully controllable by other software). Just download it and save ii toa location: you will need to pass the path of the downloaded driver as argument of pygce.

Usage

A simple pygce -h from the terminal should result in this output

usage: -u <username (email) to login to Garmin Connect> -p <password to login to Garmin Connect> -c <path to chromedriver to use> -d <days to save. e.g -d 2017-12-30 or -d 2016-01-01 2017-12-30> -o <path to output file>

optional arguments:
  -h, --help            show this help message and exit
  -user                 username (email) to login to Garmin Connect
  -password             password to login to Garmin Connect
  -url URL              url to connect to (e.g https://connect.garmin.com)
  -chrome               path to chromedriver to use
  -d [DAYS [DAYS ...]]  days to save. e.g -d 2017-12-30 or -d 2016-01-01 2017-12-30
  -gpx                  download .gpx files too [y/n]
  -out                  path to output file

e.g.: pygce -u [email protected] -p myBe@Ut1fulP@550rd -c /home/foga/Downloads/chromedriver -d 2019-07-01 2019-07-04 -o /home/foga/pygce/out/2019-07-01.json

When called with appropriate args pygce saves .csv (.json) data dumps of your Garmin Connect timeline data. The files will look like

{
  "2017-04-09": {
    "steps": {
      "avg": "12948.0",
      "goal": "11197.0",
      "distance": "19.7",
      "total": "16902.0"
    },
    "sleep": {
      "total_sleep_time": "00:08:23",
      "light_sleep_time": "00:03:16",
      "awake_sleep_time": "00:00:19",
      "deep_sleep_time": "00:04:48",
      "wake_time": "07:34:00",
      "bed_time": "23:11:00",
      "nap_time": "00:00:00",
      "night_sleep_time": "00:08:23"
    },
    "activities": [

You can browse a full json output for a single day. If you instead chose the csv output format you should see something like this:

SUMMARY:comment,SLEEP:deep_sleep_time,ACTIVITIES:distance,STEPS:goal,STEPS:distance,BREAKDOWN:sleeping,SUMMARY:likes,SLEEP:bed_time,STEPS:avg,SLEEP:light_sleep_time
sweet dinner,00:04:55,22.82,10000,22.8,29,6,22:22:00,12087,00:03:36
met girlfriend in river park,00:02:47,8.92,10200,8.9,22,9,01:05:00,11940,00:01:42
droven all around ny,00:03:41,3.38,10272,3.4,27,2,03:01:00,10045,00:02:33

Browse a sample csv output for 1 day.

Sample analysis output

As of now, the analysis has not been included in the main cli program, nor has a mature command line parser: you can play with it as you want! There is lots of machine-learning stuff already done, and you can browse some samples here. Mainly the focus is on clustering, best features selection and regression. Feel free to contribute!

3d clusters correlation matrix k best features

Contributing

Fork | Patch | Push | Pull request

Feedback

Suggestions and improvements welcome!

Thanks to

Authors

sirfoga
Stefano Fogarollo

License

MIT License

pygce's People

Contributors

lozingaro avatar sirfoga avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

pygce's Issues

Could not find suitable distribution for Requirement.parse('pyhal')

Hi.!

I try to install pygce. After running **python3 setup.py install --force ** i got following Error

Installed /usr/lib/python3.4/site-packages/pygce-0.1.2-py3.4.egg Processing dependencies for pygce==0.1.2 Searching for pyhal Reading https://pypi.python.org/simple/pyhal/ No local packages or download links found for pyhal error: Could not find suitable distribution for Requirement.parse('pyhal')

System: openSUSE Leap 42.2

BYe
Harald

No module named 'hal.internet.selenium_bots'

Expected Behavior

"pygce -h" returns a help screen

Actual Behavior

"pygce -h" returns the following:

ModuleNotFoundError: No module named 'hal.internet.selenium_bots'

Traceback (most recent call last):
File "/Users/paredes/venv.pygce/bin/pygce", line 7, in
from pygce.cli import main
File "/Users/paredes/venv.pygce/lib/python3.7/site-packages/pygce/cli.py", line 9, in
from pygce.models.bot import GarminConnectBot
File "/Users/paredes/venv.pygce/lib/python3.7/site-packages/pygce/models/bot.py", line 10, in
from hal.internet.selenium_bots import SeleniumForm
ModuleNotFoundError: No module named 'hal.internet.selenium_bots'
(venv.pygce)

Steps to Solve the Problem

  1. Reinstalled Python
  2. Reinstalled Selenium
  3. Reinstalled chromedriver
  4. Reinstalled pygce

Specifications

  • pygce version (check pip3 show pygce):
    1.9.8
  • Platform (e.g Windows 8, Mac OSX Sierra, Ubuntu 18.04 ...):
    Mac OSX High Sierra ver 10.13.6
  • Url of connection (e.g https://connect.garmin.com, if you don't know it,
    then it's the stock one: https://connect.garmin.com):
    Https://connect.garmin.com/en-US/signin
  • Python version (check python --version):
    Python 3.7.0
    -Selenium ver 3.141.0; urllib3 ver 1.24.1

Garmin step details output off by varying amounts

Expected Behavior

Garmin step details output for the day

Actual Behavior

However, the 15 min bin counts are off by varying amounts. When comparing the pygce json generated 15 min counts against what is displayed on the Garmin step details screen, I see that the generated json files are always missing the last 4 or 5 hours of data.

The 15 min bin output for day 2018-05-10:

{"15-min bins": [{"time": "2018-05-10 04:00:00", "steps": 0}, {"time": "2018-05-10 04:15:00", "steps": 0}, . . .etc etc etc}

Steps to Reproduce the Problem

  1. login
  2. view step details for a day

Specifications

  • pygce version (check pip3 show pygce): 2.0.6
  • Platform (e.g Windows 8, Mac OSX Sierra, Ubuntu 18.04 ...): Ubuntu 18.04
  • Url of connection (e.g https://connect.garmin.com, if you don't know it,
    then it's the stock one: https://connect.garmin.com):
  • Python version (check python --version): 3.6.7

module not found...

Hi! I get the following on my Raspi when running pygce:

pi@raspberrypi:~/pygce $ pygce
Traceback (most recent call last):
  File "/usr/local/bin/pygce", line 11, in <module>
    load_entry_point('pygce==2.1.2', 'console_scripts', 'pygce')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 561, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.5/dist-packages/pygce-2.1.2-py3.5.egg/pygce/cli.py", line 9, in <module>
  File "/usr/local/lib/python3.5/dist-packages/pygce-2.1.2-py3.5.egg/pygce/models/bot.py", line 19, in <module>
ImportError: No module named 'models'

Any idea what it might be and how to fix it?

Cheers,
Jens

Example usage shows -u username, but using -u username results in an error

Expected Behavior

The output from -h "usage: -u <username (email) to login to Garmin Connect>" should match how arguments are actually parsed.

Actual Behavior

-h says to use -u to specify the username, but actually running pygce with -u username fails because -u is ambiguous and could mean -url.

Steps to Reproduce the Problem

  1. Run pygce -h and observe the first line of output
  2. Attempt to run pygce according to the "usage" line, using -u
  3. Observe that the application fails to run, stating that -u is ambiguous.

Specifications

  • pygce version (check pip3 show pygce): cd3ebec
  • Platform (e.g Windows 8, Mac OSX Sierra, Ubuntu 18.04 ...): ubuntu 16.04
  • Url of connection (e.g https://connect.garmin.com, if you don't know it,
    then it's the stock one: https://connect.garmin.com): default
  • Python version (check python --version): 3.5.2

Related issue: #13.

'NoneType' object has no attribute 'date'

Hi,
So 1st, I wanted to say a Massive Thankyou.
I've been pestering the Garmin people for API and they turn you away saying - Pay the Money, or Jog on!

So - I have got Virtualbox, and installed Ubuntu LTS 18.04.1.

I have got Python 3.6 on Ubuntu - This satisfies some of the min installation critera of Numpy,
https://stackoverflow.com/questions/42662104/how-to-install-pip-for-python-3-6-on-ubuntu-16-10

Anyway - I have run the Setup for pygce as you have suggested in your detailed Readme.
I have given the correct parameters, to the pygce application.

It starts Chromium in Automation mode. It logs in successfully as me.
It goes to "https://connect.garmin.com/modern/" and then stops with the following exception.

Getting day 2018-10-02 00:00:00
Getting user dashboard from https://connect.garmin.com/modern/
Message:
Traceback (most recent call last):
File "/usr/local/bin/pygce", line 11, in
load_entry_point('pygce==1.9.8', 'console_scripts', 'pygce')()
File "/home/eastabrooka/.local/lib/python3.6/site-packages/pygce/cli.py", line 133, in main
bot.save_json_days(days[0], days[1], path_out)
File "/home/eastabrooka/.local/lib/python3.6/site-packages/pygce/models/bot.py", line 227, in save_json_days
print("Parsing day", str(d.date))
AttributeError: 'NoneType' object has no attribute 'date'

Expected Behavior

The pygce to pull the daily-summary for yesterday

Actual Behavior

Exception.

Steps to Reproduce the Problem

  1. pygce -user [email protected] -p password -c /home/eastabrooka/Downloads/chromedriver -d 2018-10-02 -f json -o ./output.txt
    1.exception occurs.

Specifications

  • pygce version (check pip3 show pygce):
    Name: pygce
    Version: 1.9.8
    Summary: pygce is an unofficial Garmin Connect data exporter.
    Home-page: https://github.com/sirfoga/pygce
    Author: sirfoga
    Author-email: [email protected]
    License: UNKNOWN
    Location: /home/eastabrooka/.local/lib/python3.6/site-packages
    Requires: bs4, pyhal, lxml, numpy, sklearn

  • Platform (e.g Windows 8, Mac OSX Sierra, Ubuntu 18.04 ...):
    Ubuntu LTS 18.04.1.

  • Url of connection (e.g https://connect.garmin.com):
    Stock - Whatever Pygce uses as default

  • Python version (check python --version):

eastabrooka@GarminScraper:~/pygce$ python3 --version
Python 3.6.6

Thank you again for posting Pygce, its very useful !

Daily summary content failing to load

Expected Behavior

Daily summary content successfully loads in chromedriver and is able to be scraped.

Actual Behavior

Daily summary content does not load (menu bars do), as shown in screenshot below. This page will load in a normal browsing tab (ie this same URL has been copied across to a normal chrome window and loads instantly).

Note that the timeout length has been extended to 60s and the same issue still occurs.

Screen Shot 2020-06-23 at 2 37 43 pm

Steps to Reproduce the Problem

  1. Install pygce
  2. Run with single date.
  3. Wait for page to not load.

Specifications

  • pygce version (check pip3 show pygce):
    Version: 2.1.2

  • Platform (e.g Windows 8, Mac OSX Sierra, Ubuntu 18.04 ...):
    macOS Catalina (10.15.4)

  • Url of connection (e.g https://connect.garmin.com, if you don't know it,
    then it's the stock one: https://connect.garmin.com):
    "https://connect.garmin.com"

  • Python version (check python --version):
    Python 3.8.3

Usage instructions indicate -f option to specify format, but using that option results in an error

Expected Behavior

Adding "-f json" to execution should specify that the output format of the file should be json.

Actual Behavior

Specifying "-f json" results in an error:

unrecognized arguments: -f json

Steps to Reproduce the Problem

  1. Run pygce -h
  2. Read the usage summary: "-f <format of output file [json, csv]>"
  3. Attempt to use the flag mentioned

Specifications

  • pygce version (check pip3 show pygce):2.1.2
  • Platform (e.g Windows 8, Mac OSX Sierra, Ubuntu 18.04 ...):ubuntu 16.04
  • Url of connection (e.g https://connect.garmin.com, if you don't know it,
    then it's the stock one: https://connect.garmin.com): default
  • Python version (check python --version): 2.7.12/3.5.2

daily step format

Expected Behavior

Is it possible to output the 15 min bin step counts without formatting such that "1.897" is outputted as "1897" or "835.0" is outputted as "835" ?

Actual Behavior

"1897" steps is output as "1897"
"835" is output as "835.0"

Steps to Reproduce the Problem

  1. login
  2. download daily steps details

Specifications

  • pygce version (check pip3 show pygce): 2.0.6
  • Platform (e.g Windows 8, Mac OSX Sierra, Ubuntu 18.04 ...): Ubuntu 18.04
  • Url of connection (e.g https://connect.garmin.com, if you don't know it,
    then it's the stock one: https://connect.garmin.com):
  • Python version (check python --version): 3.6.7

cannot import SeleniumFormFiller

Steps to Reproduce the Problem

  1. git clone the repository
  2. install with given command "python3 setup.py install --force"
  3. run pygce -h which gives following output

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/bin/pygce", line 11, in
load_entry_point('pygce==2.0.6', 'console_scripts', 'pygce')()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/init.py", line 487, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/init.py", line 2728, in load_entry_point
return ep.load()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/init.py", line 2346, in load
return self.resolve()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/init.py", line 2352, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pygce-2.0.6-py3.7.egg/pygce/cli.py", line 9, in
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pygce-2.0.6-py3.7.egg/pygce/models/bot.py", line 10, in
ImportError: cannot import name 'SeleniumFormFiller' from 'hal.internet.selenium' (/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/hal/internet/selenium/init.py)

Solution on my machine

change one line in bot.py
from:
from hal.internet.selenium import SeleniumFormFiller
to:
from hal.internet.selenium.forms import SeleniumFormFiller

not sure if this is a quirk on my machine of more "widespread" as im not that familiar with python yet

Capture 15-min bin steps

Expected Behavior

Cmd line option should allow download 15-min bin steps

Actual Behavior

No such option

Details

e.g this page shows such data:

...
{"startGMT":"2018-12-13T23:00:00.0","endGMT":"2018-12-13T23:15:00.0","steps":0,"primaryActivityLevel":"sedentary","activityLevelConstant":true},{"startGMT":"2018-12-13T23:15:00.0","endGMT":"2018-12-13T23:30:00.0","steps":0,"primaryActivityLevel":"sedentary","activityLevelConstant":true},{"startGMT":"2018-12-13T23:30:00.0","endGMT":"2018-12-13T23:45:00.0","steps":0,"primaryActivityLevel":"sedentary","activityLevelConstant":true},{"startGMT":"2018-12-13T23:45:00.0","endGMT":"2018-12-14T00:00:00.0","steps":0,"primaryActivityLevel":"sedentary","activityLevelConstant":true},{"startGMT":"2018-12-14T00:00:00.0","endGMT":"2018-12-14T00:15:00.0","steps":0,"primaryActivityLevel":"sedentary","activityLevelConstant":true},{"startGMT":"2018-12-14T00:15:00.0","endGMT":"2018-12-14T00:30:00.0","steps":0,"primaryActivityLevel":"sedentary","activityLevelConstant":true},{"startGMT":"2018-12-14T00:30:00.0","endGMT":"2018-12-14T00:45:00.0","steps":0,"primaryActivityLevel":"sedentary","activityLevelConstant":true},{"startGMT":"2018-12-14T00:45:00.0","endGMT":"2018-12-14T01:00:00.0","steps":0,"primaryActivityLevel":"sedentary","activityLevelConstant":true},{"startGMT":"2018-12-14T01:00:00.0","endGMT":"2018-12-14T01:15:00.0","steps":0,"primaryActivityLevel":"sedentary","activityLevelConstant":false},{"startGMT":"2018-12-14T01:15:00.0","endGMT":"2018-12-14T01:30:00.0","steps":0,"primaryActivityLevel":"sleeping","activityLevelConstant":true},{"startGMT":"2018-12-14T01:30:00.0","endGMT":"2018-12-14T01:45:00.0","steps":0,"primaryActivityLevel":"sleeping","activityLevelConstant":true},{"startGMT":"2018-12-14T01:45:00.0","endGMT":"2018-12-14T02:00:00.0","steps":0,"primaryActivityLevel":"sleeping","activityLevelConstant":true},{"startGMT":"2018-12-14T02:00:00.0","endGMT":"2018-12-14T02:15:00.0","steps":0,"primaryActivityLevel":"sleeping","activityLevelConstant":true},{"startGMT":"2018-12-14T02:15:00.0","endGMT":"2018-12-14T02:30:00.0","steps":0,"primaryActivityLevel":"sleeping","activityLevelConstant":true},{"startGMT":"2018-12-14T02:30:00.0","endGMT":"2018-12-14T02:45:00.0","steps":0,"primaryActivityLevel":"sleeping","activityLevelConstant":true},{"startGMT":"2018-12-14T02:45:00.0","endGMT":"2018-12-14T03:00:00.0","steps":0,"primaryActivityLevel":"sleeping","activityLevelConstant":true},{"startGMT":"2018-12-14T03:00:00.0","endGMT":"2018-12-14T03:15:00.0","steps":0,"primaryActivityLevel":"sleeping","activityLevelConstant":true},{"startGMT":"2018-12-14T03:15:00.0","endGMT":"2018-12-14T03:30:00.0","steps":0,"primaryActivityLevel":"sleeping","activityLevelConstant":true}
...

Specifications

  • pygce version (check pip3 show pygce): 2.0.6
  • Platform (e.g Windows 8, Mac OSX Sierra, Ubuntu 18.04 ...): any
  • Url of connection (e.g https://connect.garmin.com, if you don't know it,
    then it's the stock one: https://connect.garmin.com):
  • Python version (check python --version): 3.6

'WebDriver' object has no attribute 'fill_form_field'

Expected Behavior

Scrape data from Garmin Connect.

Actual Behavior

It gets stuck on the login screen. The exact error is as follows:

2020-07-29 20:52:29,376 - pygce - **ERROR - 'WebDriver' object has no attribute 'fill_form_field'**
2020-07-29 20:52:29,376 - pygce - DEBUG - GET https://connect.garmin.com/modern/
2020-07-29 20:52:32,031 - pygce - DEBUG - attempt 0

So it looks as though it has something to do with the pyhal package? I was getting a different error before, but changed the imports to:

from hal.internet.selenium.forms import SeleniumFormFiller

which fixed that issue. And now I am getting the issue that 'WebDriver' object has no attribute 'fill_form_field'.

Steps to Reproduce the Problem

  1. pygce -user [email protected] -p redacted -c /c/Users/name/documents/folder/fitness/scrapesteps/chromedriver.exe -d 2018-09-05 2020-07-28 -o /c/Users/name/documents/folder/fitness/scrapesteps -f json

Specifications

  • pygce version (check pip3 show pygce): 1.9.8
  • Platform (e.g Windows 8, Mac OSX Sierra, Ubuntu 18.04 ...): Windows 10
  • Url of connection (e.g https://connect.garmin.com, if you don't know it,
    then it's the stock one: https://connect.garmin.com): Default connection.
  • Python version (check python --version): 3.8.3

Missing docs on what a chromedriver is and what to specify

Expected Behavior

Documentation should include what a chromedriver is and what to specify for that option, and should have a default if possible like the url does.

Actual Behavior

Neither the readme nor the -h output indicates what goes in this field.

Steps to Reproduce the Problem

  1. Try to run it without specifying a chromedriver
  2. Try to run it with -h
  3. Try to read the readme file

Specifications

  • pygce version (check pip3 show pygce): 2ca3adc
  • Platform (e.g Windows 8, Mac OSX Sierra, Ubuntu 18.04 ...): Ubutun 16.04
  • Url of connection (e.g https://connect.garmin.com, if you don't know it,
    then it's the stock one: https://connect.garmin.com): default
  • Python version (check python --version): 2.7.12/3.5.2

Pygce crashes with "No such file or directory" error

Expected Behavior

pygce should run and produce an output file, or a reasonable error should be shown so the user can take corrective action.

Actual Behavior

Program crashes with backtrace and error. No indication is provided as to what directory is missing or why the application expects it to be there, nor what the user can do to fix it.

Steps to Reproduce the Problem

1.pygce -user (redacted email address) -c /path/to//Downloads/chromedriver/chromedriver -o foo.json -d 2019-07-01 -p (redacted password)

  1. Observe crash output:

Traceback (most recent call last):
File "/usr/local/bin/pygce", line 9, in
load_entry_point('pygce==2.1.2', 'console_scripts', 'pygce')()
File "/usr/local/lib/python3.5/dist-packages/pygce-2.1.2-py3.5.egg/pygce/cli.py", line 124, in main
File "/usr/local/lib/python3.5/dist-packages/pygce-2.1.2-py3.5.egg/pygce/cli.py", line 115, in check_args
File "/usr/lib/python3.5/os.py", line 241, in makedirs
mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''

Specifications

  • pygce version (check pip3 show pygce): 2.1.2
  • Platform (e.g Windows 8, Mac OSX Sierra, Ubuntu 18.04 ...): ubuntu 16.04
  • Url of connection (e.g https://connect.garmin.com, if you don't know it,
    then it's the stock one: https://connect.garmin.com): default
  • Python version (check python --version): 2.7.12/3.5.2

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.