floydwch / kaggle-cli Goto Github PK

Hi, it seems kg command hangs or takes longer then at least 5 minutes to run. I had to stop it and it seems got stuck with this expression 'st = os.stat(s)'.

Here's the full traceback log.

Traceback (most recent call last):
  File "/usr/local/bin/kg", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/kaggle_cli/main.py", line 19, in main
    return app.run(argv)
  File "/usr/local/lib/python2.7/site-packages/cliff/app.py", line 226, in run
    result = self.run_subcommand(remainder)
  File "/usr/local/lib/python2.7/site-packages/cliff/app.py", line 346, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python2.7/site-packages/cliff/command.py", line 55, in run
    self.take_action(parsed_args)
  File "/usr/local/lib/python2.7/site-packages/kaggle_cli/config.py", line 32, in take_action
    if os.path.isdir(prefix + '.kaggle-cli'):
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/genericpath.py", line 49, in isdir
    st = os.stat(s)
KeyboardInterrupt

Is this problem caused by the Python distribution that I installed via home-brew on OSX?

Compared to the downloading and uploading directly from the kaggle interface website, when we use kg seems to be quite slow, why is that?

Hello again, I guess there must be other users like me who participate at least one competition at a time, which makes me think of having one configure file per competition, and the easy way to implement it is to set the default configuration to ./.kaggle_cli.

Hi there,

when I attempt to download the data from a competition, I get:

Starting new HTTPS connection (1): www.kaggle.com [Errno 1] _ssl.c:507: error:1411809D:SSL routines: SSL_CHECK_SERVERHELLO_TLSEXT:tls invalid ecpointformat list
I was not able to find a solution as my protocol knowledge is very limited, any suggestions?

I can "kg download" in my home folder. However if I am in a folder on a different drive it just hangs. And typically I want to put my data on a separate data volume.

Ran from ubuntu host-
kg download -u 'user -p password' -c 'grupo-bimbo-inventory-demand'

https://pypi.python.org/pypi/progressbar2 ?

Hi, there's three way to run kg, 1) load username/password from user configuration file, 2) load username/password from current working directory, 3) set username/password on command line. I wonder if it is worth making the third method deprecated, then we could refactor the code to have a single module for loading username/password.

This can cause confusion b/c users will just think their models are not changing the scores at all.

I think there should be some warning (maybe with the remaining time to wait) and no scores should be reported.

(Maybe the response has some information we can use to detect whether there's been an error?)

While trying to make a submission by running `submit <my_csv_file> -m "some notes", I get the following printout.

Traceback (most recent call last):
  File "/usr/local/bin/kg", line 9, in <module>
    load_entry_point('kaggle-cli==0.10.0', 'console_scripts', 'kg')()
  File "/usr/local/lib/python3.5/dist-packages/kaggle_cli/main.py", line 21, in main
    return app.run(argv)
  File "/usr/local/lib/python3.5/dist-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/local/lib/python3.5/dist-packages/cliff/app.py", line 363, in run_subcommand
    subcommand = self.command_manager.find_command(argv)
  File "/usr/local/lib/python3.5/dist-packages/cliff/commandmanager.py", line 83, in find_command
    cmd_factory = cmd_ep.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2235, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.5/dist-packages/kaggle_cli/submit.py", line 3, in <module>
    from . import common
  File "/usr/local/lib/python3.5/dist-packages/kaggle_cli/common.py", line 3, in <module>
    from mechanicalsoup import Browser
  File "/usr/local/lib/python3.5/dist-packages/mechanicalsoup/__init__.py", line 2, in <module>
    from .browser import Browser
  File "/usr/local/lib/python3.5/dist-packages/mechanicalsoup/browser.py", line 3, in <module>
    import bs4
  File "/usr/lib/python3/dist-packages/bs4/__init__.py", line 30, in <module>
    from .builder import builder_registry, ParserRejectedMarkup
  File "/usr/lib/python3/dist-packages/bs4/builder/__init__.py", line 314, in <module>
    from . import _html5lib
  File "/usr/lib/python3/dist-packages/bs4/builder/_html5lib.py", line 70, in <module>
    class TreeBuilderForHtml5lib(html5lib.treebuilders._base.TreeBuilder):
AttributeError: module 'html5lib.treebuilders' has no attribute '_base'

It wasn't clear to me that we should be using single quotes for the password,
as formatting the string with double quotes is accepted (and works for username),
and a common sense check of running kg config returns
the password as '****' with single quotes making it look like it was successful.

May I suggest this is added to documentation somewhere?

It would be convenient if linked account creds could also log a person in from the CLI. It does not appear to work now.

OS: Windows 10

Python environment: Anaconda2

Command used: pip install kaggle-cli

Error received:

   Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "c:\users\marc\appdata\local\temp\pip-build-_0jqmc\kaggle-cli\setup.py", line 4, in <module>
        from kaggle_cli.main import VERSION
      File "kaggle_cli\main.py", line 2, in <module>
        from cliff.app import App
    ImportError: No module named cliff.app

Running ubuntu on a p2.xlarge on AWS.

Python version Python 2.7.12 :: Anaconda 4.2.0 (64-bit)

kaggle-cli version 0.12.3

"kg config" returns proper settings [('username', u'[email protected]'), ('password', '********'), ('competition', u'dogs-vs-cats-redux-kernels-edition')]

command I'm running:
kg submit submissions.csv -m "Initial Submission"

The resulting error is 'Browser' object has no attribute '_Browser__raise_on_404'

Any ideas? Thanks!

Hi! I get error "list index out of range" in mehod take_action of script submit.py.
This error occures at lines

team_id = submit_page.soup.select(
    '.comp-content-inside h4 strong a'
)[0].attrs['href'].split('/')[2]

I found the fast solution - commenting out the following two blocks:
1)

team_id = submit_page.soup.select(
    '.comp-content-inside h4 strong a'
)[0].attrs['href'].split('/')[2]

        status_url = (
            'https://www.kaggle.com/'
            'c/{}/submissions/status.json'
            '?apiVersion=1&teamId={}'.format(competition, team_id)
        )

        while True:
            status = browser.get(status_url).json()
            if status['submissionStatus'] == 'pending':
                time.sleep(1)
                continue
            elif status['submissionStatus'] == 'complete':
                print(status['publicScoreFormatted'])
                break
            else:
                print('something went wrong')
                break

After that you won't see submission score in console, but submissions will success.

On my GPU server, I've started getting 400 response when submitting Kaggle entries.

$ kg submit -v "sample_submission.csv" -u <my_username> -p <my_password> -c digit-recognizer -m "test submission"
Starting new HTTPS connection (1): www.kaggle.com
https://www.kaggle.com:443 "GET /c/digit-recognizer HTTP/1.1" 200 15896
https://www.kaggle.com:443 "POST /blobs/inbox/submissions HTTP/1.1" 200 303
https://www.kaggle.com:443 "POST /blobs/inbox/submissions/59d0cf127a134bc2a0c0a746aede19cf/240909/1512782907983 HTTP/1.1" 200 525
https://www.kaggle.com:443 "POST /c/digit-recognizer/submission.json HTTP/1.1" 400 235
list indices must be integers, not str

I wanted to make sure it wasn't my submission that was malformed, so I checked with the sample_submission.csv that came with the digit-recognizer competition. Same error. The sample submission file and my own submission file look fine. To be extra careful, I download one of my recently-successfully-submitted entries to Kaggle, and that submission CSV file gives me the exact same error message.

I try with both Python 3 and Python 2.7. I am using kaggle-cli 0.12.13. I'm SSH'd into a remote machine and I am using a VPN.

Python 2.7.12 |Anaconda 4.2.0 (64-bit)| (default, Jul 2 2016, 17:42:40)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2

> pip install kaggle-cli
...
> Successfully installed kaggle-cli-0.12.10

>kg config -g -u xxx -p xxx -c xxx
>kg download
>Starting new HTTPS connection (1): www.kaggle.com
>list index out of range

like above, when type kg download, it will show the error and quit, here is the debug version:

Starting new HTTPS connection (1): www.kaggle.com
list index out of range
Traceback (most recent call last):
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/cliff/command.py", line 137, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/kaggle_cli/download.py", line 32, in take_action
    browser = common.login(username, password)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/kaggle_cli/common.py", line 33, in login
    login_form = login_page.soup.select("#login-account")[0]
IndexError: list index out of range
Traceback (most recent call last):
  File "/home/ubuntu/anaconda2/bin/kg", line 11, in <module>
    sys.exit(main())
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/kaggle_cli/main.py", line 21, in main
    return app.run(argv)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/cliff/command.py", line 137, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/kaggle_cli/download.py", line 32, in take_action
    browser = common.login(username, password)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/kaggle_cli/common.py", line 33, in login
    login_form = login_page.soup.select("#login-account")[0]
IndexError: list index out of range

any ideas? thanks!

error when downloading the dataset
The files are downloaded but they are very small less than 50KB.

I've tried to download with the following command. The competition rules were excepted.
kg download -u my_username -p my_password -c dogs-vs-cats-redux-kernels-edition

and the out put was the following:

test.zip N/A% |                                                                                                                                                                                                     | ETA:  --:--:--   0.0 s/BWarning:download url for file test.zip resolves to an html documentrather than a downloadable file.
See the downloaded file for details.Is it possible you have notaccepted the competition's rules on the kaggle website?

test.zip 100% |####################################################################################################################################################################################################| Time: 0:00:00  15.0 KiB/s
downloading https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/download/train.zip

what is done wrong?

data download and config break when passing -p <password> ; I suspect because it includes special characters such as : !?*-[]. Error: config: error: argument -p/--password: expected one argument

Hi,
is it possible to add a proxy support ?

I've added "export https_proxy=my_proxy", but it fails to download

HTTPSConnectionPool(host='www.kaggle.com', port=443): Max retries exceeded with url: /account/login (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f8e70ef6f50>: Failed to establish a new connection: [Errno 110] Connection timed out',))

I just tried to submit to https://www.kaggle.com/c/facebook-v-predicting-check-ins/ and got an error

 kg config -c facebook-v-predicting-check-ins -u {USER} -p {PASS}
 kg submit data/Notebooks/submission2016.06.08.01.38.csv.zip -m '2nd submit'
no form matching nr 0

Use Case: Shared data-sets used for creating course projects.

Trying to download competition files gives me authentication errors:

$ kg download -u <my-username> -p <my-password> -c <competition>
Starting new HTTPS connection (1): www.kaggle.com
The username or password provided is incorrect.
'NoneType' object has no attribute 'get'

I'm using the same user and password that lets me login via the browser with no problems.

I don't want my password to be seen when I work in public spaces.

Usually, linux utilities ask the user to input password (without echoing) if it is not provided.

I am trying to submit to the whats-cooking competition as a late submission. I continually get a competition not found message, even though I do successfully retrieve my past submissions. The same happens when I try a dummy submission to carvana-image-masking-challenge (though my submissions returns nothing).
The submit_url in submit.py always 404's if I got to [competition_url]/submissions/attach, but if I go to [competition_url]/submit there is no longer a form with the id tags that submit.py looks for.

I got error while running the download command.

kg download -u '******' -p '******' --competition 'https://www.kaggle.com/c/data-science-bowl-2017'

I got the following error:

Starting new HTTPS connection (1): www.kaggle.com
'NoneType' object has no attribute 'find_all'

Any suggestions??

Hi,
Is there any way to download our old submission from Kaggle My submissions page from a competition using kaggle-cli?
Thanks!

Maybe it's something specific to my installation, but kg download -c titanic (or any competition) is hanging in config.py. It's stuck in the loop:

def get_config_candidates(curdir):
    # derive parent dir of homedir
    cnt = 0
    while curdir != os.path.dirname(os.path.expanduser('~')):
        config_path = os.path.join(
            curdir, CONFIG_DIR_NAME, CONFIG_FILE_NAME
        )
        cnt += 1
        print 'config_path:', config_path, cnt
        if os.path.isfile(config_path):
            config = ConfigParser(allow_no_value=True)
            config_file = open(config_path)
            config.read_file(config_file)
            yield config
        curdir = os.path.dirname(curdir)  # derive parent dir

I added the cnt and print as a debug. It just sits there constantly increasing the cnt. My OS is Ubuntu 16.04 and I'm using Python 2. The config_path it prints is /.kaggle-cli/config. Over and over.

I'm trying to fix it but I guess I'm a bit confused what this function is supposed to do. What do you mean by 'derive parent dir of homedir'? Something like /ubuntu if the homedir is /ubuntu/home?

The traceback is:

config_path: /.kaggle-cli/config 43027
^CTraceback (most recent call last):
  File "/usr/local/bin/kg", line 11, in <module>
    load_entry_point('kaggle-cli==0.8.1', 'console_scripts', 'kg')()
  File "/usr/local/lib/python2.7/dist-packages/kaggle_cli/main.py", line 21, in main
    return app.run(argv)
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/cliff/command.py", line 90, in run
    return self.take_action(parsed_args) or 0
  File "/usr/local/lib/python2.7/dist-packages/kaggle_cli/download.py", line 22, in take_action
    config = get_final_config(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/kaggle_cli/config.py", line 55, in get_final_config
    get_working_config(get_config_candidates(os.getcwd())),
  File "/usr/local/lib/python2.7/dist-packages/kaggle_cli/config.py", line 41, in get_working_config
    map(lambda config: dict(config['user']), configs),
  File "/usr/local/lib/python2.7/dist-packages/kaggle_cli/config.py", line 22, in get_config_candidates
    print 'config_path:', config_path, cnt
KeyboardInterrupt

Hi,

I'm running it on windows 10 with python installed and the Kaggle-cli being installed using the pip command.
After the installation I'm getting:
C:>kg config
'module' object is not callable

Any idea what should I do?

Thanks,
joao

When submission is made using
kg submit -u -p -c -m ""

'NoneType' object has no attribute 'find'

is thrown.

Hello,

I am trying to make a submission through kg however, whenever I run a command, I seem to get No module named dummy_thread'error. However if I usekg config` it displays config details.

Thanks.

I keep getting this error.
I have config filled with info and launch the command with option -c ""
Tried with the competition name only and it worked now :)

I just ran a kg download with flag "-c ultrasound-nerve-segmentation", and the .zip files that downloaded were just html files rather than actually zip files. It looks like Kaggle might be doing some javascript magic to prevent your cli tool from working.

[ec2-user@ip-10-236-17-221 test]$ ls -l
total 64
-rw-rw-r-- 1 ec2-user ec2-user 15424 Jul 31 22:14 sample_submission.csv.zip
-rw-rw-r-- 1 ec2-user ec2-user 15390 Jul 31 22:14 test.zip
-rw-rw-r-- 1 ec2-user ec2-user 15412 Jul 31 22:14 train_masks.csv.zip
-rw-rw-r-- 1 ec2-user ec2-user 15392 Jul 31 22:14 train.zip

I tried to use the CLI to download a dataset,
kg download -c ml-project-1 -u username -p password
But received this error:
[Errno 2] No such file or directory: '/home/user/.kaggle-cli/browser.pickle'

This is in a miniconda environment. I have access to the competition online but am unable to download the data using the CLI.
It would be great help if this issue could be resolved.

Thanks,
Ankit

Hi,

I run into an error when using 'kg download' to get data to my EC2.

$ kg download -u username -p passwprd -c grupo-bimbo-inventory-demand
no form matching nr 0

Thanks.

I get this error when I do "kg submit out.csv"
config is:

[('username', u's******@gmail.com'), ('password', '********'), ('competition', u'cats-vs-dogs-redux-kernels-edition')]

I use kaggle-cli to download the data in private competition in school, but no response and not get anything.
Can anyone help me?

Following is the error message, please advice.

downloading https://www.kaggle.com/c/carvana-image-masking-challenge/download/metadata.csv.zip

Something wrong here, Incorrect file !
downloading https://www.kaggle.com/c/carvana-image-masking-challenge/download/train_masks.zip

train_masks.zip N/A% | | ETA: --:--:-- 0.0s
Warning:download url for file train_masks.zip resolves to an html documentrather than a downloadable
file.
See the downloaded file for details.Is it possible you have notaccepted the competition's rules on the kaggle website?

train_masks.zip 4% |#### | ETA: 0:00:54 408.8 B/sValue 23552 is out of range, should be between 0 and 23337

Hi, thanks for writing this package. I start to use it and it works brilliantly. I wonder difficult it is to enable this package read an encrypted configuration file, like config.gpg?

I first did

kg config -g -u USERNAME -p PASSWORD -c COMPETITION

then

kg download

and it returned the error

There was an error logging in: The User name field is required.
The Password field is required.

when i try to use it i get this error :
[Errno 2] No such file or directory: '$HOME/python/lib/python3.6/site-packages/ptyprocess-0.5.2.dist-info/top_level.txt'
any ideas how to solve it ? it used to work fine but then i installed something which i forgot and it seems like it deleted that file

I needed to install additional packages to make it work as follows:

pip install stevedore PrettyTable cmd2

I got this when trying to submit anything after a clean install (python 3)

kaggle-cli==0.12.8
html5lib==0.999999999

DUMP:

Traceback (most recent call last):
  File "/home/felipe/tf-venv3/bin/kg", line 11, in <module>
    sys.exit(main())
  File "/home/felipe/tf-venv3/lib/python3.5/site-packages/kaggle_cli/main.py", line 21, in main
    return app.run(argv)
  File "/home/felipe/tf-venv3/lib/python3.5/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/home/felipe/tf-venv3/lib/python3.5/site-packages/cliff/app.py", line 363, in run_subcommand
    subcommand = self.command_manager.find_command(argv)
  File "/home/felipe/tf-venv3/lib/python3.5/site-packages/cliff/commandmanager.py", line 83, in find_command
    cmd_factory = cmd_ep.resolve()
  File "/home/felipe/tf-venv3/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2415, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/felipe/tf-venv3/lib/python3.5/site-packages/kaggle_cli/submit.py", line 8, in <module>
    from . import common
  File "/home/felipe/tf-venv3/lib/python3.5/site-packages/kaggle_cli/common.py", line 5, in <module>
    from mechanicalsoup import Browser
  File "/home/felipe/tf-venv3/lib/python3.5/site-packages/mechanicalsoup/__init__.py", line 2, in <module>
    from .browser import Browser
  File "/home/felipe/tf-venv3/lib/python3.5/site-packages/mechanicalsoup/browser.py", line 3, in <module>
    import bs4
  File "/usr/lib/python3/dist-packages/bs4/__init__.py", line 30, in <module>
    from .builder import builder_registry, ParserRejectedMarkup
  File "/usr/lib/python3/dist-packages/bs4/builder/__init__.py", line 314, in <module>
    from . import _html5lib
  File "/usr/lib/python3/dist-packages/bs4/builder/_html5lib.py", line 70, in <module>
    class TreeBuilderForHtml5lib(html5lib.treebuilders._base.TreeBuilder):
AttributeError: module 'html5lib.treebuilders' has no attribute '_base'

Hello,

I've set the global config as:

kg config -g -u obeavers -p password -c dogs-vs-cats-redux-kernel-edition

When I go to call :

kg download -v

It returns:

Starting new HTTPS connection (1): www.kaggle.com
https://www.kaggle.com:443 "GET /account/login HTTP/1.1" 200 3452
list index out of range

Am I doing something wrong here? I have accepted the terms of the contest.

Thx.

Hi,

The command line program works for me to do a submission to kaggle. However, I would like to get the result this submission made on the leaderboard.

Is there an easy way to do this?

reduce() of empty sequence with no initial value

If a username or password is incorrect and someone tries to download files, the files will still be downloaded with their names, but the size will be much smaller than it should be and it will be an HTML file. I tried logging in with the CLI using my Google creds, and instead of downloading files, it downloaded HTML responses. It would be nice if the CLI could catch this error.

Maybe it could also try to first successfully login (or maybe try to download a small file from some competition) when the config username and password are set, and then report if the username or password is wrong. In the case of downloading a file, this could try to unzip the file, then if it fails, check if it's actually HTML, and if it is HTML, report an error 'incorrect password and/or username'.

An example of the returned HTML is contained in the attached zip.
train.html.zip

Hi,

I tried kg download -u <login> -p <psw> dogs-vs-cats but it says unrecognized arguments: dogs-vs-cats

How else to refer to this competition?

Assuming that I'm not completely confused, I think Kaggle just changed its HTML code and as a result, (at the very least) downloading datasets doesn't work any more; for example, https://www.kaggle.com/c/the-nature-conservancy-fisheries-monitoring/data contains no data-files div any more, leading to the stack trace

https://www.kaggle.com:443 "GET /account/login HTTP/1.1" 200 5203

https://www.kaggle.com:443 "POST /account/login HTTP/1.1" 302 133
https://www.kaggle.com:443 "GET /account/welcome HTTP/1.1" 302 118
https://www.kaggle.com:443 "GET / HTTP/1.1" 200 9978
https://www.kaggle.com:443 "GET /c/the-nature-conservancy-fisheries-monitoring/data HTTP/1.1" 200 25109

  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/kaggle_cli/download.py", line 30, in take_action
    links = data_page.soup.find(id='data-files').find_all('a')
AttributeError: 'NoneType' object has no attribute 'find_all'