cs229's Introduction

CS229

CS229 Project: Analysis of Code Submissions in Competitive Programming Contests

by Wenli Looi

Description of files

dl.py: Custom scraper to download contest submissions from Codeforces for 10 contests. Maintains a cache so that the downloading can be easily resumed.
clang.jl: Processes the downloaded submissions using Clang to generate features.
clang.sh: Shell script to run 4 parallel instances of clang.jl to process the data faster. (Clang seems single-threaded.)
tokensproc.jl: Processes the output of clang.jl to extract bigrams and perform normalization/scaling.
learnlib.jl: Library functions related to reading data, printing data stats, processing Clang tokens, training models with ScikitLearn.jl, converting ratings to ranks, computing weights, evaluating models.
learnlibtf.jl: Library functions related to training and evaluating models (logistic regression and neural network) with TensorFlow.
learn.jl: Learns TensorFlow models and writes models to a file.
learntopbigrams.jl: Interprets GDA model to find the most common unigrams/bigrams for a specific class.
learncv.jl: Generates stats for 10-fold cross validation. Uses TensorFlow models written to the file. The linear regression and GDA models are trained on-demand (as they are fast to train).

cs229's People

Contributors

Stargazers

Watchers

cs229's Issues

file not found

No such file or directory: 'cs229_project/data2.json.gz'
i am studing this project,but found this file lost. would you like to add the file for the whole project?

Running into API limits downloading code

Hi,

When running dl.py, I'm able to fetch standings but not submissions. fetch_submission fails because the result of fetch is FAIL, more concretely:

http://codeforces.com/data/submitSource b'submissionId=46209699&csrf_token=' {'_full_url': 'http://codeforces.com/data/submitSource', 'fragment': None, 'type': 'http', 'host': 'codeforces.com', 'selector': '/data/submitSource', 'headers': {'User-agent': '', 'X-csrf-token': '', 'Cookie': ''}, 'unredirected_hdrs': {'Content-type': 'application/x-www-form-urlencoded', 'Content-length': '33', 'Host': 'codeforces.com'}, '_data': b'submissionId=46209699&csrf_token=', '_tunnel_host': None, 'origin_req_host': 'codeforces.com', 'unverifiable': False, 'timeout': <object object at 0x7f5169d121f0>} FAIL

I then changed the code to use:

req = requests.get('http://codeforces.com/contest/'+str(contest)+
                '/submission/'+str(submissionId))
        tree2 = lxml.html.fromstring(req.text)
        code = tree2.xpath('//*[@id="pageContent"]/div[3]/pre/text()')[0]

but I'm running into limits of how many codes I can download per time (seems to be about 45 every 5 or 10 minutes)