This is the first version released of next-gen-scrapy. This repo of was built to allow users to extract all pass locations - completions, incompletions, interceptions, and touchdowns - from the regular season and postseason pass charts provided by Next Gen Stats beginning in 2017.

The pass charts are scraped from the NFL's official Next Gen Stats website, https://nextgenstats.nfl.com/charts/list/pass. After obtaining all of the pass chart images from the website, for every pass chart, the field is then undistorted, and all pass locations on the field relative to the line of scrimmage are extracted in (x,y) coordinate format.

The file pass_and_game_data.csv is the final version of all pass location data for the 2017 regular season and postseason, and the 2018 regular season. After all Python and R scripts are run, for every available pass chart, the data contains Game ID, home team, away team, week, season, player, type of pass, and pass location from the line of scrimmage. This repo will be maintained regularly for bug fixes and new, exciting features and updates - including wide receiver route locations coming soon! Thank you to Sam Ventura, Kostas Pelechrinis, and Ron Yurko for all your help and guidance with this project!

Requires Python 2.7 and R

Python 2.7:

pip install --upgrade pip
pip install requests bs4 lxml Pillow opencv-python scipy numpy pandas scikit-learn

R:

install.packages(c("devtools", "dplyr"))
devtools::install_github(repo = "ryurko/nflscrapR")

From the command line:

python scrape.py 	# Scrape images and html data, store in folder Pass_Charts

python clean.py 	# Clean images and html data, store in folder Cleaned_Pass_Charts

python main.py 		# Extract pass information from images and data, output to pass_locations.csv

Rscript game_data_from_nflscrapR.R 	# Use nflscrapR to match pass information to game information, output to pass_and_game_data.csv

• For a significant amount of pass charts on Next Gen Stats, the number of incomplete passes given in the HTML data, does not match the actual number of incomplete passes depicted in the Pass Charts. Example image here: https://nextgenstats.nfl.com/charts/list/all/kansas-city-chiefs/2017/wild-card/alex-smith/SMI031126/2017/wild-card/pass (there are supposed to be 33-24=9 incompletes in the pass chart, but there are only 8 shown on the field.) For these rows representing incomplete pass locations not present in a pass chart, pass_type is equal to 'INCOMPLETE', and x_coord and y_coord are both NA.
• NA values for x_coord and y_coord if pass locations could not be extracted.
• A row of NA values follwing a Game ID if no pass charts are provided for a game. As of the end of the 2018 regular season, there are 33 games without pass charts for either team.