https://docs.github.com/en/discussions/quickstart

Add comments to dataset building process and code more generally to enable new contributors to more quickly get up to speed

Add data loader for Excel files in data_loaders.py. See data_loaders.py for examples of other data_loaders and data.py for their usage

Deaths in custody data at https://openjustice.doj.ca.gov/data and NYPD stop and frisk data use Excel.

We will strive to not increase access to data that is not appropriate to share online like people's names, addresses, or other identifying information. Indicate in documentation how users can flag inappropriate data shared in this package

Our repo is currently dependent on geopandas. In Windows, it is easier to install geopandas using conda. If we add our package and required packages (as needed) to conda-forge, our installation instructions could be simplified to: conda install -c conda-forge openpolicedata

https://conda-forge.org/docs/user/introduction.html
https://conda-forge.org/docs/maintainer/adding_pkgs.html#the-staging-process

Some of our packages (at least Socrata) aren't available through conda, but it looks like we could add them to conda-forge: "You don’t have to be the upstream maintainer of a package in order to contribute it to conda-forge. "

Example installation using conda-forge package: https://github.com/cenpy-devs/cenpy

Standard way of filtering is not working with this data set.

Also tried running a query here: https://gis.charlottenc.gov/arcgis/rest/services/CMPD/CMPD/MapServer/14/query

Setting WHERE to Month_of_Stop >= '2020-01-01' AND Month_of_Stop < '2021-01-01' should give dates in 2020.

1st 1000 results are all either Month_of_Stop: 2021/09 or Month_of_Stop: 2021/10.

Same with WHERE = Month_of_Stop >= '2020-01' AND Month_of_Stop < '2021-01'

dataset has been removed from package until this is resolved

Code can be searched for the phrase "TODO" to find all cases

This is a big issue that is being included as a single issue for now for simplicity. This should be broken into multiple issues once it is started.

Data standardization consists of the following:

• Conversion of raw data fields (column names) to standard ones
• Conversion of raw data values (i.e. column contents) to standard ones

This might include:

• Creation of tools that help identify if data standardization works on new data sets that are added
• Tools for informing the user which fields and data values have changed during standardization
• Tools or documentation for informing the user what the definitions of standardized data fields and data values are

• Provide a single source for datasets from many different jurisdictions
• Provide a standard way to access datasets by the name of the jurisdiction rather than by website
• Provide a standard data type (pandas or geodataframe) rather than CSV, JSON, PDF, etc.
• Standardize data field names since different jurisdictions use different field names
• Standardize data since different jurisdictions record data in different ways

Include:

• How to use package
• How to contribute new datasets to package
• Table of available data including PDs where no data was found. Include how to contribute to this table as well (i.e. suggested datasets or PDs without data)

What is the best host for the documentation?

Use PyTest to validate when complete

Create pytest tests to ensure that exporting from a Table to csv works and that importing from csv to a Table also works. This should be tested for datasets with different parameters.

Data from the Socrata source requires an app token if you don't want to be throttled when making data requests.

1. Get an App Token here: http://dev.socrata.com/docs/app-tokens.html
2. Copy the app token
3. Create an environment variable SODAPY_API_KEY and set it equal to the app token

• Setting environment variable in Linux: https://phoenixnap.com/kb/linux-set-environment-variable
• Windows: https://www.wikihow.com/Create-an-Environment-Variable-in-Windows-10

@potto216 Below is additional data that you can use to test your ArcGIS data loader if you feel it is needed. If you navigate down to "View API Resources". The GeoService link can be used to access FCPD's citations data from 2020.

https://policedata-fcpdgis.hub.arcgis.com/datasets/FCPDGIS::issued-citations-2020-layer/about

Note: This is only for testing the ArcGIS data loader. Current plan is to use the GeoJSON to access the table in the package.

Currently, data sets are listed in a Python file so when new data sets are added, an update is required. Change location of data set information to an online location

This has a tutorial that walks you through how to package a simple Python project. It will show you how to add the necessary files and structure to create the package, how to build the package, and how to upload it to the Python Package Index.

https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/about-issue-and-pull-request-templates

https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/creating-a-default-community-health-file

Add mention of template in Reporting a bug section of Contributing.MD

https://www.reddit.com/r/Python/comments/1bbbwk/whats_your_opinion_on_what_to_include_in_init_py/

Add data loader for open datasoft data in data_loaders.py. See data_loaders.py for examples of other data_loaders and data.py for their usage.

Burlington, VT data uses it
https://data.burlingtonvt.gov/explore/dataset/bpd-traffic-stops/information/

Documentation
https://help.opendatasoft.com/apis/ods-search-v1/#search-api-v1
https://data.burlingtonvt.gov/api/v2/console

Release = Publicly available on pypi

Convert initial code to follow Python naming conventions to encourage others to follow suit in the future

https://www.python.org/dev/peps/pep-0008/#package-and-module-names

It could provide export to CSV capability. Maybe analysis including analysis by geography

Columns:

• Date checked
• Who checked
• Source name
• Website of source
• Website of data (for checking for updates in the future)
• Which dataset types (i.e. traffic stops, arrests, use of force, etc.) are available from this source

https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/setting-guidelines-for-repository-contributors

One possible version: https://github.com/pyproj4/pyproj/blob/acaee8e06f05525cfc3ad43b20c87488304367b2/CONTRIBUTING.md

Many websites have CSV files available in addition to APIs. Use CSV file load as validation of API data load

So the date time issue with the Denver data is because this line
table = table.astype({date_field: 'datetime64[ns]'})
in
https://github.com/sowdm/openpolicedata/blob/345ea62ab021dedd45773ffc7f9eb325607a87f2/openpolicedata/data.py#L291

needs to be milliseconds
table = table.astype({date_field: 'datetime64[ms]'})

But this doesn't seem to be standard for most datetime types which use nanoseconds or seconds. What should be done?!?!? :)

https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/about-issue-and-pull-request-templates

https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/creating-a-default-community-health-file

Add mention of template in Reporting a bug section of Contributing.MD

Suggestion:

from datasets import datasets

All dataset information is held in a pandas Dataframe. Return a list of available datasets. Columns of dataset include id (that we generate for making loading a specific dataset easier) , state, department, year, type of table (traffic stops, use of force, arrests, etc.), URL, and a description.

datasets.list()

Return all the datasets for Virginia as a pandas Dataframe

datasets.list(state="Virginia")

Return all the datasets for Fairfax County, Virginia as a pandas Dataframe

datasets.list(state="Virginia", county="Fairfax County Police Department")

Load dataset correspond to this id

ds = datasets.load(id)

Export to CSV. Filename uses standard structure so that it can be imported later.

ds.to_csv(outputDir=folder)

Load from CSV.

ds = datasets.from_csv(id, outputDir=folder)

dataset is stored in a pandas or geopandas dataframe

df = ds.df

This is necessary for posting the package on pypi

https://setuptools.pypa.io/en/latest/userguide/declarative_config.html

When data is requested for download, it can take a while depending on the size of the request. Progress bars and/or print statements should be used to indicate to the user that downloading is in progress so that they don't think the code is hanging.

Maybe this? https://www.geeksforgeeks.org/progress-bars-in-python/

geopandas seems to be what keeps our package from being easily installable.

We could either remove all use or only create geopandas DataFrames if the user has geopandas and create pandas DataFrames otherwise.

Current method for adding datasets requires adding each dataset individual. For departments with datasets that only contain a single year and a new dataset is added annually for the latest year, can the URL be predicted? For example, if the URL was whatever.com/police-data-2019 for 2019 and whatever.com/police-data-2020 for 2020, perhaps, we could add code that searches over the years to find all the datasets.

Example from Baltimore Calls for Service Data:

_builder.add_data(state="Maryland", jurisdiction="Baltimore", table_type=TableTypes.CALLS_FOR_SERVICE, url=["https://opendata.baltimorecity.gov/egis/rest/services/Hosted/911_Calls_For_Service_2017_csv/FeatureServer/0", "https://opendata.baltimorecity.gov/egis/rest/services/Hosted/911_Calls_For_Service_2018_csv/FeatureServer/0", "https://opendata.baltimorecity.gov/egis/rest/services/Hosted/911_Calls_For_Service_2019_csv/FeatureServer/0", "https://opendata.baltimorecity.gov/egis/rest/services/Hosted/911_Calls_For_Service_2020_csv/FeatureServer/0"], data_type=DataTypes.ArcGIS, description=" Police Emergency and Non-Emergency calls to 911", years=[2017,2018,2019,2020], lut_dict={"date_field" : "calldatetime"})

https://medium.com/@jl_ruiz/plot-maps-from-the-us-census-bureau-using-geopandas-and-contextily-in-python-df787647ef77
https://github.com/darribas/contextily
https://contextily.readthedocs.io/en/latest/places_guide.html