Scraping policy about site-policy HOT 4 CLOSED

Thanks, @KOLANICH.

Web scraping is a tricky topic. We worked very hard to write a balanced policy that allows some scraping, but limits abuses and requires scrapers to enact certain privacy-protective policies.

We appreciate your suggestions, but at this time, the site-policy repository is only for discussions of policies. We are not able to approve changes to GitHub functionality, such as new APIs.

from site-policy.

I came here googling for scraping policy information. This policy text about scraping "Researchers may scrape..." "Archivists may scrape..." is that present in live github policy docs anywhere? I guess it used to be, but was since rearranged. I can only find a GitHub Acceptable Use Policies - Spam and inauthentic activity section which seems relevant. "excessive automated bulk activity" is disallowed, but that's less specific, not mentioning "scraping" directly, and certainly no "You may scrape the website for the following reasons"

Over on this github-crawler-lib repo I see they've lifted out the same text about researchers and archivist, but is that text old?

from site-policy.

@harry-wood yes, it's in this repo at https://github.com/github/site-policy/blob/f05aeb95464408a8d854250b2033302e11cf395f/Policies/github-acceptable-use-policies.md#7-information-usage-restrictions which is rendered two sections below the one you linked at https://docs.github.com/en/site-policy/acceptable-use-policies/github-acceptable-use-policies#7-information-usage-restrictions but yes the language was made more general in #309

from site-policy.

is there a data dump for public research somewhere? preferably a .torrent file?

from site-policy.