rafaeljusto / gddoexp Goto Github PK

As we are today using token bucket strategy to limit the number of requests and the Github API uses requests per hour strategy, we can fall into rate limit when running the scanner. To solve this we could:

• Change the token bucket parameters and sleep the go routines (until X-Ratelimit-Reset) in case that we reach the rate limit (status 403 Forbidden). The problem here is how to stop filling tokens in the bucket while in rate limit.
• Rethink our rate limit strategy to query Github API, maybe something controlled by Github HTTP response headers (X-Ratelimit-Limit, X-Ratelimit-Remaining, X-Ratelimit-Reset).

I suggest saying that a package is "removed from search results" or "not indexed" instead of "archived". The server is not going to store the identified packages in an archive.
@garyburd

When we request for a repository in Github API and receives a 404 Not Found we should also consider to archive this package, as the repository and/or the user account were removed.

Probably a change in github.go :: doGithub function would solve this. But to avoid changing the low level function interfaces, we could create a error code for this related issue (ErrorCodeRemovedPackage?), that would be analyzed in gddoexp.go :: ShouldArchivePackage and gddoexp.go :: IsFastForkPackage.

On the last scan we got 837 of 132277 packages that received an unknown status from Github (!= 200 OK). So this change could archive a significant number of packages (0.63%).