GitHub rate limit triggered when syncing a lot of repositories about repo-file-sync-action HOT 9 CLOSED

Awesome - appreciate it @BetaHuhn. Do ping me if you have something you want me to test.

from repo-file-sync-action.

@BetaHuhn no problems running with the develop branch - although I can't prove that it hit the rate limit and backed off :-)

from repo-file-sync-action.

Great, I will merge it into master and release a new version!

from repo-file-sync-action.

Hey @grahammcculloch, thanks for creating the Issue!

I'm not sure if this means the GitHub user associated with GH_PAT has been rate-limited or whether repo-file-sync-action or the workflow itself has been rate-limited?

The GitHub PAT was probably rate-limited.

I wonder if it would help to add a RATE_LIMIT option or similar that could be used to throttle the action's requests to the GitHub API to mitigate this error?

What would the RATE_LIMIT option do? Would it be a fixed delay which we would wait for between requests or repositories?

We could also automatically throttle the number of requests when we encounter the rate limit. I will have to take a closer look at GitHub's Rate Limits, maybe they return a value indicating when we can make a request again.

I don't think we would need the RATE_LIMIT option if we can detect the rate limit automatically.

What do you think?

from repo-file-sync-action.

@BetaHuhn yeah, I agree with your analysis. The GitHub docs on 'dealing with abuse rate limits' are quite useful with their suggestions.

I guess you could handle repos serially instead of concurrently.

Or you could enqueue all GitHub requests through a queue that throttles them to wait at least a second between requests.

I guess this is an edge-case. But presumably, others will have similar issues in due course.

from repo-file-sync-action.

Thanks for taking a closer look @grahammcculloch!

It might be an edge case, but as you mentioned more people are probably going to run into it. This definitely needs to be fixed.

I will take a look at your proposed solutions and will implement one of them as soon as I have time this weekend.

from repo-file-sync-action.

I implemented a fix on the develop branch.

The action now uses plugin-throttling.js to throttle API requests based on the mentioned recommended request throttling best practices.

I also refactored (ec2d589) the GitHub part of the action to simplify the code, but that shouldn't have any effects on the action itself.

@grahammcculloch you can try it out by using the action like this:

uses: BetaHuhn/repo-file-sync-action@develop

I don't have that many repos available right now to test this myself, so please report back if you are still hitting any rate limits.

from repo-file-sync-action.

@BetaHuhn awesome - I will test this out today or tomorrow and get back to you with the results!

from repo-file-sync-action.

🎉 This issue has been resolved in version 1.7.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

from repo-file-sync-action.