BackgroundSync being rate limited about wuffle HOT 10 CLOSED

A few insights of my preliminary analysis: GitHub rate-limits to 5000 requests per hour. Looking into your setup, we get decently close to that boundary with a single sync already.

There are a few ways to work around this limitation:

• Only sync open issues (will still fail / conflict with rate limits at some point)
• Find a clever sync strategy that adjusts the synchronization rate to never hit the rate limit

Additional idea:

• We should always prioritize open issues when synchronizing

from wuffle.

A few more details on how background synchronization works that relates to this issue:

Background synchronization handles open and closed issues differently: Open issues are always synchronized, closed issues are only synchronized if they have been updated in the last 30 days. This is obviously a problem in test setups or super huge organizations that have 1000 or more issues open or closed in thirty days.

31c2747 and 9dd0c63 ship some additional changes:

• Details (checks, statuses, reviewers) are only being synchronized if issues have recently been updated (in last four hours for closed issues, in the last day for open issues)
• Updating an issue on GitHub or on the board ensures that it is being picked up by background sync and synched including their details with the next sync check.
• Those things can be fine-tuned via environment variables to suite specific needs and balance open versus closed issue synchronization and initial sync behavior (cf. commits mentioned above).

As the result of this change, initial is going to fetch details for recently updated issues only. As issues are being worked on (i.e. changed via the board or GitHub) details will eventually be background synched.

I expect this to be a good change and look forward for your feedback how this behaves in your organizations. I got a few additional ideas for optimization, however I'd rather see the impact of this change in bigger setups (yours, ours) before dumping huge amounts of additional work in this matter.

Closing this issue for now, looking forward for your feedback.

from wuffle.

This also happens when there is no storedump.json file present

from wuffle.

Did you add the new required permissions on status, checks and so on? Please check your log for WARN entries.

from wuffle.

Check this diff for the newly required permissions.

from wuffle.

Yes, I need to add the webhook permission but I already had the checks and status added .
I created a new app to check and compare the permissions to make sure they matched.

I have my logs outputted to a file attached. The config is valid and there is no warning except for the abuse limits
wuffle.log

from wuffle.

Thanks for sharing the log, is it helpful.

from wuffle.

I've recently run into this as well in a large organization. Prioritizing open issues as well as newer issues would be a great help.

from wuffle.

Thanks for that I have just updated to v0.29.0 and I have been rate limited yet which is great.
I have tried it against with an original storedump.json present and with no storedump.json
Thanks for getting that sorted

from wuffle.

Glad to hear!

from wuffle.