Comments (10)
A few insights of my preliminary analysis: GitHub rate-limits to 5000 requests per hour. Looking into your setup, we get decently close to that boundary with a single sync already.
There are a few ways to work around this limitation:
- Only sync open issues (will still fail / conflict with rate limits at some point)
- Find a clever sync strategy that adjusts the synchronization rate to never hit the rate limit
Additional idea:
- We should always prioritize open issues when synchronizing
from wuffle.
A few more details on how background synchronization works that relates to this issue:
Background synchronization handles open and closed issues differently: Open issues are always synchronized, closed issues are only synchronized if they have been updated in the last 30 days. This is obviously a problem in test setups or super huge organizations that have 1000 or more issues open or closed in thirty days.
31c2747 and 9dd0c63 ship some additional changes:
- Details (checks, statuses, reviewers) are only being synchronized if issues have recently been updated (in last four hours for closed issues, in the last day for open issues)
- Updating an issue on GitHub or on the board ensures that it is being picked up by background sync and synched including their details with the next sync check.
- Those things can be fine-tuned via environment variables to suite specific needs and balance open versus closed issue synchronization and initial sync behavior (cf. commits mentioned above).
As the result of this change, initial is going to fetch details for recently updated issues only. As issues are being worked on (i.e. changed via the board or GitHub) details will eventually be background synched.
I expect this to be a good change and look forward for your feedback how this behaves in your organizations. I got a few additional ideas for optimization, however I'd rather see the impact of this change in bigger setups (yours, ours) before dumping huge amounts of additional work in this matter.
Closing this issue for now, looking forward for your feedback.
from wuffle.
This also happens when there is no storedump.json
file present
from wuffle.
Did you add the new required permissions on status
, checks
and so on? Please check your log for WARN
entries.
from wuffle.
Check this diff for the newly required permissions.
from wuffle.
Yes, I need to add the webhook permission but I already had the checks and status added .
I created a new app to check and compare the permissions to make sure they matched.
I have my logs outputted to a file attached. The config is valid and there is no warning except for the abuse limits
wuffle.log
from wuffle.
Thanks for sharing the log, is it helpful.
from wuffle.
I've recently run into this as well in a large organization. Prioritizing open issues as well as newer issues would be a great help.
from wuffle.
Thanks for that I have just updated to v0.29.0
and I have been rate limited yet which is great.
I have tried it against with an original storedump.json
present and with no storedump.json
Thanks for getting that sorted
from wuffle.
Glad to hear!
from wuffle.
Related Issues (20)
- Cards not displayed on wuffle pod restart until next background sync HOT 7
- Moving a draft PR to review column should mark it as ready for review
- Filter `reviewer` displays PRs where mentioned person is the author HOT 4
- Support GitHub style task breakdowns to create epics
- Add a filter that shows all cards I work on (issues + PRs) HOT 1
- Improve developer getting started experience HOT 6
- Comments in task lists are treated as dependent issues
- Add keyboard / mouse shortcuts quick documentation
- Card task list hides completed items HOT 1
- Self referencing dependency created on issue comment link
- Indicate missing required event subscriptions and missing required permission HOT 3
- It's not possible to configure the BASE_URL for a domain starting with board HOT 1
- Board fails to load after login (refresh required) HOT 1
- Cannot find module 'hbs' HOT 2
- Clicking on a filter suggestion does not select it HOT 1
- Support repository transfer
- Support issue transfer
- Issue transferred to different repository is not being cleaned up
- Synchronize issue comments
- Add <mentions:someuser> filter
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wuffle.