Comments (2)
Here is the result (after my local code changes) to both use cases mentioned above...
2024/05/23 19:20:32 will delete runner: 7a8d1181-4452-49d4-93d4-272bada8dc76
2024/05/23 19:20:32 7a8d1181-4452-49d4-93d4-272bada8dc76 is idle and not running 6h0m0s, so not will delete (created_at: 2024-05-23 19:14:55 +0000 UTC, now: 2024-05-23 19:20:32.330051792 +0000 UTC)
2024/05/23 19:20:32 7a8d1181-4452-49d4-93d4-272bada8dc76 is idle and not running 6h0m0s, a recent cancel was found; we will shut down
2024/05/23 19:20:32 will delete runner with GitHub: 7a8d1181-4452-49d4-93d4-272bada8dc76
2024/05/23 19:16:46 7a8d1181-4452-49d4-93d4-272bada8dc76 is not running MustRunningTime
2024/05/23 19:15:46 7a8d1181-4452-49d4-93d4-272bada8dc76 is not running MustRunningTime
2024/05/23 19:14:15 instance create successfully! (job: 7a8d1181-4452-49d4-93d4-272bada8dc76, cloud ID: i-04d5c846a5d0c4adb)
2024/05/23 19:14:14 start create instance (job: 7a8d1181-4452-49d4-93d4-272bada8dc76)
2024/05/23 19:14:14 start job (job id: 7a8d1181-4452-49d4-93d4-272bada8dc76)
the line that indicates new behavior, and hints at how it's implemented is...
2024/05/23 19:20:32 7a8d1181-4452-49d4-93d4-272bada8dc76 is idle and not running 6h0m0s, a recent cancel was found; we will shut down
Basically, I had to expand the webhook code to catch the canceled job case and I chose to put the event details into a Sync.Map.
I then do two things...
- periodically clear the cancelled sync.map when no runners exist (prevent memory leak)
- if we notice that a host has been idle and is in that 6hr category, we walk the cancelled map and find the first entry that matches the same repo that the runner was created for,
- we then remove that entry from the map.
Problems...
So, I know that GH does scheduling and you don't get to control which runner runs which job, so there's no perfection here. We're just doing best effort to match up a cancel with a stale runner.
So far so good but it's only been running for a couple days.
Let me know you want actual code examples and I can put them in another comment.
from myshoes.
I just discovered a race-condition I thought would help anyone else that might be trying to address this problem in a similar way.
I had a case where
- a cancel happened
- the existing delete_runner.go logic detects that a runner has been idle for to long
[...] is idle and not running [...]
- my suggestion above kicks in
[...]a recent cancel was found [...]
- delete_runner.go logic continues and attempts to perform the
deleteRunnerWithGitHub(...)
behaviour - failure happens do to race-condition, we a 422 from GH
422 Bad request - Runner "myshoes-<uuid>" is still running a job"
I went through the GH logs and found that indeed that runner was chosen at the same time and beat the delete code to the finish line.
In my case the ramifications were, we consumed the cancel but didn't reduce the runner count, so we remained in a over-producing state.
One thought I'm having now is to track "in-flight-cancels" and if this race fails again we put the in-flight-cancel back into the cancel pool so it can hopefully consome another one of the idle runners.
from myshoes.
Related Issues (20)
- GetClient() call after getSetupScript()
- Need to delete exist runners_running but not exist in GitHub
- Set ETag and If-Modified-Since
- Input validator
- Delete record of runners_running if not found from shoes
- Support multiple image ID HOT 2
- Metrics for stuck job
- Support case-insensitive in REST API
- Migrate to workflow_job
- Terraform provider
- Delete LOCK from MySQL HOT 1
- Become status is active if use ephemeral and no queue runner
- dependabot label in GHES
- Configure GitHub Apps after running app
- myshoes_memory_runner_queue_concurrency_deleting become negative values
- Delete job when invalid input from AddInstanceRequest
- Implement more metrics HOT 1
- Httpcache is causing excessive runner creation HOT 1
- GitHub http 503 response lead to a job not running HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from myshoes.