whywaita / myshoes Goto Github PK

View Code? Open in Web Editor NEW

132.0 6.0 13.0 15.31 MB

Auto-scaling VirtualMachine runner 🏃 for GitHub Actions

License: MIT License

Makefile 0.36% Go 98.09% JavaScript 1.31% Dockerfile 0.25%

myshoes github-actions github-actions-runner auto-scaling-runner virtual-machine

myshoes's Issues

datastore LOCK for race condition

need to set docker image tag that same git tag

Currently, the docker tag only sets latest.

Cannot found runner if multi pages

DELETE /target/{uuid} is not working

related: #31

use directory cache if already extracted

now, myshoes skip downloading runner's file if found cache.

myshoes/pkg/starter/scripts.go

Lines 230 to 232 in 2edd288

 elif [ -f "/usr/local/etc/${runner_file}" ]; then 

  echo "${runner_file} cache is found. skipping download." 

  mv /usr/local/etc/${runner_file} ./

But this logic executes gzip -d always. It has occurrs many i/o tasks.

So myshoes need to add directory cache that extracted gzip.

Delete record of runners_running if not found from shoes

need to pass "resource_type" to shoes provider

Input validator

Exist runner_version in update / create target
input github.com in ghe_domain

blocking creating runner if exist slow shoes-provider

text file busy in remote provider

2021/04/20 19:20:40 failed to bung: failed to get plugin client: failed to get shoes client: fork/exec /tmp/shoes-lxd: text file busy

Support case-insensitive in REST API

Now

https://github.com/octo-org/hello-world-javascript-action

OK: octo-org/hello-world-javascript-action
NG: OCTO-ORG/hello-world-javascript-action

Expected

OK: octo-org/hello-world-javascript-action OCTO-ORG/hello-world-javascript-action

Migrate to workflow_job

GitHub released workflow_job as a new webhook event.

https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#workflow_job

myshoes use check_run in

myshoes/pkg/web/webhook.go

Line 46 in b0f514d

case *github.CheckRunEvent:

.
workflow_job provides a label for runs-on in the workflow definition.

check installed to GitHub when deleting a job

step

Create a job
Create an instance for the runner
failed to register runner as a self-hosted
myshoes has dust in runners_runnning

validate more in REST API

check permission before creating
- need to error response if mistake token or scope
check status before deleting
- maybe not delete runner if status is active

Mode change API

#87 becomes checking registration before deleting a job.
But actions/runner has a problem with race. So myshoes needs API for mode chenge.

need to update resource_type when recreating target

myshoes/pkg/web/target_create.go

Line 75 in 8edab30

// TODO: need to update resource

The target error status is always overwritten to active by the lines below

myshoes/pkg/runner/runner.go

Lines 167 to 184 in 98029fd

 for _, r := range sanitizedRunners { 

 if err := m.deleteRunner(ctx, client, r.ds, *r.github.ID, owner, repo, r.status); err != nil { 

 logger.Logf(false, "failed to delete runner: %+v\n", err) 

 if err := datastore.UpdateTargetStatus(ctx, m.ds, t.UUID, datastore.TargetStatusErr, ""); err != nil { 

 logger.Logf(false, "failed to update target status (target ID: %s): %+v\n", t.UUID, err) 

 } 

 continue 

 } 

 } 

 if err := datastore.UpdateTargetStatus(ctx, m.ds, t.UUID, datastore.TargetStatusActive, ""); err != nil { 

 logger.Logf(false, "failed to update target status (target ID: %s): %+v\n", t.UUID, err) 

 return fmt.Errorf("failed to update target status: %w", err) 

 } 

 return nil

When the deleteRunner function is failed, target status becomes "error", but the lines below (L179~182) always set the status as "active".
Therefore the target status will be always "active" even though all the runner deletion was failed.

myshoes can't delete targets if runners is exist

mysql> delete from targets where uuid = "UUID";
ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`myshoes`.`runners`, CONSTRAINT `runners_ibfk_1` FOREIGN KEY (`target_id`) REFERENCES `targets` (`uuid`) ON DELETE RESTRICT)

detect an event that removed queueing runner

GitHub will remove offline runner if connected last 30 days.

https://docs.github.com/en/actions/hosting-your-own-runners/removing-self-hosted-runners

A self-hosted runner is automatically removed from GitHub if it has not connected to GitHub Actions for more than 30 days.

if remove runner for queueing by GitHub, myshoes not remove a runner created by myshoes.

The runner_detail doesn't have audit information from a job.

runner_detail add columns

resourece_type
Repository URL
workflow ID and job ID

failed to get target from repo: %!w(<nil>) when already deleted

configure organization (example)
configure repository (example/repo)
delete repository (example/repo)

Correct

Use config for organization

Now

No queuing job and output error log

failed to process check_run event: failed to search registered target: failed to get target from repo: %!w(<nil>)

Operation API: Cleanup dust runner

The stadium VM has dust runner because bug from myshoes.

can not check reason of delete runner

systemd unit file

Secret datastore for GitHub personal token

Set ETag and If-Modified-Since

https://docs.github.com/en/rest/overview/resources-in-the-rest-api#conditional-requests

Support multiple image ID

This project is great!
Thank you for developing it.

By the way, I would like to be able to change an image ID for each target.
Are there any plans to support multiple image ID?

support remote image

sanitize sql.NullString from response of REST API

Blocking reconcile loop in not return

myshoes/pkg/starter/starter.go

Line 164 in 4d7c105

wg.Wait()

~~found dust runner~~ #72
- stadium has a runner, but not registered on GitHub
- no events if not registered on GitHub, but need to delete runner
#91

Add parameter in target

runner version
- Newest GitHub Runner can't run in the old GitHub Enterprise Server if broken backward compatibility.
provider URL
- e.g.) /shoes-lxd

searchRepo return deleted target

myshoes/pkg/web/webhook.go

Line 148 in 8c3ef91

 func searchRepo(ctx context.Context, ds datastore.Datastore, gheDomain, repo string) (*datastore.Target, error) { 

searchRepo() not check the status of the target.

registered scope repository octocat/example
registered scope organization octocat
delete scope repository octocat/example

So, searchRepo return octocat/example, but this target is deleted.

add host pool
- has many hosts
select runner host
execute a job to selected host

 client, teardown, err := shoes.GetClient() 

 if err != nil { 

 return "", "", "", fmt.Errorf("failed to get plugin client: %w", err) 

 } 

 defer teardown() 

 script, err := s.getSetupScript(ctx, target) 

 if err != nil { 

 return "", "", "", fmt.Errorf("failed to get setup scripts: %w", err) 

 }

Divide ticker

https://github.com/whywaita/myshoes/blob/master/pkg/runner/runner.go#L56-L70

m.doTargetToken need to start in a some interval, But this code block due m.do .

	elif [ -f "/usr/local/etc/${runner_file}" ]; then
	echo "${runner_file} cache is found. skipping download."
	mv /usr/local/etc/${runner_file} ./

	for _, r := range sanitizedRunners {
	if err := m.deleteRunner(ctx, client, r.ds, *r.github.ID, owner, repo, r.status); err != nil {
	logger.Logf(false, "failed to delete runner: %+v\n", err)

	if err := datastore.UpdateTargetStatus(ctx, m.ds, t.UUID, datastore.TargetStatusErr, ""); err != nil {
	logger.Logf(false, "failed to update target status (target ID: %s): %+v\n", t.UUID, err)
	}

	continue
	}
	}

	if err := datastore.UpdateTargetStatus(ctx, m.ds, t.UUID, datastore.TargetStatusActive, ""); err != nil {
	logger.Logf(false, "failed to update target status (target ID: %s): %+v\n", t.UUID, err)
	return fmt.Errorf("failed to update target status: %w", err)
	}

	return nil

	client, teardown, err := shoes.GetClient()
	if err != nil {
	return "", "", "", fmt.Errorf("failed to get plugin client: %w", err)
	}
	defer teardown()

	script, err := s.getSetupScript(ctx, target)
	if err != nil {
	return "", "", "", fmt.Errorf("failed to get setup scripts: %w", err)
	}

whywaita / myshoes Goto Github PK

myshoes's Issues

Now

Expected

Correct

Now

Recommend Projects

Recommend Topics

Recommend Org