frantjc / sequence Goto Github PK

if a step is a github actions step, sequence runtimes should mount a volume somewhere under /tmp and point the environment variable $RUNNER_TEMP to that volume

secrets should be able to be read in from the config file e.g.

[secret]
  dockerhub_password = 1234

https://github.com/lima-vm/lima

would make more sense for future stuff like sync prune which should interact with the runtime "around" the step, job, and workflow services

GitHub Actions output some garbage like:

::save-state name=isPost::true
::debug::GITHUB_WORKSPACE = '/sqnc/workspace'
::debug::qualified repository = 'frantjc/containuity'
::debug::ref = 'refs/heads/6d60944b4c4bf5e1ac5207890b40165cfa1df36a refs/heads/pivot'
::debug::commit = '6d60944b4c4bf5e1ac5207890b40165cfa1df36a'
::debug::clean = true
::debug::fetch depth = 1
::debug::lfs = false
::debug::submodules = false
::debug::recursive submodules = false

to stdout/sterr which is supposed to mean something to the runner.

for example, lines beginning with ::debug:: only get printed if some GitHub secret is set to "true"

we'll need some implementation of a ReadWriter (?) that will parse out such things and do the proper action

sqnc prune should delete stuff in the working directory and any containers/volumes that the runtime created that are directly related to sequence

GitHub Actions sometimes have pre, post steps around the "main" step that we are not currently running

e.g. the RunStep, RunJob, RunWorkflow endpoints should accept

verbose  bool // whether or not to log debug stuff
github   stuff // current context from git repository
// probably much more

the latter is a pretty major bug atm as the server looks in its own path to find the git repo which won't work if we are on a remote machine

I have a desire to extract all (or non-default, not sure) runtimes to plugins and use go's plugin lib to initialize them dynamically at runtime. then some stuff like sqnc install runtime kubernetes that puts it in the plugin dir

depends on #2, #3, #4 and #7

if a step is a uses step, pull sequence's image, run it with the cmd plugin uses [uses] [path/to/action/volume]

sequence should at this point pick up the response emitted from the uses plugin and run the action as a step automatically (assuming all the depends on are satisifed)

this image should be used if one isn't provided and should try to mimic a GitHub Actions runner as closely as possible

there should always be at least one volume in a step mounted to the host machine so that users can emit an artifact from a step and get ahold of it.

by default, this volume should probably be mounted on the container somewhere cool like /dev/sqnc, but should be configurable via cli flags

the host mount should probably be somewhere under /tmp, but should also be configurable via cli flags

if a step is a github action step, sequence runtimes should mount a volume somewhere in the step's container and point the environment variable $RUNNER_TOOL_CACHE to that volume

figure out how to us https://github.com/golangci/golangci-lint-action

some sequence steps such as concourse resources and the action-git-cloning portion of a uses step need to relay information back to sequence.

the uses step specifically will need to do this to dynamically run containers depending on what it finds. to achieve this, the step will emit json on stdout that looks something like

{
  "step": {
    "image": "alpine",
    "entrypoint": [...],
    "cmd": [...]
  }
}

sequence's runtime.Run functions should anticipate which containers it runs will do this and parse stdout from such containers, returning the parsed response. sequence should then check the response for a step and, if its present, run that step immediately

figure out how to use https://github.com/codecov/codecov-action

i would like for the sqnc cli to have the following functionality:

sqnc plugin uses --git=/path/to/git ACTION [PATH]

should:

• clone the given action to the given path.
  • an example action might be charge/update-yaml-action/pr@v3. pseudo code for this example:
    • git clone --branch=@v3 https://github.com/charge/update-yaml-action > /dev/null
    • cat pr/action.yml || cat pr/action.yaml | convert_to_step
  • path should default to .
  • this command may assume that git is on the $PATH, but the path to the git binary should be an overridable flag
  • dont worry about assuming that the repo exists in the official github.com; will look into searching enterprise github domains sometime in the future

when converting an action to a step, ignore the existence of actions that have runs.using set to "composite", "Dockerfile" or runs.plugin to anything -- we will deal with those later if at all

additionally, ignore the case of an action using contexts e.g. ${{ github.sha }} -- we will deal with that later as well

if runs.using is "node12" or "node16", then the image should be node:12 or node:16 respectively :)

will try to make a non-mess of this later

stream of consciousness:

• structs Workflow, Job, Step to marshall yaml into.
• some kind of Planner that takes said structs and turns them into some interface, say []Runnable
  • this is because someday I dream of parsing "code" files into a []Runnable rather than yaml/config files, so keeping it generic
• a step can be multiple tasks across multiple containers, as any step with uses must clone the repo being used, potentially build it, and the run the action
• some tasks (e.g. the first task of a uses step, a get step or put step) need a way to communicate info back to sequence, so Runnables need a way to tell sequence if stdout should be parsed into some kind of response that sequence can then use. this is not new for get or put (concourse already does this for its resources), but is a new (and better imo) way of approaching running actions
• need more than just the workdir mounted as a volume
  • /dev/out or something optionally mounted if the user wants the output from a Runnable somewhere on their machine
  • some kind of tool cache for "setup" actions (see setup-go, runner tool cache)
  • special volume for cloning actions into
    • should just be name the actions uses ref so it can be reused
• how to implement "special" tasks like cloning and parsing an action? I'm thinking sequence cli should have such things built right into it, and sequence mounts itself into a container and invokes itself to do them

e.g. ${{ github.sha }} should be replaced with the current sha

currently, we map an action to 2 steps: 1 to get the action and its metadata, and another to execute it.

composite actions throw a wrench in that; we will need 1 step to get the action and its metadata, and an arbitrary number of steps to execute it

make it very simple for now, e.g. runs.using should be an 'docker://image_ref', 'node12' or 'node16', and no usage of any contexts

if a step is a github actions step, sequence runtimes should mount a volume to the working directory and additionally point the environment variable $GITHUB_WORKSPACE to that volume

GitHub Actions steps have access to the docker daemon, so we should, too

figure out how to use https://github.com/goreleaser/goreleaser-action as it seems to do homebrew stuff and add an install.sh script to our repository

currently, the github.com/frantjc/sequence/runtime/github.com/frantjc/sequence/workflow relationship has the workflow owning the creation if the directories that get bound to the step containers. This has lead to the workflow package making some assumptions about the runtime having access to a local directory

if we desire a Kubernetes runtime (and we do) that will have to change

• a Runtime should set up its own mounts
• we can do away with the concept of mount Types e.g. bind, volume, tmpfs. We only really have 1 use for volumes

• runtime
• github vars e.g. GITHUB_TOKEN

expand the step schema to allow to concourse resources to be used

this will also require the workflow schema to have resource_types and resources fields

if a step is a github actions step using, sequence runtimes should mount a volume somewhere on the step's container and point the environment variable $GITHUB_ACTIONS_PATH to that volume

the sequence cli should try to parse information from the .git and .github/ folders in the current directory to fill in the default environment variables that github actions supplies: https://docs.github.com/en/actions/learn-github-actions/environment-variables#default-environment-variables

such variables should be overridable on all sqnc run commands via flags

since sequence will have to implement some of its own step functionality (mostly surrounding github actions uses steps and we want to run all steps as containerized workloads, sequence will need to be built and pushed as a container

reqs:

• alpine 3.14
• install docker-cli and git (others?)
• binary /usr/local/bin/sqnc
• entrypoint sqnc
• cmd --help ?