Floe

Overview

Floe is a runner for Amazon States Language workflows with support for Docker resources and running on Docker, Podman, or Kubernetes.

Installation

Install the gem and add to the application's Gemfile by executing:

$ bundle add floe

If bundler is not being used to manage dependencies, install the gem by executing:

$ gem install floe

Usage

Floe can be run as a command-line utility or as a ruby class.

Command Line

bundle exec ruby exe/floe --workflow examples/workflow.asl --inputs='{"foo": 1}'

By default Floe will use docker to run docker:// type resources, but podman and kubernetes are also supported runners. A different runner can be specified with the --docker-runner option:

bundle exec ruby exe/floe --workflow examples/workflow.asl --inputs='{"foo": 1}' --docker-runner podman
bundle exec ruby exe/floe --workflow examples/workflow.asl --inputs='{"foo": 1}' --docker-runner kubernetes --docker-runner-options namespace=default server=https://k8s.example.com:6443 token=my-token

If your workflow has Credentials you can provide a payload that will help resolve those credentials references at runtime.

For example if your workflow had the following Credentials field with a JSON Path property:

"Credentials": {
  "RoleArn.$": "$.roleArn"
}

You can provide that at runtime via the --credentials parameter:

bundle exec ruby exe/floe --workflow my-workflow.asl --credentials='{"roleArn": "arn:aws:iam::111122223333:role/LambdaRole"}'

Or if you are running the floe command programmatically you can securely provide the credentials via a stdin pipe via --credentials=-:

echo '{"roleArn": "arn:aws:iam::111122223333:role/LambdaRole"}' | bundle exec ruby exe/floe --workflow my-workflow.asl --credentials -

Or you can pass a file path with the --credentials-file parameter:

bundle exec ruby exe/floe --workflow my-workflow.asl --credentials-file /tmp/20231218-80537-kj494t

If you need to set a credential at runtime you can do that by using the "ResultPath": "$.Credentials" directive, for example to user a username/password to login and get a Bearer token:

bundle exec ruby exe/floe --workflow my-workflow.asl --credentials='{"username": "user", "password": "pass"}'

{
  "StartAt": "Login",
  "States": {
    "Login": {
      "Type": "Task",
      "Resource": "docker://login:latest",
      "Credentials": {
        "username.$": "$.username",
        "password.$": "$.password"
      },
      "ResultPath": "$.Credentials",
      "Next": "DoSomething"
    },
    "DoSomething": {
      "Type": "Task",
      "Resource": "docker://do-something:latest",
      "Credentials": {
        "token.$": "$.bearer_token"
      },
      "End": true
    }
  }
}

Ruby Library

require 'floe'

workflow = Floe::Workflow.load("workflow.asl")
workflow.run!

You can also specify a specific docker runner and runner options:

require 'floe'

Floe::Workflow::Runner.docker_runner = Floe::Workflow::Runner::Podman.new
# Or
Floe::Workflow::Runner.docker_runner = Floe::Workflow::Runner::Kubernetes.new("namespace" => "default", "server" => "https://k8s.example.com:6443", "token" => "my-token")

workflow = Floe::Workflow.load("workflow.asl")
workflow.run!

Non-Blocking Workflow Execution

It is also possible to step through a workflow without blocking, and any state which would block will return Errno::EAGAIN.

require 'floe'

workflow = Floe::Workflow.load("workflow.asl")

# Step through the workflow while it would not block
workflow.run_nonblock

# Go off and do some other task

# Continue stepping until the workflow is finished
workflow.run_nonblock

You can also use the Floe::Workflow.wait class method to wait on multiple workflows and return all that are ready to be stepped through.

require 'floe'

workflow1 = Floe::Workflow.load("workflow1.asl")
workflow2 = Floe::Workflow.load("workflow2.asl")

running_workflows = [workflow1, workflow2]
until running_workflows.empty?
  # Wait for any of the running workflows to be ready (up to the timeout)
  ready_workflows = Floe::Workflow.wait(running_workflows)
  # Step through the ready workflows until they would block
  ready_workflows.each do |workflow|
    loop while workflow.step_nonblock == 0
  end
  # Remove any finished workflows from the list of running_workflows
  running_workflows.reject!(&:end?)
end

Docker Runner Options

Docker

Options supported by the Docker docker runner are:

network - What docker to connect the container to, defaults to "bridge". If you need access to host resources for development you can pass network=host.
pull-policy - Pull image policy. The default is missing. Allowed values: always, missing, never

Podman

Options supported by the podman docker runner are:

identity=string - path to SSH identity file, (CONTAINER_SSHKEY)
log-level=string - Log messages above specified level (trace, debug, info, warn, warning, error, fatal, panic)
network=string - What docker to connect the container to, defaults to "bridge". If you need access to host resources for development you can pass network=host.
noout=boolean - do not output to stdout
pull-policy=string - Pull image policy. The default is missing. Allowed values: always, missing, never, newer
root=string - Path to the root directory in which data, including images, is stored
runroot=string - Path to the 'run directory' where all state information is stored
runtime=string - Path to the OCI-compatible binary used to run containers
runtime-flag=stringArray - add global flags for the container runtime
storage-driver=string - Select which storage driver is used to manage storage of images and containers
storage-opt=stringArray - Used to pass an option to the storage driver
syslog=boolean - Output logging information to syslog as well as the console
tmpdir=string - Path to the tmp directory for libpod state content
transient-store=boolean - Enable transient container storage
volumepath=string - Path to the volume directory in which volume data is stored

Kubernetes

Options supported by the kubernetes docker runner are:

kubeconfig - Path to a kubeconfig file, defaults to KUBECONFIG environment variable or ~/.kube/config
kubeconfig_context - Context to use in the kubeconfig file, defaults to "default"
namespace - Namespace to use when creating kubernetes resources, defaults to "default"
pull-policy - Pull image policy. The default is Always. Allowed values: IfNotPresent, Always, Never
server - A kubernetes API Server URL, overrides anything in your kubeconfig file. If set KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT will be used
token - A bearer_token to use to authenticate to the kubernetes API, overrides anything in your kubeconfig file. If present, /run/secrets/kubernetes.io/serviceaccount/token will be used
ca_file - Path to a certificate-authority file for the kubernetes API, only valid if server and token are passed. If present /run/secrets/kubernetes.io/serviceaccount/ca.crt will be used
verify_ssl - Controls if the kubernetes API certificate-authority should be verified, defaults to "true", only vaild if server and token are passed

Development

After checking out the repo, run bin/setup to install dependencies. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and the created tag, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/ManageIQ/floe.

Pod names might overload the 63 character limit

Extracted from #132 (comment)

I think there's technically a prior bug here since the pod_name can only be 63 characters, and it's possible to have an image name longer than (63 - 2 - 4 - 36 =) 21 characters. Before this change, we could afford 26 characters, which is still pretty small.

I don't think we need a full UUID either - thinking we might be able to use, say, 8 or 12 random hex bytes.

SecureRandom.uuid is overly large for what we require, so I'm thinking we can use something significantly smaller, but wanted to discuss before I wrote a PR.

I'm thinking something like SecureRandom.urlsafe_base64(6) which should give us more than enough entropy in a tiny payload.

method	chars	bits	example	notes
`SecureRandom.uuid`	36	128	`b6bb2535-808a-40bc-a3b1-cd71213c4d4d`	36 = 32 chars + 4 `-`
`SecureRandom.urlsafe_base64(16)`	22	128	`VKQxaJwgrmJkDTrjCj6NFA`
`SecureRandom.hex(6)`	12	48	`f788b7d2f448`
`SecureRandom.urlsafe_base64(8)`	11	64	`sBfQEpPiFVA`	I think this is what YouTube uses
`SecureRandom.urlsafe_base64(6)`	8	48	`PHEghZQl`	*my choice
`SecureRandom.hex(4)`	8	32	`c3b17c47`

We don't actually need uuid's full 128 bits of entropy (2¹²⁸ or 340 undecillion), since we're just trying to avoid collisions on very short-lived things. IMO 48 bits of entropy (2⁴⁸ or 281 trillion) is way more than enough.

@agrare @kbrock Just want to come to an agreement here, then I'll write up the PR.

manageiq / floe Goto Github PK

floe's Introduction

Floe

Overview

Installation

Usage

Command Line

Ruby Library

Non-Blocking Workflow Execution

Docker Runner Options

Docker

Podman

Kubernetes

Development

Contributing

floe's People

Contributors

Watchers

Forkers

floe's Issues

Detected dependencies

Top-level fields

States

Transitions

Timestamps

Data

Reference Paths

Payload Template

Intrinsic Functions

Recommend Projects

Recommend Topics

Recommend Org