opctl / opctl Goto Github PK

If an ops entrypoint returns a non-zero exit code, the exit code of the op will incorrectly be set to 0.

This is a regression as of v0.1.1

User Story:
As a user of the CLI
I would like the ability to query the output
In order to filter to only the records and fields I need

Reference:
Azure CLI
AWS CLI

ContainerStd*WrittenToEvent.Data doesn't represent a single write to stdErr/stdOut

This causes the consumer to have no way to correctly reconstruct the data

Registry:
op/collection list/search
op/collection push
op/collection pull
authorization
authentication

see opspec-io/spec#1

Story:
As a user of the engine
I would like support for opspec 0.1.3
In order to integrate with the latest version of the spec

Current Behavior Example:

subOps:
- url: init
- url: buildProjA
- url: buildProjB
  isParallel: true
- url: buildProjC
  isParallel: true
- url: debug-node

The above means that buildProj{A,B,C} will run together. I believe this could be confusing and propose a different format:

subOps:
- url: init
- url: buildProjA
  isParallel: true
- url: buildProjB
  isParallel: true
- url: buildProjC
  isParallel: true
- url: debug-node

The behavior of the proposed format would be more readable (subjectively speaking here, others should feel free to weigh in), as I would understand that buildProj{A,B,C} are all running together as one parallel "block". In the current behavior, I'm led to think that only buildProj{B,C} are designated to run in parallel.

A desirable use case is to listen for all log entries associated with a given root op run.

Currently this is not supported as log entries do not contain the root op run id

Creating an explicit OpRunLogEntryEmitted event type would enable this use case.

opspec-io/spec#1

When running many parallel ops 20-30, the engine will become unresponsive and eventually crash.

$ docker logs opctl_engine
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0xc0 pc=0x570f07]

goroutine 8 [running]:
panic(0x860d40, 0xc82000a090)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/opctl/engine/tcp/models.NewEventMsg(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/opctl/engine/tcp/models/eventMsg.go:16 +0x107
github.com/opctl/engine/tcp.getEventStreamHandler.ServeHTTP.func3(0xc8202800e0, 0xc8201c6de0, 0x7fb8c300d0e0, 0xc82027a270, 0xc82006fc00)
    /go/src/github.com/opctl/engine/tcp/getEventStreamHandler.go:66 +0xf5
created by github.com/opctl/engine/tcp.getEventStreamHandler.ServeHTTP
    /go/src/github.com/opctl/engine/tcp/getEventStreamHandler.go:80 +0x2cd

As a consumer of the API
I would like to obtain the state of, & references between, all ops and/or containers in a given op graph
in order to visualize/debug it

User Story:
As an user of the spec
I want to declare retry behavior for a call
So that I can handle transient failure of calls

Example:

run:
  retry:
    strategy: exponential-backoff
    limit: 5
  op:
    ref: xyz

Error text: "failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network"

Fix: docker network rm (all non-default networks)

Presumably, the client should error out or do something other than appearing to work.

repro:

1. open a powershell terminal on Windows
2. clone a fresh copy of https://github.com/opspec-io/spec
3. cd examples/nodejs
4. opctl run install-deps

expected:
op runs to completion, displays ContainerExited & OpEnded events, and exits

actual:
op never shows ContainerExited or OpEnded events and never exits; ctrl +c doesn't work either

Tried to run the sample at: https://github.com/opspec-io/spec/blob/master/examples/azure/create-resource-group/op.yml with the following:

opctl run -c azure/ -a location=asia create-resource-group

this is an issue to track socket input/output support

If I have sibling ops with isParallel flag set, when one of them starts with an outcome of failed, the siblings should be killed immediately.

setup

1. create an op w/ an input constraint
2. set an environment var w/ same name as input and value not meeting defined constraint
3. run op

expected
display msg value from environment invalid & prompt for value

actual
validation error displayed over and over again in infinite loop

User Story:
As a user of the CLI
I would like the ability to specify the format of the output
In order to use it directly from tools supporting particular formats

Reference:
Azure CLI
AWS CLI
Docker CLI
k8s kubectl

OpEncounteredError Id='09404bcc0352415b6f56c4b5e72ee505' OpRef='/c/\Users\chris\repos\github.com\opspec-io\spec\examples\nodejs/.opspec/test-unit' Timestamp='2017-02-22T16:14:08Z'
Msg='open /c/\Users\chris\repos\github.com\opspec-io\spec\examples\nodejs/.opspec/test-unit/op.yml: The system cannot find the path specified.'

OpEnded Id='09404bcc0352415b6f56c4b5e72ee505' OpRef='/c/\Users\chris\repos\github.com\opspec-io\spec\examples\nodejs/.opspec/test-unit' Outcome='FAILED' Timestamp='2017-02-22T16:14
:08Z'

notice the paths are not valid

when docker-compose errors on building a service, the OpRun's exit code is not correct.

http://json-schema.org/latest/json-schema-validation.html#rfc.section.3.1

User Story:
As the author of an op
I would like the ability to declare the version of opspec my op uses in
In order to formalize the version being used

Reference:
Azure Resource Manager Template Format

Currently events are just held in memory.
A proper event store needs to be implemented

User Story:
As the author of an op
I would like the ability to declare the allowed values of a string input
In order to formalize its set of allowed values

Reference:
AWS Cloud Formation Template Parameters
Azure Resource Manager Template Parameters

@poppuyo

User Story:
As a user of the CLI
I would like node creation to be automatic
In order save me from having to create a node

setup

1. create an op w/ a container
2. open a terminal and run opctl events; don't close it
3. open a different terminal and run the op you created
4. run op
5. compare output of both terminals

expected
output should be the same

actual
output from run cmd is incomplete

there are many common types and interactions between sdk-golang & engine-sdk-golang. Also, it is simpler conceptually for consumers if there is just one SDK

repro:

1. start a node via opctl node create
2. kill the node via opctl node kill

expected:
the node process created from 1) is killed by 2)

actual:
the node process keeps running

specify serial/parallel via block

run:
  subOps:
    - serial:
      - url: serial-1_1
      - url: serial-1_2
    - parallel:
      - url: parallel-1
      - serial:
        - url: serial-2_1
        - url: serial-2_2
      - url: parallel-2
    - serial:
      - url: serial-3_1

setup

1. create an op w/ no inputs
2. create a second op w/ an input constraint
3. reference the second op (child) from the first (parent)
4. run the first (parent) op

expected
both ops fail and input validation error displayed

actual
child fails but parent succeeds

fix: pass the same env vars to docker-compose down as were passed to docker-compose up

PS C:\WINDOWS\system32> opctl events
error while listing nodes: findErr was: OpenProcess: The parameter is incorrect.
error while killing node: findErr was: OpenProcess: The parameter is incorrect.
created node w/ PID: 7056

this results in the node being re-created every time you run the events or run command

Currently if events contain secret params the secrets are visible in plain text.

Now that the context package is part of standard lib; we should update all existing code paths to pass/make use of it

it is desirable to run multiple ops simultaneously with the same name.

currently this is unsupported as the first one to finish will result in the others prematurely exiting with code 137

according to docker recommendations;
engine container should be started with --network host arg

When an error occurs in a sub-op; rather than repeating the error from the child, the parent should have a clearer message noting a child failed and giving reference to the child.

proposed message: "child op w/ id: xyz failed"

In order to enable running ops in parallel, we need a way to indicate they should be:

Something like adding an isParallel flag to
https://github.com/opspec-io/spec/blob/master/op-file.md#sub-op

Containerd has a pretty enticing advertisement:

• embeddable
• both windows and linux support
• only does single host (good for p2p story where multi-host orchestration should be left up to opctl)

It is a narrower, more purposeful dependency than all that is and will be docker

If an op is running and an op from the same collection is run, it will cause both ops to end immediately with non-zero exit codes.

setup

1. create an op
2. create a second op w/ an input, a container, and the input bound to an env var set in the container
3. reference the second op (child) from the first (parent)
4. create a third op w/ the output from the second op passed as an input, a container, and the input echo'd to stdout within the container
5. run the first (parent) op

expected
the env var value is printed

actual
nothing is printed because the env var has no value