Use OpenRefine as much as possible to go from raw --> SQL.

Then work on Wiz Tasks/ETL to go from SQL to FoundationDB

Log each request, and optionally the request body and response body

1. Add a -kor -kill option to all manager commands that kill and restart the manager if is running
2. Track the PID of the manager process in wix/default.yaml and check in maybeStartManager
3. Move the generated pipeline into the Manager's State pipelines map so it gets serialized. Don't serialize Graph
4. Extract UUID generation to the utils package so we can replace and slot with any set of ID libraries or even abstract the GenID function to later call a ID service (probably in the Manager API) so it has sole ID generation responsibilities. However, the design of our current UID is fine for distributed use. Also, we could mock the GenID function during testing to make tests more understandable with basic IDs like procID-1 etc.

Don't just move for complexity.

Abstract the simple ideas in k8s into Go library, then use that as base.

Needs strong resource support and table views to describe them.
Support actions and generic verbs

Should be able to:

1. when called with pipeline spec
2. create the pipeline, configure all processors, and if in local mode, enable auto-forwarding
3. if in local mode, write basic state data to a file

Add basic functionality under
wiz tasks or wiz pipelines
create, get, delete, describe? - kubectl like API for managing them

Should be able to create a pipeline from a standalone pipeline file.

Also, create one when called via wiz get <package> as in #2

• HTTP
• Git - done!
• FTP
• object storage (could use Go CDK for this)
• BitTorrent or similar
• Aria2

Should be quick, already marked the spot with a TODO.

Need a few processors for testing though, and all need to be configured before can test.

Setup CI so I get automatic coverage reports + tests are run automatically

• python
• filtery dsl
• Go filters on native data aren't great
• some DSL which is frontend to the Apache Arrow/FoundationDB ETL layer

Processor API extensibility/chaining/merging
Use multiple ProcAPIs -- e.g. use the regular executor one, but an additional special one for the ETL layer -- all kernels exposed as processors with their own configs etc

1. check for "first-time readability"
2. centralize the docs so that they aren't scattered and necessary implementation details are in the code dirs while main user-facing/guide docs are in the docs/ dir
3. Have a friend or someone else try to read and review the code after this. Ask for questions, suggestions etc on APIs + design, readability, understandability of functions/breaking up even more, etc

• json
• prototxt
• protobufs
• hcl

• install
• run
• search
• info

• Wait on package install pipeline, don't quit CLI until it completes.
  • show error on exit, data on success
• view running pipelines
• optionally, as manager option, save JSON/raw output of pipeline to file as well as using Filesystem API
• build out FS API to request files? nah, then external impls of ProcAPI will have a dependency.
  • assume FS API is setup above orchestration env layer
  • for Wiz executor though could explicitly track and allocate file locations

I'm really bad about this

This satisfies all basic fetching reqs and more.

https://github.com/hashicorp/go-getter

It supports:

• Local files
• Git
• Mercurial
• HTTP
• Amazon S3
• Google GCP

Main issue is that not all config options are exposed/supported, but it is really good for quick fetching.

This can handle all our fetching needs for v0.1.0. Velocity!!

Use zerolog or similar.

Plugin/interface with Gin? have Gin logs go into our main logger?

Or build a pure Go client

We need a Kubectl-like API to interact with:

• Managers - a daemon or tool to manage pipelines
• Executors - an environment which runs the Wiz Executor or other processors
• Pipelines - the logical pipeline, a graph of connected processors
• Processors - an individual operation which processes data
• Registries - an endpoint/API/server that contains a set of packages
• Package - a logical Wiz Package

all of these should be able to be created, deleted, viewed, and possibly modified via the CLI.

Some of the commands may be aliased for ease of use but they should all have their own sections.

With Kubectl they chose get <resource>, create <resource> because each resource is just an HTTP endpoint/object that has the same CRUD API on the API server.
Docker however chose <resource> create etc because each resource is managed differently

Since our resources are heterogenous and may be interacting with different client API libraries, API specs, and endpoints, we should be careful about this. Additionally, some resources may not support all operations.