People are asking questions about how to use the router with Digital Ocean since the router does not create an ELB with that particular provider.

On this page, what is now workflow is still referred to as "controller." This should probably change, but this probably also relates to #7 about clarifying doc scope. i.e. It would be odd to list workflow as a component of itself, which perhaps changes the nature of this page to a "dependencies on other components" page?

The "store" component is also listed. Is that how we want to officially refer to minio? Probably? But I am just checking.

The following describes builder inaccurately.

https://github.com/deis/docs-v2/blame/master/src/understanding-deis/components.md#L23-L29

Among the inaccuracies-- in the case of buildpack apps, builder does not produce Docker images, but rather slugs, and builder does not incorporate last-mile config into the applications.

Depends on:

• deis/builder#224
• deis/slugbuilder#32
• deis/slugrunner#32
• deis/postgres#71

Where should we document the prerequisites of v2? Such as daemonsets. It seems like a separate section

The e2e ps:restart tests are consistently hitting 503 Service Unavailable. (See recent jenkins run for all related failures.) I'm hitting this locally when focusing on these tests.

------------------------------
Processes with a deployed app can restart processes 
  restarts all of 1
  /go/src/github.com/deis/workflow-e2e/vendor/github.com/onsi/ginkgo/extensions/table/table_entry.go:46
$ deis login http://deis.10.135.245.137.xip.io --username=test-81 --password=asdf1234
Logged in as test-81
$ deis ps:scale web=1 --app=test-260918254
Scaling processes... but first, coffee!
...���done in 0s
=== test-260918254 Processes
--- web:
test-260918254-v2-web-ap44e up (v2)
$ deis ps:restart  --app=test-260918254
Restarting processes... but first, coffee!
...���Error: 
503 Service Unavailable
detail: tuple index out of range


• Failure [2.388 seconds]
Processes
/go/src/github.com/deis/workflow-e2e/tests/ps_test.go:161
  with a deployed app
  /go/src/github.com/deis/workflow-e2e/tests/ps_test.go:160
    can restart processes
    /go/src/github.com/deis/workflow-e2e/vendor/github.com/onsi/ginkgo/extensions/table/table.go:96
      restarts all of 1 [It]
      /go/src/github.com/deis/workflow-e2e/vendor/github.com/onsi/ginkgo/extensions/table/table_entry.go:46

      No future change is possible.  Bailing out early after 0.046s.
      Got stuck at:
           coffee!
          ...���
      Waiting for:
          done in \d+s

      /go/src/github.com/deis/workflow-e2e/tests/ps_test.go:124
------------------------------

A sampling of the controller pod logs show the following when this occurs:

...
10.196.2.1 "POST /v2/apps/test-387025991/pods/web/test-387025991-v2-web-xhmps/restart/ HTTP/1.1" 200 114 "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/apps/test-387025991/pods/?limit=1000 HTTP/1.1" 200 701 "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/ HTTP/1.1" 401 58 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/auth/login/ HTTP/1.1" 200 52 "Deis Client v2.0.0-dev"
DEBUG create test-387025991
INFO test-387025991: test-81 scaled containers web=6
INFO [test-387025991]: test-81 scaled containers web=6
DEBUG create test-387025991
10.196.2.1 "POST /v2/apps/test-387025991/scale/ HTTP/1.1" 204 - "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/apps/test-387025991/pods/?limit=100 HTTP/1.1" 200 701 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/apps/test-387025991/pods/restart/ HTTP/1.1" 503 37 "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/ HTTP/1.1" 401 58 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/auth/login/ HTTP/1.1" 200 52 "Deis Client v2.0.0-dev"
DEBUG create test-387025991
INFO test-387025991: test-81 scaled containers web=6
INFO [test-387025991]: test-81 scaled containers web=6
DEBUG create test-387025991
10.196.2.1 "POST /v2/apps/test-387025991/scale/ HTTP/1.1" 204 - "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/apps/test-387025991/pods/?limit=100 HTTP/1.1" 200 701 "Deis Client v2.0.0-dev"
10.196.2.1 "GET /healthz HTTP/1.1" 200 2 "Go 1.1 package http"
10.196.2.1 "GET /healthz HTTP/1.1" 200 2 "Go 1.1 package http"
10.196.2.1 "POST /v2/apps/test-387025991/pods/web/restart/ HTTP/1.1" 503 37 "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/ HTTP/1.1" 401 58 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/auth/login/ HTTP/1.1" 200 52 "Deis Client v2.0.0-dev"
DEBUG create test-387025991
INFO test-387025991: test-81 scaled containers web=6
INFO [test-387025991]: test-81 scaled containers web=6
DEBUG create test-387025991
10.196.2.1 "POST /v2/apps/test-387025991/scale/ HTTP/1.1" 204 - "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/apps/test-387025991/pods/?limit=100 HTTP/1.1" 200 701 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/apps/test-387025991/pods/cmd/restart/ HTTP/1.1" 503 37 "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/ HTTP/1.1" 401 58 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/auth/login/ HTTP/1.1" 200 52 "Deis Client v2.0.0-dev"
DEBUG create test-387025991
INFO test-387025991: test-81 scaled containers web=0
INFO [test-387025991]: test-81 scaled containers web=0
DEBUG scale test-387025991-v2-web, img http://10.199.250.82:9000/git/home/test-387025991:git-779ca3d7/push/slug.tgz, params {'num': 0, 'build_type': 'buildpack', 'envs': {}, 'memory': {}, 'app_type': 'web', 'version': 'v2', 'tags': {}, 'healthcheck': {}, 'cpu': {}}, cmd "start web"
DEBUG scaling RC test-387025991-v2-web in Namespace test-387025991 from 6 to 0 replicas
DEBUG waiting for RC test-387025991-v2-web to get a newer resource version than 36118 (30s timeout)
DEBUG RC test-387025991-v2-web has a new resource version 36135
DEBUG waiting for 6 pods in test-387025991 namespace to be terminated (120s timeout)
10.196.2.1 "GET /healthz HTTP/1.1" 200 2 "Go 1.1 package http"
10.196.2.1 "GET /healthz HTTP/1.1" 200 2 "Go 1.1 package http"
DEBUG 6 pods in namespace test-387025991 are terminated
DEBUG create test-387025991
10.196.2.1 "POST /v2/apps/test-387025991/scale/ HTTP/1.1" 204 - "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/apps/test-387025991/pods/?limit=100 HTTP/1.1" 200 24 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/apps/test-387025991/pods/restart/ HTTP/1.1" 503 37 "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/ HTTP/1.1" 401 58 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/auth/login/ HTTP/1.1" 200 52 "Deis Client v2.0.0-dev"
DEBUG create test-387025991
INFO test-387025991: test-81 scaled containers web=0
INFO [test-387025991]: test-81 scaled containers web=0
DEBUG create test-387025991
10.196.2.1 "POST /v2/apps/test-387025991/scale/ HTTP/1.1" 204 - "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/apps/test-387025991/pods/?limit=100 HTTP/1.1" 200 24 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/apps/test-387025991/pods/web/restart/ HTTP/1.1" 503 37 "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/ HTTP/1.1" 401 58 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/auth/login/ HTTP/1.1" 200 52 "Deis Client v2.0.0-dev"
DEBUG create test-387025991
INFO test-387025991: test-81 scaled containers web=0
INFO [test-387025991]: test-81 scaled containers web=0
DEBUG create test-387025991
10.196.2.1 "POST /v2/apps/test-387025991/scale/ HTTP/1.1" 204 - "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/apps/test-387025991/pods/?limit=100 HTTP/1.1" 200 24 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/apps/test-387025991/pods/cmd/restart/ HTTP/1.1" 503 37 "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/ HTTP/1.1" 401 58 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/auth/login/ HTTP/1.1" 200 52 "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/apps/?limit=100 HTTP/1.1" 200 231 "Deis Client v2.0.0-dev"
DEBUG destroy test-387025991
10.196.2.1 "GET /healthz HTTP/1.1" 200 2 "Go 1.1 package http"
10.196.2.1 "GET /healthz HTTP/1.1" 200 2 "Go 1.1 package http"
WARNING test-387025991: Error deleting existing application logs: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: /logs/test-387025991 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ffae4306be0>: Failed to establish a new connection: [Errno 111] Connection refused',))
WARNING [test-387025991]: Error deleting existing application logs: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: /logs/test-387025991 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ffae4306be0>: Failed to establish a new connection: [Errno 111] Connection refused',))
INFO test-387025991: domain test-387025991 removed
INFO [test-387025991]: domain test-387025991 removed
10.196.2.1 "DELETE /v2/apps/test-387025991/ HTTP/1.1" 204 - "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/ HTTP/1.1" 401 58 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/auth/login/ HTTP/1.1" 200 52 "Deis Client v2.0.0-dev"
10.196.2.1 "DELETE /v2/auth/cancel/ HTTP/1.1" 204 - "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/ HTTP/1.1" 401 58 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/auth/login/ HTTP/1.1" 200 52 "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/apps/?limit=100 HTTP/1.1" 200 52 "Deis Client v2.0.0-dev"
10.196.2.1 "GET /v2/ HTTP/1.1" 401 58 "Deis Client v2.0.0-dev"
10.196.2.1 "POST /v2/auth/login/ HTTP/1.1" 200 52 "Deis Client v2.0.0-dev"
10.196.2.1 "DELETE /v2/auth/cancel/ HTTP/1.1" 204 - "Deis Client v2.0.0-dev"
10.196.2.1 "GET /healthz HTTP/1.1" 200 2 "Go 1.1 package http"
10.196.2.1 "GET /healthz HTTP/1.1" 200 2 "Go 1.1 package http"
...

These 9 tests are the consistent ones failing:

Summarizing 9 Failures:

[Fail] Processes with a deployed app can restart processes [It] restarts all of 1
/Users/vaughndice/work/go/src/github.com/deis/workflow-e2e/tests/ps_test.go:124

[Fail] Processes with a deployed app can restart processes [It] restarts all of 1 by type
/Users/vaughndice/work/go/src/github.com/deis/workflow-e2e/tests/ps_test.go:124

[Fail] Processes with a deployed app can restart processes [It] restarts all of 1 by wrong type
/Users/vaughndice/work/go/src/github.com/deis/workflow-e2e/tests/ps_test.go:122

[Fail] Processes with a deployed app can restart processes [It] restarts all of 6
/Users/vaughndice/work/go/src/github.com/deis/workflow-e2e/tests/ps_test.go:124

[Fail] Processes with a deployed app can restart processes [It] restarts all of 6 by type
/Users/vaughndice/work/go/src/github.com/deis/workflow-e2e/tests/ps_test.go:124

[Fail] Processes with a deployed app can restart processes [It] restarts all of 6 by wrong type
/Users/vaughndice/work/go/src/github.com/deis/workflow-e2e/tests/ps_test.go:122

[Fail] Processes with a deployed app can restart processes [It] restarts all of 0
/Users/vaughndice/work/go/src/github.com/deis/workflow-e2e/tests/ps_test.go:122

[Fail] Processes with a deployed app can restart processes [It] restarts all of 0 by type
/Users/vaughndice/work/go/src/github.com/deis/workflow-e2e/tests/ps_test.go:122

[Fail] Processes with a deployed app can restart processes [It] restarts all of 0 by wrong type
/Users/vaughndice/work/go/src/github.com/deis/workflow-e2e/tests/ps_test.go:122

Milestone: v2.0-beta2

Installation / Configuration

As an operator, I should be able to quickly configure object storage
consistently across Deis Workflow when running helm install deis/workflow-...

Most of the work is described and tracked in: deis/deis#4966

Scenarios to test and validate:

• On-cluster: minio, default
• Off-cluster: AWS
  • Bucket in us-east-1 (s3 us-standard)
  • Bucket in us-west-2 (s3 regional endpoint)
• Off-cluster: GCS w/ S3 compat mode
  • us-central1-x
  • us-east1-x
• Bucket for database backups slug_bucket
• Bucket for slugs slug_bucket
• Bucket for registry registry_bucket

Recoverability

Operators need to have confidence that they can recover from cluster disasters.
We need to test and validate cluster recovery from off-cluster object storage
to working platform.

Reproducible Helm charts: deis/charts#199

Given a set of buckets, used by a previous installation of the platform. An
Operator should be able to quickly recover their instance of Deis Workflow.

Assumes a brand-new k8s cluster, with no conflicting or lingering application
namespaces.

This should be tested on both S3 and GCS-backed clusters.

Champagne

Learn by doing! We need to host our own applications. First two applications
are deis slack inviter and workflow-manager API backend.

https://github.com/deis/deployment/issues/8

Delivery Process

@sgoings details here!

There are several places in the documentation that reference our IRC channel. These should be corrected to reference our Slack channel instead.

I think this section requires more details.

I know @technosophos recently submitted and merged an awesome PR (deis/charts#123) to hypothetically add a consolidated point of configuration for object storage platform-wide. (In practice, I don't think all components are making use of this yet, but they should.)

I think no section on configuring object storage could be complete without covering this.

The welcome page introduces "Deis," but then proceeds to describe the PaaS (Workflow), without ever using the word "Workflow."

We need to clarify the scope of the documentation. Does it document all of Deis as a broader, more general set of reusable components? Or does it document Workflow and other components matter only insofar as Workflow depends upon them?

Ref deis/deis#4962

Depends on deis/charts#167

Currently, some docs (such as object storage) point to https://github.com/deis/charts/tree/master/workflow-dev, but they should probably point to https://github.com/deis/charts/tree/master/workflow-beta1

Similar to the note about the controller managing the slugrunner, the slugbuilder's docs should indicate that the builder manages it, and that all config instructions are for standalone use.

Requires a section in the components doc.

Currently it's the same as that for deis/workflow.

Eventually when we move onto Deis v3, we should still technically be able to use this repository to store our own documentation. Therefore I propose we rename this to plain ol' docs.

Most diagrams are inaccurate. Some contain mentions of things such as etcd, fleet, and deisctl. Some express relationships between components that no longer exist. For instance, the logger does not depend on the store anymore.

This is for those wishing to stick with the notion of "stateless platform" and utilize RDS or similar.

#322

Similar to #110, the docs should add a note saying that credentials need not be base64-encoded in the objectstorage.toml file - only in the minio secret.

Under "Installing Deis," we have separate pages for DigitalOcean and bare metal. Both of these pages have unresolved TODOs on them.

My question is why bother with this? The Quick start page already lists, inline, bulleted links for bootstrapping k8s on AWS, GKE, and Vagrant. If we care about DO and bare metal, we should just add to that bulleted list and not do more than that.

This page (a placeholder) may not be relevant at all. There is no procedure for adding or removing Workflow hosts that is above or beyond the procedure for adding or removing hosts to/from the underlying k8s cluster.

https://github.com/deis/docs-v2/blob/master/src/managing-workflow/adding-or-removing-hosts.md

I propose it be removed. All in favor?

Each Credentials section mentions the paths that the component looks in, but does not explain practically how to create those paths. Each section should reference the secret (i.e. minio-user) to change in the chart in order to make the necessary changes.

After deis/controller#523, the controller will read the minio-user secret from the k8s API, but it won't read any object storage endpoint data. It simply passes the secret on to slugrunner so that the slugrunner can download the slug (note that the builder passes the slug URL through workflow directly to the slugrunner).

Instructions for how to configure this secret for use in the controller/slugrunner should be added to https://github.com/deis/workflow/blob/master/src/installing-workflow/configuring-object-storage.md#deisdatabase

gabrtv 10:42 AM if anyone has spare cycles to move the warning at the top of the docs page to somewhere that looks less god-awful, i would be very thankful :)

I agree that it does stick out like a sore thumb.

The concepts page talk about Docker, but Docker actually seems, increasingly like an implementation detail. Describing the relationship between Deis dependency on k8s and articulating that Deis components run as k8s pods seems like more cogent information.

In the "Customizing Workflow" section, we currently have very little-- just a tiny bit about CLI plugins.

Formerly, we had extensive documentation on how each component could be customized. I think given the complexity of some components, including all of that here might be way overkill.

I would, however, propose that we expand this section two include two things:

1. Links to component-specific documentation for those who really do want to dig down and tweak things.
2. A very brief tutorial on customizing Workflow's Helm chart.

This page needs some rework:

https://github.com/deis/docs-v2/blob/master/src/managing-workflow/configuring-load-balancers.md

This was all, mostly, inherited from the old docs. Parts of it are not entirely accurate and in general, this could stand to be reworked a bit. Load balancing is, unfortunately, a more nuanced topic now than it was previously since there are now multiple viable ways to achieve this.

Ultimately, I'll lift some of what's in the router's docs and reference the router doc for readers wanting further details.

#360

See http://docs-v2.readthedocs.org/en/latest/installing-deis/digitalocean/

installing-the-client.md still points to alpha 'install-v2-alpha.sh', we have cut beta already

With the new architecture being what it is, slugrunner isn't merely a base image for applications, it is its own thing that fetches and runs app slug. It's probably significant enough that it bears describing.

Dockerbuilder uses object storage, so it should be added to https://github.com/deis/workflow/blob/master/src/installing-workflow/configuring-object-storage.md#credentials

As part of the Deis v2 release process, designated registry images will move from the deis to deisci organization, and be re-tagged: e.g. quay.io/deis/controller:v2.0.0-beta1.

Each component's repository web page shows a "Description" field as documentation, for example https://quay.io/repository/deis/controller. These were originally seeded from project README.md files when Deis used Docker Hub for automated builds, but now they must be manually updated and have gone stale.

These description fields need to be updated as be component landing pages, since they will contain registry tags from (at least) v1.9.0 through v2.0.0 and onward.

Since all state is either entirely ephemeral or stored off-cluster, I would posit this (placeholder) page isn't relevant and would propose it be removed.

https://github.com/deis/docs-v2/blob/master/src/managing-workflow/backing-up-and-restoring-data.md

All in favor?

Blocked by deis/postgres#24 and deis/postgres#44

Since the credentials are base64 encoded by helm already, the encoding isn't needed here: https://github.com/deis/workflow/blob/master/src/installing-workflow/configuring-object-storage.md#helm-chart-4

If/when deis/controller#497 is merged, we need to make a corresponding update to the v2 API docs.

Monitor needs some love.

The DB config section should specify exactly how to configure it for minio, S3 and Google Cloud Storage (in compatibility mode)

This issue was created in response to the discussion in deis/postgres#72.

As a vestige of the v1 documentation, the docs still reference things like "control plane" and "data plane."

Are these classification still useful to us?

It references control plane, data plane, and etcd.

You used to be able to send people links to a specific section, such as http://docs.deis.io/en/latest/troubleshooting_deis/#troubleshooting-etcd

This page is currently a placeholder: https://github.com/deis/docs-v2/blob/master/src/managing-workflow/production-deployments.md

I want to start a simple checklist here of recommendations we must document for anyone wishing to run Workflow in production. I invite others to edit this.

• Must: Use off-cluster object storage (or else everything is ephemeral!). Should include a link to object storage configuration page.
• Recommended: Use off-cluster Postgres?
• Highly recommended: Secure the Workflow cluster using an SSL wildcard cert
• Recommended: Generate a unique dhparam for the router (to further harden Diffie Hellman key exchange)

Do we want to make any recommendations re: the k8s clusters themselves? Or is this not our concern? Example: speaking to @slack today, I believe we intend in our dog-fooding cluster to have the k8s worker nodes live in private subnets. I don't know of any pre-baked zero-to-k8s solutions that account for that... so is it worth mentioning? Or maybe is it worth having a whole separate checklist for things like this and articulating that "A Deis Workflow cluster is only as good as the Kubernetes cluster you run it on. Here are some considerations..."

Discuss... discuss...

This line about being able to deploy from a private registry probably isn't entirely correct-- at least not without additional steps having been taken:

https://github.com/deis/docs-v2/blame/master/src/using-deis/using-docker-images.md#L46-L47

The question of deployment from private registries or from private repositories in public registries has been raised before. Just one such example: deis/deis#4756

Per the thread I referenced, I think we can support such scenarios, but that it requires the docker daemon on each host to be pre-authenticated to the registries in question. This isn't documented anywhere currently.

This is definitely an endeavor only for more advanced operators. Do we want to document it? Or does this fall into the realm of things we don't officially support or endorse? Should we just change "private" to "public" here and then wait until we've build Docker auth into workflow itself?

To reflect that this is Deis Workflow documentation?

There are a few references to deis/docs-v2, but they all redirect to deis/workflow so it's okay for now.

I believe we need to call out a newer version of helm. (Docs currently show 0.2) and that we also need to include the helm generate step in the install process.