At the moment we don't discover parameter errors until we create ServiceInstance/Binding objects.

OSB providers can/should have json schemas for their parameters. However, we can't trivially validate against these if we have any Smith references.

Unclear how to solve this. One possible solution is to make smith references include type information sufficient to generate something to pass the json schema validation (or alternatively remove or simplify the json schema in some way - e.g. strip all non-type information, or delete those fields, etc.).

Related problem: validating the type of outputs. But that's not in the OSB spec yet...

Smith hopefully will be relying on Kubernetes' Garbage Collector (GC) + finalizers to handle deletion of resources in case of Bundle deletion. At the moment GC does not support TPRs (Bundle is a TPR) and has other various issues unfortunately:

• kubernetes/kubernetes#44507
• kubernetes/kubernetes#43430
• kubernetes/kubernetes#33612

Finalizers do not work for TPRs:

• kubernetes/kubernetes#40715

Once GC and finalizers are suitable for this use case we should add a finalizer to each Bundle to make sure all resources provisioned by a Bundle are deleted before Smith removes the finalizer from it to let GC delete the Bundle object.
Actually, we don't need finalizers for the Bundle - it will be added and removed by GC itself. We just need GC to handle references properly.

If you define an explicit secret in the bundle such as:

 - name: b
    spec:
      object:
        metadata:
          name: b
        apiVersion: v1
        kind: Secret
        stringData:
          mysecret2: notreallysecret

It updates the existing secret (i.e. leaving earlier keys there) due to kubernetes compatible processing of stringData. This means on resource renames etc. old data is left around. Suggest that we make stringData do a complete replacement rather than an update, given that Smith has different requirements than raw kube.

Given that the usage of explicit secrets is minimal, and this is less of an issue with generated secrets, this is probably low-priority.

Currently objects are indexed by Bundle namespace+name and then controller reference is checked. This can be simplified.

https://github.com/atlassian/smith/blob/master/pkg/plugin/types.go#L32

This will allow us to do version bump actions when secrets change (workaround for service instances not updating when secret inputs change).

This is a step in allowing us to change the 'shape' of secrets, which is particularly important due to the way parametersFrom works in Service Catalog:

https://github.com/kubernetes-incubator/service-catalog/blob/master/docs/parameters.md

(i.e. the secret ends up merged directly into the inputs without the ability to customise the field name)

In the longer term, this functionality would allow:

• plugins to explicitly declare secret dependencies
• objects (e.g. service instances) to directly have non-secret parameters from secrets
• new secrets be created in smith from the outputs of other secrets (however, at the moment this is impossible due to the limited templating language used)

Per recommendation on writing controllers Bundle object should have ObservedGeneration in it's status. Some discussions about it:

• kubernetes/kubernetes#7328
• kubernetes/enhancements#95 (comment)

It is blocked on splitting spec/status for CustomResources (item in list kubernetes/enhancements#95). Because Smith cannot update status without also bumping up metadata.generation (is it even incremented automatically in 1.7?).

See here:

This can use this:

  go_image( 
     name = "smith", 
     embed = [":go_default_library"], 
     importpath = "github.com/atlassian/smith/cmd/smith", 
     visibility = ["//visibility:public"], 
 ) 

(the goal is to make containerizing as simple as s/lang_binary/lang_image/)

You might also TAL at rules_k8s, feel free to reach out if you have any problems.

#194 would be substantially more useful if we could directly reference the secrets output by bindings (which at the moment live outside the bundle, so even if #194 is implemented would not be accessible).

If a resource in a bundle is misspecified, smith hot loops.

E1004 04:09:38.667737       1 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: Object 'Kind' is missing in '.....'

Use case: generate resource names based on the bundle name (see name: {{#metadata.name}}-sleeper1 for example):

apiVersion: smith.atlassian.com/v1
kind: Bundle
metadata:
  name: bundlex
spec:
  resources:
  - name: sleeper1
    spec:
      apiVersion: tpr.atlassian.com/v1
      kind: Sleeper
      metadata:
        name: {{#metadata.name}}-sleeper1
      spec:
        sleepFor: 3
        wakeupMessage: Hello 1 from bundle {{#metadata.name}}!
  - name: sleeper2
    dependsOn:
    - sleeper1
    spec:
      apiVersion: tpr.atlassian.com/v1
      kind: Sleeper
      metadata:
        name: {{#metadata.name}}-sleeper2
      spec:
        sleepFor: 4
        wakeupMessage: "{{sleeper1.status.message}}"
  - name: sleeper3
    dependsOn:
    - sleeper2
    spec:
      apiVersion: tpr.atlassian.com/v1
      kind: Sleeper
      metadata:
        name: sleeper3
      spec: "{{{sleeper2.spec}}}"

Some "global variables" defined within bundle would also work.

At the moment, its behaviour is confusing (and I believe incorrect).

Consider the following case:

• A
• B
• C depends on B
• D depends on B
• E depends on D
• F

(assume this is the topologically sorted order)

A, B, D, and E, F could successfully provision, leaving us with:

• A - Ready
• B - Ready
• C - In Progress
• D - Ready
• E - Ready
• F - Ready

Some time later, C fails. This causes the bundle processing step to then mark D as 'BlockedOnError' (even though it was already provisioned!), E to be blocked on dependencies (since D is now in error), and F to be marked 'BlockedOnError'.

I think we should take 'BlockedOnError' out as a processing state, and just keep trying to process anything that's unresolved.

Alternatively, we can only block on error after actually doing the comparison and seeing if an update is necessary (but this is tricky due to Smith logic, I believe).

If this made no sense, talk to @amckague while I'm away.

Use code generation tools from client-go project. This is a prerequisite for migration to the newest apimachinery.

Right now delete operation only uses UID precondition but it should also use resource version precondition once (if) available. See https://github.com/kubernetes/kubernetes/issues/30698#issuecomment-319955082

In addition to handling defaults (#86), we should handle server-set fields.

Blocked by Service Catalog using 1.8.

Informers panic on errors so we need to ensure that our indexing functions do not return error on invalid Bundles/etc. I.e. only return errors when something really unexpected/bad happens.

Use code generation tools from client-go project. Move convenient for Smith and for dependent projects.

This is necessary to prevent storage of invalid bundles.
https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation

Detect situations where object, returned from update call, does not match the spec and hence notification about update will cause an update again, leading to an infinite loop.
Should catch issues like #101 and prevent hot looping.

Provide ClusterRole that grants necessary permissions.

That way it is more convenient to use - any resource type can be used to populate the field.
MarshalJSON() and UnmarshalJSON() for Resource should be converting it to/from unstructured.Unstructured.

Need to verify that owner references with "block owner deletion" flag actually work as expected. If not, Bundle should have a finalizer to block deletion.

Validate bundle before processing it:

• Plugins can do validation of the spec via json schema/etc
• Check that each resource in the bundle is either an object or a plugin (not both, exactly one of them)
• Check that each used plugin is actually known to Smith
• ...?

It is ok to run multiple copies of Smith because it performs all mutations via concurrency safe operations (except deletes - see #118). However there is benefit in avoiding unnecessary load on api servers.
Should use leader election code available in client-go once we are on 5.0. Depends on #111.

If some resource or a tree of resources is removed from a Bundle (via an update of that Bundle), Smith should ensure removed objects are deleted.

Correct deletion order should be achieved and enforced by Owner References and finalizers. I.e. it should be enough to issue synchronous delete commands for all removed objects in any order and let GC+finalizers do the job.

See the list of blockers in #55.

The implementation should be able to recover if Smith fails in the middle of the deletion process - all orphaned objects (not in a Bundle and not marked for deletion and has BundleName label) can be found by inspecting their bundle.smith.atlassian.com/BundleName label, which has the name of the Bundle that should contain the object.

Can we somehow use k8s.io/apimachinery/pkg/util/strategicpatch or whatever is used in kubectl apply instead of our custom defaulting+cleanup code?

Currently Smith adopts an object (i.e. considers it belonging to a particular bundle) if an object with same name is defined in the bundle within the same namespace. It does not take into account whether that object has a "controller owner reference" to some other bundle or not. Behaviour should be as described in ControllerRef proposal. Only adopt (consider part of the bundle) an object if it does not have any "controller owner references". Adoption means adding a "controller owner reference" pointing to the owner bundle. If an object that is mentioned in a bundle A has a controller (another bundle B) that owns it, then that bundle A should be marked with ERROR condition and IN PROGRESS and READY conditions should be set to false.

To add to the above - a decision needs to be made whether to implement adoption at all or not. What if the user wants to delete the bundle but keep some objects? Remove controller owner reference and "detach" an object from the bundle - should it adopt the object back or not? Or just point the controller owner reference to the object itself, delete the bundle and remove the controller reference?

In #83 we added a temporary workaround for spec-vs-actual comparison. Now integration tests are failing. Obviously this need to be fixed asap.
Perhaps should address the issue properly. The only fix that comes to mind is applying defaults locally to the spec and then comparing with the "actual" object.

Many resource kinds have array of conditions in their status. Smith should support telling if resource is in ready state by looking at some condition, defined via annotation on TPR (for TPRs). This is similar to existing annotations.
One way to solve this is to add json path support to existing annotations so that it is possible to filter though conditions to check the specific one.

TPRs are deprecated and are going to be removed in Kubernetes 1.8.

The current assumption is that GC takes care of the ordering but it is not right. We need to delete the resources that have been removed from the Bundle in reverse creation order. Otherwise Service Catalog is not happy:

    - lastTransitionTime: 2018-01-31T02:22:51Z
      message: Delete instance failed. Delete instance blocked by existing ServiceBindings
        associated with this instance.  All credentials must be removed first
      reason: DeprovisionBlockedByExistingCredentials
      status: "False"
      type: Ready

#194 and #195 can be useful, but expose the secret in the service instance object. Surprisingly, this is usually ok - most binding outputs are not truly 'secret'. But obviously not always, and the only way to not expose the secret is to use parametersFrom.

To make 'secret references' truly useful, then, it would be good to automatically generate secrets to allow parameter passing. Possible approaches:

1. automatically detect when a 'secret reference' is inside a parameter, and convert that parameter to a secret

• has the advantage that it's impossible to inadvertently leak secrets, but doesn't expose the 'non-secret' parts of the binding outputs (the majority) AND means that one can't easily use a generic templating language (imagining future changes...), since you need to apply the template yourself to know where the secrets are in the JSON structure. It also means that if you have a single non-secret 'secret' in a big parameter block, the whole lot will be converted to a secret

2. allow one to mark a particular parameter as 'secret' (e.g. interpreting an '!' at the end of the parameter as indicating it should be converted to a secret)

• opposite pros/cons cf (1)

Both of these approaches mean that it's possible to easily pass secrets into the midst of a complex JSON object, which Service Catalog makes difficult.

Downsides to this:

• we're building in more Service Catalog knowledge into Smith (but that ship has sailed with secret updates?)
• in reality, I think this belongs in Service Catalog (ability to have secret refs inside parameter block vs. parametersFrom)
• although this allows passing secrets outside the top-level, one must make the whole top-level parameter secret if any sub property is secret, which means that large blocks of the JSON can be 'infected' by a secret (even worse in (1))

Nevertheless, I think this is worth doing, as it makes a bunch of things easier (no need for complex secret generating plugins) and gets us much closer to schema based validation (no 'magical' set of variables coming from parametersFrom).

Issues with current approach where Smith ensures Bundle CRD exists before it starts doing anything:

• Smith needs extra permissions to create/update CRDs. This may not be desirable.
• This complicates code a bit. It is a bit ugly to attach CRD listener after the CRD informer has started.

As part of this issue documentation should be added, explaining what needs to be done to provision CRD for Bundle.

Otherwise objects might be deleted too early.

Currently Smith watches all namespaces but it may not always be desirable/permitted. Make it possible to configure a list of namespaces to watch. Perhaps make Smith discover namespaces to watch dynamically via labels/annotations.

minikube 1.9.1 introduced a way to run kube cluster without virtualization. See if it can be used with Travis to run integration tests.
https://github.com/kubernetes/minikube/blob/v0.19.1/CHANGELOG.md

Smith currently checks for conflicts in its own resource names, but allows the user to set the object names in the bundle. Among other things, this means:

• two resources in the same bundle can refer to the same underlying objects, leading to Smith fighting itself if their specs are different
• users can rename the resource, but not the metadata name, which may lead to delete/recreate fight over resource
• resource creation can fail late due to ownerReference clash with object elsewhere

Use the standard work queue pattern described in https://github.com/kubernetes/community/blob/master/contributors/devel/controllers.md

Current approach is to run a goroutine per bundle while there is work to do for that bundle (i.e. worker goroutines only block to do back off sleep for retries). This approach was chosen to maximize concurrency but avoid running a goroutine per bundle all the time. In practice non-blocking mode of operation means there is not much value in having a quite complex implementation to achieve perfect concurrency. Fixed number of workers will do the job too. Actually, a single queue poller can spawn a goroutine per work item to do the work because work queue makes sure that at most one worker is handling work for each object. But number of running goroutines should be capped anyway.

==================
WARNING: DATA RACE
Write at 0x00c4205d00cc by main goroutine:
  internal/race.Write()
      /private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/bazel-sandbox/1936151865359060414/execroot/__main__/bazel-out/host/bin/external/go_stdlib_darwin_amd64_cgo_race/src/internal/race/race.go:41 +0x38
  sync.(*WaitGroup).Wait()
      /private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/bazel-sandbox/1936151865359060414/execroot/__main__/bazel-out/host/bin/external/go_stdlib_darwin_amd64_cgo_race/src/sync/waitgroup.go:129 +0xf4
  github.com/ash2k/stager/wait.(*Group).Wait()
      vendor/github.com/ash2k/stager/wait/group.go:14 +0x3a
  github.com/atlassian/smith/pkg/controller.(*BundleController).Run()
      pkg/controller/controller.go:104 +0x51f
  github.com/atlassian/smith/cmd/smith/app.(*App).Run()
      cmd/smith/app/app.go:148 +0x169f
  main.runWithContext()
      cmd/smith/main.go:81 +0x417
  main.run()
      cmd/smith/main.go:36 +0xd9
  main.main()
      cmd/smith/main.go:26 +0x33

Previous read at 0x00c4205d00cc by goroutine 115:
  internal/race.Read()
      /private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/bazel-sandbox/1936151865359060414/execroot/__main__/bazel-out/host/bin/external/go_stdlib_darwin_amd64_cgo_race/src/internal/race/race.go:37 +0x38
  sync.(*WaitGroup).Add()
      /private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/bazel-sandbox/1936151865359060414/execroot/__main__/bazel-out/host/bin/external/go_stdlib_darwin_amd64_cgo_race/src/sync/waitgroup.go:71 +0x16f
  github.com/ash2k/stager/wait.(*Group).Start()
      vendor/github.com/ash2k/stager/wait/group.go:35 +0x43
  github.com/ash2k/stager/wait.(*Group).StartWithChannel()
      vendor/github.com/ash2k/stager/wait/group.go:20 +0xd0
  github.com/atlassian/smith/pkg/controller.(*crdEventHandler).ensureWatch()
      pkg/controller/controller_crd_event_handler.go:111 +0xaaf
  github.com/atlassian/smith/pkg/controller.(*crdEventHandler).OnAdd()
      pkg/controller/controller_crd_event_handler.go:36 +0x57
  k8s.io/client-go/tools/cache.(*processorListener).run()
      vendor/k8s.io/client-go/tools/cache/shared_informer.go:545 +0x3b2
  k8s.io/client-go/tools/cache.(*processorListener).(k8s.io/client-go/tools/cache.run)-fm()
      vendor/k8s.io/client-go/tools/cache/shared_informer.go:381 +0x41
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
      vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x5c

Goroutine 115 (running) created at:
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start()
      vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x6f
  k8s.io/client-go/tools/cache.(*sharedProcessor).addAndStartListener()
      vendor/k8s.io/client-go/tools/cache/shared_informer.go:381 +0x2dd
  k8s.io/client-go/tools/cache.(*sharedIndexInformer).AddEventHandlerWithResyncPeriod()
      vendor/k8s.io/client-go/tools/cache/shared_informer.go:331 +0x281
  k8s.io/client-go/tools/cache.(*sharedIndexInformer).AddEventHandler()
      vendor/k8s.io/client-go/tools/cache/shared_informer.go:267 +0x67
  github.com/atlassian/smith/pkg/controller.(*BundleController).Run()
      pkg/controller/controller.go:89 +0x2e3
  github.com/atlassian/smith/cmd/smith/app.(*App).Run()
      cmd/smith/app/app.go:148 +0x169f
  main.runWithContext()
      cmd/smith/main.go:81 +0x417
  main.run()
      cmd/smith/main.go:36 +0xd9
  main.main()
      cmd/smith/main.go:26 +0x33
==================
Found 1 data race(s)

WaitGroup's Wait() and first Add() method calls should be ordered but they are not. Should fix that.

Blocked by Service Catalog using 1.7 apimachinery.

To improve visibility/reporting and aid troubleshooting Smith should have status of each resource as part Bundle status.

Decide what events and when to emit.

If we support #194 and #195, and we want to do #186 and #187, then one option would be to stop supporting plugins as a separate top-level 'thing' and instead allow references to call user provided functions.

e.g. rather than doing:

  - name: compute--secretenvvar
    dependsOn:
    - compute--basic--binding
    - compute--mykinesis--binding

    spec:
      plugin:
        name: secretenvvar
        objectName: compute--secretenvvar
        spec:
          outputJsonKey: secretEnvVars
          outputSecretKey: ec2ComputeEnvVars

One would instead do:

  - name: compute--secretenvvar
    dependsOn:
    - compute--basic--binding
    - compute--mykinesis--binding
    spec:
      metadata:
        name: b
      apiVersion: v1
      kind: Secret
      stringData:
        ec2ComputeEnvVars: {{buildSecretObject(*#*)}}

i.e. plugins simply provide some set of simple functions which you can pass parameters to. Here # is shorthand for all outputs of all dependencies, but for most plugins you're likely to want more specific values (for either deps or outputs).

Why is this useful?

• it's considerably clearer what's going on in terms of kubernetes objects than using plugins as they stand
• we go back to the state of Smith only supporting 'objects' in the spec (no additional nesting/confusion)
• we're closer to supporting schema based validation (if using functions in ServiceInstance objects)

Yes, we do make the references more complex, and yes, this is looking more like what RPS does (in terms of jmespath functions and more complex references)... we could just use jmespath here ;) Just saying.

When doing make minikube-run:

INFO: Running command line: bazel-bin/cmd/smith/smith-race -disable-service-catalog '-pprof-address=:6060'
2017/10/13 16:26:35 Waiting for apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition informer to sync
2017/10/13 16:26:35 Waiting for CustomResourceDefinition bundles.smith.atlassian.com it to become established
2017/10/13 16:26:35 Waiting for informers to sync
==================
WARNING: DATA RACE
Read at 0x00c42035c930 by goroutine 52:
  runtime.mapaccess1()
      /private/var/tmp/_bazel_mmazurskiy/5b7f41401ec9baa31e49cd17aa7eef5f/external/go1_9_darwin_amd64/src/runtime/hashmap.go:341 +0x0

Previous write at 0x00c42035c930 by goroutine 50:
  runtime.mapassign()
      /private/var/tmp/_bazel_mmazurskiy/5b7f41401ec9baa31e49cd17aa7eef5f/external/go1_9_darwin_amd64/src/runtime/hashmap.go:521 +0x0

Goroutine 52 (running) created at:
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start()
      vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x61

Goroutine 50 (running) created at:
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start()
      vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x61
==================
==================
WARNING: DATA RACE
Read at 0x00c42035cb70 by goroutine 52:
  runtime.mapiterinit()
      /private/var/tmp/_bazel_mmazurskiy/5b7f41401ec9baa31e49cd17aa7eef5f/external/go1_9_darwin_amd64/src/runtime/hashmap.go:709 +0x0

Previous write at 0x00c42035cb70 by goroutine 50:
  runtime.mapassign()
      /private/var/tmp/_bazel_mmazurskiy/5b7f41401ec9baa31e49cd17aa7eef5f/external/go1_9_darwin_amd64/src/runtime/hashmap.go:521 +0x0

Goroutine 52 (running) created at:
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start()
      vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x61

Goroutine 50 (running) created at:
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start()
      vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x61
==================
==================
WARNING: DATA RACE
Read at 0x00c42035d6b0 by goroutine 52:
  runtime.mapaccess1()
      /private/var/tmp/_bazel_mmazurskiy/5b7f41401ec9baa31e49cd17aa7eef5f/external/go1_9_darwin_amd64/src/runtime/hashmap.go:341 +0x0

Previous write at 0x00c42035d6b0 by goroutine 50:
  runtime.mapassign()
      /private/var/tmp/_bazel_mmazurskiy/5b7f41401ec9baa31e49cd17aa7eef5f/external/go1_9_darwin_amd64/src/runtime/hashmap.go:521 +0x0

Goroutine 52 (running) created at:
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start()
      vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x61

Goroutine 50 (running) created at:
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start()
      vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x61
==================
2017/10/13 16:26:36 Starting Bundle controller
2017/10/13 16:26:36 [CRDEH] Configuring watch for CRD sleepers.crd.atlassian.com

Attached pprof. Looks like these are the goroutines (from a different run, numbers were matching, I just lost the original logs ^):

READ:
goroutine 54 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc420360000, 0x268c6c0, 0xc4204e3470, 0xc4205a1b78, 0xc420073aa0, 0xc420154060, 0x0, 0x0)
	vendor/k8s.io/client-go/tools/cache/reflector.go:366 +0x288
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc420360000, 0xc420154060, 0x0, 0x0)
	vendor/k8s.io/client-go/tools/cache/reflector.go:332 +0xed0
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	vendor/k8s.io/client-go/tools/cache/reflector.go:204 +0x33
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc420033718)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4205a1f18, 0x3b9aca00, 0x0, 0x1b5f801, 0xc420154060)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
k8s.io/apimachinery/pkg/util/wait.Until(0xc420033718, 0x3b9aca00, 0xc420154060)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc420360000, 0xc420154060)
	vendor/k8s.io/client-go/tools/cache/reflector.go:203 +0x164
k8s.io/client-go/tools/cache.(*Reflector).Run-fm(0xc420154060)
	vendor/k8s.io/client-go/tools/cache/controller.go:122 +0x34
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 +0x31
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc420364030, 0xc42000dc00)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62


WRITE:
goroutine 49 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc420144550)
	vendor/k8s.io/client-go/tools/cache/shared_informer.go:511 +0x1d8
k8s.io/client-go/tools/cache.(*processorListener).(k8s.io/client-go/tools/cache.pop)-fm()
	vendor/k8s.io/client-go/tools/cache/shared_informer.go:382 +0x2a
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc42015a758, 0xc4203202e0)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

Running make test on my machine produced:

There were tests whose specified size is too big. 
Use the --test_verbose_timeout_warnings command line option to see which ones these are.

This diff adds in the flag to the make file

diff --git a/Makefile b/Makefile
index b923821..0a2c8a0 100644
--- a/Makefile
+++ b/Makefile
@@ -149,6 +149,7 @@ test-ci:
 	bazel test \
 		--test_env=KUBE_PATCH_CONVERSION_DETECTOR=true \
 		--test_env=KUBE_CACHE_MUTATION_DETECTOR=true \
+		--test_verbose_timeout_warnings \
 		-- //... -//cmd/... -//vendor/...
 
 .PHONY: check

This is the resulting output

//examples/sleeper/pkg/apis/sleeper/v1:go_default_test          (cached) PASSED in 1.5s
  WARNING: //examples/sleeper/pkg/apis/sleeper/v1:go_default_test: Test execution time (1.5s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//pkg/apis/smith/v1:go_default_test                             (cached) PASSED in 1.0s
  WARNING: //pkg/apis/smith/v1:go_default_test: Test execution time (1.0s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//pkg/client/smart:go_default_test                              (cached) PASSED in 1.1s
  WARNING: //pkg/client/smart:go_default_test: Test execution time (1.1s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//pkg/controller:go_default_test                                (cached) PASSED in 1.6s
  WARNING: //pkg/controller:go_default_test: Test execution time (1.6s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//pkg/resources:go_default_test                                 (cached) PASSED in 1.5s
  WARNING: //pkg/resources:go_default_test: Test execution time (1.5s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//pkg/speccheck:go_default_test                                 (cached) PASSED in 1.5s
  WARNING: //pkg/speccheck:go_default_test: Test execution time (1.5s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//pkg/util/graph:go_default_test                                (cached) PASSED in 1.0s
  WARNING: //pkg/util/graph:go_default_test: Test execution time (1.0s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".

Perhaps a REST endpoint that accepts POST with a Bundle object and returns detailed explanation of what is going to happen if this object is stored:

• Which objects are going to be created
• Which objects are going to be updated
  • A diff for each one?
• Which object are going to be deleted
• What status will the Bundle have
  • Not a final status, but the one after the very first pass. This is to catch issues with the Bundle object itself.
• What else?