currently solo kit only generates code for resources for the package in which the solo-kit.json is found.
it would be nice if it will also generate the client code for resources in sub-packages.

related to #44

We implicitly depend on protoc 3.6.1 for generation. If you run protoc 3.7.x, you'll get a substantial generated diff. We should probably validate this version before running solo kit generation.

right now we need the following import somewhere:

	_ "k8s.io/client-go/plugin/pkg/client/auth/gcp"

which is accessible both to tests and main.go for any kube-based project (that might run on GCP).

currently it's added to test packages by solo-kit; we probably want to put that in a centralized place where it's easy to access from test code as well as binaries

In order to watch kube cofig maps from event loops requires a solo-kit resource for configmaps.
Pod and namespace already exist, and now configmaps.

generate new event loops WithBackoff for backoff syncers

this results in a closing a closed channel panic when the cache is cancelled.

Our protos are written in snake case, but the yaml is persisted in Kube as camel case. Because the docs are written based on the protos, they are persisted as snake case. This makes it confusing to read the docs and try to update kube config via yaml.

test keeps failing

this would make solo-kit able to support a generic go struct that implements the Resource interface to utilize resource clients and event loops.

create the ability to aggregate watches for resources into a single stream of lists on a single Watch function

now that configmaps are being actively written/read by solo-kit projects, need to implement the same caching logic for the kube configmap client as we do with kube crd client

After reinstalling the the protoc-gen-solo-kit plugin I was not able to generate the expected code for the vcs project. The generate.sh script would exit with a 0 code, but the files had not been generated. After doing some investigation I determined the cause to be the content of the project.json file. The one in the vcs project had

{
  "name": "gloo",
  "version": "v1"
}

instead of the now required full package name:

{
  "name": "gloo.solo.io",
  "version": "v1"
}

We should update the code generation plugin in order to exit with an error if incorrect metadata was supplied.

This is 10 seconds for an in-memory resource client test

Step #3: • [SLOW TEST:10.198 seconds]
Step #3: EndpointClient
Step #3: /workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/endpoint_client_test.go:18
Step #3:   resource client backed by memory-based
Step #3:   /workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/endpoint_client_test.go:31
Step #3:     CRUDs Endpoints
Step #3:     /workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/endpoint_client_test.go:45

for multicluster to be supported, we need a new type of SnapshotEmitter that simply aggregates Lists from different watch channels into a single list of that resource type. the emitter will not know which namespaces (including the NamespaceAll) the resource lists originated from.

rather than creating a V2Snapshot i plan to remove the ByNamespaces abstraction from solo-kit. this will cause an easily resolvable compile error in most places where resource snasphots are referenced.

all calls to <TypeByNamespace>.List() will become redundant, as here: https://github.com/solo-io/solo-kit/blob/master/test/mocks/v1/testing_snapshot.sk.go#L46

func (s TestingSnapshot) hashMocks() uint64 {
	return hashutils.HashAll(s.Mocks.List().AsInterfaces()...)
}

will change to

func (s TestingSnapshot) hashMocks() uint64 {
	return hashutils.HashAll(s.Mocks.AsInterfaces()...)
}

this warning is very verbose and appears all over the logs of solo kit projects at times

what it indicates is that storage events from kubernetes are being dropped (as solo-kit only buffers up to 10 events at a time)

not super important so long as the last event is captured, i.e. we are eventually consistent

we should convert this warning to a metric as it is quite verbose

1. currently, resource groups must be specified on the proto messages themselves (in the comments).

• this is annoying when you need to add resources from another project (like discovery upstreams) to your event loop.
• what we probably want is a reference to all the resource groups for the project in the solo-kit.json (ProjectConfig)

2. currently we hueristically detect proto messages to be resources based on the presence of the metadata field which is a core.solo.io.Metadata. this can cause weird behavior if a proto message contains a metadata but is not intended to be a resource

• can be fixed by improving the semantics for our magic comments as well as the way other things get included in solo kit can be refined

Very minor suggestion:

export errors strings like this:

const UpstreamListErrorTag = "list did not find upstream"

as was done in:
https://github.com/solo-io/gloo/pull/651/files#diff-8eec5c179347890490d95491acb44602R3

what do you think @ilackarms ?
seems like it could be useful

I will implement if you think it's worth having

ran into this issue in glooshot

summary:

• reconciler on API-A watches A and updates resources of API-B
• A gets created, reconciler creates B
• reconciler process crashes
• All As get deleted by some means
• reconciler process is restarted
• Stale Bs do not get cleaned up until something triggers another snapshot of A

Fix:

• add ability for event loop to force an emitter to emit

@yuval-k

Currently solo-kit can either watch all, or specific namespaces on an event loop level.
However, it would be very useful in many situations to be able to watch specific namespaces for specific resources.

When go-utils was created the decision was made not to immediately removed duplicated code from solo-kit in an attempt to not break existing code. However, enough time has passed now, and accidentally using the wrong version is a very real possibility.

Squash history once #2 merges, then pull in new revision of solo-kit to dependent repos

It seems that solo kit generates code for other projects, if the protos of those other projects are included.
i.e. solokit invocation on supergloo will generate code for gloo as well.

need to remove a few stray namespace = clients.DefaultNamespaceIfEmpty(namespace)

We should have environment variable like SKIP_RESOURCE_CLIENT_TESTS=true, to skip running expensive tests that aren't really necessary to verify the generated code (because solo-kit should run those instead)

Right now we have a message-level option (skip_docs_gen). If a proto file contains a message with that option set to true, the doc generator will skip that file. This approach is convoluted and not self-explanatory. It makes more sense to have this behavior determined by a file-level option.

Add a mode to watch all namespaces

Should also be able to add/remove namespaces from the watch when they are created/deleted

This will involve creating a way to create a new event loop and drain the old event loop

Several gotchas w.r.t. magic comments and generation:

• If the top-level message definition is missing Status and Metadata, generate won't work
• If the magic comment resource group name doesn't include the full package name at the end, the event loop won't get generated

currently the shared cache only allows one registration per <namespace, resourceType> combination

this means that if different clients of the same resourceType re-register with the same cache, they will error with 'errors.Errorf("Shared cache already contains informer for resource [%v] and namespace [%v]", resourceType, ns)'

we should have a way to either:
1 - ensure that caches are not shared between resource clients that independently Register
2 - ensure caches can support multiple registrations

currently this isn't a problem outside of tests, as, in most projects (that i'm aware) clients are always initialized with a fresh kube cache in the root SetupSyncer.

@marcogschmidt

add a new type of emitter for use with multicluster

accepts a single aggregated resourcewatch and produces snapshots from it

In #83 it was revealed that the GKE test cluster was accumulating mock resources and after awhile the all namespaces test was failing.

This commit shouldn't have been necessary: 7487ebc

for supergloo, need a way to read/write a prometheus configmap yaml as a map[string]string

this works for plainsecrets, but not (yet) for configmaps

in order to include other projects go resources in event loops requires the ability to import other projects solo-kit.json

currently solo-kit performs name validation on the assumption that kubernetes requires names to be a DNS_LABEL string. turns out, after experimenting/researching, DNS_SUBDOMAIN is the required format for names in k8s, which allows dots.

source: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/identifiers.md#general-design

mockgen mocks are incredibly useful for unit testing, so in an effort to make them more accessbile we will automatically generate them for all solo-kit resources.

right now if the reconciler doesn't receive a transition function, it always performs an update on existing resources, even if they are equal (causing unnecessary writes)

we should prevent re-writing the same resource from happening by leveraging the Hash() function generated for sk resources

Currently to implement kubernetes resources as solo-kit resources, boilerplate resource client interface code has to be created at each location. Create some generic functions which can be used for any resource type

right now the KubeCoreCache implicitly starts watches on:

• namespaces
• pods
• configmaps
• secrets
• services

solo-kit dependent projects may only depend on one or some of these, and we don't want to force RBAC configuration as well as the overhead of watching/caching additional resources

the kuebcorecache should accept an opts in its constructor which allows en(dis)abling the informers/caches wrapped by the KubeCoreCache

create a base reconciler that supports reconciling resources for multiple clusters

crd client can panic when a List or Watch is called but the crd has not been registered yet:

https://console.cloud.google.com/cloud-build/builds/5328909b-bc3c-4e04-aea4-cc334802db58?project=solo-public

in order to protect against this, we should verify during client construction that the crd has been registered