simple local test-net using docker compose about test-infra HOT 7 CLOSED

Why docker compose over an alternative automated environment setup?

My preference would be to use a Kubernetes based environment with either minikube, k3s, or Kind for a local cluster. This has the benefit of more closely resembling a production environment. I almost have a working setup on Minikube. It should be done by EOD Monday.

from test-infra.

Why docker compose over an alternative automated environment setup?

Mostly because it's probably way easier to spin up locally and does not require any additional dependencies for tools (at least on mac os docker compose is included in the docker bundle IIRC). Docker compose feels the most docker native and syntax is extremely simple and straight forward. Another reason is that we have a bunch of scripts and dockercompose files (in various cosmos repos) that could be used as the basis for this.

But if you have sth with minikube almost working that is also perfectly fine with me 👌🏼 Kubernetes often felt like a bit of overkill to me, especially for local testing. The only project I really worked with kubernetes a bit was keytransparency and there we used kubernetes for cloud deployment but dockercompose for local testing. I thought that was still the most common way today.

from test-infra.

In general Kubernetes can be heavy weight for local testing. But if what we're testing is multiple containers networking between each other I think there are some advantages to the more robust configuration possibilities.

Regarding the effort to spin up locally, minikube is a 68MB self contained go binary. Once it's spun up it is persistent until shutdown, so the startup time is amortized over all the necessary tests or deployments.

from test-infra.

Makes sense. I think the only reservations I still have for kubernetes is that it is cloud-native and doesn't necessarily have the best support for local development (and that was the main use-case my request was coming from for now).

Maybe these are non-issues:

• local deployment using kubernetes does not seem to have a local container registry? (annoying for local development testing as minikube constantly pushs/pulls images?)
• Cognitive overhead: most devs will know docker and docker-compose, while kubernetes is mostly only used in cloud settings (maybe some scripts can help with that).

from test-infra.

I am against using kubes for testing infra generally as no one besides @jbowen93 can maintain it. There are > 3 people on the team that have worked with docker before in a more serious capacity, and docker is much easier to onboard to than kubes.

If we decide to use kubes at all, it should be much later down the line when we have a more mature infra that demands something more serious like kubes to emulate a prod env, in which case we'll dedicate some engineering effort to it. Regardless, there are easier approaches to automation we can use like terraform or ansible.

I really want the the infra to be accessible to most devs on the team and kubes isn't quick or easy to use.

my 2c.

from test-infra.

I forgot that this issue existed. It's a duplicate of: celestiaorg/devops-test#28 which I'll close in a little bit after writing some more documentation.

from test-infra.

celestiaorg/devops-test#28 has been closed. So this issue should be closed.

from test-infra.