are you going to use kubernetes for orchestration? about abstruse HOT 4 CLOSED

hi @veeramarni.
thanks for --privileged notice, we will remove it right away. the reason to use this flag was about Google Chrome not wanted to run in xvfb without that flag. since now its possible to run chrome headless this can be easily removed.

about orchestration, yes, we are using RxJS to handle concurrent jobs. we have a plan to do a 1.0 release (at least beta) this week or at least in the beginning of next.
about second release that we have in plan where distribution jobs will be possible across servers, kubernetes seems like a perfect tool for this (I think).

if you have any other suggestions please feel free to raise it.

from abstruse.

It might be interesting how https://www.portainer.io/ handles integration for Docker Swarm and Azure Container Instances.
Azure Container Instances can be useful to provide infinite scalable cloud-native build environment ( https://docs.microsoft.com/en-us/azure/container-instances/container-instances-jenkins ). There is also Amazon Fargate (see https://serverless.com/blog/serverless-application-for-long-running-process-fargate-lambda/ ). In the future I would be interested to integrate my container launch environment (based on my own API), which will be publicly rented.

I mention to think about worker environments, not just Docker / Kubernetes.

from abstruse.

Hi Adam!
Distributed processing with workers is something we talk and think about alot. We think it is essential to have this feature. With our limited resources we scheduled this for 3.0.0 release now and as you can see I work hard to release 2.0.0 in the last days.
I investigated about this already and I think best way to communicate between main instance with workers is via gRPC but not sure if we can generalize or standarize this to be compatible with all mentioned services. Seems like you investigated about that also, can you give us some input here? Would be highly appreciated.

from abstruse.

GitLab CI has concept of runner for that: https://docs.gitlab.com/runner/
There is cool plugin to auto-scalable workers on that: https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/index.html
Jenkins has agent concept: https://jenkins.io/doc/book/pipeline/syntax/
CircleCI has concept executor type: https://circleci.com/docs/2.0/executor-types/#section=configuration

1. What kind of tasks in your opinion should such an worker perform? Do you can list them anyway?
2. Where should this gRPC agent work?
3. Do we want to support, for example, our own Docker images (this prevents us from placing anything inside the container)? I support your product, but I do not use it yet.
4. Do we want to support cache?
5. Do we want to support many different type of workers (AWS Fargate & Azure Cointainer Instances togerther) at the same time? Portainer handles a multiple endpoints at the same time.
6. How to handle group of containers (aka services container)?
7. Should each agent support the same possibilities? I think not, so we must support the validation that, for example, this agent does not support building using services or SSH connection to container.

Docker Engine provides REST endpoint for example:

• create container: https://docs.docker.com/engine/api/v1.37/#operation/ContainerCreate
• get logs of container: https://docs.docker.com/engine/api/v1.37/#operation/ContainerCreate
  There is also Docker Swarm which distribute jobs over docker multiple-hosts.

I think you should take awareness how works endpoint settings in portainer:

There is also documentation of Azure Container Instances good to read: https://docs.microsoft.com/en-us/rest/api/container-instances/

from abstruse.