vmagamedov / pi Goto Github PK

https://github.com/Yelp/dumb-init

Basically do this:

$ docker run --rm image:version sleep 3600
$ docker exec container-id cmd *args

Instead of:

$ docker run --rm image:version cmd *args

Benefits:

$ time docker exec 952fd2f07aaa true

real	0m0.082s
user	0m0.012s
sys	0m0.009s

Instead of:

$ time docker run --rm alpine:3.6 true

real	0m0.966s
user	0m0.013s
sys	0m0.008s

===> More than 10x faster

Something like tox-travis, but with containerized environments

Prototype:

- !Image
  name: test-py{3.5,3.6}
  from: python:{3.5,3.6}-alpine
  repository: reg.local/project.{{name}}
  tasks:
  - run: pip3 install {{packages|join(" ")}}
    packages:
    - pytest==3.0.3
    - flake8==3.0.4

It will help modify files in the container for "print"-style debugging and other types of hacks

I'll start with brief description of vagga (I know @vmagamedov already knows that but somebody visiting a page might not):

Vagga does very similar thing to what pi is supposed to do: it allows to build containers and run commands inside. It keeps list of commands in yaml config, including all advanced things like mounting volumes, building containers and so on, including the possibility to add a description for the command :) (which make lacks for unknown reason)

The key difference is: pi uses docker as a containerization, but vagga doesn't, it implements containers using raw linux system calls. And that largely limits feature set of both utilities. Let me explain.

Why Vagga Doesn't Use Docker?

There are two major reasons for this, that are relevant to this discussion:

1. We want security: vagga test in a gitlab-ci has access only to project dir (+ containers and cache), and doesn't run with elevated privileges (as a user that runs gitlab-ci-runner). While any tool around docker can docker run --privileged. [*]
2. We want IDE accessibility: python libraries of the container are in .vagga/containername/usr/lib/python3.5/site-packages. So you can just point IDE there. For docker you need some hacks to achieve this.

[*] To run docker securely on a shared server like gitlab-ci you need to set up a separate virtual machine for every project and use docker socket of that machine in docker run. And yes we need security in CI system because we deploy from it.

How You Can Use Vagga on Mac OS?

Vagga runs in docker on linux perfectly fine. So should run in docker for mac too. We just need a small script that runs docker run with some preconfigured settings. Here is a vagga issue for the topic, that has no feedback so far. We just need to figure out:

1. If we can keep .vagga folder on host system and how expensive it is (to allow IDEs accessing files inside the container) [*]
2. If (1) doesn't work figure out two more things:
   a) add something like --volume=/work/.vagga:/vagga
   b) a way for vagga to copy /usr/lib/python3.5/site-packages outside of the container on each build
3. Figure out how ports are published. Probably adding exposed ports to the metadata in vagga.yaml so this script can look at them (you only need ports you want to access from your host system, e.g. a browser, interconnection between processes/containers will work)

[*] Files in the project dir will definitely be accessible from a host filesystem. We only question files of the container filesystem, like system libraries, python packages...

Answers on few obvious questions:

1. Will vagga in docker run slower than running by docker? No, we just adding one layer of containers. Containers are cheap in linux. Like if you mount another hard-drive will it slow the system? No.
2. Will vagga in docker be less or more secure that docker? Well, it isn't a useful question: we don't care about security of our own code on our own laptops :)
3. Will there be any cross-platform issues for vagga? Previously vagga was run on linux only, so it was easier to make vagga.yaml. I think they will be minimal if at all (mostly dealing with filesystems shared with a host).
4. Does it requires too much OS X specific configuration? I think it does not, probably (a) exposed ports and (b) exposed part of the filesystem (and maybe none of them). I think they might be useful for vagga on linux too: (a) we may isolate the network in vagga and only allow those ports (note we already have pass-tcp-socket option), and (b) might be cool when we make layers and deduplication (i.e. to disable optimizations for certain folders).

What Are The Benefits?

Well, vagga is almost 2 years old (the first commit of bare readme file was at 28th June). So it already got some things. Here is what you are going to loose:

1. You need to define your own configuration format, reiterate, keep backward compatibility
2. You need to support documentation of this thing
3. You need all kinds of tutorials for all kinds of usage
4. You loose security on CI systems as described above
5. Vagga's tools for testing network resilience
6. And higher level build tools that vagga supports. If you think that relying on Dockerfile's makes your life easier, here is a link that it's not
7. And the most important thing, you introduce a community fragmentation

Isn't All This Insulting?

Well, I hope not. It's open source. We try to contribute and collaborate as much as possible to not to waste too much resources on duplicate efforts. Sometimes forks and rewrites are justified sometimes not. It happens that owners are just uninformed. And occasionally, projects join the effort again. Hopefully, we can make another successful case for that.

Add ability to define Image in the Command definition, when this image will not be reused by another command, to place them together.

Prototype:

- !Command
  name: version
  run: python3 --version
  image: !Image
    name: version
    from: !DockerImage alpine:3.4
    repository: reg.local/project.version
    tasks:
    - run: apk add --no-cache python3