Makefile 14.94% Go 80.55% Shell 4.51%

communitytools-image2docker-linux's Introduction

v2c - An image analysis and provisioning workflow

Documentation

Demo

This codebase ships with a demo of the proof of concept. The goal of the proof of contept was to demonstrate a workflow that shares the contents of a VMDK with a set of detective components which contribute material to a set of referenced image provisioners. Those provisioners transform the detective contributed materials and contribute Dockerfile segments. All of these contributions tar streamed via tar. The workflow finally stiches together contributed Dockerfile segments into a single Dockerfile and persists an expanded build context. The proof of concept does not perform final image assembly or actually use a real input image, detectives, or provisioners. Those components are crafted to demonstrate material contribution flow.

Start off by downloading this file:

https://s3-us-west-2.amazonaws.com/allingeek-public-transport/for-export-flat.vmdk

Then clone this repo

git clone https://github.com/docker/communitytools-image2docker-linux

Prepare the demo by running:

make prepare
make build
make builtin-prep

Run on OSX with:

sudo bin/v2c-darwin64 build -n PATH-TO_for-export-flat.vmdk

Run the demo on Linux with:

sudo bin/v2c-linux64 build -n PATH-TO_for-export-flat.vmdk

communitytools-image2docker-linux's People

Contributors

Stargazers

Watchers

communitytools-image2docker-linux's Issues

Cleanup crufty transport volume / container

If your previous run of i2d fails, it leaves the glassfish transport container and volume lying around which causes subsequent runs to fail. Running i2d should be idempotent and any old volumes / containers should be cleaned up on subsequent runs.

Support building on Windows

Current Makefile doesn't work on Windows so we need another way to compile it, or simply a set of Docker images and a compose file if that would work better.

why not have input/output volumes

Currently, it seems the way that one has to transfer data from the input VM image image to the output container image is for the detectives to tar up data and for then have it be extracted in the triggered provisioner.

The issue is that this only lets one easily copy data that is within a single tree, not something where one has to pick and choose files.

It further prevents easy modification of the artifact that is created.

For instance, take the demo case with Apache. If one wants to modify the Apache instance, one either has to extract the configuration tarball, modify it and recreate it, or add another layer just to handle that configuration change.

Instead of dealing with output tarballs, wouldn't it be easier to create output volumes where data can be copied to during the detective stage and leveraged during the provisioning stage (i.e. instead of an ADD <tarball> / it be ADD <directory that was volume> /

I this way, one could

more easily programatically decide what will be fetched
enable one's created artifact to be checked into a repo, modified as needed over time, and rebuilt on demand.

thoughts?

Fix imports in go files to refer to the correct project

The import lines in several of the go source files refers to "docker/v2c" which makes it difficult to build things correctly. We should use the canonical name for any imports.

Demo VM not containerizing completely - No operating system detected or provisioned

I ran the transformation on the vm provided by the project and got the following results:

ubuntu@worker-node1:~/transformation$ sudo ../communitytools-image2docker-linux/bin/v2c-linux64 build /vagrant/for-export-flat.vmdk 
Running image transformation.
Using existing unpacked image.
Creating container for v2c/centos-detective:v6.8
Creating container for v2c/init.tomcat-detective:1
Creating container for v2c/app.tomcat8.5.5-detective:1
Creating container for v2c/runit-detective:ubuntu-v16.10
Creating container for v2c/init.apache2-sysv.detective:2
Creating container for v2c/ubuntu-detective:v16.04.1
Creating container for v2c/ubuntu-detective:v16.04
Creating container for v2c/runit-detective:ubuntu-v16.04
Creating container for v2c/runit-detective:ubuntu-v16.04.1
Creating container for v2c/ubuntu-detective:v14.04.5
Creating container for v2c/ubuntu-detective:v16.10
Creating container for v2c/conf.mysql5-data.detective:1
Creating container for v2c/runit-detective:ubuntu-v14.04.5
Creating container for v2c/conf.apache2-etc.detective:1
Creating container for v2c/conf.apache2-var-www.detective:1
Creating container for v2c/app.apt-repl.detective:1
No results for v2c/init.tomcat-detective:1 code: 1
No results for v2c/app.tomcat8.5.5-detective:1 code: 2
No results for v2c/ubuntu-detective:v16.10 code: 2
No results for v2c/runit-detective:ubuntu-v16.10 code: 2
No results for v2c/ubuntu-detective:v16.04 code: 2
No results for v2c/centos-detective:v6.8 code: 2
No results for v2c/runit-detective:ubuntu-v16.04 code: 2
No results for v2c/conf.mysql5-data.detective:1 code: 1
No results for v2c/ubuntu-detective:v16.04.1 code: 2
No results for v2c/runit-detective:ubuntu-v16.04.1 code: 2
No results for v2c/runit-detective:ubuntu-v14.04.5 code: 2
No results for v2c/init.apache2-sysv.detective:2 code: 1
No results for v2c/conf.apache2-var-www.detective:1 code: 1
No results for v2c/conf.apache2-etc.detective:1 code: 1
No results for v2c/ubuntu-detective:v14.04.5 code: 2
Creating container for v2c/app.apt-repl.provisioner:1
No operating system detected or provisioned

This is what I get out as ouput:

ubuntu@worker-node1:~/transformation/application$ ls
7fbdc5da45a7f545bd13fb9677009be9551bf2a0eb7d92e33bd2f771c3dbd955.manifest  7fbdc5da45a7f545bd13fb9677009be9551bf2a0eb7d92e33bd2f771c3dbd955.tar

Opening up the Dockerfile in the tar file shows this in the Dockerfile:

ubuntu@worker-node1:~/transformation/application$ more Dockerfile 
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install -y
 \

make volume name should be dynamic

Currently volume name is statically named,v2c-transport. The current solution checks if a volume is already mounted via the docker client. if volume is mounted, it will use the existing volume - ignoring the new one specified in the command.

Error message also doesn't make this clear.

Support `go get` for installation

λ  go get github.com/docker/communitytools-image2docker-linux
# cd .; git clone https://github.com/docker/v2c c:\code\go\src\github.com\docker\v2c
Cloning into 'c:\code\go\src\github.com\docker\v2c'...
remote: Repository not found.
fatal: repository 'https://github.com/docker/v2c/' not found
package github.com/docker/v2c/api: exit status 128
package github.com/docker/v2c/system: cannot find package "github.com/docker/v2c/system" in any of:
        C:\Go\src\github.com\docker\v2c\system (from $GOROOT)
        c:\code\go\src\github.com\docker\v2c\system (from $GOPATH)
package github.com/docker/v2c/workflow: cannot find package "github.com/docker/v2c/workflow" in any of:
        C:\Go\src\github.com\docker\v2c\workflow (from $GOROOT)
        c:\code\go\src\github.com\docker\v2c\workflow (from $GOPATH)

golang 1.7 causes IsAlias error

Ubuntu 16.04 LTS

git clone https://github.com/docker/communitytools-image2docker-linux
cd communitytools-image2docker-linux
make prepare

Sending build context to Docker daemon  51.82MB
Step 1/2 : FROM golang:1.7
 ---> 225948c9a8b1
Step 2/2 : RUN go get -u github.com/rancher/trash &&     go get -u github.com/golang/lint/golint
 ---> Running in 1cc869ecca54
# golang.org/x/tools/go/internal/gcimporter
src/golang.org/x/tools/go/internal/gcimporter/bexport.go:212: obj.IsAlias undefined (type *types.TypeName has no field or method IsAlias)
The command '/bin/sh -c go get -u github.com/rancher/trash &&     go get -u github.com/golang/lint/golint' returned a non-zero code: 2

I fixed it with:
tooling/tooling.df: FROM golang:1.11

Add installation script

The installation script should:

Download and install binary
Add binary to PATH
Install Docker Images

Capture existing network information

such as /etc/hosts and other settings necessary to access other services.

CentOS 6.8 detected as Ubuntu 14.04

CentOS 6.8 is detected as ubuntu 14.04.
Kindly let me know the files which are needed for debugging.

Update detectives/provisioners to Alpine 3.6

Current detectives/provisioners are all using Alpine 3.4. We should change these to use a more up-to-date alpine.

Cleanup destination directory after failure

If the build fails, it does not cleanup the destination file path

SSL Certificates

Capture SSL Certificates

Apache Tomcat 8.5.5 not detected

Installed Apache tomcat 8.5.5 on Ubuntu 14.04 and then deployed a simple website, created a .vmdk file out of the image. Image2Dokcer was not able to detect tomcat and it created the docker file similar to the demo image, No entries for tomcat in docker file.

.vmdk Image creation:

Imported a .ova file from Oracle virtual box
Converted .ova file to .ovf using VMware OVF tool. A .vmdk file was generated after conversion process.
Used the newly created .vmdk file for Image2Docker.

Kindly let me know if this approach is correct.

Display a proper error message when guestfish can't read the VMDK file

When running image2docker against a clean VM created in VMware or VirtualBox, I get the following error:

Current Result

$ i2d build ../vmware-test.vmdk
Running image transformation.
Creating container for v2c/guestfish-export:1
The packager failed with code: 1

Expected Result

Successfully created Dockerfile

Repro Steps

Setup ubuntu 16.04 VM in VirtualBox or VMware Fusion
Shutdown VM
Run I2D against .vmdk file

Ubuntu 16.04.2 not working

The demo seems to be failing because Ubuntu 16.04.2 isn't yet supported. There's currently no provisioner/detective for 16.04.2 at all.

SSH Keys

Add flag to specify output directory

Currently the output directory is always the $(pwd). This should ideally be configurable using a flag

Doesn't work on Windows

This is with a vmdk that works on Mac

λ  .\..\v2c-windows64.exe build -t test C:\Users\friism\Downloads\for-export-flat.vmdk
Running image transformation.
Using existing unpacked image.
Creating container for v2c/centos-detective:v6.8
Creating container for v2c/app.apt-repl.detective:1
Creating container for v2c/init.tomcat-detective:1
Creating container for v2c/app.tomcat8.5.5-detective:1
Creating container for v2c/init.apache2-sysv.detective:2
Creating container for v2c/runit-detective:ubuntu-v16.10
Creating container for v2c/runit-detective:ubuntu-v16.04.1
Creating container for v2c/conf.mysql5-data.detective:1
Creating container for v2c/runit-detective:ubuntu-v14.04.5
Creating container for v2c/ubuntu-detective:v16.10
Creating container for v2c/ubuntu-detective:v14.04.5
Creating container for v2c/conf.apache2-etc.detective:1
Creating container for v2c/ubuntu-detective:v16.04.1
Creating container for v2c/conf.apache2-var-www.detective:1
Creating container for v2c/ubuntu-detective:v16.04
Creating container for v2c/runit-detective:ubuntu-v16.04
No results for v2c/runit-detective:ubuntu-v14.04.5 code: 2
No results for v2c/conf.apache2-etc.detective:1 code: 1
No results for v2c/centos-detective:v6.8 code: 2
No results for v2c/ubuntu-detective:v16.04 code: 2
No results for v2c/ubuntu-detective:v14.04.5 code: 2
No results for v2c/runit-detective:ubuntu-v16.04 code: 2
No results for v2c/init.apache2-sysv.detective:2 code: 1
No results for v2c/runit-detective:ubuntu-v16.10 code: 2
No results for v2c/ubuntu-detective:v16.10 code: 2
No results for v2c/ubuntu-detective:v16.04.1 code: 2
No results for v2c/conf.mysql5-data.detective:1 code: 1
No results for v2c/runit-detective:ubuntu-v16.04.1 code: 2
No results for v2c/app.tomcat8.5.5-detective:1 code: 2
No results for v2c/init.tomcat-detective:1 code: 1
No results for v2c/conf.apache2-var-www.detective:1 code: 1
Creating container for v2c/app.apt-repl.provisioner:1
No operating system detected or provisioned.

Failure when creating Dockerfile from Tomcat Ubuntu VM

I got the following error when running image2docker on a ubuntu vm with tomcat 8 installed:

ubuntu@worker-node1:~/transformation$ sudo ../communitytools-image2docker-linux/bin/v2c-linux64 build ../communitytools-image2docker-linux/tomcat.vmdk 
Running image transformation.
Using existing unpacked image.
Creating container for v2c/centos-detective:v6.8
Creating container for v2c/init.tomcat-detective:1
Creating container for v2c/app.tomcat8.5.5-detective:1
Creating container for v2c/runit-detective:ubuntu-v16.10
Creating container for v2c/init.apache2-sysv.detective:2
Creating container for v2c/runit-detective:ubuntu-v16.04.1
Creating container for v2c/runit-detective:ubuntu-v16.04
Creating container for v2c/runit-detective:ubuntu-v14.04.5
Creating container for v2c/conf.mysql5-data.detective:1
Creating container for v2c/conf.apache2-etc.detective:1
Creating container for v2c/conf.apache2-var-www.detective:1
Creating container for v2c/app.apt-repl.detective:1
Creating container for v2c/ubuntu-detective:v14.04.5
Creating container for v2c/ubuntu-detective:v16.10
Creating container for v2c/ubuntu-detective:v16.04.1
Creating container for v2c/ubuntu-detective:v16.04
No results for v2c/runit-detective:ubuntu-v16.04 code: 2
No results for v2c/ubuntu-detective:v16.04 code: 2
No results for v2c/conf.apache2-var-www.detective:1 code: 1
No results for v2c/ubuntu-detective:v16.04.1 code: 2
No results for v2c/ubuntu-detective:v14.04.5 code: 2
No results for v2c/init.tomcat-detective:1 code: 1
No results for v2c/ubuntu-detective:v16.10 code: 2
No results for v2c/conf.mysql5-data.detective:1 code: 1
No results for v2c/conf.apache2-etc.detective:1 code: 1
No results for v2c/centos-detective:v6.8 code: 2
No results for v2c/runit-detective:ubuntu-v14.04.5 code: 2
No results for v2c/app.tomcat8.5.5-detective:1 code: 2
No results for v2c/init.apache2-sysv.detective:2 code: 1
No results for v2c/runit-detective:ubuntu-v16.10 code: 2
No results for v2c/runit-detective:ubuntu-v16.04.1 code: 2
Creating container for v2c/app.apt-repl.provisioner:1
Unable to remove the transport volume due to: Error response from daemon: unable to remove volume: remove v2c-transport: volume is in use - [71a71ba3721bd778630290f41f0d565de5be03a234640d57bb4ac6a76e9de17c]

System and Environment variables

Capture and set System and Environment variables

User Configuration & Shadow file

We need to lift and shift user configuration and login data. You're taking a risk any time you're pulling critical files like /etc/shadow into an environment where access controls might be subverted. You're taking a further risk by committing those files into a layered image. You're also making it very difficult to change those passwords after the fact because mutations will revert every time you restart a container from a shifted image.

We have to build this component pair because many systems have been specialized with user configuration. User education is going to be really important for this tooling.

Future Plans

Hi, I tried the demo application and seems working, It is creating a single image for all detected applications like Apache2 and MySQL, which is not acceptable for production environment specially scalability point of view. So just wanted to check future development plans for this tool, are you planning to create single image for each application and orchestrate them using docker compose? Please suggest.

Running cron jobs

Add detectives for running cron jobs

Hi Guys,

I am getting the below error while trying to create a dockerfile on Ubuntu 16.04.3 LTS .

Running image transformation.
Unpacked input will not be cleaned up upon completion.
Using existing unpacked image.
The transport volume remains intact.
No components were detected.

Originally posted by @ashwinipraveen in #11 (comment)

docker-archive / communitytools-image2docker-linux Goto Github PK