Giter Club home page Giter Club logo

jail-task-driver's Introduction

FreeBSD Jail Task Driver

Task driver for FreeBSD jails.

Requirements

Installation

Install(and compile) the jail-task-driver binary and put it in plugin_dir and then add a plugin "jail-task-driver" {} line in your nomad config file.

go get github.com/cneira/jail-task-driver
cp $GOPATH/bin/jail-task-driver YOURPLUGINDIR

Then in your nomad config file, set

plugin "jail-task-driver" {}

In developer/test mode(nomad agent -dev) , plugin_dir is unset it seems, so you will need to mkdir plugins and then copy the jail-task-driver binary to plugins and add a plugins_dir = "path/to/plugins" to the above config file. then you can run it like:

nomad agent -dev -config nomad.config

For more details see the nomad docs.

Parameters

Parameters used by the driver support most of JAIL(8) functionality, parameter names closely match the ones in JAIL(8).

Parameters documentation

Examples

Basic jail

job "test" {
  datacenters = ["dc1"]
  type        = "service"

  group "test" {
    restart {
      attempts = 0
      mode     = "fail"
    }

    task "test01" {
      driver = "jail-task-driver"

      config {
        Path    = "/zroot/iocage/jails/myjail/root"
	Persist  = true
      }
    }
  }
}

Non vnet jail

job "non-vnet" {
  datacenters = ["dc1"]
  type        = "service"

  group "test" {
    restart {
      attempts = 0
      mode     = "fail"
    }

    task "test01" {
      driver = "jail-task-driver"

      config {
        Path              = "/zroot/iocage/jails/myjail/root"
        Ip4               = "new"
        Allow_raw_sockets = true
        Allow_chflags     = true
        Ip4_addr          = "em1|192.168.1.102"
        Exec_start        = "/usr/local/bin/http-echo -listen :9999 -text hello"
      }
    }
  }
}

Vnet jail example

job "vnet-example" {
  datacenters = ["dc1"]
  type        = "service"

  group "test" {
    restart {
      attempts = 0
      mode     = "fail"
    }

    task "test01" {
      driver = "jail-task-driver"

      config {
        Path    = "/zroot/iocage/jails/myjail/root"
	Persist  = true
 	Host_hostname = "mwl.io"
	Exec_clean = true	
	Exec_start = "sh /etc/rc"
	Exec_stop = "sh /etc/rc.shutdown"
	Mount_devfs = true
	Exec_prestart = "logger trying to start "	
	Exec_poststart = "logger jail has started"	
	Exec_prestop = "logger shutting down jail "	
	Exec_poststop = "logger has shut down jail "	
	Exec_consolelog ="/var/tmp/vnet-example"
	Vnet = true
	Vnet_nic = "e0b_loghost"
	Exec_prestart = "/usr/share/examples/jails/jib addm loghost em1"
	Exec_poststop = "/usr/share/examples/jails/jib destroy loghost "
      }
    }
  }
}

Setting resource limits

job "vnet-example2" {
  datacenters = ["dc1"]
  type        = "service"

  group "test" {
    restart {
      attempts = 0
      mode     = "fail"
    }

    task "test01" {
      driver = "jail-task-driver"

      config {
        Path            = "/zroot/iocage/jails/myjail/root"
        Host_hostname   = "mwl.io"
        Exec_clean      = true
        Exec_start      = "sh /etc/rc"
        Exec_stop       = "sh /etc/rc.shutdown"
        Mount_devfs     = true
        Exec_prestart   = "logger trying to start "
        Exec_poststart  = "logger jail has started"
        Exec_prestop    = "logger shutting down jail "
        Exec_poststop   = "logger has shut down jail "
        Exec_consolelog = "/var/tmp/vnet-example"
        Vnet            = true
        Vnet_nic        = "e0b_loghost"
        Exec_prestart   = "/usr/share/examples/jails/jib addm loghost em1"
        Exec_poststop   = "/usr/share/examples/jails/jib destroy loghost "

        Rctl = {
          Vmemoryuse = {
            Action = "deny"
            Amount = "1G"
            Per    = "process"
          }
          Openfiles = {
            Action = "deny"
            Amount = "500"
          }
        }
      }
    }
  }
}

Demo

asciicast

Support

ko-fi

It's also possible to support the project on Patreon

References

TODO:

  • Implement exec interface
  • Implement RecoverTask interface
  • Test All jail options
  • Refactor to match parameters as closely as JAIL(8)
  • Create jails using docker images

jail-task-driver's People

Contributors

cneira avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

peacetara

jail-task-driver's Issues

jail resource allocation report

Currently running top inside a jail, it will report the host total memory instead of the memory allowed for the jail to use.
nomads need to know how much memory is currently the container using to be able to calculate
to which node the allocation will be send.
This is not a problem on illumos as a zone has it own memory counter to keep track of this.

stderr and stdout is not sent on to nomad.

it is captured in the nomad agent logs

    2019-07-10T21:01:43.596-0700 [INFO ] client.driver_mgr.jail-task-driver: Error Creating Jail: driver=jail-task-driver @module=jail-task-driver d
river_initialize_container="Jailcmd error args=[-cmr name=bj-task-bdc991f1-3a1e-3c6a-92fb-7cef3414e140 host.hostname=bj-task-bdc991f1-3a1e-3c6a-92fb
-7cef3414e140 path=/tmp/NomadClient237305148/bdc991f1-3a1e-3c6a-92fb-7cef3414e140/bj-task ip4.addr=10.99.8.229 command=/usr/local/bin/http-echo] err
=exit status 1 out=Missing -text option!
jail: /usr/local/bin/http-echo: failed
" timestamp=2019-07-10T21:01:43.596-0700
    2019-07-10T21:01:43.596-0700 [INFO ] client.driver_mgr.jail-task-driver: Error starting jail task: driver=jail-task-driver driver_cfg="Calling j
ail failed Jailcmd error args=[-cmr name=bj-task-bdc991f1-3a1e-3c6a-92fb-7cef3414e140 host.hostname=bj-task-bdc991f1-3a1e-3c6a-92fb-7cef3414e140 pat
h=/tmp/NomadClient237305148/bdc991f1-3a1e-3c6a-92fb-7cef3414e140/bj-task ip4.addr=10.99.8.229 command=/usr/local/bin/http-echo] err=exit status 1 out=Missing -text option!

this is using go get github.com/hashicorp/http-echo

This is probably out of my golang comfort zone. I might be able to hack that but I don't really understand how Nomad drivers work, so might be best to leave to someone smarter than me.

driver doesn't inform nomad about jail status

the jail is running, but nomad alloc status shows it still 'Pending'.

zsh@userfbsd:~/testjobs nomad run basicjail.nomad
==> Monitoring evaluation "b6bd4ae3"
    Evaluation triggered by job "bj-job"
    Allocation "dc4fa081" created: node "3e5c4135", group "example"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "b6bd4ae3" finished with status "complete"
zsh@userfbsd:~/testjobs nomad alloc status dc4fa081
ID                  = dc4fa081
Eval ID             = b6bd4ae3
Name                = bj-job.example[0]
Node ID             = 3e5c4135
Node Name           = userfbsd
Job ID              = bj-job
Job Version         = 0
Client Status       = pending
Client Description  = No tasks have started
Desired Status      = run
Desired Description = <none>
Created             = 37s ago
Modified            = 37s ago

Task "bj-task" is "pending"
Task Resources
CPU      Memory   Disk     Addresses
100 MHz  300 MiB  300 MiB  http: X.X.X.X:5678

Task Events:
Started At     = N/A
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type                   Description
2019-07-11T10:27:53-07:00  Downloading Artifacts  Client is downloading artifacts
2019-07-11T10:27:53-07:00  Task Setup             Building Task Directory
2019-07-11T10:27:53-07:00  Received               Task received by client
zsh@userfbsd:~/testjobs jls
   JID  IP Address      Hostname                      Path
   132  X.X.X.X     bj-task-dc4fa081-8a1d-4084-80 /tmp/NomadClient026711311/dc4fa081-8a1d-4084-80cf-d2a05dc511c1/bj-task
zsh@userfbsd:~/testjobs

as we can see, the nomad status thinks it's still Pending, but jls clearly shows it running (and so does ps auxww). So I imagine there is some status that has to be sent back to Nomad?

Awesome! :)

I happened to run across this repo while searching github & saw that it was very recently published. I was contemplating building something like this for myself, but yours is obviously much further along. I don't know the details for implementing Nomad task drivers, but I'll read the documentation & review your codebase to get the idea so that hopefully I can contribute some code.

I'm going to test this out this week and see how it works for my uses. I saw the TODO list and have a couple of questions/comments:

  • How are you thinking about implementing docker support? The only ways I can think of doing it are by either implementing docker & augmented Linuxulator for it (big undertaking), updating the community port (very outdated), or using a bhyve VM inside the jail and using the up-to-date ported docker client to control it from the jail. Another option could be to grab the docker image tarball, extract it, & patch it up to run within the jail, but this is not an area I'm super familiar with yet. I'm wondering how automated the transformation could be for converting a docker image to OCI.

  • Would exploring CloudABI be of any use here? It's supposed to be able to run containers as part of a Kubernetes cluster via Scuba.

  • I'd love to be able to ZFS send/recv datasets, much like artifacts in Nomad jobs are for downloading files, before starting jails. This is something that I might actually be willing to implement and submit a PR for, if possible. Any advice?

Thanks & looking forward to further development on this! :)

Path is not getting expanded properly.

jail-task-driver is seeing:
"{Path:/ local/..
but the config is set:
Path = "${NOMAD_TASK_DIR}/.."

I think if we set:

jailparams["path"] = taskConfig.Path

for the default to be TaskConfig.AllocDir if Path is unset, that would fix my issue.

I tried that:

diff --git a/driver/jail.go b/driver/jail.go
index 1d4480a..37ebf88 100644
--- a/driver/jail.go
+++ b/driver/jail.go
@@ -82,7 +82,11 @@ func (d *Driver) initializeContainer(cfg *drivers.TaskConfig, taskConfig TaskCon

        jailparams["name"] = fmt.Sprintf("%s-%s", cfg.Name, cfg.AllocID)
        jailparams["host.hostname"] = fmt.Sprintf("%s-%s", cfg.Name, cfg.AllocID)
-       jailparams["path"] = taskConfig.Path
+       if len(taskConfig.Path) > 0 {
+               jailparams["path"] = taskConfig.Path
+       } else {
+               jailparams["path"] = taskConfig.AllocDir
+       }

        if len(taskConfig.Jid) > 1 {
                jailparams["jid"] = taskConfig.Jid

but I get a
driver/jail.go:88:34: taskConfig.AllocDir undefined (type TaskConfig has no field or method AllocDir)
but the docs seem to say it exists:
https://godoc.org/github.com/hashicorp/nomad/plugins/drivers#TaskConfig

I don't know enough go to understand why AllocDir doesn't seem to exist..

driver does not stop jail after nomad stop job

after running nomad stop, the job is stopped, but the jail is still running.

zsh@userfbsd:~/testjobs nomad stop bj-job
==> Monitoring evaluation "0a1517d4"
    Evaluation triggered by job "bj-job"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "0a1517d4" finished with status "complete"
zsh@userfbsd:~/testjobs jls
   JID  IP Address      Hostname                      Path
   132  X.X.X.X     bj-task-dc4fa081-8a1d-4084-80 /tmp/NomadClient026711311/dc4fa081-8a1d-4084-80cf-d2a05dc511c1/bj-task
zsh@userfbsd:~/testjobs sudo ps auxww | grep http-echo
root    13811   0.0  0.0  11132  2656  0  I+   10:28        0:00.00 jail -cmr ip4.addr=X.X.X.X exec.start=/usr/local/bin/http-echo -listen :5678 -text hello name=bj-task-dc4fa081-8a1d-4084-80cf-d2a05dc511c1 host.hostname=bj-task-dc4fa081-8a1d-4084-80cf-d2a05dc511c1 path=/tmp/NomadClient026711311/dc4fa081-8a1d-4084-80cf-d2a05dc511c1/bj-task
root    13885   0.0  0.1 122828  7340  0  I+J  10:28        0:00.01 /usr/local/bin/http-echo -listen :5678 -text hello
user   15122   0.0  0.0  11320  2728  3  S+   10:30        0:00.00 grep http-echo
zsh@userfbsd:~/testjobs

bhyve support

It would be nice to be able to orchestrate bhyve instances and run them inside jails.

jail's Command does not handle command line arguments correctly.

when Command = "/usr/local/bin/http-echo -listen :5678 -text hello"
I get this:

    2019-07-10T20:58:18.328-0700 [INFO ] client.driver_mgr.jail-task-driver: starting jail task: driver=jail-task-driver @module=jail-task-driver dr
iver_cfg="{Path: Jid: Ip4_addr:10.99.8.229 Ip4_saddrsel:false Ip4: Ip6_addr: Ip6_saddrsel:false Ip6: Vnet: Host_hostname: Host: Securelevel: Devfs_r
uleset: Children_max:0 Children_cur:0 Enforce_statfs:0 Persist:false Osrelease: Osreldate: Allow_set_hostname:false Allow_sysvipc:false Allow_raw_so
ckets:false Allow_chflags:false Allow_mount:false Allow_mount_devfs:false Allow_quotas:false Allow_read_msgbuf:false Allow_socket_af:false Allow_res
erved_ports:false Allow_mlock:false Allow_mount_fdescfs:false Allow_mount_fusefs:false Allow_mount_nullfs:false Allow_mount_procfs:false Allow_mount
_linprocfs:false Allow_mount_linsysfs:false Allow_mount_tmpfs:false Allow_mount_zfs:false Allow_vmm:false Linux: Linux_osname: Linux_osrelease: Linux_oss_version: Sysvmsg: Sysvsem: Sysvshm: Exec_prestart: Exec_prestop: Exec_created: Exec_start: Exec_stop: Command:/usr/local/bin/http-echo -listen
 :5678 -text hello Exec_poststart: Exec_poststop: Exec_clean:false Exec_jail_user: Exec_system_jail_user: Exec_system_user: Exec_timeout:0 Exec_cons
olelog: Exec_fib: Stop_timeout:0 Nic: Vnet_nic: Ip_hostname: Mount:false Mount_fstab: Mount_devfs:false Mount_fdescfs:false Depend: Rctl:{Cputime:0
Datasize:0 Coredumpsize:0 Stacksize:0 Memoryuse:0 Memorylocked:0 Maxproc:0 Openfiles:0 Vmemoryuse:0 Pseudoterminals:0 Swapuse:0 Nthr:0 Msgqqueued:0
Msgqsize:0 Nmsgq:0 Nsemop:0 Nshm:0 Shmsize:0 Wallclock:0 Pcpu:0 Readbps:0 Writebps:0 Readiops:0 Writeiops:0}}" timestamp=2019-07-10T20:58:18.328-0700
    2019-07-10T20:58:18.332-0700 [INFO ] client.driver_mgr.jail-task-driver: Error Creating Jail: driver=jail-task-driver driver_initialize_containe
r="Params map[command:/usr/local/bin/http-echo -listen :5678 -text hello host.hostname:bj-task-10cad86f-299f-5338-2eac-5a0ba067040c ip4.addr:10.99.8
.229 name:bj-task-10cad86f-299f-5338-2eac-5a0ba067040c path:/tmp/NomadClient237305148/10cad86f-299f-5338-2eac-5a0ba067040c/bj-task]" @module=jail-ta
sk-driver timestamp=2019-07-10T20:58:18.331-0700
    2019-07-10T20:58:18.332-0700 [INFO ] client.driver_mgr.jail-task-driver: Error Creating Jail: driver=jail-task-driver @module=jail-task-driver d
river_initialize_container="Jailcmd error args=[-cmr name=bj-task-10cad86f-299f-5338-2eac-5a0ba067040c host.hostname=bj-task-10cad86f-299f-5338-2eac
-5a0ba067040c path=/tmp/NomadClient237305148/10cad86f-299f-5338-2eac-5a0ba067040c/bj-task ip4.addr=10.99.8.229 command=/usr/local/bin/http-echo -lis
ten :5678 -text hello] err=exit status 1 out=jail: exec /usr/local/bin/http-echo -listen :5678 -text hello: No such file or directory
jail: /usr/local/bin/http-echo -listen :5678 -text hello: failed
" timestamp=2019-07-10T20:58:18.331-0700

When I set Command = "/usr/local/bin/http-echo"
then it works, but of course http-echo requires arguments to function properly.

So the arguments are not getting added/called correctly I think. if go's exec function is calling through the shell, the answer is, the Command just needs to be double quoted.. but I imagine Go isn't that crazy.. to exec() through a shell. I'm not up to speed on how Go's exec() does arguments so I'm not sure what the right answer here is.

Obviously a work-around is to use a template {} block and write a little shell script and call that. I just haven't gotten that far yet.

Docker image is not working from examples/docker-jail

There is no explanation on how to setup the a docker container using jail-task-driver.
Also there is a hard dependency on gtar to uncompress docker images that could be handled on the driver code.
There is a need to add documentation on the driver to which are the requirements on FreeBSD
to be able to execute Linux binaries.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.