Error out of disk space when install in a LXC container about microk8s HOT 8 CLOSED

Hi @raguilera82 ,

After some experimentation with LXC and zfs I got microk8s working. Here is what I had to do:

1. Create the following profile:

$ cat microk8s.profile 
name: microk8s
config:
  boot.autostart: "true"
  linux.kernel_modules: ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh,nf_conntrack_ipv4,ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
  raw.lxc: |
    lxc.apparmor.profile=unconfined
    lxc.mount.auto=proc:rw sys:rw
    lxc.cgroup.devices.allow=a
    lxc.cap.drop=
  security.nesting: "true"
  security.privileged: "true"
description: ""
devices:
  aadisable:
    path: /sys/module/nf_conntrack/parameters/hashsize
    source: /sys/module/nf_conntrack/parameters/hashsize
    type: disk
  aadisable1:
    path: /sys/module/apparmor/parameters/enabled
    source: /dev/null
    type: disk
  aadisable2:
    path: /dev/zfs
    source: /dev/zfs
    type: disk
$ lxc profile copy default microk8s
$ cat microk8s.profile | lxc profile edit microk8s

2. Launch a container:

$ lxc launch -p default -p microk8s ubuntu:18.04 

3. Install zfsutils-linux in the container. This package should be part of microk8s so I also opened a new issue.

# apt install zfsutils-linux

4. And finally install microk8s

# snap install microk8s --edge --classic

Most of the work is on the lxc profile so there is not really much to do for microk8s. Here is an ascii cast of the above steps: https://asciinema.org/a/198964

Thanks

from microk8s.

Hi @raguilera82 ,

What is your lxc profile look like?

I had partial success in running microk8s with this one:

name: microk8s-profile
config:
  boot.autostart: "true"
  linux.kernel_modules: ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
  raw.lxc: |
    lxc.apparmor.profile=unconfined
    lxc.mount.auto=proc:rw sys:rw
    lxc.cap.drop=
  security.nesting: "true"
  security.privileged: "true"
description: ""
devices:
  aadisable:
    path: /sys/module/nf_conntrack/parameters/hashsize
    source: /dev/null
    type: disk
  aadisable1:
    path: /sys/module/apparmor/parameters/enabled
    source: /dev/null
    type: disk

from microk8s.

@raguilera82 , seems docker inside lxc containers does not play well with ZFS https://github.com/lxc/lxd/issues/2305 could you use dir instead?

from microk8s.

Hi @ktsakalozos

Thanks for your answers. My experience with LXC containers (version 3.2) is that they work properly with Docker and ZFS (this is the recommend storage) if you enabled nesting and privileged properties of security. For example, I have a LXC container with an instance of GitLab and I can work with Docker and docker-compose without problems into the container. The issue that you have mentioned is quite old.

Now, I can not change the system storage, maybe for that reason I will get back to Minikube until this issue will be fixed.

Thank you very much for your time.

from microk8s.

Thank you for giving microk8s a try

from microk8s.

Thank you @ktsakalozos

Your steps work perfectly! Also, I have tried with ubuntu:16.04 with your microk8s profile and all works like a charm.

Thank you very much for your time!

from microk8s.

@ktsakalozos Thanks for the info above! I have a question, I see we're removing a lot of the security constraints from lxc .. Can you comment on how contained is that lxc container now? Is it still a little hard to break out of a lxc container with that configuration, or does it basically have full host control without much effort? Basically can I let fairly hostile guests in that lxc with some level of guarantee it's not too easy to break out

Thanks!

from microk8s.

Hi @kim0 ,

I am not entirely sure what you are asking me. I would say it is a little hard to reach the host but not hard enough.

MicroK8s, can get out of the lxc confinement and can access the host machine and this is the purpose of the profile we apply. MicroK8s is expected to run on the host and you are expected to trust it. What you might not trust is the user workload that runs inside Kubernetes and that is constrained by docker containers. You should be aware that MicroK8s is not supposed to be deployed in production or in untrusted environments. Its purpose is mainly to assist in application development locally. If security is of concern you should look into production grade Kubernetes solutions.

Having said that here are a few ways to harden MicroK8s.

• Refine the lxc profile
• Use MicroK8s inside a VM
• Set AppArmor and firewall rules that apply to your usecase
• Test and contribute to this PR #88

Thanks

from microk8s.