systrap: parallel containers have inadequate overhead about gvisor HOT 8 OPEN

Can you provide the output of cat /proc/cpuinfo, as well as details on what environment you are running this in? Specifically, is there a CPU limit set on the containers, and is runsc itself running within a cgroup that limits its max CPU usage?

from gvisor.

@EtiennePerot I am attaching the log with the /proc/cpuinfo cpuinfo.log. The environment is as follows:

• local bare metal machine with the latest Arch Linux on board, Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz
• I execute runsc inside of the --privileged docker container, which means that it has no of any security/performance restrictions. I probably can try running the same thing outside of docker. I just faced it by measuring runtime costs for my supervisor daemon that basically just does exec() calls to runsc create + start in parallel on request. (docker run -it --privileged --rm MY-IMAGE-WITH-GVISOR)
• original tests were done with having CPU & Memory cgroups set to 1000000 and 16777216reserve/268435456limit, but I have just rechecked it with all cgroups removed completely and together with the -ignore-cgroups flag.

Specifically, is there a CPU limit set on the containers

On the docker privileged container cgroup in which I am running gvisor has no constraints.

and is runsc itself running within a cgroup that limits its max CPU usage?

It was the case for the runsc containers yesterday (1000000), but I have rechecked everything without cgroups at all, the results are the same.

I can try to build a reproducer, but it will take some time if I will have any success. For now, I hope I made a mistake somewhere.

• trial2_1.log
• trial2_2.log

To put a bit of more context the container that I am starting with the gvisor is:

• a default Ruby distro 2.7.8
• with the script that
• loads a tree of usual gems dependencies (thousands of syscalls)
• blocks on the stdin reads

so the executions are pretty much deterministic in steps.

from gvisor.

I can't reproduce this issue. I wrote a small program that execute the specified number of system calls. Here are steps that I used to reproduce the issue: https://gist.github.com/avagin/1dbd7736083e9fe1ad0983832858d7aa.

from gvisor.

@avagin thank you for your gist! I will try to check everything and maybe also write a reproducer.

from gvisor.

I have rechecked the effect on my app. The effect is still observable, but only when the number of simultaneously starting VMs tends to exceed the number of available CPU cores. And only with SysTrap platform. So the question by @EtiennePerot was a good catch:

Specifically, is there a CPU limit set on the containers, and is runsc itself running within a cgroup that limits its max CPU usage?

I didn't set the CPU limit (with --cpuset-cpus 4,5) this time. My CPU has 8 physical cores, and I observe the effect with 7 VMs, with 8 its already full double. With 6 VMs there is no effect.

@EtiennePerot, @avagin does it makes sense for you? you were asking about the CPU limit, maybe there is a contention I am not aware of ...

I will now check the reproducer by @avagin .

from gvisor.

@avagin testing your reproducer with 8 instances of runsc on my local 8 cores CPU:

• ptrace - ~22s/done in both 1 and 8 instances cases (26s/done)
• systrap - 10s/done when 1, 60s/done with 8 instances : 8 cores

https://gist.github.com/sitano/e2ff4a0463bc1fa545c6102110a1ae69

from gvisor.

@EtiennePerot does it makes sense for you? you were asking about the CPU limit, maybe there is a contention I am not aware of ...

I was asking because the way Systrap works involves the Sentry and the sandboxed processes sometimes spin-looping while waiting on each other, as a means to avoid context-switching and process scheduling overhead during system call handling. When the number of such spinloops exceeds the number of available cores, some contention is to be expected. There are mitigations in place to throttle this behavior when it detects some thrashing, but as this thread shows, it's not perfect. This is something we should probably document in the Systrap README.

Does this problem affect your setup in a significant manner, and if so, can you explain why?

from gvisor.

Does this problem affect your setup in a significant manner, and if so, can you explain why?

@EtiennePerot yes. I am not sure if what we are doing is the gvisor primary use case but basically, we are using it to protect our host environment from the malicious user code that we are executing. You can think of AWS-Lambda-like serverless compute system that takes user code in different forms and runs it as fast as it can. Currently, we are running it on gvisor/ptrace platform.

Those code pieces are not big and usually do not live longer than minutes and sometimes can span just seconds. The primary concern for this system is to ensure the isolation of running code and its security - the inability to escape jails and escalate privileges. The second concern is performance in terms of startup latency and implied overhead.

So speaking of performance the system has almost always more running tasks (gvisor containers) than the available cores. We allow CPU overcommit there as it perfectly balances out the existing workload and the pushback. Considering that you can't usually tell in advance what the workload will be bounded by primarily - CPU, disks, or network. Basically, we are good, if all executed tasks equally receive a fair amount of CPU time that there is allocated to that system. That changes if the platform (systrap ie) starts to suffer from contention.

In our case, the host environment is even worse as we are not just not allocating the whole CPU cores to the gvisor system but we are running nested gvisor containers inside of the Kubernetes without dedicated CPU cores but with CPU Shares and Limits (that can enable artificial throttling sometimes), and those cores are also shared with all the other Pod external processes that exist on the host.

That's why it is very important in our case that the gvisor execution platform does not degrade if there is contention on the available CPU cores to the working processes. At least not that much.

Hope this helps.

from gvisor.