容器 with fixed IP label

• Create -> docker -> minions 判断有这个 label 在另外一个地方存一个 flag -> 正常 assign IP 流程
• Stop -> docker -> minions 判断是否有 label
  • 没有 -> 正常释放
  • 有 -> 输出日志不释放
• Start -> docker -> minions 判断是否有 label
  • 没有 -> 正常 assign
  • 有 -> 从另外一个地方存的 flag 中取出 assign 的老 IP，返回给 docker
• Remove -> barrel -> docker
  • 有 label -> 清理 flag -> 正常 remove 行为转发给 docker

容器 without fixed IP label

一切照旧

容器 Replace

old without fixed IP labels

• inherit network

  • get IP from old one
  • old one stop -> 老流程，IP 释放回了 calico pool
  • minions 需要判断，不管有没有 fixed IP label，有则忽略 fixed IP label 逻辑，直接 assign 老 IP 出去。
  • start new one
    • succ -> remove old one
    • fail -> remove new one and start old one

• no inherit network

  • old one stop -> 取决于老的是否带 label 带则走 minions 新流程，不带则走老流程
  • start new one
    • 不带 label -> 老流程
    • 带 label -> 新流程

old with fixed IP labels

• no inerit network

  • old one stop -> 新流程
  • start new one
    • 不带 label -> 老流程
    • 带 label -> 新流程

• inerit network

  • get IP from old one
  • old one stop -> 新流程，这时候 IP 没释放回 calico pool
  • minions 需要判断，不管有没有 fixed IP label，有则忽略 fixed IP label 逻辑，并且在这里因为是 reserved IP 需要操作 calico IPAM 和 workload endpoints 使得能 IP 能 assign 出去。
    • start new one
    • succ -> remove old one -> barrel 删除 flag -> minions 需要对 IP 判断，如果存在另外一组 weps，则 minions 不把 IP 放回 calico pool。
    • fail -> remove new one -> IP 是配置中的，同样需要 minions 对 IP 判断，如果存在另外一组 weps，则 minions 不把 IP 放回 calico pool -> start old one -> 新流程

Background

Currently, the barrel will return a bad gateway error to the client when failed. For example, when it failed to allocate a new address when creating a container with a fixed-ip label, it will show a bad gateway error.

Related code:

func writeErrorResponse(res http.ResponseWriter, logger utils.Logger, err error, label string) {
	logger.Errorf("%s failed %v", label, err)
	if err := utils.WriteBadGateWayResponse(
		res,
		utils.HTTPSimpleMessageResponseBody{
			Message: label + " error",
		},
	); err != nil {
		logger.Errorf("write %s error response failed %v", label, err)
	}
}

Logging:

Aug 25 19:02:51 bash[23490]: time="2021-08-25 19:02:51" level=info msg="Ran out of existing affine blocks for host"
Aug 25 19:02:51 bash[23490]: time="2021-08-25 19:02:51" level=info msg="Auto-assigned 0 out of 1 IPv4s: []" 
Aug 25 19:02:51 bash[23490]: time="2021-08-25 19:02:51" level=error msg="[containerCreateHandler::Handle] check and request fixed-ip failed Unexpected number of assigned IP addresses. A single address should be assigned. Got []"

But the error message is not explicit to read on the client-side:

Error response from daemon: check and request fixed-ip error

Improvement

The Barrel should be compatible with the docker client and act like a normal dockerd.

Extra

Actually, it's not my first time helping others (like QA colleagues) and I have already found it's difficult to identify the root cause. I am recalling my memory of it, if I made it, I will update it here.

Update:

I have recalled it. On day QA came to ask me why he kept getting an error The requested subnet must match the CIDR of a configured Calico IP Pool when creating new containers using the calico network in his own test cluster.

It looked like the subnet was not right in the configuration. But after confirming this and others, it wasn't the cause. And I created a new one with calico, it worked normally. That really confused me. After spending time to reading code and logs, I found the problem finally. The related code was:

	f := false
	networkName := ""
	for _, p := range pools.Items {
		if nid, ok := p.Annotations[dockerLabelPrefix+"network.ID"]; ok && nid == request.NetworkID {
			f = true
			networkName = p.ObjectMeta.Name
			log.Debugf("Find ippool : %v\n", p.Name)
			break
		}
	}
	if !f {
		log.Errorln(types.ErrCIDRNotInPool)
		return nil, types.ErrCIDRNotInPool
	}

Though I didn't know how it happened, the network.ID didn't match that in the configuration of docker. So the error message was totally misleading. I suggest we output appropriate and explicit logs at least or good to have a more explicit error message.

Assume that a user had bind several directory to a container, that parent of those directory also belongs to the container:

 Mounts":[
{
"Type": "bind" ,
"Source": "/data/biz/cluster01/container01/data","Destination" : " /data/my_data" ,
"Mode" : "rw" ,
"RW" : true,
"Propagation" :"rprivate"
,
"Type" : "bind" ,
" Source" : " /data/biz/cluster01/container01/1og","Destination" : "/data/my_log",
"Mode" : "rw" ,
"RW":true,
"Propagation" :"rprivate"
],

The root directory for bindings of a single container is like "/data/biz/cluster01/container01", If user want to remove the directory, we need more info for us to conclude that the path is some sort of resource for the container. Thus we have plans:

A. User mount /data/biz/cluster01/container01 as well
B. Recording the infomation in label:

"Labels": {
  “BARREL”: "{\"Mounts\: [\"/data/biz/cluster01/container01\"]"}"
}

For plan A, we need to parse the mounts to find out the directories need to remove and directories need not, and there will be a useless mount for the container. For plan B it's more simple but there is a risk that a container declares unrelated resources in the label, however we can do the same parsing to the mount declares inside label to eliminate the risk, or using a regex to safeguard it.