Giter Club home page Giter Club logo

Comments (16)

hainesc avatar hainesc commented on August 16, 2024

@694982827 No configuration needed for kubelet.

It seems that the CNI config on node 100.64.15.228 is incorrect. Please check the files in your CNI config directory which defaults /etc/cni/net.d.

from anchor.

694982827 avatar 694982827 commented on August 16, 2024

@hainesc
I found the /etc/cni/net.d/10-anchor.conf created by DaemonSet in anchor.yaml file,

{
    "name": "anchor",
    "cniVersion": "0.3.1",
    "type": "",
    "master": "",
    "octopus": {__OCTOPUS__},
    "policy": {
        "type": "k8s",
        "k8s_api_root": "https://172.1.0.1:443",
        "k8s_auth_token": "xxxxxxxx"
    },
    "kubernetes": {
        "kubeconfig": "/etc/cni/net.d/anchor-kubeconfig"
    },
    "ipam": {
        "type": "anchor",
        "etcd_endpoints": "https://XX:2379,https://XX:2379,https://XX:2379",
        "etcd_key_file": "/etc/etcd/etcdSSL/etcd-key.pem",
        "etcd_cert_file": "/etc/etcd/etcdSSL/etcd.pem",
        "etcd_ca_cert_file": "/etc/etcd/etcdSSL/ca.pem",
        "service_ipnet": "172.1.0.0/24",
        "node_ips": [],
        "policy": {
            "type": "k8s",
            "k8s_api_root": "https://172.1.0.1:443",
            "k8s_auth_token": "XXXXXX"
        },
        "kubernetes": {
            "kubeconfig": "/etc/cni/net.d/anchor-kubeconfig"
        }
    }
}

from anchor.

hainesc avatar hainesc commented on August 16, 2024

@694982827 Is there any other files except 10-anchor.conf?

from anchor.

694982827 avatar 694982827 commented on August 16, 2024

@hainesc
no

root@ubuntu-1:/etc/cni/net.d# ls
10-anchor.conf  anchor-kubeconfig  anchor-tls
root@ubuntu-1:/etc/cni/net.d# tree
.
├── 10-anchor.conf
├── anchor-kubeconfig
└── anchor-tls
    ├── etcd-ca
    ├── etcd-cert
    └── etcd-key

1 directory, 5 files
root@ubuntu-1:/etc/cni/net.d#

from anchor.

hainesc avatar hainesc commented on August 16, 2024

Please show me the ConfigMap in your anchor.yaml which you used to deploy anchor.

from anchor.

694982827 avatar 694982827 commented on August 16, 2024

I set the anchor_mode of configmap to octopus,and the 10-anchor.conf as follows:

root@ubuntu-1:/etc/cni/net.d# cat 10-anchor.conf 
{
    "name": "anchor",
    "cniVersion": "0.3.1",
    "type": "octopus",
    "octopus": {},
    "policy": {
        "type": "k8s",
        "k8s_api_root": "https://172.1.0.1:443",
        "k8s_auth_token": "XXX"
    },
    "kubernetes": {
        "kubeconfig": "/etc/cni/net.d/anchor-kubeconfig"
    },
    "ipam": {
        "type": "anchor",
        "etcd_endpoints": "https://XXX:2379,https://XXX:2379,https://XXX:2379",
        "etcd_key_file": "/etc/etcd/etcdSSL/etcd-key.pem",
        "etcd_cert_file": "/etc/etcd/etcdSSL/etcd.pem",
        "etcd_ca_cert_file": "/etc/etcd/etcdSSL/ca.pem",
        "service_ipnet": "172.1.0.0/24",
        "node_ips": [],
        "policy": {
            "type": "k8s",
            "k8s_api_root": "https://172.1.0.1:443",
            "k8s_auth_token": "XXX"
        },
        "kubernetes": {
            "kubeconfig": "/etc/cni/net.d/anchor-kubeconfig"
        }
    }
}
root@ubuntu-1:/etc/cni/net.d#

And my ConfigMap as follows:

kind: ConfigMap
apiVersion: v1
metadata:
  name: anchor-config
  namespace: kube-system
data:
  # Configure this with the location of your etcd cluster.
  etcd_endpoints: "https://xxx:2379,https://xxx:2379,https://xxx:2379" # "https://10.0.1.2:2379,https://10.0.1.3:2379"
  # Configure the Secret below if etcd TLS enabled.
  etcd_ca: "/etc/etcd/etcdSSL/ca.pem"   # "/anchor-secrets/etcd-ca"
  etcd_cert: "/etc/etcd/etcdSSL/etcd.pem" # "/anchor-secrets/etcd-cert"
  etcd_key: "/etc/etcd/etcdSSL/etcd-key.pem"  # "/anchor-secrets/etcd-key"

  # Config this as the value you specify in kube-apiserver.
  service_cluster_ip_range: "172.1.0.0/24" # "10.96.0.0/12"
  anchor_mode: "octopus" # "macvlan or octopus"
  # Create a macvlan interface at the node, Only centos 7.2+ tested.
  create_macvlan: "true" # "true" or "false"
  # Configure it only when create_macvlan is true.
  # Fields: hostname,master_interface,ip,gateway,mask. Use semicolon(;)
  # to seperate multi items. Recently we only support one item per node
  cluster_network: "100.64.15.228,enp130s0f0,100.64.15.228,100.64.15.254,20;100.64.15.229,enp130s0f0,100.64.15.229,100.64.15.254,20" # "node01,eth0,10.0.2.8,10.0.2.1,24;node02,eth3.2,10.0.12.3,10.0.12.1,24"
  # The CNI network configuration to install on each node.
  cni_network_config: |-
    {
        "name": "anchor",
        "cniVersion": "0.3.1",
        "type": "__ANCHOR_MODE__",
        "master": "__MACVLAN_INTERFACE__",
        "octopus": {__OCTOPUS__},
        "policy": {
            "type": "k8s",
            "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
            "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
        },
        "kubernetes": {
            "kubeconfig": "__KUBECONFIG_FILEPATH__"
        },
        "ipam": {
            "type": "anchor",
            "etcd_endpoints": "__ETCD_ENDPOINTS__",
            "etcd_key_file": "__ETCD_KEY_FILE__",
            "etcd_cert_file": "__ETCD_CERT_FILE__",
            "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
            "service_ipnet": "__SERVICE_CLUSTER_IP_RANGE__",
            "node_ips": [__NODE_IPS__],
            "policy": {
                "type": "k8s",
                "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
                "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
            },
            "kubernetes": {
                "kubeconfig": "__KUBECONFIG_FILEPATH__"
            }
        }
    }
---

from anchor.

hainesc avatar hainesc commented on August 16, 2024

I was confused. The type field in 10-anchor.conf is not the same in your last two comments. And the hostname, ubuntu-1 or 100.64.15.228, which one is correct?

from anchor.

694982827 avatar 694982827 commented on August 16, 2024

@hainesc
I am Sorry,

  1. ubuntu-1 is the hostname od 100.64.15.228 , and I have set kubelet config KUBELET_HOSTNAME="--hostname-override=100.64.15.228".

  2. And the type last is blank ,I have just update the configmap's anchor_mode set it's value to octopus ;then the 10-anchor.conf type change to octopus.

now,the 10-anchor.conf as follows:

{
    "name": "anchor",
    "cniVersion": "0.3.1",
    "type": "octopus",
    "octopus": {},
    "policy": {
        "type": "k8s",
        "k8s_api_root": "https://172.1.0.1:443",
        "k8s_auth_token": ""
    },
    "kubernetes": {
        "kubeconfig": "/etc/cni/net.d/anchor-kubeconfig"
    },
    "ipam": {
        "type": "anchor",
        "etcd_endpoints": "https://xxx:2379,https://xxx:2379,https://xxx:2379",
        "etcd_key_file": "/etc/etcd/etcdSSL/etcd-key.pem",
        "etcd_cert_file": "/etc/etcd/etcdSSL/etcd.pem",
        "etcd_ca_cert_file": "/etc/etcd/etcdSSL/ca.pem",
        "service_ipnet": "172.1.0.0/24",
        "node_ips": [],
        "policy": {
            "type": "k8s",
            "k8s_api_root": "https://172.1.0.1:443",
            "k8s_auth_token": ""
        },
        "kubernetes": {
            "kubeconfig": "/etc/cni/net.d/anchor-kubeconfig"
        }
    }
}

and the configmap :

root@k8s-master:~/network# cat install.yaml 
kind: ConfigMap
apiVersion: v1
metadata:
  name: anchor-config
  namespace: kube-system
data:
  # Configure this with the location of your etcd cluster.
  etcd_endpoints: "https://xxx:2379,https://xxx:2379,https://xxx:2379" # "https://10.0.1.2:2379,https://10.0.1.3:2379"
  # Configure the Secret below if etcd TLS enabled.
  etcd_ca: "/etc/etcd/etcdSSL/ca.pem"   # "/anchor-secrets/etcd-ca"
  etcd_cert: "/etc/etcd/etcdSSL/etcd.pem" # "/anchor-secrets/etcd-cert"
  etcd_key: "/etc/etcd/etcdSSL/etcd-key.pem"  # "/anchor-secrets/etcd-key"

  # Config this as the value you specify in kube-apiserver.
  service_cluster_ip_range: "172.1.0.0/24" # "10.96.0.0/12"
  anchor_mode: "octopus" # "macvlan or octopus"
  # Create a macvlan interface at the node, Only centos 7.2+ tested.
  create_macvlan: "true" # "true" or "false"
  # Configure it only when create_macvlan is true.
  # Fields: hostname,master_interface,ip,gateway,mask. Use semicolon(;)
  # to seperate multi items. Recently we only support one item per node
  cluster_network: "100.64.15.228,enp130s0f0,100.64.15.228,100.64.15.254,20;100.64.15.229,enp130s0f0,100.64.15.229,100.64.15.254,20" # "node01,eth0,10.0.2.8,10.0.2.1,24;node02,eth3.2,10.0.12.3,10.0.12.1,24"
  # The CNI network configuration to install on each node.
  cni_network_config: |-
    {
        "name": "anchor",
        "cniVersion": "0.3.1",
        "type": "__ANCHOR_MODE__",
        "master": "__MACVLAN_INTERFACE__",
        "octopus": {__OCTOPUS__},
        "policy": {
            "type": "k8s",
            "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
            "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
        },
        "kubernetes": {
            "kubeconfig": "__KUBECONFIG_FILEPATH__"
        },
        "ipam": {
            "type": "anchor",
            "etcd_endpoints": "__ETCD_ENDPOINTS__",
            "etcd_key_file": "__ETCD_KEY_FILE__",
            "etcd_cert_file": "__ETCD_CERT_FILE__",
            "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
            "service_ipnet": "__SERVICE_CLUSTER_IP_RANGE__",
            "node_ips": [__NODE_IPS__],
            "policy": {
                "type": "k8s",
                "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
                "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
            },
            "kubernetes": {
                "kubeconfig": "__KUBECONFIG_FILEPATH__"
            }
        }
    }
---


root@k8s-master:~/network# cat anchor-2048.yaml 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: anchor-2048
  labels:
    app: anchor-2048
spec:
  template:
    metadata:
      name: anchor-2048
      labels:
        app: anchor-2048
      annotations:
        cni.anchor.org/subnet: 100.64.0.0/20
        cni.anchor.org/gateway: 100.64.15.254
    spec:
      nodeSelector:
        kubernetes.io/hostname: "100.64.15.228"
      containers:
      - name: anchor-2048
        image: alexwhen/docker-2048
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: anchor-2048
spec:
  type: NodePort
  ports:
  - port: 80
  selector:
    app: anchor-2048
root@k8s-master:~/network# kubectl create -f anchor-2048.yaml 

root@k8s-master:~/network# kubectl get pods -o wide
NAME                           READY     STATUS    RESTARTS   AGE       IP             NODE
anchor-2048-5989c879c6-8fmw8   1/1       Running   0          20m       172.17.0.13    100.64.15.228
root@k8s-master:~/network#

from anchor.

hainesc avatar hainesc commented on August 16, 2024

It's a sad story I am not sure the root cause and the config file 10-anchor.conf is wrong. The octopus field and node_ip should not empty. The project has only been tested on Centos 7.x and maybe the install script works incorrectly on Ubuntu.

from anchor.

694982827 avatar 694982827 commented on August 16, 2024

oh, thank you.

from anchor.

hainesc avatar hainesc commented on August 16, 2024

My advice:

  1. Set a hostname that looks normally.
  2. Set the command to sleep 3600 for the daemonset in anchor.yaml, then run into the Pod, and run install-cni.sh manually and check the output.

Good luck, boy.

from anchor.

694982827 avatar 694982827 commented on August 16, 2024

ok.

from anchor.

694982827 avatar 694982827 commented on August 16, 2024

I have succeed！
I fond the cluster_network should config the hostname of the node and could not be the ip of the node. and I config the ip of the node so it cannot create the macvlan interfance successfully.

I have another problem,for now I should add ips to etcd for every namespace in k8s ,is there any way to add ips to all namespaces once ?

from anchor.

hainesc avatar hainesc commented on August 16, 2024

Good job.

Frankly, there is no way to allocate IPs to all namespaces. Powder monkey is a WebUI that can help you allocate IPs, show etcd data but it is half ready.

BTW, may I know your use case?

  1. What's the kubernetes cluster used for?
  2. Why anchor?
  3. How do you know anchor?

from anchor.

694982827 avatar 694982827 commented on August 16, 2024

oh,

  1. We used kubernetes cluster for Machine Learning and HPC.
  2. I found the default macvlan plugin could not work well for us ,for example the pod could not communicate with the host and pod could not visit k8s's cluster ip by drfault. Anchor solved these problems
  3. I just search macvlan in github and found anchor

from anchor.

hainesc avatar hainesc commented on August 16, 2024

Since you have installed anchor and it works correctly, I will close this issue.

If any problem in future, please feel free to open a new issue.

from anchor.

Related Issues (5)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.