kubernetes / cloud-provider-aws Goto Github PK

View Code? Open in Web Editor NEW

370.0 370.0 299.0 13.01 MB

Cloud provider for AWS

Home Page: https://cloud-provider-aws.sigs.k8s.io/

License: Apache License 2.0

Go 91.81% Makefile 1.40% Shell 5.88% Dockerfile 0.44% Smarty 0.01% Python 0.45%

k8s-sig-cloud-provider

cloud-provider-aws's Introduction

Kubernetes (K8s)

Kubernetes, also known as K8s, is an open source system for managing containerized applications across multiple hosts. It provides basic mechanisms for the deployment, maintenance, and scaling of applications.

Kubernetes builds upon a decade and a half of experience at Google running production workloads at scale using a system called Borg, combined with best-of-breed ideas and practices from the community.

Kubernetes is hosted by the Cloud Native Computing Foundation (CNCF). If your company wants to help shape the evolution of technologies that are container-packaged, dynamically scheduled, and microservices-oriented, consider joining the CNCF. For details about who's involved and how Kubernetes plays a role, read the CNCF announcement.

To start using K8s

See our documentation on kubernetes.io.

Take a free course on Scalable Microservices with Kubernetes.

To use Kubernetes code as a library in other applications, see the list of published components. Use of the k8s.io/kubernetes module or k8s.io/kubernetes/... packages as libraries is not supported.

To start developing K8s

The community repository hosts all information about building Kubernetes from source, how to contribute code and documentation, who to contact about what, etc.

If you want to build Kubernetes right away there are two options:

You have a working Go environment.

git clone https://github.com/kubernetes/kubernetes
cd kubernetes
make

You have a working Docker environment.

git clone https://github.com/kubernetes/kubernetes
cd kubernetes
make quick-release

For the full story, head over to the developer's documentation.

Support

If you need support, start with the troubleshooting guide, and work your way through the process that we've outlined.

That said, if you have questions, reach out to us one way or another.

Community Meetings

The Calendar has the list of all the meetings in the Kubernetes community in a single location.

Adopters

The User Case Studies website has real-world use cases of organizations across industries that are deploying/migrating to Kubernetes.

Governance

Kubernetes project is governed by a framework of principles, values, policies and processes to help our community and constituents towards our shared goals.

The Kubernetes Community is the launching point for learning about how we organize ourselves.

The Kubernetes Steering community repo is used by the Kubernetes Steering Committee, which oversees governance of the Kubernetes project.

Roadmap

The Kubernetes Enhancements repo provides information about Kubernetes releases, as well as feature tracking and backlogs.

cloud-provider-aws's People

Contributors

Stargazers

Watchers

Forkers

nckturner bertinatto mirake thekubebro nikhita balan2010 poothia ra489 justaugustus renan mailjunze mooncak anypm sathish-thangaraj namingwaysway oded-b jsarkar ayoneeyee not-a-dev0 joewrightss sivagnanamm zacharya leakingtapan akutz injeti-manohar joelsmith d-kuro nashim220 mjrmyke somatorio jaypipes amdonov preskton bordeuax matyix m1losz krish7919 mrsipan neerajb janitha09 sathyanglg kenzo777 phanama niksonx briskgopesh taipanbox minsheng-fintech-corp-ltd tinaoberoi rfranzke gardener-attic manvindar lgs kireledan flant ialidzhikov jigish netflix-skunkworks benluteijn albertofem mbahvw krisnova kevincloud c3y1huang antonionc strongmonkey hawkins6423 jgreat itay andrewsykim noreply10 harsha415 w7374520 sargun mqasimsarfraz wistia galal-hussein redcarpetup etsangsplk matthiasr sudhakar470 jeffwan nicolehanjing windayski ayberk vjestin ruizmarlon clix-dev-llc marvel-works dineshkumar181094 jeswinkninan saranbalaji90 davar-playgrounds krickwix tidbcloud pmmalinov01 openshift sfowl wish oguzhanun b2jrock

cloud-provider-aws's Issues

load-balancer-source-ranges should support managing multiple SG's

K8's only handles managing a single security group.

Some people require a larger number of security group rules, than what would fit on a single security group for a node. K8's should scale out to using every SG tagged as an owned.shared security group for rules, and only fail when it exceeds the total number of SG's for all groups.

/kind feature

Use Cross-Account Roles

What would you like to be added:
The ability to use cross-account roles within my Kubernetes cluster.

Why is this needed:
I am managing a cluster in an AWS account that is not mine. I will soon not have credentials to that account. Instead, I will use a user from my own account to assume a role in the other account to perform any actions I need to in that account. I'd like the Kubernetes cluster to also assume a role for the purposes of managing load balancers and volumes. The alternative is that the role I assume has the ability to create and manage IAM users, which kind of defeats the purpose of using cross-account roles.

/kind feature

POD want real client ip

/triage support

hi all

In our business scenario, POD expects to obtain the real client IP address.

what can i do ?

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  name: xxx
  namespace: xxxxx
spec:
  clusterIP: 192.168.93.139
  externalTrafficPolicy: Cluster
  ports:
  - name: gate
    nodePort: 32410
    port: 20011
    protocol: TCP
    targetPort: 20011
  selector:
    serviceType: client-gate
  sessionAffinity: None
  type: LoadBalancer

Unable to deploy cluster onto AWS us-east-1 region

What happened:
I was in the process of running the kubeadm init command to initialize the first control plane for the cluster, but I ran into an issue when running this command as it times out. I had no issues deploying a k8s cluster with kubeadm prior to this, the only difference now is that I am using the aws cloud provider.

In syslog, I am seeing the following:

Nov 10 04:15:11 ip-XXXXXX kubelet[24132]: E1110 04:15:11.285928   24132 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: Get https://XXXXXX.elb.us-east-1.amazonaws.com:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
Nov 10 04:15:11 ip-XXXXXX kubelet[24132]: E1110 04:15:11.323234   24132 kubelet.go:2267] node "ip-XXXXX.ec2.internal" not found
Nov 10 04:15:11 ip-XXXXXX kubelet[24132]: E1110 04:15:11.423449   24132 kubelet.go:2267] node "ip-XXXXX.ec2.internal" not found
Nov 10 04:15:11 ip-XXXXXX kubelet[24132]: E1110 04:15:11.484386   24132 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://XXXXXXX.elb.us-east-1.amazonaws.com:6443/api/v1/nodes?fieldSelector=metadata.name%3Dip-XXXX.ec2.internal&limit=500&resourceVersion=0: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
Nov 10 04:15:11 ip-XXXXXX kubelet[24132]: E1110 04:15:11.523611   24132 kubelet.go:2267] node "ip-XXXXX.ec2.internal" not found
Nov 10 04:15:11 ip-XXXXXX kubelet[24132]: E1110 04:15:11.623781   24132 kubelet.go:2267] node "ip-XXXXX.ec2.internal" not found

Reaching out for help on Slack, I got this response:
https://kubernetes.slack.com/archives/C2P1JHS2E/p1573444966468300

Looking into this further, the private DNS hostname does not match up with the format expected. This seems to only affect the us-east-1 region. Per:

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-hostnames

I found this:

An Amazon-provided private (internal) DNS hostname resolves to the private IPv4 address of the instance, and takes the form ip-private-ipv4-address.ec2.internal for the us-east-1 Region, and ip-private-ipv4-address.region.compute.internal for other Regions (where private-ipv4-address is the reverse lookup IP address). You can use the private DNS hostname for communication between instances in the same network, but we can't resolve the DNS hostname outside the network that the instance is in.

With this, I went ahead to test a deployment in another region. I ran the same steps in the us-east-2 region and I had no issues there, using the aws cloud provider.

What you expected to happen:
Be able to deploy a Kubernetes cluster, using Kubeadm and the aws cloud provider, in the us-east-1 region.

How to reproduce it (as minimally and precisely as possible):
sudo kubeadm init --upload-certs --config=aws.yml

aws.yml file:

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
  extraArgs:
    cloud-provider: aws
clusterName: kube-test
controlPlaneEndpoint: XXXXXXXXXXXXX
controllerManager:
  extraArgs:
    cloud-provider: aws
    configure-cloud-routes: "false"
kubernetesVersion: stable
networking:
  podSubnet: 192.168.0.0/16
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    cloud-provider: aws

Anything else we need to know?:
Haven't tried to deploy in other regions. But, I am guessing that this is only affecting us-east-1 per AWS' documentation.

Environment:

Kubernetes version (use kubectl version): v1.16.2
Cloud provider or hardware configuration: AWS
OS (e.g. from /etc/os-release): 18.04.3 LTS (Bionic Beaver)
Kernel (e.g. uname -a): Linux 4.15.0-1052-aws #54-Ubuntu SMP Tue Oct 1 15:43:26 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Install tools: NA
Others: NA

/kind bug

Status of project and documentation

Following the announcement of v1.15.0, my understanding is that the future development of cloud providers are being moved out of the kubernetes core.

Existing core including AWS has been moved to https://github.com/kubernetes/legacy-cloud-providers.
Which pretty much breaks the link in note and also seems like in-tree providers will be completely removed in the next couple releases.
We're currently on 1.14.3 and have tested 1.15.0 and can verify that the in-tree provider still works.

But the point I wanted to make is, this repo doesn't seem to have much activity compared to the other major cloud providers.
You also don't have any sample manifests for deploying the cloud-controller-manager and I haven't had any luck getting the aws provider to work as an external cloud provider ever since I first attempted at 1.13..
Asking on kubernetes slack also haven't work well so far, which I suspect either no one had got this working, or simply no one cares.
Doing a google search and reading docs of the CCM and other cloud providers also got me no where close to a working example of this external provider.

So these are my requests:

Please do clarify on the status, or any plans on work with this project.
Please update documentation and provide a close to working sample manifest to deploy this provider

Activity in this repo for the past 6+ months had mostly been cosmetic changes and I haven't seen anything related to feature updates, bug fixes or even documentations.

Please do forgive me if my tone is offending and I might just be paranoid for no good reason, but if you can at least clarify 1 for me, that can help me decide how to approach in the future. Either fork this repo and work on it on my own, or some other paths.

FYI:
We don't use EKS and have no plans on doing so in the future.

Thanks!

/triage support

Where is the latest documentation for this?

I've been trying for weeks to set up a vanilla Kubernetes cluster on AWS and still unable to pinpoint an official and latest tutorial to use this plugin.

Every guide on the internet uses --cloud-provider=aws flag, which we know will be deprecated very soon.

Also, on the half-baked instructions in this GitHub README, it says :

The flag --cloud-provider=external needs to be passed to kubelet, kube-apiserver, and kube-controller-manager.

But when you look at Kubernetes official documentation at https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ :

kube-apiserver and kube-controller-manager MUST NOT specify the --cloud-provider flag.

So which is it??

AWS NLB Deletion Protection

What would you like to be added:

An annotation in the format of

service.beta.kubernetes.io/aws-load-balancer-deletion-protection: [true|false]

that will set the Deletion Protection of the ELB to True or False. (Default is False)

Why is this needed:

To protect the loadbalancer from accidental deletion by external sources.

/kind feature

Kick off meeting for cloud-provider-aws

On Friday, April 13 9:00 AM PDT we will discuss our approach to using this repository. This may include repository structure, scope, migration strategy, any specifics about the components that people would like to discuss, and finally owners/contributors for areas of work. (CSI, flex, CCM, ...)

Links:

AWS Cloud Providers in k8s usable sample/example

Regarding the AWS Cloud Providers in k8s

We are trying to use the cloud providers in the kubernetes 1.14.0 looks like we are there are a few issues in creating the cloud providers with the cluster.

Can we get a usable sample/example for AWS Cloud Providers in the k8s. Any help on this would be helpful.

Missing CONTRIBUTING.md file

All K8s subrepositories should have a CONTRIBUTING.md file, which at the minimum should point to https://github.com/kubernetes/community/blob/master/contributors/guide/README.md. Care should be taken that all information is in sync with the contributor guide.

Subrepositories may also have contributing guidelines specific to that repository. They should be explicitly documented and explained in the CONTRIBUTING.md

Ref: kubernetes/community#1832

EKS 1.15 - NLB TLS termination is failing when multiple ACM certificate ARN's are used

What happened:
EKS 1.15 - NLB TLS termination is not supporting multiple ACM certificateARN's

What you expected to happen:
When a service with NLB annotation is deployed, kube-controller-manager (AWS Cloud provider integration) should create/update Network Load Balancer listener with all the certificates listed as part of annotation "service.beta.kubernetes.io/aws-load-balancer-ssl-cert".

Service deployment fails with following error:

    Warning  SyncLoadBalancerFailed  6s service-controller  Error syncing load balancer: failed to ensure load balancer: Error creating load balancer listener: "CertificateNotFound: Certificate 'arn:aws:acm:us-east-1:xxxxxx:certificate/xxxxxxxx,arn:aws:acm:us-east-1:xxxxxxx:certificate/xxxxxxxxx' not found\n\tstatus code: 400, request id: 28c67c03-cf14-4dd3-bde7-28daab6dd524"

How to reproduce it (as minimally and precisely as possible):

Deploy a pod and expose it as service type LoadBalancer with following annotation. Make sure you pass two ACM ARN's as value.

annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <ACM_CERT_ARNS_HERE>  #use atleast two ARN's

Test pod manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
    run: nlb-tls-deploy
name: nlb-tls-deploy
spec:
replicas: 1
selector:
    matchLabels:
    run: nlb-tls-deploy
strategy: {}
template:
    metadata:
    creationTimestamp: null
    labels:
        run: nlb-tls-deploy
    spec:
    containers:
    - image: nginx
        name: nlb-tls-deploy
        resources: {}
status: {}
---
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
    run: nlb-tls-deploy
name: nlb-tls-deploy
annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <ACM_CERT_ARN_HERE> #"arn:aws:acm:us-east-1:xxxxxxxx:certificate/xxxx,xxxxx "
spec:
ports:
- port: 443
    protocol: TCP
    targetPort: 80
selector:
    run: nlb-tls-deploy
type: LoadBalancer
status:
loadBalancer: {}

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version): EKS 1.15
Cloud provider or hardware configuration: AWS

/kind bug

Tighten IAM Permissions

What would you like to be added:
Cluster API Provider AWS attempts to use least privileges whereever possible.

The project maintains a copy of the IAM policies used by cloud-provider-aws, but these are permissive compared to the use of IAM conditions in the Cluster API AWS.

If there is consistent tagging in use, then these permissions can be scoped down.

Why is this needed:

Enhanced security posture.

/kind feature

Create a SECURITY_CONTACTS file.

As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS
file.

The template for the file can be found in the kubernetes-template repository[2].
A description for the file is in the steering-committee docs[3], you might need
to search that page for "Security Contacts".

Please feel free to ping me on the PR when you make it, otherwise I will see when
you close this issue. :)

Thanks so much, let me know if you have any questions.

(This issue was generated from a tool, apologies for any weirdness.)

[1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE
[2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS
[3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md

Support TargetType: ip for NLB - Issue with SRC/DEST packet resets using NLB from within the cluster

What happened:
When accessing NLB from within kubernetes cluster intermittent packets are dropped due to src/dest being the same in some traffic flows.

This issue occurs when TargetType is set to instance, when set to IP the issue is resolved.

It looks like at present it is hardcoded to instance, happy to provide a pull request unless more discussions are needed around this?

What you expected to happen:
Access NLB pointing at the cluster from within the cluster, all connections to services succeed.

How to reproduce it (as minimally and precisely as possible):

Create a NLB pointing at a service in kubernetes cluster
from within the cluster (node or pod) run multiple curls to the NLB

Anything else we need to know?:
May be related to the following post but did not resolve the issue:
https://kubernetes.io/blog/2019/03/29/kube-proxy-subtleties-debugging-an-intermittent-connection-reset/

This change resolved the issue for us
https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/

/kind bug

Kubernetes 1.11.x => 1.12.x AWS Cloud Provider fails to align AWSInstances <=> Kubernetes Instances

Hello,

i have chef + terraform + kubeadm glued together to raise quickly a Kubernetes cluster in any version inside of AWS with nginx ingress + ELB installed via default.

Starting of Kubernetes Version 1.11.x => this stops working.

The AWS Cloud provider doesn't populate the ELB with the EC2 Nodes anymore.

The Only hint i have is that this Log lines appears in the Kubernetes-Controller pod.

node ".....compute.internal" did not have ProviderID set
node ".....compute.internal" did not have ProviderID set
node ".....compute.internal" did not have ProviderID set

has some tagging changed or needs to be adapted ?

Reference:
kubernetes/kubeadm#973

control ICMP rules created with LoadBalancer type service

What would you like to be added:
Currently if a k8 service of type LoadBalancer is created without loadBalancerSourceRanges, it adds 2 SG rules to the loadbalancer SG, one is for the actual service port which is expected. But the other SG rule is also added that opens ICMP rule for all which comes into security team's radar.

Manually editing SG is a short fix, if the same k8 service is re-applied then the ICMP SG rule appears back.

Why is this needed:
Security risk

/kind feature

Migrate from glog to klog

Migrate all usages of glog to the vendored klog introduced in kubernetes/kubernetes#70889

Don't create an additional SecurityGroups if it's already defined by aws-load-balancer-security-groups annotation

What happened:
AWS Cloud provider creates an additional AWS Security Group per LoadBalancer defined in Kubernetes Service configuration in case if it's already defined by service.beta.kubernetes.io/aws-load-balancer-security-groups annotation

What you expected to happen:
Don't create an additional SG and don't rewrite rules on SG provided by aws-load-balancer-security-groups annotation

How to reproduce it (as minimally and precisely as possible):

Create SG with rules for ELB
Create a K8S service with existing SG defined by annotation service.beta.kubernetes.io/aws-load-balancer-security-groups and Type: LoadBalancer
For example,

$ kubectl describe service some-service
Name:  some-service

Annotations: service.beta.kubernetes.io/aws-load-balancer-security-groups: sg-some-id
....
Type:  LoadBalancer

Then AWS Cloud provider creates an additional SG and replaces custom rules to the existing Security Group
Anything else we need to know?:
Creates SG by cloud provider:

My custom SG after several seconds after service creation:

Environment:

Kubernetes version (use kubectl version):
v1.14.8-eks-b7174d
Cloud provider or hardware configuration:
AWS

/kind bug

NLBs don't seem to support tagging

What happened: I created a Service with service.beta.kubernetes.io/aws-load-balancer-type: nlb and service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: Environment=sandbox-main,Name=sandbox-main-ingress.

What you expected to happen: I expected to get an NLB with the given tags. Instead I got an NLB with no tags.

How to reproduce it (as minimally and precisely as possible): Create a Service as described above.

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-07T09:57:54Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6-eks-5047ed", GitCommit:"5047edce664593832e9b889e447ac75ab104f527", GitTreeState:"clean", BuildDate:"2019-08-21T22:32:40Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: AWS on EKS
OS (e.g. from /etc/os-release): Amazon Linux
Kernel (e.g. uname -a): whatever the EKS-optimized AMI provides
Install tools:
Others:

/kind bug

Allow load balancer name to be provided via service annotation

What would you like to be added:
Would like to add an additional annotation type for the load balancer name.

Why is this needed:
Currently balancer can only be created with a machine-generated name making it very hard to distinguish in AWS if multiple balancers have been created within an account.

Additionally the DefaultLoadBalancerName from the cloud-provider package which is currently in use here has been deprecated and needs a replacement.

/kind feature

NLB use private static IP address

Network Load Balancers can have static private IPs assigned to them using subnet mappings. In 1.16 the allocation of EIPs to LoadBalancer service types was made available, this is to enable the allocation of private IP addresses.

Move beta annotations to GA

AWS cloud provider uses couple of load balancer annotations that are still marked as beta: https://github.com/kubernetes/kubernetes/blob/076dbc6fd645f66488d5adb374fd338ed5f9a439/pkg/cloudprovider/providers/aws/aws.go#L98

Some of them are more than 2 years old. They should be stable by now. Now that cloud provider is being moved is a great time to clean them up.

Note that according to API deprecation policy all beta annotations are declared as stable API and we can't remove them. The code needs to support both beta and stable annotations, advertising only the stable ones in docs.

Deleting LoadBalancer also deletes Security Groups specified in extra-security-group annotation

What happened:
Security group referenced in extra-security-groups annotation gets deleted when k8s LoadBalancer is deleted.

What you expected to happen:
Extra Security Group should not be deleted. This security group was managed by terraform, and is useful for multiple purposes.

How to reproduce it (as minimally and precisely as possible):

Create an nginx-ingress load balancer with an attached security group using extra-security-groups annotation
Delete nginx-ingress load balancer created in step 1.
Notice the extra security group gets deleted

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):

$  k version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.12-eks-c500e1", GitCommit:"c500e11584c323151d6ab17526d1ed7461e45b0c", GitTreeState:"clean", BuildDate:"2019-10-22T03:11:52Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration:
AWS, EKS
Install tools:
helm for nginx-ingress install

/kind bug

Proper Node Names -- Config Option on How This is Set

What would you like to be added:

A configuration option to choose node naming:

do not change existing node names
change but use the requested hostname or resolved hostname from the server (might not be an option)
keep current behavior (use internal DNS)

Why is this needed:

IMO, This is unexpected behavior and a bad user experience.

I currently use Rancher 2 and various tooling to bring up my nodes. I also go to great lengths to name them, populate DNS and keep them organized, but if I want to use the cloud provider for AWS to get the benefits of EBS volumes, for example, all my nodes get renamed to the internal DNS, which is NOT desired.

When using another tool like terraform or Rancher 2 to launch nodes that set the requested hostname to value X but then the value gets changed to Y by the plugin it can cause labels and annotations not to line and break other tooling that expect values to match.

Additional Information:

I use the cloud plugin purely for the EBS feature. I do not use it to launch or control any other resources. If i could disable the other features I would.

/kind feature

Document use of custom endpoints

What would you like to be added:
Documentation regarding the use of AWS Custom/VPC Endpoints for services like EC2.

Why is this needed:
I am deploying in an airgapped region of AWS and need to be able to access the custom endpoints as the traditional ones are unavailable to me. I can't seem to find documentation on how to accomplish this.

/kind feature

How to disable http listener of classic load balancer ?

Hello,

I am looking for way to omit/disable http listener on the created classic load balancer at aws eks v1.13.7-eks-c57ff8 environment.

It looks okay when I deleted the listener on the management console.
But, I cannot find any way or any similar annotations for this case.

Is there any way to use only https listener for classic load balancer (not with ingress) ?

Thanks in advance.

Publish versioned Docker images

What would you like to be added:
Currently I'm not able to find any official Docker image. We need a versioned Docker images published.

Why is this needed:

First of all the aws-cloud-controller-manager is supposed to run as Pod in a cluster. So Docker image is the ideal/expected format.
Right now there are some unofficial published images, but nobody wants to run any unofficial images within clusters with such privileges.
There is a number of supported versions of Kubernetes. The concrete image should support some concrete version, eg. For K8s 1.16 -> should use 1.16 K8s API.
cloud-controller-manager is also deprecated since 1.16. So there is no other replacement and this repository is the only official source now.

/kind feature

what is the aws command that we can use to add a K8s pod's ip address into an NLB. As well as how to remove the IP from the existing NLB.

Would this command work?
aws --region us-east-1 elb register-instances-with-load-balancer --load-balancer-name --instances

NLB HealthCheckIntervalSeconds cannot be overridden with annotations

What happened:
The docs and the code [here](const ServiceAnnotationLoadBalancerHCInterval = "service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval") clearly state that we can use annotations to override the healthcheck interval:

// ServiceAnnotationLoadBalancerHCInterval is the annotation used on the
// service to specify, in seconds, the interval between health checks.
const ServiceAnnotationLoadBalancerHCInterval = "service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval"

What you expected to happen:

When I set the annotation to, for example, 5, I want the healthchecks to happen every 5 seconds.

How to reproduce it (as minimally and precisely as possible):
Create a service of type LoadBalancer with the NLB annotation in AWS.

Anything else we need to know?:
This code sets the default HealthCheckIntervalSeconds to 30 seconds, which is then never over-written by the user provided value.

		input := &elbv2.CreateTargetGroupInput{
			VpcId:                      aws.String(vpcID),
			Name:                       aws.String(name),
			Port:                       aws.Int64(mapping.TrafficPort),
			Protocol:                   aws.String("TCP"),
			TargetType:                 aws.String("instance"),
			HealthCheckIntervalSeconds: aws.Int64(30),
			HealthCheckPort:            aws.String("traffic-port"),
			HealthCheckProtocol:        aws.String("TCP"),
			HealthyThresholdCount:      aws.Int64(3),
			UnhealthyThresholdCount:    aws.Int64(3),
		}

Environment:

Kubernetes version (use kubectl version): v1.11.9
Cloud provider or hardware configuration: AWS EKS
OS (e.g. from /etc/os-release): Amazon Linux

/kind bug

NLB services with "externalTrafficPolicy: Local" route traffic to nodes that cannot handle it for a short time when a node joins the cluster

What happened: In our EKS cluster we have an nginx ingress as an NLB service with externalTrafficPolicy set to Local. It uses an NLB instead of ELB/ALB because it is part of an API Gateway VPC Link, which can only use an NLB.

We are seeing the following behavior: When a new node (even one that does not/cannot run nginx) joins the kubernetes cluster, sometimes (maybe only when the new node is in the same AZ as nginx?), there will be a period of about 30 seconds when hitting our app via the API gateway returns a 500 error with a body of {"message": "Internal server error"}. This error does not show up in our app logs or the nginx logs. That body seems to be generated by API Gateway. Checking flow logs I was able to see what is happening.

Every node in the kubernetes cluster, even nodes that do not have an nginx pod, is a target in the NLB's target group. Only nodes that are running an nginx pod are considered healthy by the NLB. All other nodes in the target group are marked as unhealthy. When a new node not running nginx joins the cluster and the 500 errors happen, flow logs confirmed that traffic between the NLB and the new node on the NLB's health check port flows in both directions. After a while, traffic on the "real traffic" port occurs from the NLB to the new node, but not from the new node to the NLB.

If I SSH onto a non-nginx node in the cluster and run curl http://<non-nginx node IP>:<health check port>, I get a response from nginx. The non-nginx node routes the health check traffic to the nginx node. If I run curl http://<non-nginx node IP>:<main traffic port>, I do not get any response. The non-nginx node does not route "primary" traffic to the nginx node.

This means that when a new non-nginx node comes up, it passes the "active" health check that the NLB does by doing an HTTP request to /healthz on the health check port, because the node routes the request to an nginx node that is able to respond. But after it passes the initial health checks and enters the healthy state, the NLB is never able to connect to the new node on the "primary" traffic port, causing it to fail the NLB's "passive" health checks and the node enters the unhealthy state after failing to serve traffic for a bit.

What you expected to happen: I expected only nodes that actually run a pod belonging to the nginx service to be in the NLB's target group. Failing that, I would expect nodes to not forward health check traffic to the service when externalTrafficPolicy is Local. I could not see any way to configure those behaviors, please let me know if I am mistaken.

How to reproduce it (as minimally and precisely as possible):

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx-gateway
  labels:
    app.kubernetes.io/name: ingress-nginx-gateway
    app.kubernetes.io/part-of: ingress-nginx-gateway
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx-gateway
  labels:
    app.kubernetes.io/name: ingress-nginx-gateway
    app.kubernetes.io/part-of: ingress-nginx-gateway
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx-gateway"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-gateway-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx-gateway
    app.kubernetes.io/part-of: ingress-nginx-gateway
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx-gateway
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx-gateway
  labels:
    app.kubernetes.io/name: ingress-nginx-gateway
    app.kubernetes.io/part-of: ingress-nginx-gateway
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx-gateway
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx-gateway
  labels:
    app.kubernetes.io/name: ingress-nginx-gateway
    app.kubernetes.io/part-of: ingress-nginx-gateway
data:
  use-proxy-protocol: "false"
  use-forwarded-headers: "true"
  proxy-real-ip-cidr: "0.0.0.0/0"
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx-gateway
  labels:
    app.kubernetes.io/name: ingress-nginx-gateway
    app.kubernetes.io/part-of: ingress-nginx-gateway
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx-gateway
  labels:
    app.kubernetes.io/name: ingress-nginx-gateway
    app.kubernetes.io/part-of: ingress-nginx-gateway
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx-gateway
  labels:
    app.kubernetes.io/name: ingress-nginx-gateway
    app.kubernetes.io/part-of: ingress-nginx-gateway
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx-gateway
  labels:
    app.kubernetes.io/name: ingress-nginx-gateway
    app.kubernetes.io/part-of: ingress-nginx-gateway
  annotations:
    flux.weave.works/automated: 'true'
    fluxcd.io/tag.nginx-ingress-controller: semver:~0.26
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx-gateway
      app.kubernetes.io/part-of: ingress-nginx-gateway
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx-gateway
        app.kubernetes.io/part-of: ingress-nginx-gateway
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        group: misc
      containers:
      - name: nginx-ingress-controller
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.2
        resources:
          requests:
            cpu: 250m
            memory: 2Gi
          limits:
            memory: 2Gi
        args:
        - /nginx-ingress-controller
        - --configmap=$(POD_NAMESPACE)/nginx-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx
        - --annotations-prefix=nginx.ingress.kubernetes.io
        - --ingress-class=nginx-gateway
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
            # www-data -> 33
          runAsUser: 33
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: metrics
          containerPort: 10254
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
---
kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx-gateway
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx-gateway
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: Application=nginx-ingress
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
  externalTrafficPolicy: Local
  selector:
    app.kubernetes.io/name: ingress-nginx-gateway
    app.kubernetes.io/part-of: ingress-nginx-gateway
  ports:
  - name: http
    port: 80
    targetPort: http
  type: LoadBalancer

Anything else we need to know?: I believe we can prevent the errors by setting externalTrafficPolicy to Cluster instead of Local. We were hoping to avoid the latency of an extra hop but will use Cluster if we have to.

Environment:

Kubernetes version (use kubectl version): Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-502bfb", GitCommit:"502bfb383169b124d87848f89e17a04b9fc1f6f0", GitTreeState:"clean", BuildDate:"2020-02-07T01:31:02Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider or hardware configuration: AWS EKS
OS (e.g. from /etc/os-release): Amazon Linux 2
Kernel (e.g. uname -a): 4.14.154-128.181.amzn2.x86_64

/kind bug

Allow for hostnames other than private dns

What would you like to be added:
Does the out-of-tree AWS cloud provider allow for hostname overrides or at least hostnames that aren't the private DNS name? There's a few issues in kubernetes/kubernetes regarding limitations in the kubelet where it can only assign the private DNS name as a node name, it would be great if the out-of-tree provider allows for more flexiblity here without breaking anything.

Why is this needed:
More descriptive node names are generally useful.

/kind feature

Override load-balancer healthcheck target via annotations

What would you like to be added:
Have a kubernetes service annotation like "service.beta.kubernetes.io/aws-load-balancer-healthcheck-target", that can override the default healthcheck target and follow the format from aws-sdk for Go lang , for more info see Target field https://docs.aws.amazon.com/sdk-for-go/api/service/elb/#HealthCheck.

Why is this needed:
I have two uses cases

One of our component require client certificate authentication and all the traffic is handled by LB as TCP , HealthCheck from LB doing simple TCP connections that generate a lot of logs with SSL errors in ours pods. To avoid this it's will be nice to deploy the K8S service with :

apiVersion: v1
kind: Service
metadata:
  labels:
    project: 
  annotations:
       service.beta.kubernetes.io/aws-load-balancer-healthcheck-target:  "SSL:3356"

if the port is not specifyed then by default is from application

apiVersion: v1
kind: Service
metadata:
  labels:
    project: 
  annotations:
       service.beta.kubernetes.io/aws-load-balancer-healthcheck-target:  "SSL"

The healthcheck of application is accessible via different port(comparing with applicaiton port) and have a specific path, in this case the solution can be :

apiVersion: v1
kind: Service
metadata:
  labels:
    project: 
  annotations:
     service.beta.kubernetes.io/aws-load-balancer-healthcheck-target:  "HTTP:3636/app/status/lb"

/kind feature
/remove-lifecycle stale

Override automatic creation of security group rules for the NLB

What would you like to be added:
Overrride the creation of an AWS security group rule, e.g. allow 0.0.0.0/0 inbound to the service node port, with a rule that allows you to specify a range of node ports or create an annotation that effectively disables the automatic creation of a security group rule.
Why is this needed:
There are hard limits as to the number of SG rules you can create per ENI. Customers who are using the NLB and creating lots of [NLB] services will hit this limit eventually. By allowing customers to specify a range of ports, e.g. all node ports, they don't have to create a rule per service.

/kind feature

Port ranges for ELB SG or allow for main SG to be managed #79279

What would you like to be added:

One of either:-

Allow a port range
Allow SG to be created and managed outside of k8s and added in, some sort of annotation like service.beta.kubernetes.io/aws-load-balancer-extra-security-groups

Why is this needed:

You can end up running into security group rule limits, due to K8S managing the SG for the ELB and adding rules for each specific port

/kind feature

fix the invalid links

#76
I found their have many invaild url in this repository！Some paths have been deleted, they no longer exist, and some are no longer valid.

ex: https://git.k8s.io/community/contributors/devel/testing.md

aws-load-balancer-security-groups annotation is updating neighbour VPC SG if its in same AWS account

What happened:
While creating service with LoadBalance type, when a Security group from different VPC (non VPC of Kubernetes Cluster deployed, same AWS account) is referenced. Security Group update is done successfully without checking its VPC and is left behind by failling ELB creation.

Error:

Warning  CreatingLoadBalancerFailed  39s  service-controller  Error creating load balancer (will retry): failed to ensure load balancer for service appmesh-system/nginx: InvalidConfigurationRequest: Security group "sg-04cb4d54" does not belong to VPC "vpc-0121ce8563b0fcb56"

What you expected to happen:

There should be a guard rail that checks if,

SG ID belongs to same VPC,
SG should have cluster tag 'kubernetes.io/cluster/'
and fail updating Security group.

How to reproduce it (as minimally and precisely as possible):

Create a deployment

$ kubectl run nginx --image nginx --dry-run -o yaml

Select a Security group Id from a different VPC (same AWS account) and describe it.
```
$ aws ec2 describe-security-groups --group-ids sg-xxxxxx > sg-describe.json
```
Note: Make sure there are some dummy Ingress rules on the SG. We will describe SG again after step 3 and will compare results.

Expose 'nginx' deployment as Service with type LoadBalancer, ref annotation service.beta.kubernetes.io/aws-load-balancer-security-groups. Update 'sg-xxxx' with security group from step 2 in below file.

apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
    run: nginx
name: nginx
annotations:
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "auto-delete=no"
    service.beta.kubernetes.io/aws-load-balancer-security-groups: sg-xxxx
spec:
ports:
- port: 80
    name: status-port1
    protocol: TCP
    targetPort: 80
- port: 81
    name: status-port2
    protocol: TCP
    targetPort: 80
selector:
    run: nginx
type: LoadBalancer

Service will be in <pending> state, it should show warnings

$ kubectl get svc -l run=nginx
$ kubectl describe svc nginx

    Events:
    Type     Reason                      Age                 From                Message
    ----     ------                      ----                ----                -------
    Warning  CreatingLoadBalancerFailed  49m (x10 over 94m)  service-controller  (combined from similar events): Error creating load balancer (will retry): failed to ensure load balancer for service appmesh-system/nginx: InvalidConfigurationRequest: Security group "sg-04cb4d54" does not belong to VPC "vpc-0121ce8563b0fcb56"
            status code: 409, request id: 2801210a-6d2e-4ea7-aaaf-bd54fe84780f
    Normal   EnsuringLoadBalancer  4m28s (x28 over 115m)  service-controller  Ensuring load balancer

Describe security group again and it should have overrided ingress rules port 80 and 81
```
$ aws ec2 describe-security-groups --group-ids sg-xxxxxx > sg-describe2.json
```
Compare sg-describe.json and sg-describe2.json should show overrided rules.

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):

$ kubectl version
    Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-13T18:06:54Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"}
    Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-c0eccc", GitCommit:"c0eccca51d7500bb03b2f163dd8d534ffeb2f7a2", GitTreeState:"clean", BuildDate:"2019-12-22T23:14:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: Cluster created using AWS EKS service.

/kind bug

Lessen the pain of outgrowing ELB subnets by allowing multiple subnets per zone.

What would you like to be added:
We've outgrown the subnets we were originally provided (from an upstream team) for ELB allocation. In the past we've always been able to get larger subnets and essentially replace them. We've now reached the point where our upstream team is having a hard time finding a contiguous subnet block large enough to allocate to us (let alone one per az).

We've noticed that in the code (

cloud-provider-aws/pkg/cloudprovider/providers/aws/aws.go

Lines 3088 to 3094 in 8f6e1e3

 // If we have two subnets for the same AZ we arbitrarily choose the one that is first lexicographically. 

 // TODO: Should this be an error. 

 if strings.Compare(*existing.SubnetId, *subnet.SubnetId) > 0 { 

 klog.Warningf("Found multiple subnets in AZ %q; choosing %q between subnets %q and %q", az, *subnet.SubnetId, *existing.SubnetId, *subnet.SubnetId) 

 subnetsByAZ[az] = subnet 

 continue 

 }

) when presented with two subnets tagged properly the provider simply chooses the one that is alphabetically smaller by id.

However if the one selected has no IPs available it should move down the line to the next subnet until it finds one with allocatable IPs.

Why is this needed:
As clusters grow it can be a challenge, especially in large enterprises, to find large blocks of unallocated ip address space. At a certain point it may be necessary to use many smaller blocks instead.

/kind feature

Allow NodeGroup/ASG targeting for ?LB instances

What would you like to be added:

Our cluster has been created with kops.

We operate multiple instance groups in our cluster to partition the different workloads we run.

I would like an annotation on the aws-load-balancer for targeted instances that will be assigned to the ELB/ALB/NLB TargetGroup using a tag.

E.g.:

service.beta.kubernetes.io/aws-load-balancer-node-groups: "asg-node-group-comma-separated-list"

Where node-group-comma-separated-list is a list of the ASG groupName tags to filter the host instance ids that will be added to the Load Balancer.

E.g. In our environment our 4 node groups have a tag: aws:autoscaling:groupName set to

nodes
ml-cpu
ml-gpu
search

Using the above list of instance group ASG names; to target Pods of the ml-* ASGs I would use the following annotation.

service.beta.kubernetes.io/aws-load-balancer-node-groups: "ml-cpu,ml-gpu"

Currently, I suspect that the tag k8s.io/role/node with a value of 1 is used to populate the instances on the LB.

Why is this needed:

When creating an ELB/ALB or NLB - the complete node instance list associated with the cluster is assigned to the TargetGroup. It is really inefficient to perform health checks against instances that will never host a Pod of the service type you are creating the LB for.

/kind feature

Add ability to provide an EBS volume snapshot ID for the EBS storage class

We have internal requirements that all EBS volumes be encrypted and based off of a certain snapshot in our accounts. Currently there is no way to pass in a snapshot ID, however the change is trivial and I have an enhancement queued up if the maintainers agree to add this functionality.

LoadBalancerIP support for Amazon LoadBalancer

What would you like to be added:
Amazon cloud support addition elastic IP as loadbalancer external IP. Screenshot attached.
But I cannot use this featured as not supported for Kubernetes 1.13. I receive error LoadBalancerIP cannot be specified for AWS ELB.

yaml content

apiVersion: v1
kind: Service
metadata:
  name: proxy-dc0-0
spec:
  selector:
    statefulset.kubernetes.io/pod-name: proxy-dc0-0
  type: LoadBalancer
  loadBalancerIP: 34.211.156.102
  ports:
  - name: sip
    port: 5060
    protocol: TCP
    targetPort: 5060

Service addition

[safarov@safarov-dell kube]$ kubectl create -f sip_service3.yaml
service/proxy-dc0-0 created

Server status

[safarov@safarov-dell kube]$ kubectl describe  service proxy-dc0-0
Name:                     proxy-dc0-0
Namespace:                ippbx
Labels:                   <none>
Annotations:              <none>
Selector:                 statefulset.kubernetes.io/pod-name=proxy-dc0-0
Type:                     LoadBalancer
IP:                       10.100.113.135
IP:                       34.211.156.102
Port:                     sip  5060/TCP
TargetPort:               5060/TCP
NodePort:                 sip  30797/TCP
Endpoints:                
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                      Age               From                Message
  ----     ------                      ----              ----                -------
  Normal   EnsuringLoadBalancer        8s (x2 over 13s)  service-controller  Ensuring load balancer
  Warning  CreatingLoadBalancerFailed  8s (x2 over 13s)  service-controller  Error creating load balancer (will retry): failed to ensure load balancer for service ippbx/proxy-dc0-0: LoadBalancerIP cannot be specified for AWS ELB

Why is this needed:
This feature need because many customers use firewalls and need use static IP on our server side.

/kind feature

Does cloud-provider-aws support dual stack?

What happened:
At present, kubernetes 1.17 and cloud-provider-openstack can support dual-stack.
When tested cloud-provider-aws with dual-stack and version 1.17, I cann't get dual-stack node address

Does cloud-provider-aws support dual-stack? If yes, where can I get the official docker image?
What you expected to happen:
cloud-provider-aws with dual-stack can work well, can get node's IPv4/IPv6 interfaces. Following is the openstack output

# kubectl get nodes henry-dual-we-01 -o go-template --template='{{range .status.addresses}}{{printf "%s: %s \n" .type .address}}{{end}}'  
InternalIP: 10.75.72.170
InternalIP: 192.16.1.15
InternalIP: 2001:282:4000:2000::9
Hostname: henry-dual-we-01

How to reproduce it (as minimally and precisely as possible):

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cloud-controller-manager
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: aws-cloud-controller-manager
  namespace: kube-system
  labels:
    k8s-app: aws-cloud-controller-manager
spec:
  selector:
    matchLabels:
      k8s-app: aws-cloud-controller-manager
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: aws-cloud-controller-manager
    spec:
      nodeSelector:
        node-role.kubernetes.io/master: ""
      securityContext:
        runAsUser: 1001
      tolerations:
      - key: node.cloudprovider.kubernetes.io/uninitialized
        value: "true"
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      - effect: NoSchedule
        key: node.kubernetes.io/not-ready
      serviceAccountName: cloud-controller-manager
      containers:
        - name: aws-cloud-controller-manager
          image: parchment/aws-cloud-controller-manager:latest
          #image: docker.io/k8scloudprovider/openstack-cloud-controller-manager:latest
          args:
            - --v=5
            - --cloud-provider=aws
            - --use-service-account-credentials=false
            - --feature-gates=IPv6DualStack=true
            - --address=127.0.0.1
          volumeMounts:
            - mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
              name: flexvolume-dir
          resources:
            requests:
              cpu: 200m
      hostNetwork: true
      volumes:
      - hostPath:
          path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
          type: DirectoryOrCreate
        name: flexvolume-dir

Logs of aws-cloud-controller-manager

[root@ip-10-0-0-217 ~]# kubectl logs -f -n kube-system   aws-cloud-controller-manager-qrq5p
I0313 08:42:38.386148       1 feature_gate.go:243] feature gates: &{map[IPv6DualStack:true]}
Flag --address has been deprecated, see --bind-address instead.
I0313 08:42:38.386274       1 flags.go:33] FLAG: --add-dir-header="false"
I0313 08:42:38.386296       1 flags.go:33] FLAG: --address="127.0.0.1"
I0313 08:42:38.386306       1 flags.go:33] FLAG: --allocate-node-cidrs="false"
I0313 08:42:38.386320       1 flags.go:33] FLAG: --allow-untagged-cloud="false"
I0313 08:42:38.386328       1 flags.go:33] FLAG: --alsologtostderr="false"
I0313 08:42:38.386334       1 flags.go:33] FLAG: --authentication-kubeconfig=""
I0313 08:42:38.386342       1 flags.go:33] FLAG: --authentication-skip-lookup="false"
I0313 08:42:38.386349       1 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="10s"
I0313 08:42:38.386358       1 flags.go:33] FLAG: --authentication-tolerate-lookup-failure="false"
I0313 08:42:38.386365       1 flags.go:33] FLAG: --authorization-always-allow-paths="[/healthz]"
I0313 08:42:38.386773       1 flags.go:33] FLAG: --authorization-kubeconfig=""
I0313 08:42:38.386921       1 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
I0313 08:42:38.387051       1 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
I0313 08:42:38.387146       1 flags.go:33] FLAG: --bind-address="0.0.0.0"
I0313 08:42:38.387294       1 flags.go:33] FLAG: --cert-dir=""
I0313 08:42:38.387434       1 flags.go:33] FLAG: --cidr-allocator-type="RangeAllocator"
I0313 08:42:38.387539       1 flags.go:33] FLAG: --client-ca-file=""
I0313 08:42:38.387687       1 flags.go:33] FLAG: --cloud-config=""
I0313 08:42:38.387827       1 flags.go:33] FLAG: --cloud-provider="aws"
I0313 08:42:38.387923       1 flags.go:33] FLAG: --cluster-cidr=""
I0313 08:42:38.388064       1 flags.go:33] FLAG: --cluster-name="kubernetes"
I0313 08:42:38.388195       1 flags.go:33] FLAG: --concurrent-service-syncs="1"
I0313 08:42:38.388298       1 flags.go:33] FLAG: --configure-cloud-routes="true"
I0313 08:42:38.388441       1 flags.go:33] FLAG: --contention-profiling="false"
I0313 08:42:38.388568       1 flags.go:33] FLAG: --controller-start-interval="0s"
I0313 08:42:38.388717       1 flags.go:33] FLAG: --controllers="[*]"
I0313 08:42:38.388836       1 flags.go:33] FLAG: --external-cloud-volume-plugin=""
I0313 08:42:38.388979       1 flags.go:33] FLAG: --feature-gates="IPv6DualStack=true"
I0313 08:42:38.389135       1 flags.go:33] FLAG: --help="false"
I0313 08:42:38.389264       1 flags.go:33] FLAG: --http2-max-streams-per-connection="0"
I0313 08:42:38.389366       1 flags.go:33] FLAG: --kube-api-burst="30"
I0313 08:42:38.389509       1 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I0313 08:42:38.389645       1 flags.go:33] FLAG: --kube-api-qps="20"
I0313 08:42:38.389745       1 flags.go:33] FLAG: --kubeconfig=""
I0313 08:42:38.389888       1 flags.go:33] FLAG: --leader-elect="true"
I0313 08:42:38.389905       1 flags.go:33] FLAG: --leader-elect-lease-duration="15s"
I0313 08:42:38.389913       1 flags.go:33] FLAG: --leader-elect-renew-deadline="10s"
I0313 08:42:38.389921       1 flags.go:33] FLAG: --leader-elect-resource-lock="endpointsleases"
I0313 08:42:38.389929       1 flags.go:33] FLAG: --leader-elect-resource-name="cloud-controller-manager"
I0313 08:42:38.389967       1 flags.go:33] FLAG: --leader-elect-resource-namespace="kube-system"
I0313 08:42:38.389978       1 flags.go:33] FLAG: --leader-elect-retry-period="2s"
I0313 08:42:38.389986       1 flags.go:33] FLAG: --log-backtrace-at=":0"
I0313 08:42:38.389996       1 flags.go:33] FLAG: --log-dir=""
I0313 08:42:38.390004       1 flags.go:33] FLAG: --log-file=""
I0313 08:42:38.390012       1 flags.go:33] FLAG: --log-file-max-size="1800"
I0313 08:42:38.390020       1 flags.go:33] FLAG: --log-flush-frequency="5s"
I0313 08:42:38.390028       1 flags.go:33] FLAG: --logtostderr="true"
I0313 08:42:38.390036       1 flags.go:33] FLAG: --master=""
I0313 08:42:38.390044       1 flags.go:33] FLAG: --min-resync-period="12h0m0s"
I0313 08:42:38.390052       1 flags.go:33] FLAG: --node-monitor-period="5s"
I0313 08:42:38.390059       1 flags.go:33] FLAG: --node-status-update-frequency="5m0s"
I0313 08:42:38.390067       1 flags.go:33] FLAG: --node-sync-period="0s"
I0313 08:42:38.390074       1 flags.go:33] FLAG: --port="0"
I0313 08:42:38.390082       1 flags.go:33] FLAG: --profiling="false"
I0313 08:42:38.390089       1 flags.go:33] FLAG: --requestheader-allowed-names="[]"
I0313 08:42:38.390103       1 flags.go:33] FLAG: --requestheader-client-ca-file=""
I0313 08:42:38.390110       1 flags.go:33] FLAG: --requestheader-extra-headers-prefix="[x-remote-extra-]"
I0313 08:42:38.390120       1 flags.go:33] FLAG: --requestheader-group-headers="[x-remote-group]"
I0313 08:42:38.390129       1 flags.go:33] FLAG: --requestheader-username-headers="[x-remote-user]"
I0313 08:42:38.390138       1 flags.go:33] FLAG: --route-reconciliation-period="10s"
I0313 08:42:38.390146       1 flags.go:33] FLAG: --secure-port="10258"
I0313 08:42:38.390153       1 flags.go:33] FLAG: --skip-headers="false"
I0313 08:42:38.390161       1 flags.go:33] FLAG: --skip-log-headers="false"
I0313 08:42:38.390167       1 flags.go:33] FLAG: --stderrthreshold="2"
I0313 08:42:38.390175       1 flags.go:33] FLAG: --tls-cert-file=""
I0313 08:42:38.390182       1 flags.go:33] FLAG: --tls-cipher-suites="[]"
I0313 08:42:38.390192       1 flags.go:33] FLAG: --tls-min-version=""
I0313 08:42:38.390199       1 flags.go:33] FLAG: --tls-private-key-file=""
I0313 08:42:38.390206       1 flags.go:33] FLAG: --tls-sni-cert-key="[]"
I0313 08:42:38.390216       1 flags.go:33] FLAG: --use-service-account-credentials="false"
I0313 08:42:38.390223       1 flags.go:33] FLAG: --v="5"
I0313 08:42:38.390230       1 flags.go:33] FLAG: --version="false"
I0313 08:42:38.390246       1 flags.go:33] FLAG: --vmodule=""
I0313 08:42:38.698871       1 serving.go:312] Generated self-signed cert in-memory
W0313 08:42:39.160232       1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::client-ca-file" due to: configmap "extension-apiserver-authentication" not found
W0313 08:42:39.160414       1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" due to: configmap "extension-apiserver-authentication" not found
W0313 08:42:39.172285       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0313 08:42:39.175457       1 controllermanager.go:120] Version: v0.0.0-master+$Format:%h$
W0313 08:42:39.175491       1 plugins.go:115] WARNING: aws built-in cloud provider is now deprecated. The AWS provider is deprecated and will be removed in a future release
I0313 08:42:39.175771       1 aws.go:1214] Building AWS cloudprovider
I0313 08:42:39.175898       1 aws.go:1180] Zone not specified in configuration file; querying AWS metadata service
I0313 08:42:39.177833       1 log_handler.go:34] AWS API Send: ec2metadata GetMetadata &{GetMetadata GET /meta-data/placement/availability-zone <nil> <nil>} <nil>
I0313 08:42:39.178253       1 log_handler.go:39] AWS API ValidateResponse: ec2metadata GetMetadata &{GetMetadata GET /meta-data/placement/availability-zone <nil> <nil>} <nil> 200 OK
I0313 08:42:39.180859       1 log_handler.go:34] AWS API Send: ec2metadata GetMetadata &{GetMetadata GET /meta-data/instance-id <nil> <nil>} <nil>
I0313 08:42:39.180929       1 log_handler.go:39] AWS API ValidateResponse: ec2metadata GetMetadata &{GetMetadata GET /meta-data/instance-id <nil> <nil>} <nil> 200 OK
I0313 08:42:39.181192       1 log_handler.go:29] AWS request: ec2 DescribeInstances
I0313 08:42:39.287589       1 log_handler.go:34] AWS API Send: ec2 DescribeInstances &{DescribeInstances POST / 0xc000238f00 <nil>} {
  InstanceIds: ["i-07b0eadfbfa7a1af7"]
}
I0313 08:42:39.287679       1 log_handler.go:39] AWS API ValidateResponse: ec2 DescribeInstances &{DescribeInstances POST / 0xc000238f00 <nil>} {
  InstanceIds: ["i-07b0eadfbfa7a1af7"]
} 200 OK
I0313 08:42:39.290853       1 log_handler.go:29] AWS request: ec2 DescribeInstances
I0313 08:42:39.347071       1 log_handler.go:34] AWS API Send: ec2 DescribeInstances &{DescribeInstances POST / 0xc0007a2190 <nil>} {
  InstanceIds: ["i-07b0eadfbfa7a1af7"]
}
I0313 08:42:39.347119       1 log_handler.go:39] AWS API ValidateResponse: ec2 DescribeInstances &{DescribeInstances POST / 0xc0007a2190 <nil>} {
  InstanceIds: ["i-07b0eadfbfa7a1af7"]
} 200 OK
I0313 08:42:39.350647       1 tags.go:79] AWS cloud filtering on ClusterID: henry
I0313 08:42:39.350701       1 healthz.go:123] Installing health checkers for (/healthz): "leaderElection"
I0313 08:42:39.352377       1 configmap_cafile_content.go:205] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0313 08:42:39.352405       1 shared_informer.go:197] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0313 08:42:39.352472       1 configmap_cafile_content.go:205] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0313 08:42:39.352499       1 shared_informer.go:197] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0313 08:42:39.352976       1 reflector.go:153] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/[email protected]/tools/cache/reflector.go:108
I0313 08:42:39.353028       1 reflector.go:188] Listing and watching *v1.ConfigMap from k8s.io/[email protected]/tools/cache/reflector.go:108
I0313 08:42:39.353851       1 reflector.go:153] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/[email protected]/tools/cache/reflector.go:108
I0313 08:42:39.353867       1 reflector.go:188] Listing and watching *v1.ConfigMap from k8s.io/[email protected]/tools/cache/reflector.go:108
I0313 08:42:39.358574       1 tlsconfig.go:179] loaded serving cert ["Generated self signed cert"]: "localhost@1584088958" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer="localhost-ca@1584088958" (2020-03-13 07:42:38 +0000 UTC to 2021-03-13 07:42:38 +0000 UTC (now=2020-03-13 08:42:39.358540827 +0000 UTC))
I0313 08:42:39.362357       1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1584088959" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1584088958" (2020-03-13 07:42:38 +0000 UTC to 2021-03-13 07:42:38 +0000 UTC (now=2020-03-13 08:42:39.362337843 +0000 UTC))
I0313 08:42:39.362423       1 secure_serving.go:178] Serving securely on [::]:10258
I0313 08:42:39.362487       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cloud-controller-manager...
I0313 08:42:39.362576       1 tlsconfig.go:219] Starting DynamicServingCertificateController
I0313 08:42:39.376348       1 leaderelection.go:352] lock is held by ip-10-0-0-217.us-east-2.compute.internal_d2057a2d-0543-4a7b-bc20-863fef995622 and has not yet expired
I0313 08:42:39.376416       1 leaderelection.go:247] failed to acquire lease kube-system/cloud-controller-manager
I0313 08:42:39.452679       1 shared_informer.go:227] caches populated
I0313 08:42:39.452711       1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0313 08:42:39.453430       1 tlsconfig.go:157] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "front-proxy-ca" [] issuer="<self>" (2020-03-12 03:30:08 +0000 UTC to 2030-03-10 03:30:08 +0000 UTC (now=2020-03-13 08:42:39.453402296 +0000 UTC))
I0313 08:42:39.454789       1 tlsconfig.go:179] loaded serving cert ["Generated self signed cert"]: "localhost@1584088958" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer="localhost-ca@1584088958" (2020-03-13 07:42:38 +0000 UTC to 2021-03-13 07:42:38 +0000 UTC (now=2020-03-13 08:42:39.454767178 +0000 UTC))
I0313 08:42:39.455684       1 shared_informer.go:227] caches populated
I0313 08:42:39.455714       1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0313 08:42:39.456167       1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1584088959" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1584088958" (2020-03-13 07:42:38 +0000 UTC to 2021-03-13 07:42:38 +0000 UTC (now=2020-03-13 08:42:39.456098292 +0000 UTC))
I0313 08:42:39.458427       1 tlsconfig.go:157] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubernetes" [] issuer="<self>" (2020-03-12 03:30:07 +0000 UTC to 2030-03-10 03:30:07 +0000 UTC (now=2020-03-13 08:42:39.458410411 +0000 UTC))
I0313 08:42:39.458573       1 tlsconfig.go:157] loaded client CA [1/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "front-proxy-ca" [] issuer="<self>" (2020-03-12 03:30:08 +0000 UTC to 2030-03-10 03:30:08 +0000 UTC (now=2020-03-13 08:42:39.458551365 +0000 UTC))
I0313 08:42:39.459916       1 tlsconfig.go:179] loaded serving cert ["Generated self signed cert"]: "localhost@1584088958" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer="localhost-ca@1584088958" (2020-03-13 07:42:38 +0000 UTC to 2021-03-13 07:42:38 +0000 UTC (now=2020-03-13 08:42:39.459894389 +0000 UTC))
I0313 08:42:39.460302       1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1584088959" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1584088958" (2020-03-13 07:42:38 +0000 UTC to 2021-03-13 07:42:38 +0000 UTC (now=2020-03-13 08:42:39.460287603 +0000 UTC))
I0313 08:42:43.006932       1 leaderelection.go:352] lock is held by ip-10-0-0-217.us-east-2.compute.internal_d2057a2d-0543-4a7b-bc20-863fef995622 and has not yet expired
I0313 08:42:43.007005       1 leaderelection.go:247] failed to acquire lease kube-system/cloud-controller-manager
I0313 08:42:45.105031       1 leaderelection.go:352] lock is held by ip-10-0-0-217.us-east-2.compute.internal_d2057a2d-0543-4a7b-bc20-863fef995622 and has not yet expired
I0313 08:42:45.105106       1 leaderelection.go:247] failed to acquire lease kube-system/cloud-controller-manager
I0313 08:42:48.237031       1 leaderelection.go:352] lock is held by ip-10-0-0-217.us-east-2.compute.internal_d2057a2d-0543-4a7b-bc20-863fef995622 and has not yet expired
I0313 08:42:48.237062       1 leaderelection.go:247] failed to acquire lease kube-system/cloud-controller-manager
I0313 08:42:51.230061       1 leaderelection.go:352] lock is held by ip-10-0-0-217.us-east-2.compute.internal_d2057a2d-0543-4a7b-bc20-863fef995622 and has not yet expired
I0313 08:42:51.230140       1 leaderelection.go:247] failed to acquire lease kube-system/cloud-controller-manager
I0313 08:42:54.035958       1 leaderelection.go:352] lock is held by ip-10-0-0-217.us-east-2.compute.internal_d2057a2d-0543-4a7b-bc20-863fef995622 and has not yet expired
I0313 08:42:54.035992       1 leaderelection.go:247] failed to acquire lease kube-system/cloud-controller-manager
E0313 08:42:56.772069       1 event.go:319] Could not construct reference to: '&v1.Lease{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cloud-controller-manager", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager", UID:"343b9a64-c9d1-4298-859e-8b5d2fc69721", ResourceVersion:"248403", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63719666986, loc:(*time.Location)(0x35096e0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.LeaseSpec{HolderIdentity:(*string)(nil), LeaseDurationSeconds:(*int32)(nil), AcquireTime:(*v1.MicroTime)(nil), RenewTime:(*v1.MicroTime)(nil), LeaseTransitions:(*int32)(nil)}}' due to: 'no kind is registered for the type v1.Lease in scheme "k8s.io/[email protected]/pkg/runtime/scheme.go:101"'. Will not report event: 'Normal' 'LeaderElection' 'ip-10-0-0-217.us-east-2.compute.internal_9814435f-ec58-489e-9742-8cbda7ce2f18 became leader'
I0313 08:42:56.772186       1 leaderelection.go:252] successfully acquired lease kube-system/cloud-controller-manager
I0313 08:42:56.776159       1 event.go:281] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"cloud-controller-manager", UID:"338ca850-07b6-4fac-a13d-a43fab193282", APIVersion:"v1", ResourceVersion:"248402", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' ip-10-0-0-217.us-east-2.compute.internal_9814435f-ec58-489e-9742-8cbda7ce2f18 became leader
I0313 08:42:56.777916       1 aws.go:771] Setting up informers for Cloud
I0313 08:42:56.777973       1 controllermanager.go:237] Starting "cloud-node"
I0313 08:42:56.779953       1 node_controller.go:110] Sending events to api server.
I0313 08:42:56.780016       1 controllermanager.go:247] Started "cloud-node"
I0313 08:42:56.780033       1 controllermanager.go:237] Starting "cloud-node-lifecycle"
I0313 08:42:56.781955       1 node_lifecycle_controller.go:77] Sending events to api server
I0313 08:42:56.785298       1 controllermanager.go:247] Started "cloud-node-lifecycle"
I0313 08:42:56.785327       1 controllermanager.go:237] Starting "service"
I0313 08:42:56.787453       1 controllermanager.go:247] Started "service"
I0313 08:42:56.787483       1 controllermanager.go:237] Starting "route"
I0313 08:42:56.787493       1 core.go:101] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
W0313 08:42:56.787506       1 controllermanager.go:244] Skipping "route"
I0313 08:42:56.787795       1 controller.go:207] Starting service controller
I0313 08:42:56.787815       1 shared_informer.go:197] Waiting for caches to sync for service
I0313 08:42:56.791111       1 log_handler.go:29] AWS request: ec2 DescribeInstances
I0313 08:42:56.794660       1 reflector.go:153] Starting reflector *v1.Service (30s) from k8s.io/[email protected]/tools/cache/reflector.go:108
I0313 08:42:56.794686       1 reflector.go:188] Listing and watching *v1.Service from k8s.io/[email protected]/tools/cache/reflector.go:108
I0313 08:42:56.794892       1 reflector.go:153] Starting reflector *v1.Node (18h12m34.191385441s) from k8s.io/[email protected]/tools/cache/reflector.go:108
I0313 08:42:56.795034       1 reflector.go:188] Listing and watching *v1.Node from k8s.io/[email protected]/tools/cache/reflector.go:108
I0313 08:42:56.804305       1 node_controller.go:312] This node ip-10-0-0-217.us-east-2.compute.internal is registered without the cloud taint. Will not process.
I0313 08:42:56.812999       1 leaderelection.go:283] successfully renewed lease kube-system/cloud-controller-manager
I0313 08:42:56.853245       1 log_handler.go:34] AWS API Send: ec2 DescribeInstances &{DescribeInstances POST / 0xc0009c87d0 <nil>} {
  InstanceIds: ["i-07b0eadfbfa7a1af7"]
}
I0313 08:42:56.853287       1 log_handler.go:39] AWS API ValidateResponse: ec2 DescribeInstances &{DescribeInstances POST / 0xc0009c87d0 <nil>} {
  InstanceIds: ["i-07b0eadfbfa7a1af7"]
} 200 OK
I0313 08:42:56.855476       1 log_handler.go:29] AWS request: ec2 DescribeInstances
I0313 08:42:56.888099       1 shared_informer.go:227] caches populated
I0313 08:42:56.888129       1 shared_informer.go:204] Caches are synced for service
I0313 08:42:56.907546       1 log_handler.go:34] AWS API Send: ec2 DescribeInstances &{DescribeInstances POST / 0xc0009b0280 <nil>} {
  InstanceIds: ["i-07b0eadfbfa7a1af7"]
}
I0313 08:42:56.907607       1 log_handler.go:39] AWS API ValidateResponse: ec2 DescribeInstances &{DescribeInstances POST / 0xc0009b0280 <nil>} {
  InstanceIds: ["i-07b0eadfbfa7a1af7"]
} 200 OK
I0313 08:42:58.826488       1 leaderelection.go:283] successfully renewed lease kube-system/cloud-controller-manager
I0313 08:43:00.840086       1 leaderelection.go:283] successfully renewed lease kube-system/cloud-controller-manager
I0313 08:43:02.854028       1 leaderelection.go:283] successfully renewed lease kube-system/cloud-controller-manager
I0313 08:43:04.866878       1 leaderelection.go:283] successfully renewed lease kube-system/cloud-controller-manager
I0313 08:43:06.879703       1 leaderelection.go:283] successfully renewed lease kube-system/cloud-controller-manager

Node info

[root@ip-10-0-0-217 ~]# kubectl get nodes ip-10-0-0-217.us-east-2.compute.internal -o go-template --template='{{range .status.addresses}}{{printf "%s: %s \n" .type .address}}{{end}}'
InternalIP: 10.0.0.217
ExternalIP: 18.191.237.94
InternalDNS: ip-10-0-0-217.us-east-2.compute.internal
Hostname: ip-10-0-0-217.us-east-2.compute.internal
ExternalDNS: ec2-18-191-237-94.us-east-2.compute.amazonaws.com

Node IPs

[root@ip-10-0-0-217 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:23:dd:5c:04:f6 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.217/24 brd 10.0.0.255 scope global noprefixroute dynamic eth0
       valid_lft 3010sec preferred_lft 3010sec
    inet6 **2600:1f16:5e4:b200:d9f2:6cd1:c3fb:41c3**/128 scope global noprefixroute dynamic
       valid_lft 397sec preferred_lft 97sec
    inet6 fe80::23:ddff:fe5c:4f6/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

[root@ip-10-0-0-217 ~]# kubectl get po -A
NAMESPACE     NAME                                                               READY   STATUS    RESTARTS   AGE
kube-system   aws-cloud-controller-manager-qrq5p                                 1/1     Running   0          44m
kube-system   calico-kube-controllers-5b644bc49c-jjxbl                           1/1     Running   0          4h5m
kube-system   calico-node-cjf25                                                  1/1     Running   0          4h5m
kube-system   coredns-6955765f44-6gfv8                                           1/1     Running   0          29h
kube-system   coredns-6955765f44-tnml9                                           1/1     Running   0          29h
kube-system   etcd-ip-10-0-0-217.us-east-2.compute.internal                      1/1     Running   0          29h
kube-system   kube-apiserver-ip-10-0-0-217.us-east-2.compute.internal            1/1     Running   0          29h
kube-system   kube-controller-manager-ip-10-0-0-217.us-east-2.compute.internal   1/1     Running   0          29h
kube-system   kube-proxy-cgbbr                                                   1/1     Running   0          29h
kube-system   kube-scheduler-ip-10-0-0-217.us-east-2.compute.internal            1/1     Running   0          29h

Anything else we need to know?:

I already enable IPv6DualStack
Environment:

[root@ip-10-0-0-217 ~]#  cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.7 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.7"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.7 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.7:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.7
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.7"
[root@ip-10-0-0-217 ~]# uname -a
Linux ip-10-0-0-217.us-east-2.compute.internal 3.10.0-1062.12.1.el7.x86_64 #1 SMP Thu Dec 12 06:44:49 EST 2019 x86_64 x86_64 x86_64 GNU/Linux

/kind bug

Port ranges for ELB SG or allow for main SG to be managed

What happened:

Run out of security group rules, due to K8S managing the SG for the ELB and adding rules for
each specific port

What you expected to happen:

One of either:-

Allow a port range
Allow SG to be created and managed outside of k8s and added in, some sort of annotation like service.beta.kubernetes.io/aws-load-balancer-extra-security-groups

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):
1.12.6
Cloud provider or hardware configuration:
EKS
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

/kind bug

NLB LoadBalancer doesn't update security groups

I've changed loadBalancerSourceRanges for the LoadBalancer service with service.beta.kubernetes.io/aws-load-balancer-type: nlb annotation.

What happened: Well, nothing.

What you expected to happen: SG for node should be updated with corresponding ranges from loadBalancerSourceRanges.

How to reproduce it (as minimally and precisely as possible):

Create LoadBalancer service with service.beta.kubernetes.io/aws-load-balancer-type: nlb annotation and 1 element in loadBalancerSourceRanges.
Check SG in the cloud.
Add 2nd element to loadBalancerSourceRanges.
Check SG in the cloud.

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-26T00:05:06Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.8-eks-7c34c0", GitCommit:"7c34c0d2f2d0f11f397d55a46945193a0e22d8f3", GitTreeState:"clean", BuildDate:"2019-03-01T22:49:39Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: AWS EKS
OS (e.g. from /etc/os-release): AWS EKS
Kernel (e.g. uname -a): AWS EKS
Install tools:
Others:

/kind bug

LoadBalancer with NLB type will not deregister healthy targets although they should be deregistered

What happened:
LoadBalancer with NLB type will not deregister healthy targets although they should be deregistered.
pkg/cloudprovider/providers/aws/aws_loadbalancer.go#L448

		actualIDs := []string{}
		for _, healthDescription := range healthResponse.TargetHealthDescriptions {
// We will not append healthy instances to actualIDs
			if healthDescription.TargetHealth.Reason != nil {
				switch aws.StringValue(healthDescription.TargetHealth.Reason) {
				case elbv2.TargetHealthReasonEnumTargetDeregistrationInProgress:
					// We don't need to count this instance in service if it is
					// on its way out
				default:
					actualIDs = append(actualIDs, *healthDescription.Target.Id)
				}
			}
		}
		actual := sets.NewString(actualIDs...)
		expected := sets.NewString(instances...)

// So when diff occurs - we will not add healthy instances to removals set
		additions := expected.Difference(actual)
		removals := actual.Difference(expected)

Reason is nil for healthy instances aws-sdk

What you expected to happen:
Code can be rewritten like that:

		actualIDs := []string{}
		for _, healthDescription := range healthResponse.TargetHealthDescriptions {
			if healthDescription.TargetHealth.Reason != nil {
				switch aws.StringValue(healthDescription.TargetHealth.Reason) {
				case elbv2.TargetHealthReasonEnumTargetDeregistrationInProgress:
					// We don't need to count this instance in service if it is
					// on its way out
					continue 
				}
			}
			actualIDs = append(actualIDs, *healthDescription.Target.Id)
		}

/kind bug

Allow to enable partial feature

What would you like to be added:
Config options to enable/disable handling of k8s resources
Why is this needed:
Adding these config options allows users to choose what k8s resources should be handled by AWS.

/kind feature

AWS node/instance security group - misuse of cluster tag

Cross post of k8s issue. Aside: Is there an official policy on where cloud provider issues should be opened?

The AWS cloud provider cluster tag, which defines ownership semantics, is presently being utilised improperly for unrelated purposes. In particular, attempting to identify which instance security groups should be updated to allow inbound load balancer traffic.

This means we (or our controllers) are unable to attach additional security groups to our instances (without "leaking" security group resources).

Please refer to the upstream issue for further details kubernetes/kubernetes#73906

Add limit of used zones/IP for service cloud LoadBalancer

What would you like to be added:
I want publish SIP service for external Internet. To do this recommended use service with type LoadBalancer. In my case to get it working properly need assign only one IP for each pod:port.
In current implementation LoadBalancer created over several zones and one private IP matched to several public IP.
Example

yaml file

apiVersion: v1
kind: Service
metadata:
  name: proxy-dc0-0
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  selector:
    statefulset.kubernetes.io/pod-name: proxy-dc0-0
  type: LoadBalancer
  ports:
  - name: sip
    port: 5060
    protocol: TCP
    targetPort: 5060

service status

[safarov@safarov-dell kube]$ kubectl describe service proxy-dc0-0
Name:                     proxy-dc0-0
Namespace:                ippbx
Labels:                   <none>
Annotations:              service.beta.kubernetes.io/aws-load-balancer-type: nlb
Selector:                 statefulset.kubernetes.io/pod-name=proxy-dc0-0
Type:                     LoadBalancer
IP:                       10.100.5.33
LoadBalancer Ingress:     abd45deeea4d911e98d500a4e92bef38-62dd1831e110ee07.elb.us-west-2.amazonaws.com
Port:                     sip  5060/TCP
TargetPort:               5060/TCP
NodePort:                 sip  32755/TCP
Endpoints:                
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age    From                Message
  ----    ------                ----   ----                -------
  Normal  EnsuringLoadBalancer  3m20s  service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   3m17s  service-controller  Ensured load balancer

public IP

[safarov@safarov-dell kube]$ nslookup abd45deeea4d911e98d500a4e92bef38-62dd1831e110ee07.elb.us-west-2.amazonaws.com
Server:		192.168.100.1
Address:	192.168.100.1#53

Non-authoritative answer:
Name:	abd45deeea4d911e98d500a4e92bef38-62dd1831e110ee07.elb.us-west-2.amazonaws.com
Address: 54.149.51.189
Name:	abd45deeea4d911e98d500a4e92bef38-62dd1831e110ee07.elb.us-west-2.amazonaws.com
Address: 35.165.172.1
Name:	abd45deeea4d911e98d500a4e92bef38-62dd1831e110ee07.elb.us-west-2.amazonaws.com
Address: 52.37.24.89
Name:	abd45deeea4d911e98d500a4e92bef38-62dd1831e110ee07.elb.us-west-2.amazonaws.com
Address: 35.155.206.55

Could you add ability publish service using cloud LoadBalancer via not more one IP for each pod:port. Like

apiVersion: v1
kind: Service
metadata:
  name: proxy-dc0-0
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  selector:
    statefulset.kubernetes.io/pod-name: proxy-dc0-0
  type: LoadBalancerPeerPod
  ports:
  - name: sip
    port: 5060
    protocol: TCP
    targetPort: 5060

Why is this needed:
This required for services that use public IP at application level (L7 OSI).
Example SIP servers use public IP in call signaling.

/kind feature

ELBs are created and deleted sequentially which take a lot of time

Not sure if it's a bug or a conscious decision, but it causes some pain for our use cases when we need to provision a lot of services with ELBs in a short period of time.

What happened:
When I create a bunch of services at the same time (like 100) it take about 6 minutes to provision all ELBs. From the controller logs it looks like it processes services sequentially and maybe even with some wait time in between. The same story happens when I destroy all services, it still takes minutes to clean ELBs. It used to take way longer when it was also creating security groups and rules which we stopped doing and now providing a shared security group, but it's still slow.

What you expected to happen:
ELBs are requested in parallel (with reasonable throttling to avoid rate limits)

How to reproduce it (as minimally and precisely as possible):
Create a bunch of Services with ELB annotations and see they are coming up slowly.

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version): 1.14
Cloud provider or hardware configuration: EKS

/kind bug

Support to use IPv6 address from ec2 meta data

What would you like to be added: Allow node use IPv6 addresses

Why is this needed: When we launch IPv6-only kubernetes cluster on AWS, kubelet etc. keep using IPv4 IP which cause problem, there should be a flag to indicate if people want to enforce IPv6 and that flag will control where to read IP address from ec2 meta data (local-ipv4s or ipv6s), example of hardcoded ipv4: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/legacy-cloud-providers/aws/aws.go#L1393

/kind feature

Kubelet w/ external cloud provider needs run with explicit provider id

What happened:

Following the readme to switch to this external cloud provider, it's either not mentioned or I'm misconfigured, nodes do not auto-detect their ProviderID which leads the aws-cloud-controller-manager to not be able to find the instance in AWS. When I manually add the --provider-id flag to kubelet in the proper aws://$az/$instanceid format, the aws-cloud-controller-manager is able to find the node.

What you expected to happen:

The ProviderID to be automatically detected. I'm not sure this is a correct expectation when using --cloud-provider=external.

How to reproduce it (as minimally and precisely as possible):

Start aws-cloud-controller-manager and set all daemons to --cloud-provider=external.
Start Kubelet with --cloud-provider=external and no --provider-id.

Observe log lines such as https://gist.github.com/mattlqx/a2f6bc5a198fb11f77cbd9adaac46ece

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:47:40Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:37:43Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: AWS
OS (e.g. from /etc/os-release): Ubuntu 18.04.3 LTS
Kernel (e.g. uname -a): 4.15.0-1056-aws
Install tools:
Others:

/kind bug

	// If we have two subnets for the same AZ we arbitrarily choose the one that is first lexicographically.
	// TODO: Should this be an error.
	if strings.Compare(existing.SubnetId, subnet.SubnetId) > 0 {
	klog.Warningf("Found multiple subnets in AZ %q; choosing %q between subnets %q and %q", az, subnet.SubnetId, existing.SubnetId, *subnet.SubnetId)
	subnetsByAZ[az] = subnet
	continue
	}

kubernetes / cloud-provider-aws Goto Github PK

cloud-provider-aws's Introduction

Kubernetes (K8s)

To start using K8s

To start developing K8s

You have a working Go environment.

You have a working Docker environment.

Support

Community Meetings

Adopters

Governance

Roadmap

cloud-provider-aws's People

Contributors

Stargazers

Watchers

Forkers

cloud-provider-aws's Issues

What you expected to happen:

Recommend Projects

Recommend Topics

Recommend Org