interface disappearing about amazon-ec2-net-utils HOT 7 CLOSED

from amazon-ec2-net-utils.

The instance retrieves the token from the EC2 Instance Metadata Service at 169.254.169.254. This is typically reachable via the default route, which would be via eth0, even when configuring a secondary interface. Are you disabling or otherwise restricting access to the IMDS somehow?

from amazon-ec2-net-utils.

I'm not changing anything IMDS. At least not that I'm aware of. It seems like network restart succeeded not because IMDS request succeeded but because the existing config files for each interface are sufficient. If I run a simple curl on the affected instance. I'm able to get the ip located at meta-data/network/interfaces/macs/xx:xx:xx:xx:xx:xx/local-ipv4s. Obviously, if the issue affects eth0 this curl will fail.

from amazon-ec2-net-utils.

Yeah, I've confirmed that the instance will still renew DHCP leases and maintain its network configuration even if IMDS goes away, which is expected as it's fairly common to disable IMDS access after initial instance setup.

Obviously, if the issue affects eth0 this curl will fail.

So you've seen this behavior on eth0 as well as on secondary interfaces? That's a relevant detail because eth0 is handled slightly differently.

Can you share the full output of ip addr, ip rule, and ip route show table all from an instance that is experiencing this issue?

The logs you originally provided seem to indicate that the DHCP lease is successfully renewed, and nothing indicates that anything is removing routes or address configuration, so we'll need to collect some more data.

You can enable tracing on the dhclient-script by creating the file /etc/dhcp/dhclient-enter-hooks.d/01_debug.sh with content exec 2>> /tmp/dhclient-debug.log ; set -x and setting the execute bit on it (chmod +x /etc/dhcp/dhclient-enter-hooks.d/01_debug.sh). If you can install that and leave it in place on a working instance until a failure occurs, then send me the output of journalctl -t ec2net -t dhclient and the content of /tmp/dhclient-debug.log, that may provide some clues.

from amazon-ec2-net-utils.

@nmeyerhans I have an open case with AWS support. can I upload the logs there? Due to security concerns I can't upload them on here.

from amazon-ec2-net-utils.

Noting for visibility: It appears that this problem is related to a dhclient bug that is triggered when the system clock moves backwards. See https://access.redhat.com/solutions/1215993 for additional details. This bug appears to remain unfixed in the current Amazon Linux 2 dhclient packages. It does not seem like this bug is triggered with the default Amazon Linux 2 chrony configuration, but @Rockawear is using a custom configuration that may be contributing to the frequency with which it occurs.

from amazon-ec2-net-utils.

This was fixed in a recent update to the dhcp packages on Amazon Linux 2:

[ec2-user@ip-10-0-0-237 ~]$ rpm -q --changelog dhclient | head -n 2
* Wed Aug 03 2022 Dirk Harms-Merbitz <[email protected]> 4.2.5-79.amzn2.1.1
- Backport fix for negative timestep. RedHat RHBA-2020:1087

from amazon-ec2-net-utils.