Azure: Failed to start CoreOS Metadata Agent about afterburn HOT 8 CLOSED

@lucab, thanks, in CoreOS v1871 it works. Waiting when it will be available in the Stable channel.

from afterburn.

It could be caused by the recent changes made by @csssuf and @sdemos in the: https://github.com/coreos/coreos-metadata/blob/master/src/providers/azure/mod.rs

from afterburn.

Those changes haven't made it into a release yet, so this is an existing bug. There are a number of unwrap()s in the Azure crypto logic which should probably be removed in favor of proper error reporting, one of which is likely the cause of this issue.

@dtretyakov, can you get a backtrace from coreos-metadata? You should be able to do so by putting the following in /etc/systemd/system/coreos-metadata.service.d/10-backtrace.conf:

[Service]
Environment=RUST_BACKTRACE=1

from afterburn.

The Azure crypto logic is definitely messy. Doing a chain_err instead of unwrap is definitely the way to go, I think I left those there initially because of assumptions (that might be wrong) about the cryptographic structures I was getting back from the API. Also, since it's Option::unwrap and not Result::unwrap, only 3 of the 5 unwrap calls apply (these: https://github.com/coreos/coreos-metadata/blob/master/src/providers/azure/crypto/mod.rs#L66, https://github.com/coreos/coreos-metadata/blob/master/src/providers/azure/crypto/mod.rs#L74, https://github.com/coreos/coreos-metadata/blob/master/src/providers/azure/crypto/mod.rs#L75)

@dtretyakov do you have this machine configured with a public ssh key? If so, what is the key type?

Also, as a funny side node, this is another situation that will be fixed by coreos/bugs#2362, since all the unwrap calls are related to retrieving the public ssh keys but it's causing the coreos-metadata --attributes to fail.

from afterburn.

@sdemos, this machine was created with username/password pair:

So it looks like it may not have expected ssh key.

from afterburn.

@dtretyakov if you have the opportunity at some point, I would be very interested in seeing what happens when you get the backtrace as @csssuf suggested. I have a guess as to which unwrap it is triggering (https://github.com/coreos/coreos-metadata/blob/master/src/providers/azure/crypto/mod.rs#L66) but it would be nice to know for sure.

As far as fixing this bug, we should audit all locations where unwrap is used, since it's likely not what we actually want to do. However, the next release of coreos-metadata should stop triggering it in your particular case.

I suspect you can work around this by providing an ssh public key in the web UI. If you still need the username and password to be created, it's possible to do this using ignition in the passwd section (see the passwd section of the ignition specification)

from afterburn.

I've verified that the case with ssh public key works fine.

When was used username/password pair I've tried to enable backtrace for rust but it looks like it does not take any effect:

> export RUST_BACKTRACE=1
> /usr/bin/coreos-metadata --cmdline --attributes=/run/metadata/coreos
Aug 03 07:57:35.480 INFO Fetching http://168.63.129.16/?comp=versions: Attempt #1
Aug 03 07:57:35.482 INFO Fetch successful
Aug 03 07:57:35.483 INFO Fetching http://168.63.129.16/machine/?comp=goalstate: Attempt #1
Aug 03 07:57:35.485 INFO Fetch successful
Aug 03 07:57:35.673 INFO Fetching http://168.63.129.16/machine/aadeb670-1ea7-40f2-bee2-a27287cbfc48/86dcad60%2Dca5b%2D45aa%2D91e7%2Df8780e4a7847.%5Fteamcity?comp=certificates&incarnation=2: Attempt #1
Aug 03 07:57:35.682 INFO Fetch successful
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /build/amd64-usr/var/tmp/portage/dev-libs/rustlib-1.26.0/work/rustc-1.26.0-src/src/libcore/option.rs:335:21
> echo $RUST_BACKTRACE
1

from afterburn.

This should be fixed in 2.0.0. For ContainerLinux, it is now released as part of 1871.0.0.

I'm going ahead and closing this. @dtretyakov feel free to followup if you are still hitting this in the latest release.

from afterburn.