myproxy FTBFS on IPV6-only build servers about gct HOT 13 CLOSED

CCing @jbasney

Strange. Comparing the Debian build log to my log on the OBS (though for MyProxy v6.2.6, but the changes in #146 and #147 don't look like they changed something relevant), it looks to me, like there's just no service listening on the IPv6 localhost address on the Debian build host, because the server seems to listen on the IPv4 localhost address:

[...]
Attempting to connect to ::1:45721 
Attempting to connect to 127.0.0.1:45721 
Successfully connected to localhost:45721
[...]

Maybe myproxy-server (commandline in https://github.com/gridcf/gct/blob/master/myproxy/source/myproxy-test#L269) doesn't listen on IPv6 addresses by default, when the env var MYPROXY_SERVER is configured to localhost only (done in https://github.com/gridcf/gct/blob/master/myproxy/source/myproxy-test#L243). On Ubuntu 20.04 for example the IPv6 localhost address has the following aliases:

• ip6-localhost
• ip6-loopback

Not sure if (1) we just would need to add one of these to MYPROXY_SERVER (@jbasney: Can this be done and what separator is needed here?) or (2) we need to add additional -l [...] arguments to the server command in https://github.com/gridcf/gct/blob/master/myproxy/source/myproxy-test#L269) to solve this.

from gct.

Additonal info:

• Failing tests are 3T, 20T and 41 (https://github.com/gridcf/gct/blob/master/myproxy/source/myproxy-test#L386)

• "Bootstrapping MyProxy server root of trust." comes from

  gct/myproxy/source/myproxy.c

  Line 497 in 5882044

  myproxy_bootstrap_trust(myproxy_socket_attrs_t *attrs)

• "Attempting to connect to [...]" comes from

  gct/myproxy/source/myproxy.c

  Line 322 in 5882044

  static int connect_socket_to_host(char *host, int port) {

from gct.

MYPROXY_SERVER can be a comma-separated list of hostnames. That tells clients where to connect.

The myproxy-server may only listen on IPv4 by default.

Otherwise, the myproxy-server --listen option can only specify a single hostname.

Since only MyProxy Test 3T fails, I think there's not a problem in general with the clients connecting to the server, but something specific to myproxy_bootstrap_trust() though that uses get_connected_myproxy_host_socket() just like the other connections, so I'm not sure what's causing a 'Connection refused' error only in that case.

Running myproxy-test -verbose may give additional clues.

from gct.

MYPROXY_SERVER can be a comma-separated list of hostnames. That tells clients where to connect.

gct/myproxy/source/myproxy.c

Line 387 in 246f1c3

static int get_connected_myproxy_host_socket(char *hostlist, int port) {

What I don't understand is, why the client tries both IPv4 and IPv6 localhost addresses with a single localhost in MYPROXY_SERVER. Aha, because of this in /etc/hosts (checked on Raspberry Pi OS/Raspbian):

127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
[...]

And here is why: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=686970

Here's what other distributions do:

• CentOS 7, CentOS 8 and Fedora 34 use:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

So our builds at Travis CI would actually have the same issue, if they'd use IPv6 in addition to IPv4.

• openSUSE Leap 15.2 uses:

127.0.0.1	localhost

# special IPv6 addresses
::1             localhost ipv6-localhost ipv6-loopback

• Ubuntu 18.04.x and 20.04.x use:

127.0.0.1	localhost
[...]
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback

But IMHO I'd have expected localhost to resolve to a singular address.

Otherwise, the myproxy-server --listen option can only specify a single hostname.

Can --listen then be used mutliple times? Possibly not: https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L963..L965

Since only MyProxy Test 3T fails, I think there's not a problem in general with the clients connecting to the server, but something specific to myproxy_bootstrap_trust() though that uses get_connected_myproxy_host_socket() just like the other connections, so I'm not sure what's causing a 'Connection refused' error only in that case.

Do the other tests try to connect to all localhost addresses the same way as the 3T test? If not, then I assume myproxy-server just creates IPv4 sockets for all (IPv4) addresses the host has (see https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L937..L938).

from gct.

Complete logs are available here:
https://buildd.debian.org/status/logs.php?pkg=myproxy&ver=6.2.8-1&arch=amd64
Both attempts say:

MyProxy Tests Complete: 53 tests passed, 3 tests failed

All 3 failed test say:
trust root bootstrap failed
Unable to connect to ::1: .......
Connection refused

from gct.

Can --listen then be used multiple times?

No, but myproxy-server --listen localhost should bind to all localhost addresses just like MYPROXY_SERVER=localhost tries to connect to all localhost addresses. Adding --listen localhost to the myproxy-server command in the myproxy-test script may be a fix for this issue.

Do the other tests try to connect to all localhost addresses the same way as the 3T test?

I think so, which is why I'm struggling to identify why only the 3T test fails. There must be some difference that I'm missing.

from gct.

Since only MyProxy Test 3T fails, I think there's not a problem in general with the clients connecting to the server, but something specific to myproxy_bootstrap_trust() though that uses get_connected_myproxy_host_socket() just like the other connections, so I'm not sure what's causing a 'Connection refused' error only in that case.

Could the problem be get_connected_myproxy_host_socket()? It tries to split a host:port string on a colon, but maybe it gets confused with an IPv6 host that has lots of colons in it (like e.g. ::1).

I still don't get why only 3 tests fail though...

from gct.

@ellert:

Complete logs are available here:
https://buildd.debian.org/status/logs.php?pkg=myproxy&ver=6.2.8-1&arch=amd64
Both attempts say:

MyProxy Tests Complete: 53 tests passed, 3 tests failed

All 3 failed test say:
trust root bootstrap failed
Unable to connect to ::1: .......
Connection refused

Thanks for clarification, I was too quick and didn't check the whole log but just the first occurance of an error, sorry. :-/ I'll update #160 (comment) accordingly.

@jbasney: So I assume all tests where a client tries to connect to the server using the address in MYPROXY_SERVER which points to both 127.0.0.1 and ::1 on Debian seem to fail.

Can --listen then be used multiple times?

No, but myproxy-server --listen localhost should bind to all localhost addresses just like MYPROXY_SERVER=localhost tries to connect to all localhost addresses. Adding --listen localhost to the myproxy-server command in the myproxy-test script may be a fix for this issue.

But it should already per default listen on all addresses - at least that's what I understand from https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L937..L938 - and the myproxy-test script doesn't use --listen nor anything address related in the configuration file.

@ellert

Since only MyProxy Test 3T fails, I think there's not a problem in general with the clients connecting to the server, but something specific to myproxy_bootstrap_trust() though that uses get_connected_myproxy_host_socket() just like the other connections, so I'm not sure what's causing a 'Connection refused' error only in that case.

Could the problem be get_connected_myproxy_host_socket()? It tries to split a host:port string on a colon, but maybe it gets confused with an IPv6 host that has lots of colons in it (like e.g. ::1).

So you assume that the myproxy-server is actually listening on ::1, too?

from gct.

@ellert:
But:

Unable to connect to ::1:45721 error:0200206F:system library:connect:Connection refused error:2008A067:BIO routines:BIO_connect:connect error error:0200206F:system library:connect:Connection refused hostname=localhost service=45721 error:20073067:BIO routines:conn_state:connect error error:140E0197:SSL routines:SSL_shutdown:shutdown while in init 
Connection refused

...seems to correctly identify the port used (service=45721).

from gct.

There's a subtle difference between myproxy-server --listen localhost and myproxy-server without any --listen option. For better or worse, my thinking at the time when I added IPv6 support was not to enable it by default for fear of breaking things but to require an explicit --listen option for server admins who knew they wanted IPv6 enabled.

Without any --listen option, the myproxy-server binds with AF_INET (and not AF_INET6):

With a --listen option, the myproxy-server binds with AF_UNSPEC so that IPv4 and/or IPv6 will be enabled based on the hostname provided:

I still don't understand why only the 3 "bootstrap" test cases are failing, but I still think changing the myproxy-test script to explicitly pass a myproxy-server --listen localhost option may fix it.

from gct.

@jbasney
Sounds good. We're in a VC now, so please don't mind us not replying for some time.

from gct.

Small update: For me make myproxy-check works through w/o an issue on openSUSE Leap 15.2 with the /etc/hosts configuration mentioned in #160 (comment) (gct at 246f1c3, configured using ./configure --disable-gridftp --disable-gram5 --enable-myproxy):

johndoe@gridftp-5:~/git-projects/gct> make myproxy-check
make[1]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
Making check in web
make[2]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source/web'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source/web'
Making check in systemd
make[2]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source/systemd'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source/systemd'
Making check in man
make[2]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source/man'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source/man'
make[2]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
make  myproxy-test-wrapper
make[3]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
make[3]: Nothing to be done for 'myproxy-test-wrapper'.
make[3]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'
make  check-TESTS
make[3]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
make[4]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
PASS: myproxy-test-wrapper
============================================================================
Testsuite summary for myproxy 6.2.8
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[4]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'
make[3]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'
make[2]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'
make[1]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'

johndoe@gridftp-5:~/git-projects/gct> cat myproxy/source/myproxy-test-wrapper.log
[...]
MyProxy Test 3T (retrieve stored credential w/ trustroots): SUCCEEDED
[...]
MyProxy Test 20T (retrieve stored credential w/ trustroots): SUCCEEDED
[...]
MyProxy Test 41 (retrieve trustroots w/o authentication): SUCCEEDED
[...]
MyProxy Tests Complete: 56 tests passed, 0 tests failed
PASS myproxy-test-wrapper (exit status: 0)

But when checking network connections with watch -n 0.1 "netstat -plantu | grep myproxy" I only ever saw myproxy-server listening to 0.0.0.0 which matches INADDR_ANY from https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L937..L938. Why the clients don't complain for me the same way as on Debian, although localhost points to both 127.0.0.1 and ::1 on that host is the question.

from gct.

Fixed in GCT 6.2.20210826 maintenance release.

from gct.