[debops.elastic_co] Proxy support in Stretch about debops HOT 11 OPEN

A quick search shows this Ansible issue where a proposed solution is to add the environment: key to the apt_key task with specific proxy variables.

DebOps playbooks have global environment: keys included by default, and through that users should be able to affect the http_proxy, https_proxy, etc. environment variables. @scibi, do you have the HTTP proxy configured through the Ansible inventory, specifically inventory__environment variables? You can check the example usage in the documentation.

If this doesn't solve the issue, I think that it should be fixed in the apt_key module itself, rather than jury-rigged on the role/task level. There are blog posts and StackExchange answers mentioning the usage of the apt-key adv --keyserver-options parameter. If use of http_proxy environment variable is not enough, perhaps updating the apt_key module to support this parameter could be a solution. @jvantuyl might be interested.

Another solution would be to set up a local OpenPGP keyserver via debops.sks role (it might need to be refreshed). You can then manually or automatically synchronize the SKS keyserver pool with your local keyserver, and point the DebOps roles to it via the core__keyserver variable. Most of the DebOps roles should use that variable via Ansible local facts to contact your custom OpenPGP keyserver.

From your proposed solutions, the third one (support for the data parameter as an alternative to id) seems like the best one to me. This could be designed as a standardized way to provide GPG keys directly instead of GPG fingerprints. I suppose that an additional empty variable in each role where you can place the GPG key and usage of it with omit as an alternative seems to be doable.

However, can you first check if usage of inventory__environment helps? Or alternatively, what do you think about setting up a local OpenPGP keyserver mirror?

from debops.

Using environmental variables works well but only on Jessie and previous releases. They use gpg 1 which uses libcurl so it takes proxy settings from environment. As I wrote it doesn't work with gpg2. In gpg2 you have to set proxy configuration in dirmngr.conf. And right now it's not possible because apt-key generates gpg configuration on the fly. It can be fixed in apt-key command, apt_key module is just a frontend to apt-key.

Setting own keyserver just to install nodejs on one machine is probably an overkill...

I agree that the third solution is probably the best.

from debops.

For one machine I would just add the key manually, then apt_key module should recognize its existence and not try and download it again. Setting up a local keyserver was meant more in regards to a bigger environment with multiple hosts. And I'm not sure if you want to set GPG keys for each upstream role you want to enable, handling that as a service might be easier in the long run.

Fixing the apt_key module so that for example you can set http_proxy as an optional parameter or better yet use the value of http_proxy environment variable if it's set sounds best to me. You could then put this in the DebOps directory, ansible/library/ which should mask the official apt_key until it's fixed in Ansible. We could even add it to debops-playbooks repository if more people are impacted and are interested in an easy solution.

As for the separate _data variable for each DebOps role... I guess it could be implemented after the merge (currently 9 PRs left). That way it can be handled in all roles at once in one PR rather than going through roles one by one.

from debops.

Ok, lets say that in some environments you are not allowed to install key server ;)

I'm afraid that it can't be fixed in apt_key. It has to be fixed int apt-key command. apt_key just runs apt-key and apt-key creates temporary directory with gpg configuration on every run , runs gpg and removes this directory. You would have to inject some commands in the middle of this...

from debops.

@scibi I don't have a proxified environment to test this in, but you have one already. Can you test something for me?

From the ansible/ansible repository, take the lib/ansible/modules/packaging/os/apt_key.py module and put it somewhere Ansible can find it, say in ansible/library/ in the DebOps project directory. Next, on these lines, change the apt-key commands to look something like this:

    if keyring:
        cmd = "%s --keyring %s adv --keyserver %s --keyserver-options http-proxy=http://localhost:3128 --recv %s" % (apt_key_bin, keyring, keyserver, key_id)
    else:
        cmd = "%s adv --keyserver %s --keyserver-options http-proxy=http://localhost:3128 --recv %s" % (apt_key_bin, keyserver, key_id)

Of course change the address of the proxy to your own HTTP proxy address. Now, check if that makes the apt_key module behave correctly, ie. retrieve the GPG keys you request through the proxy.

from debops.

It's hard to convince you that I've really checked that it can't be done this way ;)

Instead of modifying this module I've run apt-key just like you want:

# apt-key adv --keyserver-options http-proxy=http://proxy-server-name:8080/ --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 46095ACC8548582C1A2699A9D27D666CD88E42B4
Executing: /tmp/apt-key-gpghome.ATMBCQeOYw/gpg.1.sh --keyserver-options http-proxy=http://proxy-server-name:8080/ --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 46095ACC8548582C1A2699A9D27D666CD88E42B4
gpg: keyserver receive failed: No keyserver available

Here are some details about /tmp/apt-key-gpghome.ATMBCQeOYw/:

# ls -la /tmp/apt-key-gpghome.ATMBCQeOYw
total 108
drwx------ 3 root root  4096 Oct 18 21:40 .
drwxrwxrwt 9 root root  4096 Oct 18 21:40 ..
srwx------ 1 root root     0 Oct 18 21:40 S.dirmngr
drwx------ 2 root root  4096 Oct 18 21:40 crls.d
-rw-r--r-- 1 root root     0 Oct 18 21:40 empty.gpg
-rw-r--r-- 1 root root   174 Oct 18 21:40 gpg.0.sh
-rw-r--r-- 1 root root   122 Oct 18 21:40 gpg.1.sh
-rw-r--r-- 1 root root     0 Oct 18 21:40 gpgoutput.log
-rw-r--r-- 1 root root 36941 Oct 18 21:40 pubring.gpg
-rw------- 1 root root    32 Oct 18 21:40 pubring.kbx
-rw-r--r-- 1 root root 36941 Oct 18 21:40 pubring.orig.gpg
-rw------- 1 root root  1200 Oct 18 21:40 trustdb.gpg

# cat /tmp/apt-key-gpghome.ATMBCQeOYw/gpg.1.sh 
#!/bin/sh
exec sh '/tmp/apt-key-gpghome.ATMBCQeOYw/gpg.0.sh' --keyring '/tmp/apt-key-gpghome.ATMBCQeOYw/pubring.gpg' "$@"

# cat /tmp/apt-key-gpghome.ATMBCQeOYw/gpg.0.sh
#!/bin/sh
exec 'gpg' --ignore-time-conflict --no-options --no-default-keyring \
--homedir '/tmp/apt-key-gpghome.ATMBCQeOYw' --no-auto-check-trustdb --trust-model always "$@"

As you can see it's just running gpg command. On Jessie it means it's running GnuPG 1.4. In Stretch it's running 2.1. GnuPG 2.1 uses dirmngr to download keys. It's started as a daemon:

# ps ax | grep dirmngr
 1244 ?        Ssl    0:00 dirmngr --daemon --homedir /tmp/apt-key-gpghome.ATMBCQeOYw

To force it use proxy you have to add --honor-http-proxy or --http-proxy host[:port] option or add them to dirmngr.conf (https://www.gnupg.org/documentation/manuals/gnupg/Dirmngr-Options.html).

And you can't do it without modifying apt-key...

from debops.

It's hard to convince you that I've really checked that it can't be done this way ;)

It's not exactly that. I dislike the scenarios where Ansible is The Solution, instead of a Tool that gets you to The Solution. For example, DNS. Everyone's first idea is to add entries to /etc/hosts via Ansible lineinfile module because it's cheap and easy to do. For me this feels brittle, easy to break, and requires repetition all the time when new hosts are added to the cluster. My solution would be to set up a proper DNS server using Ansible; then instead of messing with /etc/hosts everywhere, you can just point all of the hosts to your DNS server via DHCP. It works and is reliable.

I was hoping that the issue with apt-key behind a proxy would be something that can be dealt with on the Ansible module level, but it seems that required code is not implemented in the package itself - it's an issue in Debian. This means that we need to deal with this for at least 4 years, basically as long as Stretch will be an oldstable release, assuming that this gets fixed in Buster. Which is worrying, because I haven't found a bug report in https://bugs.debian.org/ yet. The closest I got was bug #1433761 in Ubuntu which talks about the same situation as yours. Looking over the Internet, it seems that the preferred solution for people that encounter this is to grab the keys manually and sideload them or embed the GPG keys directly in their installer scripts. That's like beating around the bush for me.

The next step on my things better to do than handling this issue manually would be setting up a keyserver behind the proxy. This is because it solves the issue in one fell swoop - all the roles that deal with the GPG keys can just be pointed to that keyserver today, no need to mess around with the code one role at a time, it works on all your hosts in the cluster in a standardized way. If your security policy disallows it, the keyserver doesn't even need to be connected to the outside world - you can just download the whole OpenPGP keyring from SKS Keyservers (I think about 5 GB the last time I checked a few years ago), import it in your keyserver behind a proxy, and you're good to go. Coincidentally, I haven't created the debops.sks role for this exact purpose - I needed it to test usage of the Monkeysphere Project with my servers. But it seems that this solution fits nicely in your case as well.

If your policy disallows usage of a local keyserver, well - I guess its time to at least reconsider changing the policy - they shouldn't be set in stone. if it's for practical purposes, ie. you are setting up just one host with this issue, then I guess a local keyserver is an overkill. This means that we need to add these variables with key data to each role that handles it. I thought about reusing the _key_id variables as _key_data variables if the length of the contents is longer than an ID length, but that solution seems to be unnecessarily complicated.

Of course I'm wildly aware that this needs to be done in all of the roles, not just debops.elastic_co. This issue can affect multiple people, each one trying to install different upstream software behind a proxy. Since this issue will stay with us for at least 4+ years, we can just as well add this to the DebOps Code Standards Policy, so that future role creators are aware of the issue and how to deal with it. An alternative to using IDs would be to start using GPG keys directly, embedded in the role - I know that at least debops.owncloud uses this solution to distribute the keys with the role. This was done because GPG keys to the ownCloud APT repositories were unavailable through the keyserver network. This breaks the principle of using a third party for authentication, in this case the first party being essentially DebOps and second party the upstream repository. Well, maybe "breaks" is a strong word here - the GPG keys embedded in the role and distributed via a keyserver are the same - but still it sounds as a worse idea than using key IDs.

So, getting back to the issue at hand - how did you deal with it? I suppose that we could start creating PRs to ifx the issue in relevant roles, or wait for the repository merge to deal with it as a whole a bit later.

from debops.

What I did? My solution was quick and dirty ;) I've added elastic_co__key_data to defaults and changed tasks/main.yml:

 - name: Get Elastic APT GPG key
   apt_key:
-    id: '{{ elastic_co__key_id | replace(" ","") }}'
-    keyserver: '{{ ansible_local.core.keyserver
-                   if (ansible_local|d() and ansible_local.core|d() and
-                       ansible_local.core.keyserver)
-                   else "hkp://pool.sks-keyservers.net" }}'
+    data: '{{ elastic_co__key_data }}'
     state: 'present'

Unfortunately sometimes it's not your decision what is allowed on the network and what is not. Even if its stupid you have to deal with that.

from debops.

I understand, politics. :-)

BTW, I've got another solution for you. Since, most likely, you run the entire DebOps playbook on these hosts, if you add something like this to ansible/inventory/group_vars/all/apt.yml:

apt__keys:
  - keyring: '/etc/apt/trusted.gpg.d/elastic_co.gpg'
    data: '<Elastic GPG key>'

Then when you run the debops.apt role against that host, it should add the GPG key for Elastic repositories. After that, the apt_key task in debops.elastic_co should accept that the key is present and proceed as normal. If not, check what file is the key stored in normally, and use its filename. Let me know if that works, if you try it next time.

from debops.

This issue should be solved by the keyring role which allows different ways of handling GPG keys. @scibi, anything changed since this issue was created?

from debops.

DebOps now favors using extrepo for this. Elastic is supported by https://salsa.debian.org/extrepo-team/extrepo-data/-/blob/master/repos/debian/elastic.yaml. Proxying for extrepo needs to be looked into.

from debops.