Comments (11)
A quick search shows this Ansible issue where a proposed solution is to add the environment:
key to the apt_key
task with specific proxy variables.
DebOps playbooks have global environment:
keys included by default, and through that users should be able to affect the http_proxy
, https_proxy
, etc. environment variables. @scibi, do you have the HTTP proxy configured through the Ansible inventory, specifically inventory__environment
variables? You can check the example usage in the documentation.
If this doesn't solve the issue, I think that it should be fixed in the apt_key
module itself, rather than jury-rigged on the role/task level. There are blog posts and StackExchange answers mentioning the usage of the apt-key adv --keyserver-options
parameter. If use of http_proxy
environment variable is not enough, perhaps updating the apt_key
module to support this parameter could be a solution. @jvantuyl might be interested.
Another solution would be to set up a local OpenPGP keyserver via debops.sks
role (it might need to be refreshed). You can then manually or automatically synchronize the SKS keyserver pool with your local keyserver, and point the DebOps roles to it via the core__keyserver
variable. Most of the DebOps roles should use that variable via Ansible local facts to contact your custom OpenPGP keyserver.
From your proposed solutions, the third one (support for the data
parameter as an alternative to id
) seems like the best one to me. This could be designed as a standardized way to provide GPG keys directly instead of GPG fingerprints. I suppose that an additional empty variable in each role where you can place the GPG key and usage of it with omit
as an alternative seems to be doable.
However, can you first check if usage of inventory__environment
helps? Or alternatively, what do you think about setting up a local OpenPGP keyserver mirror?
from debops.
Using environmental variables works well but only on Jessie and previous releases. They use gpg 1 which uses libcurl
so it takes proxy settings from environment. As I wrote it doesn't work with gpg2. In gpg2 you have to set proxy configuration in dirmngr.conf
. And right now it's not possible because apt-key
generates gpg configuration on the fly. It can be fixed in apt-key
command, apt_key
module is just a frontend to apt-key
.
Setting own keyserver just to install nodejs
on one machine is probably an overkill...
I agree that the third solution is probably the best.
from debops.
For one machine I would just add the key manually, then apt_key
module should recognize its existence and not try and download it again. Setting up a local keyserver was meant more in regards to a bigger environment with multiple hosts. And I'm not sure if you want to set GPG keys for each upstream role you want to enable, handling that as a service might be easier in the long run.
Fixing the apt_key
module so that for example you can set http_proxy
as an optional parameter or better yet use the value of http_proxy
environment variable if it's set sounds best to me. You could then put this in the DebOps directory, ansible/library/
which should mask the official apt_key
until it's fixed in Ansible. We could even add it to debops-playbooks
repository if more people are impacted and are interested in an easy solution.
As for the separate _data
variable for each DebOps role... I guess it could be implemented after the merge (currently 9 PRs left). That way it can be handled in all roles at once in one PR rather than going through roles one by one.
from debops.
Ok, lets say that in some environments you are not allowed to install key server ;)
I'm afraid that it can't be fixed in apt_key
. It has to be fixed int apt-key
command. apt_key
just runs apt-key
and apt-key
creates temporary directory with gpg configuration on every run , runs gpg and removes this directory. You would have to inject some commands in the middle of this...
from debops.
@scibi I don't have a proxified environment to test this in, but you have one already. Can you test something for me?
From the ansible/ansible
repository, take the lib/ansible/modules/packaging/os/apt_key.py
module and put it somewhere Ansible can find it, say in ansible/library/
in the DebOps project directory. Next, on these lines, change the apt-key
commands to look something like this:
if keyring:
cmd = "%s --keyring %s adv --keyserver %s --keyserver-options http-proxy=http://localhost:3128 --recv %s" % (apt_key_bin, keyring, keyserver, key_id)
else:
cmd = "%s adv --keyserver %s --keyserver-options http-proxy=http://localhost:3128 --recv %s" % (apt_key_bin, keyserver, key_id)
Of course change the address of the proxy to your own HTTP proxy address. Now, check if that makes the apt_key
module behave correctly, ie. retrieve the GPG keys you request through the proxy.
from debops.
It's hard to convince you that I've really checked that it can't be done this way ;)
Instead of modifying this module I've run apt-key
just like you want:
# apt-key adv --keyserver-options http-proxy=http://proxy-server-name:8080/ --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 46095ACC8548582C1A2699A9D27D666CD88E42B4
Executing: /tmp/apt-key-gpghome.ATMBCQeOYw/gpg.1.sh --keyserver-options http-proxy=http://proxy-server-name:8080/ --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 46095ACC8548582C1A2699A9D27D666CD88E42B4
gpg: keyserver receive failed: No keyserver available
Here are some details about /tmp/apt-key-gpghome.ATMBCQeOYw/
:
# ls -la /tmp/apt-key-gpghome.ATMBCQeOYw
total 108
drwx------ 3 root root 4096 Oct 18 21:40 .
drwxrwxrwt 9 root root 4096 Oct 18 21:40 ..
srwx------ 1 root root 0 Oct 18 21:40 S.dirmngr
drwx------ 2 root root 4096 Oct 18 21:40 crls.d
-rw-r--r-- 1 root root 0 Oct 18 21:40 empty.gpg
-rw-r--r-- 1 root root 174 Oct 18 21:40 gpg.0.sh
-rw-r--r-- 1 root root 122 Oct 18 21:40 gpg.1.sh
-rw-r--r-- 1 root root 0 Oct 18 21:40 gpgoutput.log
-rw-r--r-- 1 root root 36941 Oct 18 21:40 pubring.gpg
-rw------- 1 root root 32 Oct 18 21:40 pubring.kbx
-rw-r--r-- 1 root root 36941 Oct 18 21:40 pubring.orig.gpg
-rw------- 1 root root 1200 Oct 18 21:40 trustdb.gpg
# cat /tmp/apt-key-gpghome.ATMBCQeOYw/gpg.1.sh
#!/bin/sh
exec sh '/tmp/apt-key-gpghome.ATMBCQeOYw/gpg.0.sh' --keyring '/tmp/apt-key-gpghome.ATMBCQeOYw/pubring.gpg' "$@"
# cat /tmp/apt-key-gpghome.ATMBCQeOYw/gpg.0.sh
#!/bin/sh
exec 'gpg' --ignore-time-conflict --no-options --no-default-keyring \
--homedir '/tmp/apt-key-gpghome.ATMBCQeOYw' --no-auto-check-trustdb --trust-model always "$@"
As you can see it's just running gpg
command. On Jessie it means it's running GnuPG 1.4. In Stretch it's running 2.1. GnuPG 2.1 uses dirmngr
to download keys. It's started as a daemon:
# ps ax | grep dirmngr
1244 ? Ssl 0:00 dirmngr --daemon --homedir /tmp/apt-key-gpghome.ATMBCQeOYw
To force it use proxy you have to add --honor-http-proxy
or --http-proxy host[:port]
option or add them to dirmngr.conf
(https://www.gnupg.org/documentation/manuals/gnupg/Dirmngr-Options.html).
And you can't do it without modifying apt-key
...
from debops.
It's hard to convince you that I've really checked that it can't be done this way ;)
It's not exactly that. I dislike the scenarios where Ansible is The Solution, instead of a Tool that gets you to The Solution. For example, DNS. Everyone's first idea is to add entries to /etc/hosts
via Ansible lineinfile
module because it's cheap and easy to do. For me this feels brittle, easy to break, and requires repetition all the time when new hosts are added to the cluster. My solution would be to set up a proper DNS server using Ansible; then instead of messing with /etc/hosts
everywhere, you can just point all of the hosts to your DNS server via DHCP. It works and is reliable.
I was hoping that the issue with apt-key
behind a proxy would be something that can be dealt with on the Ansible module level, but it seems that required code is not implemented in the package itself - it's an issue in Debian. This means that we need to deal with this for at least 4 years, basically as long as Stretch will be an oldstable
release, assuming that this gets fixed in Buster. Which is worrying, because I haven't found a bug report in https://bugs.debian.org/ yet. The closest I got was bug #1433761 in Ubuntu which talks about the same situation as yours. Looking over the Internet, it seems that the preferred solution for people that encounter this is to grab the keys manually and sideload them or embed the GPG keys directly in their installer scripts. That's like beating around the bush for me.
The next step on my things better to do than handling this issue manually would be setting up a keyserver behind the proxy. This is because it solves the issue in one fell swoop - all the roles that deal with the GPG keys can just be pointed to that keyserver today, no need to mess around with the code one role at a time, it works on all your hosts in the cluster in a standardized way. If your security policy disallows it, the keyserver doesn't even need to be connected to the outside world - you can just download the whole OpenPGP keyring from SKS Keyservers (I think about 5 GB the last time I checked a few years ago), import it in your keyserver behind a proxy, and you're good to go. Coincidentally, I haven't created the debops.sks
role for this exact purpose - I needed it to test usage of the Monkeysphere Project with my servers. But it seems that this solution fits nicely in your case as well.
If your policy disallows usage of a local keyserver, well - I guess its time to at least reconsider changing the policy - they shouldn't be set in stone. if it's for practical purposes, ie. you are setting up just one host with this issue, then I guess a local keyserver is an overkill. This means that we need to add these variables with key data to each role that handles it. I thought about reusing the _key_id
variables as _key_data
variables if the length of the contents is longer than an ID length, but that solution seems to be unnecessarily complicated.
Of course I'm wildly aware that this needs to be done in all of the roles, not just debops.elastic_co
. This issue can affect multiple people, each one trying to install different upstream software behind a proxy. Since this issue will stay with us for at least 4+ years, we can just as well add this to the DebOps Code Standards Policy, so that future role creators are aware of the issue and how to deal with it. An alternative to using IDs would be to start using GPG keys directly, embedded in the role - I know that at least debops.owncloud
uses this solution to distribute the keys with the role. This was done because GPG keys to the ownCloud APT repositories were unavailable through the keyserver network. This breaks the principle of using a third party for authentication, in this case the first party being essentially DebOps and second party the upstream repository. Well, maybe "breaks" is a strong word here - the GPG keys embedded in the role and distributed via a keyserver are the same - but still it sounds as a worse idea than using key IDs.
So, getting back to the issue at hand - how did you deal with it? I suppose that we could start creating PRs to ifx the issue in relevant roles, or wait for the repository merge to deal with it as a whole a bit later.
from debops.
What I did? My solution was quick and dirty ;) I've added elastic_co__key_data
to defaults and changed tasks/main.yml:
- name: Get Elastic APT GPG key
apt_key:
- id: '{{ elastic_co__key_id | replace(" ","") }}'
- keyserver: '{{ ansible_local.core.keyserver
- if (ansible_local|d() and ansible_local.core|d() and
- ansible_local.core.keyserver)
- else "hkp://pool.sks-keyservers.net" }}'
+ data: '{{ elastic_co__key_data }}'
state: 'present'
Unfortunately sometimes it's not your decision what is allowed on the network and what is not. Even if its stupid you have to deal with that.
from debops.
I understand, politics. :-)
BTW, I've got another solution for you. Since, most likely, you run the entire DebOps playbook on these hosts, if you add something like this to ansible/inventory/group_vars/all/apt.yml
:
apt__keys:
- keyring: '/etc/apt/trusted.gpg.d/elastic_co.gpg'
data: '<Elastic GPG key>'
Then when you run the debops.apt
role against that host, it should add the GPG key for Elastic repositories. After that, the apt_key
task in debops.elastic_co
should accept that the key is present and proceed as normal. If not, check what file is the key stored in normally, and use its filename. Let me know if that works, if you try it next time.
from debops.
This issue should be solved by the keyring
role which allows different ways of handling GPG keys. @scibi, anything changed since this issue was created?
from debops.
DebOps now favors using extrepo for this. Elastic is supported by https://salsa.debian.org/extrepo-team/extrepo-data/-/blob/master/repos/debian/elastic.yaml. Proxying for extrepo needs to be looked into.
from debops.
Related Issues (20)
- DOC: Broken links to https://docs.debops.org/en/latest/debops-keyring/docs/index.html
- Secret is undefined HOT 5
- ubuntu noble removes /etc/apt/sources.list file HOT 6
- debops.debops.kmod: Unload kernel module fails at the first time HOT 1
- Swap file deletion HOT 2
- extrepo: Why not for Ubuntu, only for Debian? HOT 2
- Nginx: Tag 'role::nginx:servers' no more applied? HOT 1
- Add a authorized_keys for a user that is not local on host HOT 2
- Dropbear initramfs : Renamed config path from /etc/dropbear-initramfs to /etc/dropbear/initramfs
- PKI: How to manually sign a CSR?
- PKI: ACME request not signed by Let's Encrypt but by our internal CA? HOT 2
- rootless docker support
- APT: the apt__repositories item signed_by doesn't work
- unattended_upgrades__mail_to does not work HOT 1
- debops.debops.apt depends on gpg
- [miniflux] Can not create PostgreSQL roles with become HOT 5
- Initial run of rsyslog playbook fails HOT 1
- rsnapshot: package module should use the "use" parameter
- Some roles not included in ansible galaxy rollout
- Consider using `community.crypto.luks_device` in `cryptsetup`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from debops.