Just want to point out to any readers of this thread that storing PATs in .env/plain text files or environment variables is a security risk and should be avoided as described in the workaround in the opening post: once attackers do get access to your system, your environment variables and .env files will be among the first things they steal from you as they are likely to contain secrets.

from artifacts-keyring.

Here's our approach on Linux and macOS. It is not dependent on Microsoft's keyring tool but instead relies on the OS keyring, meaning you don't need to install anything into your system python environment.

It's much more involved on Windows since there aren't any native keyring CLI tools available, we wrote a library in C# and PowerShell and I would rather not share it.

Linux Setup

Install prerequisites

• We'll need the command line tool secret-tool to access the OS keyring from CLI so install it with apt install libsecret-tools

Configure your shell

• Copy dev.sh (see below) to a folder you like and make it executable with chmod +x dev.sh
• Add to your shell's aliases (e.g. ~/.bash_aliases) the following line and
  customize the path: alias env-dev='env $(~/PATH/TO/dev.sh)'
• Reload your shell for the alias to become available (e.g. . ~/.bashrc)

dev.sh:

#!/bin/bash
TOKEN=$(secret-tool lookup secret MY_PAT)
echo MY_PAT=$TOKEN

Configure the credential

• Create a Personal Access Token (PAT) on Azure DevOps and give it Packaging -
  Read permission
  • You can do that here: https://dev.azure.com/<organization>/_usersSettings/tokens
• Add the key to your keyring under the name MY_PAT and set the
  password (username is irrelevant but cannot be blank; can be anything) using
  the secret-tool command line: secret-tool store --label=MY_PAT secret MY_PAT

Make sure to refresh the PAT when it expires!

MacOS Setup

MacOS ships with a keyring CLI tool security built-in so there are no prerequisites to install. You also don't need to add the secret via the CLI tool, you can just use the OS keyring UI. Otherwise the setup is the same. Here's dev.sh for macOS:

#!/bin/bash
TOKEN=$(security find-generic-password -s MY_PAT -w)
echo MY_PAT=$TOKEN

Usage

Now you can prefix any command with env-dev to expose MY_PAT as an environment variable just for the duration of that command. Typically that will only be necessary for commands such as pipenv lock, pipenv install and pipenv sync. So for example: env-dev pipenv lock is the command you would run.

from artifacts-keyring.

@rinman24 My explanation wasn't particularly clear and I see now how it reads that way.

If you already have a password for a given username for that website already stored in the Windows Credential Store that way then yeah it returns the password.

I used the website as a meaningful key; although the username could be anything like artifact_pat, and then the credential that is stored, is the PAT.

Then exporting it from the credential store as an environment variable allows for pipenv to perform string interpolation.

So if we export the PAT to ARTIFACTS_TOKEN environment variable we can add the following to our Pipfile:

[[source]]
url = "https://azure:${ARTIFACTS_TOKEN}@pkgs.dev.azure.com/<organization>/_packaging/<feed>/pypi/simple"
verify_ssl = true
name = "artifacts"

Whilst in a pipenv shell this ARTIFACTS_TOKEN must be available for the URL to resolve.

Then pipenv lock can extract the version information it needs about all packages to start the lock resolving algorithm.

Once versions are resolved, then and only then do packages get downloaded.

Running any of sync, update and install will actually check the lock status as far as I know. So if you can't lock because the URL doesn't contain the PAT, pipenv gets stuck.

References

• https://pipenv-fork.readthedocs.io/en/latest/advanced.html#injecting-credentials-into-pipfiles-via-environment-variables
• https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=preview-page

from artifacts-keyring.

ok interestingly easy:

Set Credentials
Run the CLI tool to install the creds into the Windows Credential Manager

python -m keyring set 'pkgs.dev.azure.com/myorganization' username
Password for 'username' in 'pkgs.dev.azure.com/myorganization':

Fetch Credentials
then the respective helper script to fetch the credentials from keyring

with-artifacts.sh

PAT=$(python -m keyring get 'pkgs.dev.azure.com/myorganization' username)

or with-artifacts.bat

FOR /F "tokens=* USEBACKQ" %%g in (`python -m keyring get pkgs.dev.azure.com/myorganization username`) do (SET "PAT=%g")

Reference:

• Stackoverflow: How to set commands output as a variable in a batch file

from artifacts-keyring.

Thanks for the input @neozenith. However, python -m keyring get pkgs.dev.azure.com/<organization> <username> returns the azure password and not the artifacts PAT. I don't see how this can be used to generate a valid url for pipenv. Also, the solution @Korijn proposed won't work in my case because my organization uses Windows machines for developers.

It does work as I mentioned before, the script required for Windows is just a lot bigger so I didn't feel like sharing it earlier. I figured I could use a gist though, because it's so big. Hope it's useful for you all. https://gist.github.com/Korijn/6b0e444e7c131680dacdd397f8574c50

from artifacts-keyring.

This workaround doesn't seem to work for me anymore (PAT in environment variable). If I try the index URL directly with curl the response is 404 with a very strange message in the body:
This functionality is currently not available.
Example:
curl "https://{pat}@pkgs.dev.azure.com/{org}/_packaging/{feed}/pypi/simple"

from artifacts-keyring.

We have had new devs that just started and the new UI workflow removed the step where they could easily generate a PAT.

Using keyring and artifacts-keyring with pipenv breaks.

We already have to run the following as separate steps:

pipenv shell
pipenv install

This loads in our .env that loads in our PAT so our Azure Artifacts URL will be valid for that source.

With pipenv it needs to be able to resolve our Azure Artifacts URL to extract dependency information.

But it can't do that if artifacts-keyring needs to already be installed.

Since the dependencies can not be resolved, a lock file can't be created and nothing gets installed.

The following documentation has screenshots on where the Personal Access Tokens can still be found and generated for the older workflow.

Microsoft Docs: use personal access tokens to authenticate

Could the Artifacts Pip instructions, link to this as well as the artifacts-keyring option when you get the details in the Get Tools section?

Or even a doc page for Are you using pipenv? being linked in the Get Tools section.

from artifacts-keyring.

Thanks for pointing that out @Korijn! No one wants to be the person that left the front door key under the mat.

Just so I'm clear (and we can turn this into a learning opportunity for myself and others) what would an example of with-artifacts.sh look like?

I'm presuming that you have pipenv, keyring and artifacts-keyring installed as tools in your system/user python?

You then have the with-artifacts.sh script as a boilerplate helper script in each repo?

from artifacts-keyring.

Thanks for the input @neozenith. However, python -m keyring get pkgs.dev.azure.com/<organization> <username> returns the azure password and not the artifacts PAT. I don't see how this can be used to generate a valid url for pipenv. Also, the solution @Korijn proposed won't work in my case because my organization uses Windows machines for developers.

from artifacts-keyring.

Thanks @neozenith and @Korijn. Both very helpful!

from artifacts-keyring.

You might be dealing with the new project scoped feeds which also have a {project} part in the URL...

from artifacts-keyring.

Oh wow... it's fine now. I was running an old version of pipenv on this machine. 🤦‍♂️

from artifacts-keyring.

@LTArnas I'm glad you figured it out. For reference, we are currently still using this workaround without any issues.

from artifacts-keyring.

Has anyone else experienced this workaround failing recently?

from artifacts-keyring.

Has anyone else experienced this workaround failing recently?

Not yet...

from artifacts-keyring.

For the sake of joined-up-ness, note that I think this issue is the same as pypa/pipenv#4074.

Unless more qualified people than me think differently, I believe this is really pipenv's problem to solve, not Microsoft's.

from artifacts-keyring.

Hey,

Closing due to inactivity, please reopen this if it is still an issue.

from artifacts-keyring.

I managed to get this working today =]

Be sure to replace the url:

#powershell
python -m pip install --user azure_devops_artifacts_helpers artifacts-keyring
[Environment]::SetEnvironmentVariable("VIRTUALENV_SEEDER", "azdo-pip", "Machine")
[Environment]::SetEnvironmentVariable("PIP_INDEX_URL", "https://pkgs.dev.azure.com/XXXXX/_packaging/YYYYY/pypi/simple/", "Machine")

Open a new powershell/cmd window so those environment variables get loaded and pipenv install should work. It does for me at least, let me now if it doesn't and I'll try to figure out wat else I did during all the trial and error.

Edit: our Pipfile contains the following:

[[source]]
url =  "${PIP_INDEX_URL}"
name = "azdo"

from artifacts-keyring.

@Darrstar

I managed to get this working today =]

Be sure to replace the url:

#powershell
python -m pip install --user azure_devops_artifacts_helpers artifacts-keyring
[Environment]::SetEnvironmentVariable("VIRTUALENV_SEEDER", "azdo-pip", "Machine")
[Environment]::SetEnvironmentVariable("PIP_INDEX_URL", "https://pkgs.dev.azure.com/XXXXX/_packaging/YYYYY/pypi/simple/", "Machine")

Open a new powershell/cmd window so those environment variables get loaded and pipenv install should work. It does for me at least, let me now if it doesn't and I'll try to figure out wat else I did during all the trial and error.

Edit: our Pipfile contains the following:

[[source]]
url =  "${PIP_INDEX_URL}"
name = "azdo"

I end up with this error message still. Looks like PIP_INDEX_URL isnt being converted?

from artifacts-keyring.