Describe the bug
Looks like Azure put their python package index behind an authorization mechanisms that redirects you through a login portal in such a way that this package gets confused. Pip seems to handle this scheme correctly.

unearth: Collecting links from https://pypi.org/simple/some-package/
unearth: Found index url https://pypi.org/simple
unearth: Skip https://pypi.org/simple/some-package/ because of Client Error(404): Not Found.
unearth: Collecting links from https://***@pkgs.dev.azure.com/myorg/core/_packaging/cp/pypi/simple/some-package/
unearth: Found index url https://[email protected]/myorg/core/_packaging/cp/pypi/simple/
unearth: Skip link <Link https://spsprodcca1.vssps.visualstudio.com/go/profile (from https://spsprodcca1.vssps.visualstudio.com/_signin?realm=pkgs.dev.azure.com&reply_to=https%3A%2F%2Fpkgs.dev.azure.com%myorg%2Fcore%2F_packaging%2Fcp%2Fpypi%2Fsimple%some-package%2F&redirect=1&hid=e8463d24-ca16-4e07-aeb8-7434d592857c&context=eyJodCI6MiwiaGlkIjoiOThkZWFmYjYtYmYzZC00YTE5LWFjMjQtZDk0YzY5NjhjYzYyIiwicXMiOnt9LCJyciI6IiIsInZoIjoiIiwiY3YiOiIiLCJjcyI6IiJ90)>: Not a file: profile
unearth: Skip link <Link https://spsprodcca1.vssps.visualstudio.com/_signout (from https://spsprodcca1.vssps.visualstudio.com/_signin?realm=pkgs.dev.azure.com&reply_to=https%3A%2F%2Fpkgs.dev.azure.com%myorg%2Fcore%2F_packaging%2Fcp%2Fpypi%2Fsimple%some-package%2F&redirect=1&hid=e8463d24-ca16-4e07-aeb8-7434d592857c&context=eyJodCI6MiwiaGlkIjoiOThkZWFmYjYtYmYzZC00YTE5LWFjMjQtZDk0YzY5NjhjYzYyIiwicXMiOnt9LCJyciI6IiIsInZoIjoiIiwiY3YiOiIiLCJjcyI6IiJ90)>: Not a file: _signout

To Reproduce
Add a source entry like this in pdm:

[[tool.pdm.source]]
url = "https://${TOKEN}@pkgs.dev.azure.com/coherentpath/core/_packaging/cp/pypi/simple/"
verify_ssl = true
name = "azure"

And then adding a package provided by said repository.

System (please complete the following information):

• unearth version: master
• Python version: 3.9
• OS: macOS

Link: https://peps.python.org/pep-0691/

Is your feature request related to a problem? Please describe.
I would like to automatically fetch dependnecies that we use and see when was specific version released.
I am trying to create a rule that we should upgrade 1 month after dependency is released.

Describe the solution you'd like
Package objekt should contain release date for this version.
packagefinder could have a switch fetch_metadata if this is expesnive operation.

pypi = PackageFinder(index_urls=["https://pypi.org/simple/"], fetch_metadata=True)
result = pypi.find_best_match("django==4.2.7")
print(result.best.released_date)

Describe alternatives you've considered
Fetching raw html and parse the output.

Additional context
/

Hi! This is cool. Thanks for building it!

I'm sorry if I'm missing it somewhere, but I can't seem to figure out how to get the following behavior:

I'd like for PackageFinder to find_matches from only non-pre-release candidates found from https://pypi.org/simple/.

import json
from unearth.finder import PackageFinder

finder = PackageFinder(
    index_urls=["https://pypi.org/simple/"],
)
finder.target_python.py_ver = (3, 11, 1)
m = finder.find_matches("pydantic", allow_prereleases=False)
print(json.dumps(m[0].as_json()))
# {"name": "pydantic", "version": "1.10.7", ...

Instead of 1.10.7 I get 2.0a3.

Is unearth not supposed to constrain the found packages to only pre-releases as well? I'd think passing allow_prereleases=False would limit both the supplied requirement and the found packages to non-pre-releases.

I hacked this together here.

Is your feature request related to a problem? Please describe.
When working in an environment without direct internet access, it's often hard to recognize why the program hangs indefinitely.

Describe the solution you'd like
I would like the call to timeout after a short period.

Describe alternatives you've considered

Additional context
https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
Two values for timeouts are possible with requests : connect and read. Maybe we could keep the connect to a short value like 5s and the read to longer like 30s, 60s or even None.

Timeouts can be implemented on each session.get() calls or can be implemented at the session level directly with an adapter : https://stackoverflow.com/a/62044100/6507969. This method would have the advantage of setting default timeout for other projects using PyPISession directly.

Describe the bug
PyPI (pypi.org) gets added as an index even though it's overwritten in pyproject.toml using the pypi name in unearth>=0.13

I guess it could be caused by 7d1c215 (although I don't really see how it's possible looking at the source code)...

Version 0.12.1 works fine and does not have this issue.

I'm not sure if this is a pdm or an unearth issue.

To Reproduce

[[tool.pdm.source]]
name = "pypi"
url = "https://<some_custom_artifactory_url>/artifactory/api/pypi/pypi-all/simple"
verify_ssl = true
include_packages = ["*"]

pdm install -vv --prod --no-lock --check --no-editable --no-self --fail-fast

Now the log of install looks like:

18:06:22  #12 2.285 Run command: ['/usr/local/bin/python3.11', '-m', 'virtualenv', '/<package>/.venv', '-p', '/usr/local/bin/python', '--prompt=<package>-3.11', '--no-pip', '--no-setuptools', '--no-wheel']
18:06:22  #12 2.****78 created virtual environment CPython3.11.6.final.0-6**** in 89ms
18:06:22  #12 2.****78   creator CPython3Posix(dest=/<package>/.venv, clear=False, no_vcs_ignore=False, global=False)
18:06:22  #12 2.****78   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
18:06:22  #12 2.****99 Virtualenv is created successfully at /<package>/.venv
18:06:32  #12 11.62 STATUS: Fetching hashes for resolved packages...
18:06:32  #12 11.67 Synchronizing working set with resolved packages: 100 to add, 0 to update, 0 to remove
18:06:32  #12 11.67 
18:06:32  #12 11.88 unearth.collector: Collecting links from https://pypi.org/simple/anyio/
... (repeats many times for different packages)
18:06:32  #12 11.89 unearth.auth: Found credentials in index url for pypi.org
18:06:33  #12 13.00 unearth.collector: Fetching HTML page https://pypi.org/simple/annotated-types/
18:06:33  #12 13.00 unearth.collector: Collecting links from https://<some_custom_artifactory>/artifactory/api/pypi/pypi-all/simple/annotated-types/
... (artifactory links are mixed in for the packages too, but in the end downloads from pypi are preferred)

Expected behavior
The only index should be the some_custom_artifactory_url. But instead, both PyPI and this custom index are used.

It also looks like PyPI is preferred, and the custom artifactory is used iff a package is not found in PyPI.

System (please complete the following information):

• unearth version: >0.12.1
• Python version: 3.11
• OS: Linux
• pdm version: 2.11.1

pdm is installed using these steps from https://pdm-project.org/latest/usage/advanced/#use-pdm-in-a-multi-stage-dockerfile

When installing like this, this issue appears:

pip install -U pip setuptools wheel
pip install pdm

And the issue disappears if I fix the unearth version:

pip install -U pip setuptools wheel
pip install pdm unearth==0.12.1

Additional context
Will try to provide more details if necessary. It's a bit difficult to provide an MRE in this case as there any many components at play. In fact, this issue appears inside a Docker build job running inside a Jenkins pipeline.

Describe the bug
I use PDM for Python package management and Google Artifact Registry (GAR) as a private registry and PyPi mirror using the following pyproject.toml config:

[[tool.pdm.source]]
url = "https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple"
verify_ssl = true
name = "pypi"

I'd like to use keyring and keyrings.google-artifactregistry-auth to authenticate to GAR. I can enable this by installing these packages into the PDM environment by running:

pdm self add keyring keyrings.google-artifactregistry-auth

and authentication then works:

unearth.collector: Collecting links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Getting credentials from keyring for url: https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Found credentials in keyring for europe-west4-python.pkg.dev
unearth.collector: Fetching HTML page https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/

However, when using a system-installed copy of keyring and keyrings.google-artifactregistry-auth, authentication fails:

unearth.collector: Collecting links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Getting password from keyring CLI for __token__@https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Found credentials in keyring for europe-west4-python.pkg.dev
unearth.auth: 401 Error, Credentials not correct for https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
unearth.collector: Failed to collect links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/: Client Error(401): Unauthorized

I believe this is the result of an incorrect user (__token__) being set here:

The keyring module returns the correct user (oauth2accesstoken) however:

Python 3.11.7 (main, Jan  2 2024, 11:13:25) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import keyring
>>> creds = keyring.get_credential("http://europe-west4.python.pkg.dev", None)
>>> creds.username
'oauth2accesstoken'

To Reproduce
In an environment with keyring and keyrings.google-artifactregistry-auth available globally, run:

python -m unearth --verbose --index-url 'https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/' --trusted-host 'europe-west4-python.pkg.dev' setuptools

Output:

DEBUG: Collecting links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
DEBUG: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
DEBUG: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
DEBUG: Getting password from keyring CLI for __token__@https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
DEBUG: Found credentials in keyring for europe-west4-python.pkg.dev
WARNING: 401 Error, Credentials not correct for https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
WARNING: Failed to collect links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/: Client Error(401): Unauthorized
No matches are found.

Install keyring and keyrings.google-artifactregistry-auth, .e.g.: pdm add keyring keyrings.google-artifactregistry-auth and run python -m unearth --verbose --index-url 'https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/' --trusted-host 'europe-west4-python.pkg.dev' setuptools again.

Output:

DEBUG: Collecting links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
DEBUG: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
DEBUG: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
DEBUG: Getting credentials from keyring for url: https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
DEBUG: Found credentials in keyring for europe-west4-python.pkg.dev
DEBUG: Fetching HTML page https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/

Expected behavior

Both keyring as module and as standalone CLI being able to authenticate to GAR.

System (please complete the following information):

• unearth version: 0.14.0
• Python version: 3.11.7
• OS: macOS 13.6.4

Additional context
Related to failing to run PDM as part of Renovate, see renovatebot/renovate#27454, where keyring and keyrings.google-artifactregistry-auth cannot be easily injected into the PDM environment, but only installed globally.

Is it possible to get links for platforms other than the one I'm running on? For example, I'm on MacOS but would like the links to a linux package. The --all flag returns "Return all applicable versions" but I'd like a way to get all platforms.

For example unearth --all pandas==1.5.1 returns:

macosx_11_0_arm64.whl
macosx_10_9_universal2.whl
pandas-1.5.1.tar.gz

but, I'd like to also have a way to get the windows package "win_amd64.whl". Is that possible somehow? Maybe in the API somewhere?

First of all currently it is not possible to use straight cleanly sphinx-build command to build documentation out of source tree

+ /usr/bin/sphinx-build -n -T -b man docs build/sphinx/man
Running Sphinx v5.0.2
making output directory... done
WARNING: html_static_path entry '_static' does not exist
myst v0.17.2: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions=[], linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area', disable_syntax=[], all_links_external=False, url_schemes=('http', 'https', 'mailto', 'ftp'), ref_domains=None, highlight_code_blocks=True, number_code_blocks=[], title_to_header=False, heading_anchors=None, heading_slug_func=None, footnote_transition=True, sub_delimiters=('{', '}'), words_per_minute=200)
building [mo]: targets for 0 po files that are out of date
building [man]: all manpages
updating environment: [new config] 7 added, 0 changed, 0 removed
reading sources... [100%] requirements
WARNING: autodoc: failed to import module 'evaluator' from module 'unearth'; the following exception was raised:
No module named 'unearth'
WARNING: autodoc: failed to import class 'evaluator.Evaluator' from module 'unearth'; the following exception was raised:
No module named 'unearth'
WARNING: autodoc: failed to import class 'evaluator.Package' from module 'unearth'; the following exception was raised:
No module named 'unearth'
WARNING: autodoc: failed to import class 'evaluator.TargetPython' from module 'unearth'; the following exception was raised:
No module named 'unearth'
WARNING: autodoc: failed to import function 'evaluator.evaluate_package' from module 'unearth'; the following exception was raised:
No module named 'unearth'
WARNING: autodoc: failed to import module 'finder' from module 'unearth'; the following exception was raised:
No module named 'unearth'
WARNING: autodoc: failed to import class 'finder.PackageFinder' from module 'unearth'; the following exception was raised:
No module named 'unearth'
WARNING: autodoc: failed to import class 'finder.BestMatch' from module 'unearth'; the following exception was raised:
No module named 'unearth'
WARNING: autodoc: failed to import module 'link' from module 'unearth'; the following exception was raised:
No module named 'unearth'
WARNING: autodoc: failed to import class 'link.Link' from module 'unearth'; the following exception was raised:
No module named 'unearth'
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/docs/cli_reference.md:4: ERROR: Failed to import "cli_parser" from "unearth.__main__".
No module named 'unearth'
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
writing... python-Unearth.3 { requirements api/evaluator api/finder api/link cli_reference contributing } done
build succeeded, 12 warnings.

This can be fixed by patch like below:

--- a/docs/conf.py
+++ b/docs/conf.py
@@ -10,9 +10,9 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+import os
+import sys
+sys.path.insert(0, os.path.abspath("../src"))


 # -- Project information -----------------------------------------------------

This patch fixes what is in the comment and that can of fix is suggested in sphinx example copy.py https://www.sphinx-doc.org/en/master/usage/configuration.html#example-of-configuration-file

Than .. on building my packages I'm using sphinx-build command with -n switch which shows warmings about missing references. These are not critical issues.

+ /usr/bin/sphinx-build -n -T -b man docs build/sphinx/man
Running Sphinx v5.0.2
making output directory... done
WARNING: html_static_path entry '_static' does not exist
myst v0.17.2: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions=[], linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area', disable_syntax=[], all_links_external=False, url_schemes=('http', 'https', 'mailto', 'ftp'), ref_domains=None, highlight_code_blocks=True, number_code_blocks=[], title_to_header=False, heading_anchors=None, heading_slug_func=None, footnote_transition=True, sub_delimiters=('{', '}'), words_per_minute=200)
building [mo]: targets for 0 po files that are out of date
building [man]: all manpages
updating environment: [new config] 7 added, 0 changed, 0 removed
reading sources... [100%] requirements
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.BestMatch.applicable:1: WARNING: duplicate object description of unearth.finder.BestMatch.applicable, other instance in api/finder, use :noindex: for one of them
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.BestMatch.best:1: WARNING: duplicate object description of unearth.finder.BestMatch.best, other instance in api/finder, use :noindex: for one of them
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.BestMatch.candidates:1: WARNING: duplicate object description of unearth.finder.BestMatch.candidates, other instance in api/finder, use :noindex: for one of them
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
writing... python-Unearth.3 { requirements api/evaluator api/finder api/link cli_reference contributing } /home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/evaluator.py:docstring of unearth.evaluator.evaluate_package:: WARNING: py:class reference target not found: Requirement
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/evaluator.py:docstring of unearth.evaluator.evaluate_package:: WARNING: py:class reference target not found: Requirement
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.PackageFinder:: WARNING: py:class reference target not found: PyPISession
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.PackageFinder:: WARNING: py:class reference target not found: PyPISession
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.PackageFinder.download_and_unpack:: WARNING: py:class reference target not found: Path
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.PackageFinder.download_and_unpack:: WARNING: py:class reference target not found: Path
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.PackageFinder.download_and_unpack:: WARNING: py:class reference target not found: Path
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.PackageFinder.download_and_unpack:: WARNING: py:class reference target not found: Path
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.PackageFinder.download_and_unpack:: WARNING: py:class reference target not found: Path
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.PackageFinder.download_and_unpack:: WARNING: py:class reference target not found: Path
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.PackageFinder.find_best_match:: WARNING: py:class reference target not found: Requirement
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.PackageFinder.find_best_match:: WARNING: py:class reference target not found: Requirement
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.PackageFinder.find_matches:: WARNING: py:class reference target not found: Requirement
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/finder.py:docstring of unearth.finder.PackageFinder.find_matches:: WARNING: py:class reference target not found: Requirement
/home/tkloczko/rpmbuild/BUILD/unearth-0.5.1/src/unearth/link.py:docstring of unearth.link.Link.from_path:: WARNING: py:class reference target not found: Path
done
build succeeded, 19 warnings.

You can peak on fixes that kind of issues in other projects
latchset/jwcrypto#289
click-contrib/sphinx-click@abc31069
latchset/jwcrypto#289
RDFLib/rdflib-sqlalchemy#95
sissaschool/elementpath@bf869d9e
jaraco/cssutils#21
pywbem/pywbem#2895

Describe the bug
Wheel files never seem to be chosen as the best match due to invalid requires-python versions.

To Reproduce
Steps to reproduce the behavior:

1. Create a simple PDM project without Rust installed on the machine.
2. Attempt to add ruff as a dependency.
3. Check log for the following error message:

unearth.evaluator: Skipping link <Link ...>: Invalid requires-python: py3

Expected behavior
If a wheel file is marked as py3, it should work with any python3 implementation and install that version.

System (please complete the following information):

• unearth version: 0.15.1
• Python version: 3.8
• OS: Linux

Describe the bug
In #121 failures to parse .netrc are skipped. But this leads to hard to diagnose failures when you are using .netrc for authentication

Additionally, when it falls back, it falls back to KDE wallet for some reason, despite the fact that I'm on Gnome and my main keyring is Gnome Keyring.

To Reproduce
Steps to reproduce the behavior:

1. Create a .netrc file in your home directory on Linux or macOS, with authentication credentials for a local PyPI index
2. Save it with the default permissions, which are generally 0644 (or chmod 0644 ~/.netrc if not)
3. Set up a pyproject.toml that uses tool.pdm.source with the URL for your local PyPI index, with name = "pypi"
4. Run pdm install

Expected behavior
I expect it to either use my .netrc, or give me an error why it can't be used. This used to work fine with the bad permissions (which are only really an issue on shared systems, which most developer systems are not). Newer versions of the netrc module seem to throw errors for the wrong permissions, but as long as they throw an error and I see it, I can fix it.

However, when updating my PDM environment with unearth 0.15.4, I simply get a silent failure to parse .netrc, and then it falls back to trying to use KDE Wallet, and if I cancel out of that it tries to prompt for auth credentials in the terminal. This is very confusing, and I had to downgrade unearth to actually get the error message to figure out why this was failing.

Screenshots
If applicable, add screenshots to help explain your problem.

System (please complete the following information):

• unearth version: 0.15.4
• Python version: 3.10.12
• OS: Linux

Additional context
Add any other context about the problem here.

I am trying to use this and my code base is in 3.6 unfortunately. Could you please support 3.6?

Describe alternatives you've considered
Forking and making it 3.6 compatible

Additional context
Add any other context or screenshots about the feature request here.

Describe the bug

Reported by pdm-project/pdm#1252

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
pdm-project/pdm#593

Is your feature request related to a problem? Please describe.
I'd like unearth to implicitly pick up the internal index url from pip.conf

Is your feature request related to a problem? Please describe.
I would like to use the output of pip freeze directly with the unearth cli.

Describe the solution you'd like
Add a -r, --requirements flag that takes a requirements file, and then operates on all the package descriptors in the file.

Describe alternatives you've considered
Writing a script to invoke unearth many times.

Additional context
pip download "supports" this (despite all its flaws).

Describe the bug

Hi, when I pdm install my pdm.lock file, I encounter the following error. I am able to resolve it by conda install openldap, but I'm wondering if there is a better solution (can we include ldap dependency or binaries in the unearth package)? Thank you!

Traceback (most recent call last):
  File ".venv/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 286, in
install_candidate
    self.manager.install(can)
  File ".venv/lib/python3.11/site-packages/pdm/installers/manager.py", line 34, in install
    dist_info = installer(str(prepared.build()), self.environment, prepared.direct_url())
                              ^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 403, in build
    self.obtain(allow_all=False)
  File ".venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 458, in obtain
    self._unpack(validate_hashes=not allow_all)
  File ".venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 472, in _unpack
    result = finder.download_and_unpack(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/unearth/finder.py", line 413, in download_and_unpack
    file = unpack_link(
           ^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/unearth/preparer.py", line 311, in unpack_link
    backend.fetch(link, location)
  File ".venv/lib/python3.11/site-packages/unearth/vcs/base.py", line 152, in fetch
    return self.fetch_new(location, url, rev, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/unearth/vcs/git.py", line 50, in fetch_new
    self.run_command(
  File ".venv/lib/python3.11/site-packages/unearth/vcs/base.py", line 78, in run_command
    raise UnpackError(e.output) from None
unearth.errors.UnpackError: /usr/libexec/git-core/git-remote-https: symbol lookup error: /lib64/libldap.so.2: undefined symbol: EVP_md2, version

System:

• unearth version: 0.14.0
• Python version: 3.11.7
• OS: Linux

I had found that, requires-python, platform and abi interface had been implemented, but cannot using in cli tools, why not exposing these interfaces for cli?

On a Gitlab CI, I get a traceback using PDM (pdm-project/pdm#2532). I think that the problem is related to unearth. The exception can be reproduced only with unearth:

$ python3.9 -c "from unearth import PackageFinder as F; f = F(index_urls=['https://pypi.org/simple/']); print(list(f.find_all_packages('flit-core')))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/appuser/.local/lib/python3.9/site-packages/unearth/finder.py", line 295, in find_all_packages
    self._find_packages(package_name, allow_yanked), hashes=hashes or {}
  File "/home/appuser/.local/lib/python3.9/site-packages/unearth/finder.py", line 275, in _find_packages
    return sorted(all_packages, key=self._sort_key, reverse=True)
  File "/home/appuser/.local/lib/python3.9/site-packages/unearth/collector.py", line 135, in collect_links_from_location
    yield from _collect_links_from_index(session, location)
  File "/home/appuser/.local/lib/python3.9/site-packages/unearth/collector.py", line 85, in parse_json_response
    data = json.loads(page.content)
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 10236 (char 10235)

Interestingly, (i) this code runs fine locally (and even locally in the same Docker image used for the CI) and (ii) I can install packages with pip in the Gitlab CI.

System (please complete the following information):

• unearth version: 0.12.1
• Python version: 3.9
• OS: Linux

Additional context

Cause pdm-project/pdm#2532

Describe the bug

unearth asks for the login whereas it's provided in the index url:

./venv/bin/python -m unearth  --index-url 'https://[email protected]/api/v4/projects/privateproject/p
ackages/pypi/simple' privatepackage --verbose
DEBUG: Collecting links from https://***@gitlab.privaterepo.net/api/v4/projects/privateproject/packages/pypi/simple/privatepackage/
DEBUG: Found index url https://[email protected]/api/v4/projects/privateproject/packages/pypi/simple
DEBUG: Found index url https://[email protected]/api/v4/projects/privateproject/packages/pypi/simple
User for gitlab.privaterepo.net:

We found out the root cause, see additional context

To Reproduce

Expected behavior
unearth should not ask for the login and use the provided username

Screenshots
If applicable, add screenshots to help explain your problem.

System (please complete the following information):

irrelevant

Additional context

We found out why it happens: At first, _get_new_credentials is called, with allow_netrc and allow_keyring set to false. Then unearth will receive a 401, and retry in handle_400s, but getting the url from the Response object, from which the username will have been stripped away from the url (see foot note). We found out that changing the first call to _get_new_credentials to set allow_netrc/allow_keyring to True works as expected (the username is extracted from the netrc/keyring and the query succeed without prompting the user).
So, possible fixes are :

• either _get_new_credetials should be called first with allow_netrc/allow_keyring set to True
• either check if a user was set in the url in handle_400s and if so get it back

Currently, if unearth receives a 401 when attempting to retrieve a package, it will automatically look for credentials in other locations, such as the ~/.netrc file. The relevant section of code is here

However, many private pypi artifactories will return 403 or even 404 (such as jfrog) if credentials are not provided. Currently, my group does not use PDM specifically for this reason, as it cannot automatically pick up credentials from ~/.netrc for our artifactories.

It would be a great addition to make a small update to the handling of credentials such that if one of these error codes was returned, that unearth would search for credentials and try again.

This change would be very small - essentially changing L290 here from if resp.status_code != 401:... to if resp.status_code not in [401, 403, 404]:... (and update a few names/comments)

I would be happy to make a PR to make this change if the maintainers are open to it.

Is your feature request related to a problem? Please describe.
Some keyrings from python_keyring do not support get_credential, only get_password. So if I have a repository, say pypi and want to pull packages from it and my keyring doesn't have get_credential, I'd have to set token manually.

Describe the solution you'd like
Can we try to use token if get_credential doesn't work?

Describe alternatives you've considered
Use configuration #13 somehow to set token globally or per-project

Additional context
I haven't tried this one in a bit so it may have been fixed but it was a pain with PDM and private repos w/certain keyrings.