juftin / hatch-pip-compile Goto Github PK

When the environment variable PIP_COMPILE_DISABLE is set to 1, and a lock file update is attempted, the exit code of the affected command is 0. For example: rm requirements-dev.txt; PIP_COMPILE_DISABLE=1 hatch --env dev run black /path/to/file exits with the status code 0. This could mask errors in CI/CD environments.

Currently uv pip install is supported, it would be helpful to support uv pip sync via a uv-pip-sync pip-compile-installer

Should a specific version of pip-tools be used inside the plugin or should we just continue to pip install pip-tools

This can be problematic in CI/CD where you run hatch run +py={{ matrix.python }} matrix:cov and have hatch.python set to a differing version than your current matrix version

[tool.hatch.envs.default]
pip-compile-constraint = "default"
type = "pip-compile"
python = "3.11"

[tool.hatch.envs.test]
dependencies = [
  "pytest",
  "pytest-cov"
]

[tool.hatch.envs.test.scripts]
cov = [
  "pytest --cov-config=pyproject.toml --cov {args:tests}"
]

[tool.hatch.envs.matrix]
template = "test"

[[tool.hatch.envs.matrix.matrix]]
python = ["3.8", "3.9", "3.10", "3.11", "3.12"]

In the above example you must have the 3.11 environment available to use pip-compile environments from a matrix because default is a constraint environment

AttributeError: 'NoneType' object has no attribute 'executable'

https://github.com/juftin/hatch-pip-compile/actions/runs/7117731072/job/19379122673?pr=28

See how this is handled here:

IMO this isn't a bug - but this should probably be documented

Should the --generate-hashes option be optional on the plugin, similar to pip-compile or is that desired lockfile behavior by default?

uv should only be installed once into hatch's environment instead of each virtual environment.

Hi!

pip-compile resolves dependencies for a particular Python version, and it is important because projects can define dependencies like this:

importlib-metadata >=4.3; python_version < '3.10'

To represent this, it includes a comment, which at least makes it clear, but nothing is done with it as far as I know:

# This file is autogenerated by pip-compile with Python 3.11

and in that file for the above example importlib-metadata isn't even mentioned anywhere! So it will just not work if someone tries to use this lock file with Python 3.9.

As such, I think this plugin could be improved as follows, especially because Hatch gives it direct information about the intended Python version:

1. Include the Python version into the header (currently it is stripped)

2. Perhaps warn users when an environment is invoked that the lock file might be intended for a different Python version than is currently used?

3. Anything else? Like managing separate lock files in case there is a matrix with multiple Python versions?

pip-sync has been demonstrating some issues with running lockfiles on different python versions: https://github.com/juftin/hatch-pip-compile/actions/runs/7028915350/job/19125657383

Here's an error from testing the plugin on Python 3.8:

Traceback (most recent call last):
...
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/_code/code.py", line 59, in <module>
    from exceptiongroup import BaseExceptionGroup
ModuleNotFoundError: No module named 'exceptiongroup'
Error: Process completed with exit code 1.

Basically the lockfile was generated using 3.11 but the exceptiongroup package wasn't included because it's a builtin in 3.11. The plugin uses pip-sync which mirrors the environment to the lockfile (installing and uninstalling) so the tests failed. If we were to just run pip install -r lockfile.txt instead though the missing exceptiongroup dependency would've been picked up and installed (as long as we don't pass the --no-deps argument to pip).

I can see an argument for wanting to handle this either way. To get around this issue I used a regular virtual environment in my testing matrix instead of generating a lockfile for every environment. I woul've preferred to set an option like pip-compile-installer to pip instad of pip-sync instead

This could also be handled by just doing a pip install -r in CI/CD and bypassing this hatch completely.

I thought it could be just turned into a dependency rather because it's a utility of the plugin itself, so they both should reside in hatch's origin environment, rather than the target virtual environment

super().dependencies_in_sync() ultimately calls dependencies_in_sync on self.dependencies_complex.

https://github.com/pypa/hatch/blob/482deaddb4897b4214050dc0ca4618ae8a596f2e/src/hatch/env/virtual.py#L128-L137

    def dependencies_in_sync(self):
        if not self.dependencies:
            return True

        from hatchling.dep.core import dependencies_in_sync

        with self.safe_activation():
            return dependencies_in_sync(
                self.dependencies_complex, sys_path=self.virtual_env.sys_path, environment=self.virtual_env.environment
            )

Instead of self.dependencies_complex an exhaustive list of dependencies should be read from the requirements file. pip-tools gives a nice way to access the pip API:

from typing import Iterator, List

from piptools._compat.pip_compat import InstallRequirement, PipSession, parse_requirements

reqs: Iterator[InstallRequirement] = parse_requirements(
    "requirements.txt",
    session=PipSession(),
)
piptools_dependencies_complex: List[InstallRequirement] = list(reqs)

Hi! I am hoping to be able to use Hatch for deps management outside of Python projects.

As such, I don't create a pyproject.toml but merely a hatch.toml. Which normally works but with hatch-pip-compile there is a small problem.

To reproduce:
In an empty directory, create the file hatch.toml

[env]
requires = ["hatch-pip-compile"]

[envs.docs]
type = "pip-compile"
dependencies = ["mkdocs"]

$ hatch run docs:mkdocs --version
Checking dependencies
Syncing dependencies
ERROR: file:///tmp/test does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.
#[exit status 1]

$ hatch run docs:mkdocs --version
Checking dependencies
mkdocs, version 1.5.3 from ~/.local/share/hatch/env/pip-compile/test/oZWB37hz/docs/lib/python3.11/site-packages/mkdocs (Python 3.11)

So, the first invocation manages to populate the dependencies lock file but fails before it can run the command.
And then the 2nd invocation just works.
Would be good to fix this so it works the first time 🙂

Interestingly, the created lock file mentions # via test which is just the current directory name, so I think that's another factor that supports #13.

Currently updating requirements can be tough because of the way the plugin is implemented:

Current Implementation

1. Create a temporary requirements.in file with a project's dependencies
2. Use that requirements.in file as the input to pip-compile with the output being the respective lock file
3. Since the input is actually a requirements.in file and not the actual pyproject.toml - some post-processing occurs replacing lines like # via -r /var/folders/qk/mw5mftjj0qz42nrxrnhctlzw0000gp/T/tmpo85wegym/default.in with # via hatch-pip-compile (pyproject.toml) as well as to add the custom hatch-pip-compile header

Because of that implementation there isn't currently a command you could run locally that would update the file in-place. Ideally something like this would just work:

hatch dep show requirements --all | \
  hatch run python -m piptools compile - \
    --output-file requirements.txt \
    --generate-hashes \
    --no-header \
    --upgrade-package certifi

However running that above command will blow away the hatch-pip-compile header.

At this point this issue is just a placeholder and place to brainstorm ideas for the ideal workflow with pip-compile

Hatch environments created with hatch-pip-compile are not fully removed when running hatch env prune or hatch env remove. When entering the environment (for example with hatch shell) after running removal commands, the environment is not recreated and we enter a partial environment (no dependencies are installed except for Python itself).

Hi!
Currently this plugin chooses the following path by default: .hatch/{env_name}.lock
But I naturally would've chosen this path: requirements/{env_name}.txt

This is because:

1. Maybe not necessary to hide it in .hatch, because perhaps users who for some reason hate Hatch would just discover the requirements files and use them directly with a sigh of relief.

2. I think there are many crawlers of requirements, and they look specifically for requirements*.txt. Although I'm not sure which of them allow it to be like requirements/foo.txt or if it has to be like */requirements-foo.txt.

   Anyway here are some examples:

   • https://github.com/renovatebot/renovate/blob/0cbd22006f3190974aa099fc37a4128b62d83e55/lib/modules/manager/pip_requirements/index.ts#L9
   • https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph#supported-package-ecosystems
   • https://thanks.dev/static/ecosystems

Could we think what the path should be, before it's too late to change the default 😅

Instead of raising an error if the upstream environment doesn't exist, create it instead

see #22 (comment)

Summary

This project needs unit (and integration?) tests.

Acceptance Criteria

New tests. I will keep this issue open until I feel like there's sufficient coverage though.

Notes

Some elements of this project can be tough to test without an actual hatch CLI running - hopefully I can make use of the click.CliRunner for this.

I need to confirm but I believe the latest version of hatch is skipping checks with its new use of dependency_hash (https://hatch.pypa.io/latest/blog/2023/12/11/hatch-v180/#faster-environment-usage) - this allows lockfiles to be out of date (by hand editing at least)

Faster environment usage
Spawning a shell or running commands within environments always first checks that your project's dependencies are satisfied and if not synchronizes the environment with what is defined. Previously, this had the potential to be quite slow for projects that have many dependencies.
Now the set of dependency definitions is hashed and no check is performed if the hash is the same as before, significantly speeding up environment usage in most cases.

https://github.com/pypa/hatch/blob/d3246e957584d292319e7b93301598cdf611e902/src/hatch/cli/application.py#L107-L117

When a constraint environment doesn't exist and is created by another environment its lockfile is created but not installed - the environment is left in a bad state and ImportErrors are thrown:

Reproduction

1. Blow away hatch virtual environments (.venv in my case)
2. Invoke a dependent environment first (it actually creates the constraint environment)
3. Invoke something on the constraint environment

rm -rf .venv/
hatch env run --env test -- python --version
hatch env run --env default -- python -m package_name.cli

ModuleNotFoundError: No module named 'click'

@juftin Thanks for the #12 feature and awesome logo btw.

I am just playing around a bit with it and I noticed that I can create a lock file for a test environment, which depends on the default environment, without the lock file for the default environment being created. Does that make sense?

I would assume that when running e.g. hatch run test:cov and the when the test environment is created, it should realise that there is yet no default environment from which it inherits so it should first create a lock file for default.

So if I am able to create lock file for test and maybe ours later create the environment for default, it could mean that default is not consistent with test anymore, right?

Hi!
I was just looking at the latest lock file in my env.

It had this entry, for example

markdown-callouts==0.3.0
    # via mkdocs-literate-nav

where "mkdocs-literate-nav" is the current project's name. But I was very confused - this detached env shouldn't be depending on the project itself, and mkdocs-literate-nav is not written as a dependency in it anywhere.
So I started debugging why there's a dependency.

But actually there isn't a dependency! # via [current project] just stands for "specified in pyproject.toml"

Before this change it was at least # via [current project] (pyproject.toml) and that was quite a bit clearer.
https://github.com/juftin/hatch-pip-compile/pull/9/files#diff-a17e936862775c98a1a6459602e69226471d05e2d151d52b61a73d80a54e86d0L165
I think I'd like to get back that extra clarification.

Here's what pip-compile itself shows normally:

markdown-callouts==0.3.0
    # via -r requirements/requirements-docs.in

The project's name is not mentioned and that makes a lot more sense.

I think the line should become something like # via pyproject.toml (docs). Although I suppose the config doesn't even have to be in pyproject.toml, maybe it's hatch.toml, so not so obvious...

It's possible to just add this to pyproject.toml and it seems to work without manually installing anything other than hatch:

[tool.hatch.env]
requires = [
    "hatch-pip-deepfreeze",
]

Reference - seems to have appeared in the docs only since version 1.6?

I think this should be preferred over the current manual instructions, because it's very inconvenient for people not familiar with any of this stuff (who are just trying to contribute to a project that happens to use hatch):

pipx inject hatch hatch-pip-compile

Issue

1. default environment has no dependencies
2. test environment has default as its constraint
3. test environment can't be created because default lockfile doesn't exist but also won't create itself

Hi, I'm trying to test out uv. For some reason, it worked the first time I tried, but now this is what I get:

│    74 │   │   │   │   f"Invalid pip-tools install method: {install_method} - "                   │
│    75 │   │   │   │   "must be 'pip' or 'pip-sync'"                                              │
│    76 │   │   │   )                                                                              │
│ ❱  77 │   │   │   raise HatchPipCompileError(msg)                                                │
│    78 │                                                                                          │
│    79 │   @staticmethod                                                                          │
│    80 │   def get_option_types() -> Dict[str, Any]:                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
HatchPipCompileError: Invalid pip-tools install method: uv - must be 'pip' or 'pip-sync'

It seems that the version of hatch-pip-compile I have is too old for this work. That's weird because I'm pretty sure I made it work once. I tried uninstalling and reinstalling Hatch (with different methods: pip and binary) itself, but encountered the same issue. Now, I realize that I have no clue how I'm supposed to update hatch-pip-compile itself. 🤔 Is it supposed to work automatically? If yes, when does it happen? Does the Hatch installation method matter?

Hi,

I love to make a "lock" file in my project, but when I use this plugin, I found the config setting dirs.env.virtual is not working anymore. Is there way to use hatch-pip-compile with dirs.env.virtual setting? I need this becase of VScode integration: pypa/hatch#600

Summary

Hatch doesn't lock project dependencies universally across environments

Details

In hatch a non-default environment also shares dependencies with the project dependencies.

consider the following pyproject.toml:

[build-system]
build-backend = "hatchling.build"
requires = ["hatchling"]

[project]
authors = [
  {name = "Justin Flannery", email = "[email protected]"}
]
dependencies = [
  "hatch"
]
version = "0.0.0"
license = "MIT"
name = "hatch-pip-compile"
readme = "README.md"
requires-python = ">=3.8"

[tool.hatch.env]
requires = ["hatch-pip-compile"]

[tool.hatch.envs.default]
path = ".venv"
post-install-commands = [
  "pre-commit install"
]
python = "3.11"
type = "pip-compile"
dependencies = ["pytest"]

[tool.hatch.envs.docs]
dependencies = [
  "mkdocs~=1.5.2",
  "mkdocs-material~=9.2.3",
  "mkdocstrings~=0.22.0",
  "markdown-exec[ansi]~=1.6.0"
]
type = "pip-compile"

[tool.hatch.envs.docs.scripts]
build = ["mkdocs build --clean --strict"]
deploy = ["mkdocs gh-deploy {args:}"]
serve = ["mkdocs serve --dev-addr localhost:8080 --livereload"]

We can see there are three different sets of dependencies:

• Project Dependencies
  • hatch
• Default Dependencies
  • pytest
• Docs Dependencies
  • mkdocs~=1.5.2
  • mkdocs-material~=9.2.3
  • mkdocstrings~=0.22.0
  • markdown-exec[ansi]~=1.6.0

These groups of dependencies only make up two virtual environments though, default and docs. For both environments the project dependencies are also included (as long as the environment is not "detached"):

• default environment
  • hatch
  • pytest
• docs environment
  • hatch
  • mkdocs~=1.5.2
  • mkdocs-material~=9.2.3
  • mkdocstrings~=0.22.0
  • markdown-exec[ansi]~=1.6.0

The way hatch-pip-compile currently works is that it will create two lock files for these two environments but it is possible that the hatch dependency from the project will get pinned to a different version. Instead we want them to be pinned on the same version everywhere.

Proposed Solution

hatch-pip-compile should use the default environment's lockfile as a constraints file for other environments when running pip-compile (when the environment is not detached)

Tested on python 3.12.1 & hatch 1.9.1 & hatch-pip-compile 1.9.0.

Following pip-compile settings in pyproject.toml:

[project]
dependencies = ["pydantic"]

[tool.hatch.envs.default]
python = "3.8"
features = []
type = "pip-compile"
pip-compile-constraint = "default"         # keep locks between default & others consistent
lock-filename = "locks/{env_name}.lock"
pip-compile-hashes = true
pip-compile-verbose = true
pip-compile-args = ["--no-emit-index-url"]

[tool.hatch.envs.dev]
features = ["dev"]

Then running hatch run dev:pip -V i get the following traceback:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\hatch\cli\__init__.py:221 in     │
│ main                                                                                             │
│                                                                                                  │
│   218                                                                                            │
│   219 def main():  # no cov                                                                      │
│   220 │   try:                                                                                   │
│ ❱ 221 │   │   return hatch(prog_name='hatch', windows_expand_args=False)                         │
│   222 │   except Exception:  # noqa: BLE001                                                      │
│   223 │   │   from rich.console import Console                                                   │
│   224                                                                                            │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\click\core.py:1157 in __call__   │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\click\core.py:1078 in main       │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\click\core.py:1688 in invoke     │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\click\core.py:1434 in invoke     │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\click\core.py:783 in invoke      │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\click\decorators.py:45 in        │
│ new_func                                                                                         │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\click\decorators.py:33 in        │
│ new_func                                                                                         │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\hatch\cli\run\__init__.py:79 in  │
│ run                                                                                              │
│                                                                                                  │
│   76 │   elif not env_name:                                                                      │
│   77 │   │   env_name = 'system'                                                                 │
│   78 │                                                                                           │
│ ❱ 79 │   ctx.invoke(                                                                             │
│   80 │   │   run_command,                                                                        │
│   81 │   │   args=[command, *args],                                                              │
│   82 │   │   env_names=[env_name],                                                               │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\click\core.py:783 in invoke      │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\click\decorators.py:45 in        │
│ new_func                                                                                         │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\hatch\cli\env\run.py:180 in run  │
│                                                                                                  │
│   177 │   │   │   if env_name == 'system':                                                       │
│   178 │   │   │   │   environment.exists = lambda: True                                          │
│   179 │   │   │                                                                                  │
│ ❱ 180 │   │   │   app.prepare_environment(environment)                                           │
│   181 │   │   │   app.run_shell_commands(                                                        │
│   182 │   │   │   │   environment,                                                               │
│   183 │   │   │   │   [environment.join_command_args(args)],                                     │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\hatch\cli\application.py:107 in  │
│ prepare_environment                                                                              │
│                                                                                                  │
│   104 │   │   │   │   │   with self.status('Running post-installation commands'):                │
│   105 │   │   │   │   │   │   self.run_shell_commands(environment, environment.post_install_co   │
│   106 │   │                                                                                      │
│ ❱ 107 │   │   new_dep_hash = environment.dependency_hash()                                       │
│   108 │   │   current_dep_hash = self.env_metadata.dependency_hash(environment)                  │
│   109 │   │   if new_dep_hash != current_dep_hash:                                               │
│   110 │   │   │   with self.status('Checking dependencies'):                                     │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\hatch_pip_compile\plugin.py:97   │
│ in dependency_hash                                                                               │
│                                                                                                  │
│    94 │   │   """                                                                                │
│    95 │   │   Get the dependency hash                                                            │
│    96 │   │   """                                                                                │
│ ❱  97 │   │   self.run_pip_compile()                                                             │
│    98 │   │   hatch_hash = super().dependency_hash()                                             │
│    99 │   │   if not self.dependencies:                                                          │
│   100 │   │   │   return hatch_hash                                                              │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\hatch_pip_compile\plugin.py:129  │
│ in run_pip_compile                                                                               │
│                                                                                                  │
│   126 │   │   │   │   │   _ = self.piptools_lock.compare_python_versions(                        │
│   127 │   │   │   │   │   │   verbose=self.config.get("pip-compile-verbose", None)               │
│   128 │   │   │   │   │   )                                                                      │
│ ❱ 129 │   │   │   │   self.pip_compile_cli()                                                     │
│   130 │                                                                                          │
│   131 │   def pip_compile_cli(self) -> None:                                                     │
│   132 │   │   """                                                                                │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\hatch_pip_compile\plugin.py:180  │
│ in pip_compile_cli                                                                               │
│                                                                                                  │
│   177 │   │   │   │   shutil.copy(self.piptools_lock_file, output_file)                          │
│   178 │   │   │   self.piptools_lock_file.parent.mkdir(exist_ok=True, parents=True)              │
│   179 │   │   │   self.plugin_check_command(cmd)                                                 │
│ ❱ 180 │   │   │   self.piptools_lock.process_lock(lockfile=output_file)                          │
│   181 │   │   │   shutil.move(output_file, self.piptools_lock_file)                              │
│   182 │   │   self.lockfile_up_to_date = True                                                    │
│   183                                                                                            │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\site-packages\hatch_pip_compile\lock.py:60 in  │
│ process_lock                                                                                     │
│                                                                                                  │
│    57 │   │   │   constraints_path = self.constraints_file.relative_to(self.project_root)        │
│    58 │   │   │   constraints_line = f"# [constraints] {constraints_path} (SHA256: {constraint   │
│    59 │   │   │   joined_dependencies = "\n".join([constraints_line, "#", joined_dependencies]   │
│ ❱  60 │   │   │   cleaned_input_file = re.sub(                                                   │
│    61 │   │   │   │   r"-c \S*",                                                                 │
│    62 │   │   │   │   f"-c {constraints_path}",                                                  │
│    63 │   │   │   │   cleaned_input_file,                                                        │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\re\__init__.py:186 in sub                      │
│                                                                                                  │
│   183 │   if a string, backslash escapes in it are processed.  If it is                          │
│   184 │   a callable, it's passed the Match object and must return                               │
│   185 │   a replacement string to be used."""                                                    │
│ ❱ 186 │   return _compile(pattern, flags).sub(repl, string, count)                               │
│   187                                                                                            │
│   188 def subn(pattern, repl, string, count=0, flags=0):                                         │
│   189 │   """Return a 2-tuple containing (new_string, number).                                   │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\re\__init__.py:334 in _compile_template        │
│                                                                                                  │
│   331 @functools.lru_cache(_MAXCACHE)                                                            │
│   332 def _compile_template(pattern, repl):                                                      │
│   333 │   # internal: compile replacement pattern                                                │
│ ❱ 334 │   return _sre.template(pattern, _parser.parse_template(repl, pattern))                   │
│   335                                                                                            │
│   336 # register myself for pickling                                                             │
│   337                                                                                            │
│                                                                                                  │
│ C:\Users\user\.pyenv\pyenv-win\versions\3.12.1\Lib\re\_parser.py:1075 in parse_template           │
│                                                                                                  │
│   1072 │   │   │   │   │   this = chr(ESCAPES[this][1])                                          │
│   1073 │   │   │   │   except KeyError:                                                          │
│   1074 │   │   │   │   │   if c in ASCIILETTERS:                                                 │
│ ❱ 1075 │   │   │   │   │   │   raise s.error('bad escape %s' % this, len(this)) from None        │
│   1076 │   │   │   │   lappend(this)                                                             │
│   1077 │   │   else:                                                                             │
│   1078 │   │   │   lappend(this)                                                                 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
error: bad escape \d at position 8

The same command works in the default environment hatch run pip -V

First, the package is installed with --no-deps then the dependencies are installed. This order of operations produces the following warinings which should be disregarded

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
browsr 1.17.1 requires art~=5.7, which is not installed.
browsr 1.17.1 requires pandas~=1.5.2, which is not installed.
browsr 1.17.1 requires pillow>=9.1.0, which is not installed.
browsr 1.17.1 requires pymupdf~=1.22.3, which is not installed.
browsr 1.17.1 requires rich~=13.5.2, which is not installed.
browsr 1.17.1 requires rich-click~=1.5.2, which is not installed.
browsr 1.17.1 requires rich-pixels~=2.1.1, which is not installed.
browsr 1.17.1 requires textual==0.39.0, which is not installed.
browsr 1.17.1 requires textual-universal-directorytree~=1.0.2, which is not installed.

https://github.com/astral-sh/uv

uv has a pip-compile compatible API and it's extremely fast. It might be a good option to leverage similar to pip-compile-installer

Hi,
I don't know where the issue is from but this code

[tool.hatch.env]
requires = [
    "hatch-pip-compile"
]

does not work, adding it make the pyproject.toml invalid and hatch fails to parse the file with FileNotFoundError: [Errno 2] No such file or directory: '1', I'm on hatch latest version 1.9.1.
I also tried

[build-system]
requires = ["hatchling", "hatch-pip-compile"]
build-backend = "hatchling.build"

but it seems to have no effect

Thanks for providing hatch-pip-compile, it looks really useful but I am having a strange problem. Somehow the project itself is not installed in the default environment if it has type "pip-compile" as opposed to "virtual". So in my pyproject.toml I have:

[tool.hatch.env]
requires = ["hatch-pip-compile"]  

[tool.hatch.envs.default]
type = "pip-compile"
[tool.hatch.envs.default.scripts]
upgrade-all = "pip-compile --upgrade {args}"
upgrade-pkg = "pip-compile --upgrade-package {args}"

and running hatch run python -c "import name_of_my_package" it doesn't find the package. Also hatch run pip list doesn't show it. Maybe I am missing something obvious here 🤔 .

Summary

Lockfiles can be out of date without knowing it with the pip-compile-restraints feature

Steps to Reproduce

1. You create the default lockfile
2. You create the test lockfile with the default lockfile as its constraint
3. You update the default lockfile without actually changing the requirements specified in your pyproject.toml
4. You run the test environment and it thinks it's up to date but remains pinned on the old versions

A not entirely documented (pypa/pip#11296) but extremely useful feature of pip since version 21.2 is that optional-dependencies groups can depend on each other: https://hynek.me/articles/python-recursive-optional-dependencies/

For example if your package is mypkg and it has an optional REST API which you could run with uvicorn or some other server, which you run from the uvicorn environment in hatch, and for which you'd like to lock dependencies, you could do this:

[project.optional-dependencies]
api = [
    "fastapi"
]
uvicorn = [
     "mypkg[api]",
     "uvicorn",
]
[tool.hatch.envs.uvicorn]
features = [ "uvicorn" ]
type = "pip-compile"

Unfortunately this doesn't really work with pip-compile or hatch-pip-compile, because you will end up with this in requirements/requirements-uvicorn.txt:

mypkg==1.0.0
    # via hatch.envs.uvicorn

Oh noes! It went off to PyPI and found that yes, indeed, there is a package there already called mypkg and proceeded to add it as a dependency to the requirements file. That's definitely not what you want!

Note that if you run pip-compile --extra uvicorn in the environment you'll get this instead:

mypkg[api] @ file:///my/local/path/to/mypkg
    # via file:///my/local/path/to/mypkg

Which is not what you want either, but definitely better than pulling in some possibly unrelated or out-of-date package from PyPI. In this case it would be easy to post-process the output of pip-compile to remove editable installs of recursive optional dependencies, for instance.

Responsibility for fixing this might be partly in pip-tools if there isn't a way for hatch-pip-compile to get it to do the right thing...

Changing between Linux and Windows makes hatch-pip-compile try to recreate the lockfile.

For some reason when running in Windows lockfile_up_to_date is False but lockfile_up_to_date is True when run on Linux.

See https://github.com/juftin/hatch-pip-compile/actions/runs/7450650936/job/20270034562 / #57 (comment)

In a CI/CD situation it would be useful to be able to set an environment variable that raises an error if the tooling detects an out of date environment, preventing it from regenerating a lock file without us knowing.

The environment file doesn't show up after the environment was built even though I specified lock-filename. Here is my pyproject.toml configuration:

# Default environment
[tool.hatch.envs.default]
type = "pip-compile"
lock-filename = "requirements.txt"
[tool.hatch.envs.default.scripts]
upgrade-all = "pip-compile --upgrade {args}"
upgrade-pkg = "pip-compile --upgrade-package {args}"

# Test environment
[tool.hatch.envs.test]
type = "pip-compile"
lock-filename = "requirements-test.txt"
dependencies = [
    "coverage[toml]>=6.2",
    "pytest",
    "pytest-cov",
    "pytest-mock",
    "pytest-vcr",
    "pytest-env",
    "hypothesis",
    "jupyterlab",
    "pre-commit",
    "mypy",
    "ruff",
    "ipython",
    "mktestdocs",
    "packaging",
]
[tool.hatch.envs.test.scripts]
cov = "pytest --cov-report=term-missing --cov-config=pyproject.toml --cov=src/ultimate_notion --cov=tests {args}"
no-cov = "cov --no-cov {args}"
debug =  "cov --no-cov -s --pdb --pdbcls=IPython.core.debugger:Pdb {args}"
ci = "cov --vcr-record=none --cov-report lcov {args}"
record = "cov --vcr-record=all {args}" # re-record all vcr cassettes
doctest = "pytest docs/examples/"

The file requirements-test.txt never shows up although it definitely generates the environment. Maybe also one question in this regard. I hope to actually accomplish with this that my production requirements are always pinned in requirements.txt and the test environment in requirements-test.txt. The pinned requirements in requirements-text should list the exact pinned numbers from the production environment plus the additional pinned packages only needed for testing, so they need to be consistent in the sense that if my tests run with requirements-test.txt I rest assured that in production with requirements.txt they should guaranteed to also run as it's the same pinned numbers. Will this work? Also what will happen if I use pip-compile --upgrade-package {args} in any of these environments? Will hatch-pip-compile assure that those two environments stay consistent? Maybe this is also related to #8.