Lock file of a non-default environment not showing up about hatch-pip-compile HOT 8 CLOSED

@juftin 😆 , yes, you are right! There it is :-) Thanks for being so responsive, kind and helpful ❤️

No that this is clarified. I am not sure how I would go about updating those two environments. I guess my upgrade-all = "pip-compile --upgrade {args}" will not work, right?

But regarding my other question. Will hatch-pip-compile in my setup guarantee that the pinned packages in requirements-test.txt that are also in requirements.txt will be pinned to the same version? Since the one environment inherits from the other?

from hatch-pip-compile.

Awesome! Glad that's resolved. The upcoming breaking changes on #11 will make all of this a bit more intuitive I think and they should also be the last breaking changes on this tool (🤞 ). They should be merged in the next day or two.

Before the change your pyproject.toml should look like this to locate the file where you want it:

[tool.hatch.envs.test]
type = "pip-compile"
lock-directory = "."
lock-filename = "requirements-test.txt"

With the changes are merged you could achieve this like this:

[tool.hatch.envs.test]
type = "pip-compile"
lock-filename = "requirements-{env_name}.txt"

With the new changes lock-directory is gone and lock-filename defaults to requirements/requirements-{env_name}.txt for non-default environments and we no longer use the hard to find .hatch directory.

Regarding running --upgrade, that functionality is coming. Currently the issue is that the hatch-pip-compile tool writes a header on the lockfile where it stores the installed packages to determine whether a lock file is out of sync. My current idea around a solution for this is to create a CLI that runs pip-compile --upgrade for you and re-writes the header.

To do that in the meantime you'll have to run something like this and manually restore the header:

hatch env show test --json | jq -r ".test.dependencies[]" | pip-compile - --upgrade --output-file requirements-test.txt

I know that's annoying and I hope to handle that complexity soon.

For the exact same lock between environments, I will need to build that functionality out. I think this can be done by using the default lock file as a constraint file to the others. Currently it's possible to have drift but not likely if you generate lock files at the same time.

I will need to look into exactly how hatch currently handles inheritance between environments and installing package dependencies vs. default environment dependencies vs. non-default dependencies.

The breaking changes from #11 should be merged in the next few days and the new features to run --upgrade and use cross-environment constraints should be rolled out in the next week or two. Again, thank you so much for filing the issue and helping to make the tool better. It should be in a very stable and good place soon.

from hatch-pip-compile.

FYI @FlorianWilhelm - #8 and #12 are both fully resolved.

from hatch-pip-compile.

The file requirements-test.txt never shows up although it definitely generates the environment.

I actually experienced different behavior. Can you confirm you're running a script from the test environment (hatch env create test would also work)? Here is my pyproject.toml:

[project]
name = "hatch-pip-compile-test"
version = "0.0.0"

# Default environment
[tool.hatch.envs.default]
type = "pip-compile"
lock-filename = "requirements.txt"
[tool.hatch.envs.default.scripts]
upgrade-all = "pip-compile --upgrade {args}"
upgrade-pkg = "pip-compile --upgrade-package {args}"

# Test environment
[tool.hatch.envs.test]
type = "pip-compile"
lock-filename = "requirements-test.txt"
dependencies = [
    "coverage[toml]>=6.2",
    "pytest",
    "pytest-cov",
    "pytest-mock",
    "pytest-vcr",
    "pytest-env",
    "hypothesis",
    "jupyterlab",
    "pre-commit",
    "mypy",
    "ruff",
    "ipython",
    "mktestdocs",
    "packaging",
]
[tool.hatch.envs.test.scripts]
cov = "pytest --cov-report=term-missing --cov-config=pyproject.toml --cov=src/ultimate_notion --cov=tests {args}"
no-cov = "cov --no-cov {args}"
debug =  "cov --no-cov -s --pdb --pdbcls=IPython.core.debugger:Pdb {args}"
ci = "cov --vcr-record=none --cov-report lcov {args}"
record = "cov --vcr-record=all {args}" # re-record all vcr cassettes
doctest = "pytest docs/examples/"

tree before:

.
└── pyproject.toml

1 directory, 1 file

• hatch run python --version - this created requirements.txt file
• hatch run test:doctest - this created a file at test-requirements.txt

tree after:

.
├── hatch_pip_compile_test.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
├── pyproject.toml
├── requirements-test.txt
└── requirements.txt

2 directories, 7 files

from hatch-pip-compile.

Okay, so I created a new directory with a file pyproject.toml having the corresponding content you provided above. Running hatch run test:doctest did not create a requirements-test.txt, i.e.

❯ ls -l
total 8
drwxr-xr-x  6 fwilhelm  staff   192 Nov 22 09:53 hatch_pip_compile_test.egg-info
-rw-r--r--  1 fwilhelm  staff  1070 Nov 22 09:47 pyproject.toml 

Now running hatch run ls created a requirements.txt as expected.

I am using Hatch, version 1.7.0 on MacOS, maybe it's related to my hatch config? That's my hatch config show:

mode = "local"
project = ""
shell = ""

[dirs]
project = []
python = "isolated"
data = "/Users/fwilhelm/Library/Application Support/hatch"
cache = "/Users/fwilhelm/Library/Caches/hatch"

[dirs.env]
virtual = ".direnv"
...

But also deleting this file and rerunning everything doesn't help. I'm out of ideas now why it seems to behave differently on my machine.

from hatch-pip-compile.

🤦 I've been using my local development branch from https://github.com/juftin/hatch-pip-compile/tree/breaking-changes which I think has been making this all confusing

I am going to be releasing the changes soon - but one of the breaking changes is that the tool will be moving away from the .hatch directory where I think your lock file is in favor of the lock-filename option replacing the lock-directory option and defaulting to the requirements dir for non-default locks (see #7)

mkdir hatch-test
cd hatch-test
python -m venv .venv
source .venv/bin/activate
<copy pyproject.toml>
python -m pip install hatch hatch-pip-compile

python -m hatch env create default

❯ tree -I '.venv|*.egg-info' -a
.
├── pyproject.toml
└── requirements.txt

1 directory, 2 files

python -m hatch env create test

❯ tree -I '.venv|*.egg-info' -a
.
├── .hatch
│   └── requirements-test.txt
├── pyproject.toml
└── requirements.txt

2 directories, 3 files

Is the lockfile you're looking for at .hatch/requirements-test.txt @FlorianWilhelm ?

from hatch-pip-compile.

Closing this issue as #11 was merged. I am tracking the other issues @ #8 and #12

from hatch-pip-compile.

Awesome @juftin, thanks!

from hatch-pip-compile.