juliapy / pyjuliapkg Goto Github PK

(ewatercycle_spyder) tesla-k20c@teslak20c:~$ julia --version
julia version 1.5.4
(ewatercycle_spyder) tesla-k20c@teslak20c:~$ juliaup status
 Default  Channel  Version                Update
-----------------------------------
       *  1.5      1.5.4+0.x64.linux.gnu
          1.6.7    1.6.7+0.x64.linux.gnu
(ewatercycle_spyder) tesla-k20c@teslak20c:~$ python
Python 3.10.2 | packaged by conda-forge | (main, Feb  1 2022, 19:28:35) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> **import juliapkg**
>>> juliapkg.status()
JuliaPkg Status
/home/tesla-k20c/.conda/envs/ewatercycle_spyder/julia_env/pyjuliapkg/juliapkg.json
Not resolved (resolve for more information)
Julia (^1.5)
>>> juliapkg.resolve()
[juliapkg] Locating Julia ^1.6.1
[juliapkg] Querying Julia versions from https://julialang-s3.julialang.org/bin/versions.json
[juliapkg] Using Julia 1.8.5 at /home/tesla-k20c/.conda/envs/ewatercycle_spyder/julia_env/pyjuliapkg/install/bin/julia

I need julia version 1.5, but even though julia and juliaup are in my path juliapkg installs a new version. What am I missing here? Thanks in advance.

via PythonCall/JuliaCall I came to use JuliaPkg

unfortunately sofar JuliaPkg creates more problems than it solves. I am using PythonCall on mybinder, however the JuliaPkg creates its own empty Project, not using the binder project.

Even worse: If setting PYTHON_JULIAPKG_PROJECT to the directory where the binder Project.toml lies, it overwrites the existing Project.toml file, deleting the previous environment and leaving JuliaPkg empty environment.

I'm working in a conda environment and I need julia 1.5.
Following the documentation, I ran the following:

>>> juliapkg.require_julia("1.5")
>>> juliapkg.status()
JuliaPkg Status
/home/tesla-k20c/.conda/envs/ewatercycle_spyder/julia_env/pyjuliapkg/juliapkg.json
Not resolved (resolve for more information)
Julia (^1.5)
>>> juliapkg.resolve()
[juliapkg] Locating Julia ^1.6.1
[juliapkg] Using Julia 1.8.5 at /home/tesla-k20c/.conda/envs/ewatercycle_spyder/julia_env/pyjuliapkg/install/bin/julia

How can I specify the julia version exactly, instead of only giving a minimum requirement? And why does it try to locate ^1.6.1 and not ^1.5.1?
Thanks!

Currently, $CONDA_PREFIX (and $VIRTUAL_ENV) overrides ignores $JULIA_DEPOT_PATH. This is a problem in scenarios where (Ana)conda is installed in a read-only location, like an Apptainer/Shifter container or a shared storage location on a larger multi-user compute system/cluster. Since $JULIA_DEPOT_PATH gets ignored, the user runs into trouble like this

[...]
[juliapkg] Using Julia project at /opt/anaconda3/julia_env
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/user/.local/lib/python3.9/site-packages/juliapkg/deps.py", line 278, in project
    resolve()
  File "/user/.local/lib/python3.9/site-packages/juliapkg/deps.py", line 233, in resolve
    os.makedirs(project, exist_ok=True)
  File "/opt/anaconda3/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/opt/anaconda3/julia_env'

Could we change the resolution mechanism so that $JULIA_DEPOT_PATH takes priority over $CONDA_PREFIX (and ideally $VIRTUAL_ENV as well)?

Hi, thanks for your great packages including PythonCall and this one!

I've recently got started with PythonCall, especially the JuliaCall package from Python.

As my requirement is to access an existing Julia environment instead of creating a fresh one, I tried specifying PYTHON_JULIAPKG_PROJECT. Unfortunately, juliapkg just rewrites the existing Project.toml, and I haven't got a success without modifying the source code of PythonCall.jl.

I wonder if there can be a way to specify the current project without any hardcoding.

currently, juliapkg uses its own downloader if juliaup is not found, I'm not sure if @cjdoris is aware of jill.py but it'd be much better just to let a much more mature implementation handle this.

Hello i added juliapkg.json

{
    "packages": {
        "MedPipe3D": {
            "uuid": "80492ae4-ba08-4770-a16e-2cec06b8739e",
            "version": "0.1.0"
        }
    }
}

I had written in python

from juliacall import Main as jl
import juliapkg
juliapkg.resolve()

no errors here

still in python

import MedPipe3D

do not work

but how to start using MedPipe3D now - I had no problem in calling python packages in Julia using your tool - really fantastic job, here it is probably sth simple also - but I just do not see how to import MedPipe3D as a reference

If it is set, use that as the Julia project instead of an environment-specific one.

If it is set, only look for Julia there.

... when installing packages.

It'd be nice if there was a quiet mode as an optional keyword argument

... to specify a particular Julia executable to use.

should check or just run juliaup self upgrade before juliaup add <julia version>, otherwise we will end up in this error

(base) ➜  python git:(roger/python) ✗ juliaup add 1.7.3
Error: '1.7.3' is not a valid Julia version or channel name.

because my local juliaup was not updated after 1.7.3 was released

this is just a standard python package setup as following

.
├── README.md
├── juliapkg.json
├── pdm.lock
├── pdm_build.py
├── pyproject.toml
├── src
│   └── bloqade
│       ├── __init__.py
└── tests
    ├── __init__.py
    ├── test_scalar.py
    └── test_shape.py

where in the bloqade module, it calls juliacall, directly running python test_shape.py will not use juliapkg.json but creates an empty new one. This behavior does not look right.

This is causing things like pytest not working

I think I am seeing multiple processes trying to simultaneously resolve julia versions and dependencies, mirroring JuliaPy/CondaPkg.jl#40. Maybe locking is needed here too?

Currently, at least when we are not in offline mode, we Pkg.add all dependencies on every resolve. This is rather slow and can trigger unrequested package updates to indirect dependencies. It's also unnecessary when the required packages are already installed.

Fortunately, once can use Pkg.dependencies() to check versions without doing any add or resolve. For example:

def check_julia_deps():
    import juliapkg
    from juliacall import Main as jl
    from juliacall import Pkg

    avail_jl = Pkg.dependencies()
    for p in juliapkg.deps.required_packages():
        logger.debug(f"Checking pkg '{p.name}'.")
        try:
            dep_info = avail_jl[jl.Base.UUID(p.uuid)]
        except KeyError:
            logger.debug(f"Pkg '{p.name}' not found in Julia Manifest.")
            return False
        if not dep_info.is_direct_dep:
            logger.debug(f"Pkg '{p.name}' not found in Julia Project.")
            return False
        if p.dev:
            if dep_info.is_tracking_path:
                continue
            else:
                logger.debug(f"Pkg '{p.name}' should be dev'd, but isn't.")
                return False
        # NOTE: Julia Pkg.Versions.semver_spec may not be stable API
        if p.version and dep_info.version not in Pkg.Versions.semver_spec(p.version):
            logger.debug(f"Pkg '{p.name}' at {dep_info.version} not compatible with '{p.version}'.")
            return False
    return True

This does not cover all possible cases, I think, but could be extended. Should something like this be added to pyjuliapkg?

I'm wondering if we can instead of creating a juliapkg.json, but putting a field inside pyproject.toml like

[julia.dependencies]
julia = "1.8"
Example = "0.5"
PythonCall = "0.8"

this would making packaging a lot easier because pyproject.toml will be saved in a package wheel.