build(python): alternative build approach about stencila HOT 6 OPEN

After a change has been made to the build approach it should be tested as part of the release workflow in .github/workflows/release.yml (although I'm assuming that it will work fine because we are only maturin there ?). You can do that by going to https://github.com/stencila/stencila/actions/workflows/release.yml and only turning on Python SDK builds (for the current tag v2.0.0-alpha.23)

from stencila.

Thanks Simon. @brettc I wonder if you could look at this and make a PR that improves the docs and/or refines the Python DX.

from stencila.

I had a look over this. I think there are bigger issues about poetry. Here are my first thoughts (looking for feedback).

Poetry is great for developing python projects, but has poor support for extensions. It is currently a bit of a hack.

• python-poetry/poetry#2740
• PyO3/pyo3#953 (comment)

I have written my own "build.py" script for the c++ extension to deal with this.

In the current setup, this means that poetry does not really know about maturin. A "poetry install" will not work, as it does not call the maturin build. (Actually, it currently just breaks because it does not know where the package is, but that can be fixed).

Possible paths ahead:

1. Ignore these issues and just say in the docs you need poetry installed.
2. Write a "build.py" script that correctly invokes maturin.
3. Get rid of poetry and use some of the newer, standard python building tools.

For 3, I'm thinking we could save ourselves a lot of trouble by looking at the current version of pydantic-core (the rust based support for pydantic 2.0). It has the same layout, uses Makefiles, and has much of the funtionality desired, without using poetry.

Thoughts?

from stencila.

Get rid of poetry and use some of the newer, standard python building tools.

This seems like the right approach (I thought Poetry was the best, maybe it was, but now isn't 😕 ) and yep, copying what the other projects using maturin are doing is a good idea. I think that I started doing that but things seemed over complicated and I went for the simplest approach that would work to get robust release builds:

But all done quite hastily, and probably easily improved. If you think you can do that reasonably quickly, then go for it. I can show you how to run the workflow on GH Actions (without actually creating a release).

from stencila.

I had another fairly quick look at this issue, some of the alternatives, and at pydantic-core and agree with this:

For 3, I'm thinking we could save ourselves a lot of trouble by looking at the current version of pydantic-core (the rust based support for pydantic 2.0). It has the same layout, uses Makefiles, and has much of the funtionality desired, without using poetry.

My only ask is that we keep things as simple as possible (But No Simpler ™️ ): acknowledging from the outset that dropping poetry will immediately simplify things.

from stencila.

Further investigation:

• l looked at copying the maturin setup from pydantic-core, which has some similarities.
• It was unclear to me how they pinned their non-dev dependencies (I think there just are not that many). But this is something that I suspect we will want to do, if the python library starts supporting plugins and other niceties.
• Pinning, and especially updating the pinned dependencies, is non-trivial. Poetry (and also PDM) provides solutions to this problem. A simple pip freeze does not.
• So, for the ease of our futures selves, for now, we are going to stick with poetry, cos it is already there, and it mostly works (I have added one tiny fix that cause poetry install issues in #1851).
• The remaining problem is that you need to manually run maturin because poetry will not do it for you. But the makefile does the right thing.

TLDR: if you are a developer, you need poetry installed. Use the makefile commands to build and test.
We can revisit the building in the future, but I think we should close this issue @nokome

from stencila.