Comments (11)
cargo2nix
could be a builder in dream2nix
that uses IFD, using cargo2nix
to generate a Cargo.nix
and then building that. Another way would be making a translator in dream2nix
that encodes information needed to use mkcrate
into the dream-lock
and make a builder using mkcrate
. This would also require IFD, but the IFD would be moved into the translation phase. (EDIT: though if the Cargo.lock
file had the required information for building the crates as you said in the cargo2nix
architecture issue, then it would be possible to have it build on-the-fly with no IFD.)
I would be glad to help with any of the questions etc. you might have in regards to the Rust subsystem, or if you want to implement stuff in dream2nix
using cargo2nix
code.
from dream2nix.
Hi @psionic-k
My goal is to implement independent derivation building in a way suitable for nixpkgs long-term, maximizing cache granularity.
Independent (or granular) derivation building is a goal of dream2nix as well and already implemented by the nodejs subsystem for example.
Use in nixpkgs
The long term goal of dream2nix is to bring the whole dream2nix framework into nixpkgs. This prevents an IFD import of the framework.
As this only makes sense once dream2nix reaches a stable release, I plan to add a standalone package export
feature where only the dream-lock.json and builder code required for a specific package is dumped into nixpkgs.
Regarding parsing Cargo.toml/Cargo.lock files via nix.
This is already implemented in dream2nix (see the cargo-lock translator).
You can basically forget about the parsing and only add the granular builder.
I just updated the builder template. Take a look here
I also just polished the contributor guide a bit.
Will look at the structure of dream2nix overrides before updating cargo2nix's overrides. Same for fetching.
You don't have to use the dream2nix' own overrides format. dream2nix is also compatible to the well known nixpkgs override style. I think @yusdacra uses that in nix-cargo-integration as well, if I'm not mistaken.
from dream2nix.
You don't have to use the dream2nix' own overrides format. dream2nix is also compatible to the well known nixpkgs override style. I think @yusdacra uses that in nix-cargo-integration as well, if I'm not mistaken.
Yes, nix-cargo-integration
uses them. It used to use only crate2nix
in it's infancy, so I haven't changed the overrides stuff since then.
from dream2nix.
We had some private registry support in cargo2nix and I had been meaning to at least figure out what it needed to work and get it documented, but fetching is very non-specific to cargo2nix. Are Rust private registries supported here?
from dream2nix.
We had some private registry support in cargo2nix and I had been meaning to at least figure out what it needed to work and get it documented, but fetching is very non-specific to cargo2nix. Are Rust private registries supported here?
No, haven't worked on that yet. But I would like to hear how it's done in cargo2nix.
from dream2nix.
This is already implemented in dream2nix (see the cargo-lock translator). You can basically forget about the parsing and only add the granular builder.
I'm re-iterating what other replies also show awareness of. The above statement is not quite true yet. We lose too much feature and platform information from the top-level when building crates independently. The lock is just not sufficient info yet. Cargo will forget feature activation if only using the lock in a deep dependency.
Making cargo generate enough lockfile info to avoid expression generation is one of the end goals currently identified in cargo2nix. It will require eventual upstreaming into cargo in order to live. Cargo2nix will just be the POC / prototype at that point.
No, haven't worked on that yet. But I would like to hear how it's done in cargo2nix.
I don't think it was good. Someone pretty inexperienced worked on it, and it was just for a small internal use case. AFAIK crane looks the most robust for external registry support.
https://github.com/cargo2nix/cargo2nix/blob/release-0.11.0/overlay/lib/fetch.nix#L13-L39
You don't have to use the dream2nix' own overrides format. dream2nix is also compatible to the well known nixpkgs override style.
cargo2nix derivations are as well. I think I'm just improving some plumbing for the cargo2nix user interface, so no heavy lifting will be needed anywhere.
Overall it seems like mkcrate.nix and the overrides, applied via makePackageSetInternal
, are the big pieces that will be useful to make roughly compatible with dream2nix. While I don't think that mkcrate.nix is actually good yet, it's building a lot of things in CI and has enough stability to go in a healthy direction. The main crimes I know of:
- it clobbers the .cargo/config.toml, building a new one with string concatenation, getting in the way of a lot of asked-for linking and other configuration
- remarshal + jq usage to facilitate querying and writing tomls, some querying to recover linking information necessary for downstream cargo invocations when building dependents
- using cargo may limit workflows that invoke rustc directly, although I'm unaware of these cases
- "bare metal" linking not really supported
One of the main architectural choices I'm figuring out now is if cargo2nix really needs an overlay or to expose the combined overlay. While applying overrides to nixpkgs or crates is necessary, I'm not sure that this requires giving a modified nixpkgs back to the user when a lib attribute could probably do all of the same. How nixpkgs gets configured for cross compile support is another aspect that might affect the right choice.
There's some wasm code I'm looking at now to get a better idea of what will be necessary in these cases.
from dream2nix.
Is there still any intention of integrating cargo2nix with dream2nix? Otherwise we should close this issue.
from dream2nix.
I don't have any intentions to work on this right now, so I think we can close it and if anyone else wants later we can reopen it.
from dream2nix.
I was asleep when this conversation happened. My priority is definitely cargo2nix before attempting to package pieces of it for inclusion in dream2nix, so I'm not saying anything will happen soon, but this isn't a dead issue at all. There is a zero chance that the granular derivation-for-each-crate style isn't the better solution for most use cases, so the project has an inevitable future unless one of the other solutions offers up a granular caching implementation first.
from dream2nix.
OK, thanks for the update. Then let's re-open the issue in the future once there will be collaboration.
from dream2nix.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/rust-cargo-cargo2nix-vs-crane-vs-dream2nix-vs/25001/1
from dream2nix.
Related Issues (20)
- Separate the pyEnv and devShell parts of pip module into their own module(s)
- Start shell with dependencies without creating a new flake file HOT 2
- PDM Locking fails if only config.deps.python is set HOT 1
- nodejs: native module compilation failure, need help configuring nativeBuildInputs HOT 3
- Pip installed CLI tool not in PATH HOT 3
- WIP - pdm add spyder # python3.12-pyqt5-5.15.10 - pdm.sourceSelector.<function body> is not of type 'string'. HOT 2
- WIP - pdm - Python version from d2n nixpkgs HOT 1
- [help] usage of multi-derivation-package HOT 1
- Question: Python package with libusb dependency HOT 2
- error: infinite recursion encountered # nix run github:nix-community/dream2nix#__pythonEnv extract-msg HOT 2
- realpath: command not found HOT 1
- How to make the lock building process for python proxy aware?
- Error: attribute 'overrideAttrs' missing when running nodejs-granular-v3 devshell
- bad (null) hash for package from extra pypi index
- Unable to provide nativeBuildInputs to nodejs transitive dependencies
- error: No valid filename found in the list of filenames: - PyQt5_Qt5-5.15.13-py3-none-macosx_10_13_x86_64.whl
- Can't import fsspec gcsfs
- python local developpement example using only requirements.txt and no pyproject.toml
- How to build pip package from source that requires rust toolchain HOT 2
- Support maturin pip build backend HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dream2nix.