Run dream2nix on lang / package manager ecosystems about dream2nix HOT 13 CLOSED

There is now https://github.com/yusdacra/dream2nix-crates-io for crates.io! It handles indexing (by most downloaded, newest, recently updated etc. crates), translation (to dream-lock) and exposes these locks as packages. An automatic CI job regenerates the index & lockfiles every day. Currently this is only done for the top 1000 most downloaded crates.

from dream2nix.

I have been discussing this with @tomberek a bit. For building lot's of derivations, we could simply use hydra. Hydras API will allow us to access logs of failed builds etc. We already have some small hydra infra in place for dream2nix and could potentially expand its capacity if deemed necessary.

That just leaves the question, how would we generate all the derivations for hydra.

I imagine the following generic steps for each ecosystem

1. Indexing

Simply list all package candidates (specific names and versions) to build. Then write this list to a file.
Some filters could be applied, so we don't end up listing every single package, but maybe just the 5k most popular ones, and only the last x Minor releases, etc...

2. Translating

We could introduce a new type of translator, which instead of translating from a source tree, receives a name and version and generates the dream-lock from that (using the APIs of the ecosystem)

Then we just need to run these translators against our list of candidates from the previous step and we will get a set of dream-locks.

3. Realizing

Some nix expression that reads in the dream-lock files and generates a package set for hydra.

@yusdacra I agree it makes sense to have a repo for each ecosystem where we store the index and the generated dream-lock files.

These repos would also contain a flake generating outputs for hydra.

Basically the flake for each repo could be based on the default flake for dream2nix, but instead of passing a project source, it would pass the whole package index and provide a flake app for updating the index.

from dream2nix.

I agree with that. I can get started on this, with crates.io at first and gradually support everything else dream2nix can support.

It would be nice if you could point out to a resource for what hydra expects as a package set, as I haven't used it before.

from dream2nix.

I can provide an intro to using hydra if you'd like. If using flakes, it's pretty easy, create a flake output under hydraJobs (eg: https://github.com/tomberek/ssentr/blob/master/flake.nix#L26)

from dream2nix.

I can provide an intro to using hydra if you'd like. If using flakes, it's pretty easy, create a flake output under hydraJobs (eg: https://github.com/tomberek/ssentr/blob/master/flake.nix#L26)

That would be appreciated, but I think I want to try it out myself first and see if I can get it working. Also thanks for the example flake, it looks like it should be pretty simple then.

from dream2nix.

I agree with that. I can get started on this, with crates.io at first and gradually support everything else dream2nix can support.

Sounds great!

from dream2nix.

@yusdacra I now had a first glance at dream2nix-index-lib. I still have to take a closer look, but I can already see that there is significant complexity in it.

I'm wondering what we would need to change in dream2nix to make these indexing projects more simple.
Could you summarize what were the main issues developing that and what you think needs to be improved in dream2nix.

It would be cool to create a new module type for indexers besides translators and builders.

from dream2nix.

@yusdacra I now had a first glance at dream2nix-index-lib. I still have to take a closer look, but I can already see that there is significant complexity in it.

I'm wondering what we would need to change in dream2nix to make these indexing projects more simple. Could you summarize what were the main issues developing that and what you think needs to be improved in dream2nix.

It probably looks complex because of the translateScript part, which honestly I shouldn't have implemented that, the same function already exists in dream2nix, generateResolveImpureScript and makeTranslateScript. I don't think using that was obvious to me while I was writing this though 😅 But yeah, I'll probably make it use those and it should be much more simpler...

Something else I was thinking of using recursive-nix and impure derivations to do the same thing but with "just nix" instead... Generating a derivation for each lock, and then combining these in a big derivation that would have all the lock files. This has the advantage of not needing separate scripts / programs, but it does require experimental features. But this is more of a "make it more pure-ly nix" idea I had so it's not really important here 😅

And I'm not sure about if we need anything extra in dream2nix for this. I'd like to make dream2nix-index-lib obsolete, so I've written my thoughts below.

It would be cool to create a new module type for indexers besides translators and builders.

That should be simple. Indexers would index some source maybe with some extra user parameters (as they currently do in dream2nix-crates-io). For their outputs, we can just use the same structure that the dream2nix lock file has (the sources section specifically) for that. I think we can use the combined fetcher to fetch all the sources at the same time, then discovering would just be running discoverProjects on that, and we can use those with generateResolveImpureScript. This should be pretty minimal I believe.

from dream2nix.

It probably looks complex because of the translateScript part, which honestly I shouldn't have implemented that, the same function already exists in dream2nix, generateResolveImpureScript and makeTranslateScript. I don't think using that was obvious to me while I was writing this though sweat_smile But yeah, I'll probably make it use those and it should be much more simpler...

Ok, got it. Yeah those are internals and haven't been documented so far. But it definitely makes sense to re-use them.

I agree, we should use impure-derivations for generateResolveImpureScript and makeTranslateScript. Using experimental features is not a problem for this functionality I think, because we can invoke the version of nix we want with the flags we choose if we still expose that function via a flake app. Therefore users don't have to enable the impure drv feature in their nix client, in order for #resolveImpure to work.

That should be simple. Indexers would index some source maybe with some extra user parameters (as they currently do in dream2nix-crates-io). For their outputs, we can just use the same structure that the dream2nix lock file has (the sources section specifically) for that. I think we can use the combined fetcher to fetch all the sources at the same time, then discovering would just be running discoverProjects on that, and we can use those with generateResolveImpureScript. This should be pretty minimal I believe.

I'm not sure if translating packages from their source, as we do it in the normal dream2nix is the best method here. It assumes that packages carry all necessary metadata in their source, but in some ecosystems, they don't.

With python for example, the package sources don't need to contain any parsable metadata. Sometimes they just contain a setup.pyscript that installs the package. There is no enforced standard of where exactly name and version are stored inside the source tree. Or with haskell for example, the .cabal file which contains the metadata, can be updated by the index separately, which is not reflected in the files contained in the tar.gz source. Therefore reading the metadata from the source there might be invalid.

Therefore my idea would be to introduce a new kind of impure translator, which doesn't receive a source parameter, but instead just a package identifier (name, version etc.). It then communicates with an online index, like cargo, pypi, nodejs repo, etc. to get information about that package and then builds the dream-lock.json.
This might or might not include:

• downloading and analyze the package source
• calling another impure translator that translates from source (for cargo this might be the way to go)

For most subsystems this additional impure translator might just be a wrapper around an existing impure translator that downlaods the package source and then calls the existing translator.
This also allows for a relatively simple interface for adding support for new ecosystems. One just needs to implement that extra translator plus some package selection function (an impure drv) that determines the initial list of package identifiers to translate.

from dream2nix.

If we have translators which translate from a name and version, we can also implement sugary things like this command:

dream2nix run cargo:{pname}/{version}

... which translates that package, generates a flake via makeFlakeOutputs, and then executes the default package or app.
The same could work for devShells, to get a dev shell on an arbitrary project that doesn't contain any nix files yes.

from dream2nix.

I'm not sure if translating packages from their source, as we do it in the normal dream2nix is the best method here. It assumes that packages carry all necessary metadata in their source, but in some ecosystems, they don't.

Ok yeah that makes sense. Agree with everything you said then. It does sound like it should be pretty simple (at least for crates-io and npm for now). I'll work on adding the translators for those, and also implement indexer modules for crates-io and npm (will just move over the indexers from dream2nix-crates-io and dream2nix-npm respectively). I'll also make it so that dream2nix generates apps for available indexers that index and then translate the packages and put them in a folder (like dream2nix-index-lib currently does). Does that sound good?

from dream2nix.

Sounds wonderful! The apps should probably not be exposed in the dream2nix main flake but via some function or attribute which can then be used by dream2nix-crates-io etc.

from dream2nix.

Closing this since we now have indexers and translate-index etc.

from dream2nix.