[stage1] Haxelib replacement about haxe-evolution HOT 41 CLOSED

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

from haxe-evolution.

Criteria:

• semver dependency management, and solution to handle conflicts as we can have only one version of a lib,
• global cache and local libs similarly to npm/yarn,
• support for git / file dependencies.

from haxe-evolution.

The package manager should use secure protocols (HTTPS vs. HTTP) for downloading packages (I found this to place limitations on the choice of a suitable runtime).

from haxe-evolution.

NPM isn't enough ?

from haxe-evolution.

The user should be able to use an alternative package manager (without making change to or replacing the haxe exe).

from haxe-evolution.

• Support lock files so we have reproductive build system

from haxe-evolution.

If I install a new dependency, upgrade a dependency, or change my dependencies in any way, this should show up in a way that is tracked in a file in my version control, and can be used to restore the same state on a different computer.

(Similar to a yarn.lock or haxe_libraries/*.hxml in haxeshim/lix)

from haxe-evolution.

haxelib.json should be extended, proposed new fields (non exhaustive):

• repository field (to go with the current website field)
• listing the metas/defines of the library
• haxe version compatibility

from haxe-evolution.

Configuration must be thoroughly documented and transparent. On-disk storage such as hidden .dev, and .current files should be discouraged and instead replaced by a documented configuration file.

The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.

from haxe-evolution.

I think we should deprecate haxelib run to avoid depending on neko.

Don't have to deprecate it, but it shouldn't require a run.n. It could, by convention, make haxe interpret a Run.hx script instead.

from haxe-evolution.

The following fields should be removed from haxelib.json (or its replacements) and dealt on the library repository website:

• description
• website
• contributors
• tags

since it doesn't make sense to update those by making new releases.

from haxe-evolution.

It should be possible to open a directory that a project is in, and when I type haxe build.hxml it automatically uses the configured version of Haxe for the project, and the right set of haxelibs/dependencies.

(This one might be controversial but I thought I'd throw it in!)

from haxe-evolution.

The compiler should expose an interface for frontends like CLIs, IDEs, vscode-style language servers, and so on.

from haxe-evolution.

The standard mechanism to invoke the compiler will be a first point of contact for newcomers to the language. As such it should provide an outstanding user experience and documentation can't be an afterthought.

from haxe-evolution.

From Haxe code it should still be possible to know which libs are used. so doing #if (libname > 4). It would also be useful to be able to know which classes come which library (in macros/ documentation xmls/dependency dumps etc)

from haxe-evolution.

Whatever ends up as standard mechanism to invoke the compiler for one-shot compilations must be very fast, there should be no noticeable difference to invoking the compiler directly (in other words start-up times like that of nodejs or even the JVM are out of the question)

from haxe-evolution.

A migration plan is required:

• what happens to the already installed libraries? do they need to be reinstalled?
• do we keep the haxelib online repository as is? if not what about usernames?

from haxe-evolution.

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

That means the compiler should be able to transform -lib tags into non-lib tags on its own, with the resolving logic being simple yet generic.

from haxe-evolution.

• Maintain a local cache of downloaded libs.
• Symlink cached libs to projects to save space. (flag to control behaviour and/or force local install)
• Support "globally" installing libs.
• Support licenses sub method to output all dependency licenses much like PHP's composer licenses.

from haxe-evolution.

@ibilon here is how lix deals with it, no absolute paths:

# @install: lix --silent download "haxelib:hxnodejs#4.0.9" into hxnodejs/4.0.9/haxelib
-D hxnodejs=4.0.9
-cp ${HAXESHIM_LIBCACHE}/hxnodejs/4.0.9/haxelib/src
-D nodejs
--macro allowPackage('sys')
--macro _hxnodejs.VersionWarning.include()

Here is how yarn.lock deals with it (basically, it doesn't, it just uses relative paths):

angular-cookies@^1.6.4:
  version "1.6.4"
  resolved "https://registry.yarnpkg.com/angular-cookies/-/angular-cookies-1.6.4.tgz#c28f3f6aac7a9826c1e45f1d6807240036e5b26d"

from haxe-evolution.

Beyond building projects the standard mechanism to invoke the compiler should provide a standardized way to run other common tasks like tests, benchmarks, updating dependencies, etc. (much in the spirit of cargo).

from haxe-evolution.

I have many feelings on haxelib, but quickly l want to share some "wish list" items:

• Support for multiple remote repositories
• Support for cascading local library paths
• Support for wildcard versions (IE -lib lime:3.1.*)
• Support for installing shortcut commands for tools
• Self-updating

Multiple remote repositories would allow companies to generate their own internal versions and distribute more easily, or enable "beta" or "dev" channel repositories.

Maintaining a single class path directory would not be reasonable for dealing with source files. Similarly, it would be nice to be able to add multiple custom library paths -- similar to how class paths are resolved.

Support for wildcard versions would really, really help (especially if semver is respected). You can request library version 1.0.*, allowing for patch releases. This is a great middle-solution between "Give me version 1.0.4 exactly" or "Give me any installed version", and a frequently requested feature for OpenFL

As a cherry on top, it would be nice if libraries could install command aliases, such as lime for haxelib run lime or as3hx for haxelib run as3hx

The best package manager in the world will need updates. I would like to see a library that is self-updated when you run a command similar to haxelib update

from haxe-evolution.

We shouldn't change the current workflow and so not introduce any steps between changing a hxml and running it/doing a haxe command by hand.

from haxe-evolution.

Interesting, what are the arguments against tool test, tool bench, tool update .. ?

Out of scope (except updates, but that's obvious). This is about a package manager, not a complete build tool, which would be one level higher.

from haxe-evolution.

I think we should deprecate haxelib run to avoid depending on neko.

from haxe-evolution.

I know @damoebius' suggestion to use npm has been firmly downvoted but it isn't that silly - alternatively it would be interesting to have a npm-compatible haxelib API so we could be really interesting to have.

from haxe-evolution.

The package manager should allow custom fields in haxelib.json
It would be very usefull for librairies which are tied to a specific framework or runtime.
i.e : For externs, we could provide a link to the original library and infos about how to integrate it in your haxe project
i.e : a web framework which loads plugins (haxelibs) that have various assets ( like models, controllers, templates and translation files )

from haxe-evolution.

The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.

That's too restrictive. While some package managers might agree on one database layout, others might not. A package manager using e.g. Nix as a backend will require a different layout than one based on a git repo layout, both of which would be reasonable choices to make.

It is and it isn't. While I respect the decision of writers of the extended or alternative package managers to create the solutions they think most appropriate, I believe that the canonical package manager (the one delivered as part of the Haxe toolset) must set an open standard that other tools can inter-operate with.

from haxe-evolution.

@sangohan @jcward: I removed your comments because they are way too wordy/not actionable. I've pasted them here so they are not lost: https://gist.github.com/Simn/2b13809d830cfd16dd5ba2f9c97cac6b

from haxe-evolution.

I recently discovered https://lib.haxe.org/p/hmm library which does a pretty nice job on that part with current limitations.

For me, the current haxelib version isn't really far from what I need, but here is my wishlist :

• Having a way to abstract the lib source (git, svn, local, ...)
• Having 2 scopes of lib loading : global and local
• Recursive loading (a lib depending itself on an other, should be able to load it automatically). That leads to a version conflict resolver !
• Having/Keeping a comprehensive synthax

from haxe-evolution.

@Simn @elsassph I agree it's not so much about about deprecating haxelib run itself, but rather about deprecating run.n in favor of Run.hx ;)

from haxe-evolution.

I would like to point out that it is important to consider how to handle "native code" (could be compiled ndll, C/C++/JS/... compiled libraries or source files).

One reason is that it is wasteful to download binaries of all architectures. e.g. Lime's ndll folder is 108M, but if you're only using Windows, you actually only need the 5.1M ndll/Windows folder.

We should also have some guidelines regarding to how to store the info of those "native" things, e.g. where is the original source, what param is used when building (use of -PIC/-O3/debug info), any additional dependencies that is not bundled etc.

from haxe-evolution.

Interesting, what are the arguments against tool test, tool bench, tool update .. ?

from haxe-evolution.

@clemos haxelib run isn't a problem per se, but it should be extended to other runtimes, and be configurable in haxelib.json.

from haxe-evolution.

Caching libs is good, using symlinks is bad: someone could mistakenly modify a file and it would affect the cache / other projects.
Libs should be copied locally in the project (unless optional CI optimisation to use cache).

from haxe-evolution.

That means the compiler should be able to transform -lib tags into non-lib tags on its own, with the resolving logic being simple yet generic.

Because -lib is the only thing that cannot be "consumed directly" by the compiler.

from haxe-evolution.

The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.

That's too restrictive. While some package managers might agree on one database layout, others might not. A package manager using e.g. Nix as a backend will require a different layout than one based on a git repo layout, both of which would be reasonable choices to make.

If what you mean is a common layout for a staging area, as well as for ex- and importing a package database to allow migration between package managers, OTOH, that'd be a good idea.

from haxe-evolution.

Interesting, what are the arguments against tool test, tool bench, tool update .. ?

Out of scope (except updates, but that's obvious). This is about a package manager, not a complete build tool, which would be one level higher.

I was under the impression we're talking about all the (future) components required for the workflows that haxelib is currently involved with, not just its role as a package manager.

If this is about package management only, I think we should look at the more principled approaches - so basically read A modular package manager architecture, conclude that we're probably not going to come up with something better, and hence "do whatever opam does".

from haxe-evolution.

(Similar to a yarn.lock or haxe_libraries/*.hxml in haxeshim/lix)

Can't put that in version control since it contains computer specific path into it.
EDIT: not actually the case, see @jasononeil comment below

As an alternative to a new file (like yarn.lock) maybe lock the versions directly in the hxml?

from haxe-evolution.

Build tools (hxcpp,hxjava,hxcs) should not rely on haxelib too.
They can be invoked by Context.onAfterGenerate or haxe --run

from haxe-evolution.

Stage 1 has concluded, I'll prepare the discussion part for Monday.

from haxe-evolution.