[stage2] Haxelib replacement about haxe-evolution HOT 20 CLOSED

So what's the motivation for HTTPS?

MITM attacks for one. However unlikely for anyone to spend their time on it, you just shouldn't transfer any code that will be run on your pc over http. It's just too easy to interfere with plain http data on public wifi or similar situations. Https should not be optional. I also don't see any reason for it to inhibit caching layers.

from haxe-evolution.

Is this still open? The due date was 8 days ago. I think I'll stick my $0.02 in anyway.

It's important to note that having reproducible builds is important to me, so I'm going to advocate anything that I think would bring haxe closer to having reproducible builds. I also have no clue how the internals of the haxe compiler or haxelib work, so bear with me.

• Haxeshim should definitely be the default behavior of the haxe compiler. This reduces the coupling between the two, which is important for allowing haxe to integrate into other package managers such as npm, nix, and guix in a cleaner way.
• Switchx (compiler version switching/locking) is also a good thing. Without a mechanism like this, build files and/or README's must include the version of the compiler and steps to install it.
• Lock files are a definite yes, for convenience more than anything. These should include the compiler versions. The implementation for this seems like it would be a little hairy, but it is also an essential part of making builds reproducible. A possible implementation could be having the haxe command as a script that interprets the version that needs to be (either defined in the environment, in a lock file, or passed in explicitly) that then hands off control to a specific version (e.g. haxe3-4-1).

EDIT:

• I don't like the idea of relying on lix as the standard package manager. It's nice enough, but I would prefer something that doesn't rely on the NPM/Node ecosystem. Preferably it could be compiled to C++, but that's just a personal preference.

from haxe-evolution.

The topic has been quite discussed, thank you to all the participants.
We are currently recruiting for HF (see https://haxe.org/blog/hf-is-recruiting/) and new haxelib will be one of the tasks to prioritize in our upcoming developments.

from haxe-evolution.

@ncannasse, @Simn and @back2dos isn't it the good time to start the stage 3? ;)

from haxe-evolution.

@back2dos: Which items of this feature set does lix not cover (at the moment)?

The package manager should use secure protocols (HTTPS vs. HTTP) for downloading packages (I found this to place limitations on the choice of a suitable runtime).

We will look into supporting https in the interpreter. From @ncannasse regarding that:

<nicolas> now that we have properly working ssl support for HL/Neko we should use these for OCaml/Interp as well
<nicolas> should be straightforward to backport HL ssl.hdll to ocaml FFI

Don't have to deprecate it, but it shouldn't require a run.n. It could, by convention, make haxe interpret a Run.hx script instead.

A problem with that is that it would require all the dependencies. One nice thing about run.n is that you don't need anything else. For instance, with Dox we provide a run.n so that users don't need all its dependencies to run it, only to compile it.

I'm currently working on a Haxe binary output which could help regarding that.

from haxe-evolution.

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

Either that's built-in in the compiler, and we better make it right the first time considering the compiler's release cycle (even if that's better now).

Or that falls to the user call a separate tool to obtain the information required for compilation.
Hardly an improvement in my opinion, how many time will one think "why doesn't it work, everything is good, oh I forgot to call the tool to tell the compiler I've changed my hxml file"?

Support lock files so we have reproductive build system

If I install a new dependency, upgrade a dependency, or change my dependencies in any way, this should show up in a way that is tracked in a file in my version control, and can be used to restore the same state on a different computer.

Not too familiar with the concept of lock files, do they allow that dependency change tracking?

--

@Simn it would probably be easier if the items from stage 1 had a number id.

from haxe-evolution.

Either that's built-in in the compiler (ocaml? slow release cycle?) ... Or that falls to the user call a separate tool ... Hardly an improvement in my opinion.

I had the same concern, but this "interpreting an .hx file" idea is interesting. It could alleviate these concerns.

HTTPS

Now this I don't understand. Of course you want HTTPS on the package authoring / upload side, but on the distribution / download side? HTTPS inhibits possible caching layers (reverse proxies, intermediate caches) and causes extra dependencies for the package manager. This is a public package repo, right? No login / secret info required. So what's the motivation for HTTPS?

from haxe-evolution.

Which items of this feature set does lix not cover (at the moment)?

Many of the features are actually covered just by haxeshim's stupid simple conventions, that could be trivially baked into the compiler (and should be IMO). From top to bottom, theses are the points that are yet open:

• semver dependency management, and solution to handle conflicts as we can have only one version of a lib,

  This is largely the matter of having a registry that actually exposes this information.

• support for git / file dependencies.

  Partial. See lix-pm/lix.client#19, lix-pm/lix.client#10 and lix-pm/lix.client#5 ... I haven't done much here, because lix places reproducible setups above all. Those features run counter my personal mission. If lix is considered as the "official" haxelib replacement, I am willing to compromise of course.

  One option is also to use mixed resolution strategy, which runs haxelib path for any libraries that cannot be resolved per haxeshim convention, in which case haxelib dev will work just fine.

• Don't have to deprecate it, but it shouldn't require a run.n. It could, by convention, make haxe interpret a Run.hx script instead.

  No script running. Because lix currently relies on npm and that already has it's own way of doing things, the necessity did not arise to rush towards something half baked. This is an interesting subject that should be explored in depth.

• haxelib.json should be extended, proposed new fields (non exhaustive):

  • repository field (to go with the current website field)
  • listing the metas/defines of the library
  • haxe version compatibility

  A non-exhaustive makes for a very vague feature specification :P

  Lix does not alter the haxelib.json in any way. That said, I quite like the idea of listing metas/defines. The question is whether or not we wish to process such information in the compiler or whether it's purely for documentation purposes (in which case nothing needs to be done in the package manager).

• The standard mechanism to invoke the compiler will be a first point of contact for newcomers to the language. As such it should provide an outstanding user experience and documentation can't be an afterthought.

  If "outstanding" is the benchmark here, then there's certainly more work to be done. The haxe command works as before, but the error messages could certainly be improved.

from haxe-evolution.

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

This should apply on build tools as well (e.g. hxcpp, hxjava, hxcs)

from haxe-evolution.

I have extended stage 2 time by a week because I won't be available until then anyway.

from haxe-evolution.

I've watched the London meetup, and tried the MVCoconut example, and I'm trying to digest the concepts.

To start, I think haxelib is a simple tool, it's easy to understand, and it does one thing (somewhat) well. Once you understand it and its limitations, you'll create tooling around it and live with it.

The idea of haxeshim seems fine enough -- a convention with a scoped directory structure containing library references, and a global cache of (referenced) versioned libraries. With a place for neko and haxe versions to boot. To me, the best things this provides beyond haxelib is: my dependencies are defined (and committed) with each project.

I'm less clear on lix, node, and npm joining the party. Seems like an awful lot of (JS-leaning) tooling prerequisites. Now I have to understand how NPM works and package.json and whatnot. It's a lot to take in, gotta look at it more.

Just to play devil's advocate, what could I do today with a simple script, checked into my repo, that gives me a reproducible setup with haxelib as it stands:

init_local_haxelib.sh

# Use a project-local haxelib
haxelib setup ./haxelib # or ./haxe_libraries
# [copy or symlink or unzip or wget, my choice] some cached libs to ./haxelib
# Clone some git haxelibs at exact commits:
haxelib git hxcpp-debugger https://github.com/HaxeFoundation/hxcpp-debugger

from haxe-evolution.

All this thinking about the very basics of Haxe lib management makes me wonder -- would it make haxelib better to simply remove features? What if...

• It only supported only git, and only a project-local ./haxe_libraries repository.
• The library's git repos contain haxelib.json and optionally versions.json specifying commits in history that are different versions of the lib.
• A project would define its dependencies as either
  1. git repo URL (aka, latest, whatever version is in haxelib.json), or
  2. git repo URL + major version (auto updates minors), or
  3. git rev-specific URL (obviously, this is a specific version)
• instead of haxelib path running every compile, path is always ./haxe_libraries/libName
• haxe --init-libs is only run once (to resolve, download, and optionally check for minor updates.) This interprets <haxe builtin>/NewHaxelib.hx, which sets up ./haxe_libraries, or you can override with ./NewHaxelib.hx if you like
• Global lib cache is enabled via an environment variable -- e.g. $GLOBAL_DIR/libName/version (or ~/.haxelib, I smell haxelib backward compatibility here...)

As for "version of Haxe" -- I consider this slightly out of scope. I use very simple, conventional, (and sadly OS-specific) symlinks to select versions of Haxe and it works like a charm.

from haxe-evolution.

• Supporting git only would be a pain to use some libraries that requires compilation like hxcpp

from haxe-evolution.

I'm less clear on lix, node, and npm joining the party. Seems like an awful lot of (JS-leaning) tooling prerequisites. Now I have to understand how NPM works and package.json and whatnot. It's a lot to take in, gotta look at it more.

FWIW understanding NPM is not such a bad choice ;)

What I have maintained for the past three years is that the conventions implemented in haxeshim should be built into the compiler. Beyond that point, I don't think we have to agree on anything. That's the beauty of it. People may or may not choose to use lix. If someone comes forward with a better package manager, I'm more than happy to use it, because I don't particularly enjoy working on such a thing. It's just that I have a desperate need for such a tool (and I believe that our community does share that need, even though they might not necessarily realize it ... more on that later).

Just to play devil's advocate, what could I do today with a simple script, checked into my repo, that gives me a reproducible setup with haxelib as it stands:

[...]

Uhm, nope. This is not reproducible in the slightest. Let's set aside the fact that if your machine has no git it fails and if it has cygwin git, it fails - better yet - for totally obscure reasons. The main point is: if you run that script a couple of months apart, you get two different versions of the code, where the old one might only be Haxe 3 compatible and the new one might only be Haxe 4 compatible. And that's just with one library. If you have 100 of them, the chance of winding up with compatible versions after running the script is practically 0. So you will have to write all the particular tags/commits into your setup script by hand and maintain it by hand and remember to rerun it, whenever you switch branches.

Sure, one can live with it. I documented a pretty equivalent approach years ago. Unfortunately, it just turns out that if something is optional and not automated, it doesn't get done (at least not sufficiently well).

That is why lix fully automates this part, for all dependencies, including the transitive ones.

All this thinking about the very basics of Haxe lib management makes me wonder -- would it make haxelib better to simply remove features?

I am generally a big fan of removing features. That said, if the alternative is to rely on git only, my advise would be to use git submodules as Rob does. If I din't always forget to update submodules when switching, I guess that's what I'd go with myself. But it might work perfectly for you.

As mentioned though, git is not universally supported. What's more: it's not a reliable source of 3rd party code, because the owner may alter/delete the history or the whole repo. In the coming months, lix will gain a create-only (no deletion, no alteration except deprecation) registry intended as the primary source of packages, which will prevent things like the npm fiasco to unfold in its core ecosystem.

As for "version of Haxe" -- I consider this slightly out of scope. I use very simple, conventional, (and sadly OS-specific) symlinks and it works like a charm.

Sure. It works for you, on your machine and I'm quite optimistic that on the whole many people have found some way to arrange themselves with haxe and haxelib. That's great, but it still leaves room for us to progress as a collective.

The main reason why I do see library and language version management as an indivisible unit is usage.

First, there's the matter of having a replicable setup, that will work the same on your CI and on every team member's machine (regardless of OS, regardless of which branch they are on, etc.). Big teams care about this. It's fair to say that the reason this debate was tabled is that at the foundation meeting on the summit the foundation partners agreed on such a need. And it involves the compiler version too. If a patch is landed for something we need, we want to be able to use it across all machines without much hassle - and only on a specific branch to see what other problems might occur. With lix's support for nightlies, we can.

Second, and I think this is just as important (if not more), I see much value in a solid basis for creating tutorials that don't just completely break within months. There's always talk about how Haxe needs more tutorials and what not, but very little thought is given to the technical prerequisites to make them useful, such as making sure that the code actually works for the user. I believe there's a huge difference between a code example that is slightly outdated but compiles just fine, because setting it up was straight forward, and a code example that is slightly outdated and greets you with 500 different errors, half of which come deep out of some macro, that was broken between haxe 3.2.1 (that the tutorial was written for) and 3.4.4 (that the user has on their machine). It might even be detrimental to lock versions in such a case, because the library that the example is based on most likely had a haxe 3.4.4 compatibility update in the mean time that just didn't get installed due to version locking.

So, if you look at these two version management problems not from a purely technical perspective, but through the prism of meaningful use cases, they superpose almost wholly. Given how often concerns are voiced about reducing friction for newcomers, this could make an important difference. My advise is not to waste such an opportunity.

Adding just the library convention into the compiler with fallback to haxelib borders on trivial. It's easy to do and breaks nothing, providing a clear migration path. Because of that, it's probably even worth adding to Haxe 3.

As for Haxe 4, I think the idea of supporting parallel Haxe versions at least deserves thorough consideration. If it's too hard to add, that's one thing. But it shouldn't be dismissed merely on the grounds that "it's out of scope". It isn't.

from haxe-evolution.

Thanks for the thoughtful responses, @back2dos.

This is not reproducible in the slightest. ... if you run that script a couple of months apart, you get two different versions of the code.

Oh, I meant to specify a specific commit URL.

the conventions implemented in haxeshim should be built into the compiler. Beyond that point, I don't think we have to agree on anything. That's the beauty of it.

Yes, I think I was just realizing that. So haxeshim (hence, Haxe itself) would expect those conventional directory structures, and you're free to populate them however you choose. I like this a lot (this was the core of my "simple script" idea -- just tell Haxe to look in some conventional folder, and beyond that, I'll script a reproducible setup.)

There does need to be some name / version conventions on the lib side, though, doesn't there? I suppose a library will choose to support lix, thus using lix conventions? It might be nice to be as generic as possible in that specification, so other tools can parse versions and dependencies.

Alright, well anyway, just trying to follow along. I appreciate the dialog. Cheers! 🍺

from haxe-evolution.

Who can push this subject on the next step? @ncannasse

The package manager is one of the basics in the tool chain.
We regularly reach the limit with haxelib on projects with a lot of dependencies.

I really hope it will be part of the next major release. :)

from haxe-evolution.

I would also like to bring back a point previously made by @andyli:

it is important to consider how to handle "native code" (could be compiled ndll, C/C++/JS/... compiled libraries or source files).

from haxe-evolution.

Sorry about that, that was bad timing on my part. I'll close this issue for now.

... to be continued

from haxe-evolution.

Just to make sure this is not forgotten

from haxe-evolution.

Is this happening in haxe 4?

from haxe-evolution.