Merge deps and prebuilt_dependencies about rules_haskell HOT 8 CLOSED

I forgot to add that LTS package sets are not exhaustive: the tool should be able to work on stackage2nix output because that tool can also deal with user's extra-deps &c.

Possibly we'll need to improve stackage2nix: it's currently very heavy to build (builds nearly all of stack itself…) and is quite slow (I know for sure I put sub-optimal code in there once…). Improvements here will translate directly to improvements for us.

from rules_haskell.

There is another, low-budget, way of solving this:

• instead of relying on a single global ghcWithPackages Nix instantiation, consider (unlike Cabal) that prebuilt dependencies are not so much a property of a library/binary, but rather of its environment/toolchain.

Concretely, instead of

haskell_library(
  name = "foo",
  srcs = [glob("*.hs")],
  prebuilt_deps = ["base", "bytestring", "conduit", ...]
)

We'd have,

haskell_pkgset_nixpkgs(
  name = "pkgset",
  pkgs = ["base", "bytestring", "conduit", ...],
)

haskell_library(
  name = "foo",
  srcs = [glob("*.hs")],
  pkgset = ":pkgset",
)

A haskell_pkgset_nixpkgs rule would create a new compiler using ghcWithPackages with the given set of packages. No global list needed anymore. The point of introducing the notion of pkgset is that in the future pkgsets could conceivably be created any way the user likes. That is, we could have haskell_pkgset_bazel that creates a pkgset entirely from within Bazel without relying on Nix, or even haskell_pkgset_stack that reuses the Stack snapshot cache.

The downside of this approach is that it can add a few seconds to the total compile time, because realizing a Nix derivation expressed as an application of ghcWithPackages is often slow. Doing it once instead of per-package saves a little bit of time. Won't slow down incremental builds, however.

Note: we'll probably need to name pkgsets anyways if users start created fine-grained libraries, because we'll probably want one pkgset per Cabal package, not one per BUILD file scattered across the source tree of a single package. That is - avoid stating more metadata than Cabal.

from rules_haskell.

Here's a refinement on the previous comment: we could use the toolchain mechanism to register "pkgset builders". Nixpkgs/ghcWithPackages could be one such builder. That way, we could specify pkgsets inline as we do now:

haskell_library(
  name = "foo",
  srcs = [glob("*.hs")],
  pkgs = ["base", "bytestring", "conduit", ...],
)

The pkgset builder to use would be implicit. It would depend on what pkgset builders were registered in the current workspace (there should only really be one).

from rules_haskell.

In rules_scala we face something similar, except on the JVM people want to use already compiled code from maven servers. I wrote a tool to generate the bazel targets to import these directly: https://github.com/johnynek/bazel-deps

I can imagine something similar like cabal2bazel, which could convert a cabal build into a bazel build. If we had this, and the ability to depend on in-WORKSPACE packages (which I guess you have now), that would be an ideal solution to the problem, no?

Note, bazel has build caching and external caching support, so even if it gets slow, a group of people can share a build cache as long as the rules are hermetic it should be safe.

from rules_haskell.

Right, if we had a tool to do the conversion then we could do just that. The ideas here are mostly on how to avoid having to write such a tool straight up and re-use existing tooling (like cabal2nix/stackage2nix). Part of the motivation is that we'd have to support every feature needed by most of the ecosystem: certainly the common dependency chain. This can't really happen over night so we're looking for intermediate solution which leans on existing expressions to provide the packages.

I'm sure I'll get corrected if I'm wrong.

from rules_haskell.

Such a tool sounds to me like the sensible approach long term. Related ticket: #17.

from rules_haskell.

We discussed a bit this issue with @mboes yesterday, and after a bit more of
thinking, here's a (probably quite naive) possible interface for this:

We reuse the haskell_import rule that @judah wrote, but make it depend on
a toolchain defining how to get these packages:

• The simplest case is a dummy toolchain builtin_haskell_provider which
  assumes that ghc already knows how to find the package and just adds the
  needed cli flags (so the current behavior)

• The other (or at least another) toolchain nix_haskell_provider get these
  packages using nix and outputs a an extended version of
  HaskellPrebuiltPackageInfo so that it can be used as any
  bazel-build package

The nix_haskell_provider toolchain has an interface similar to
nixpkgs_package, but instead of building a package, it returns a (nix) set of
haskell packages. A simple invokation of it would be

nix_haskell_provider(
  repository = "@nixpkgs",
  path = "haskell.packages.ghc843",
)

Once this toolchain is registered, calling haskell_import(name = "text")
calls nix-build on @nixpkgs.haskell.packages.ghc843 and returns a
HaskellPrebuiltPackageInfo with the needed informations to build and link
against this package (The idea is roughly to reimplement on the bazel side
what's done in
https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/generic-builder.nix#L284-L320,).

HaskellPrebuiltPackageInfo is defined as:

HaskellPrebuiltPackageInfo = provider(
    doc = "Information about a prebuilt GHC package.",
    fields = {
        "package": "Package name",
        "package_root": "Directory containing the package (or None if the package is bundled with ghc)",
    },
)

When building we add -L{package_root}/lib and -I{package_root}/include
to the ghc cli for each prebuilt transitive dependency, as well as
-package-db={package_root}/package.conf.d -package={package} for each direct
prebuilt dependency.

from rules_haskell.

This was fixed in #25. prebuilt_libraries is now deprecated.

from rules_haskell.