bundler downloads gem dependencies rather than Bazel downloader about rules_ruby HOT 44 CLOSED

👋 I've been following this issue, apologies for not jumping in earlier.

I agree with the sense being expressed here that we should probably just rely on bundler for resolving dependencies (i.e., writing the Gemfile.lock), and then figure out how to download those dependencies. The command to do this is bundle lock.

With respect to checksums, bundler 2.5.0.dev (which will be in the upcoming Ruby 3.3 release) does write checksums to the lock file, in a new section named CHECKSUMS. It looks like this:

CHECKSUMS
  ast (2.4.2) sha256=1e280232e6a33754cde542bc5ef85520b74db2aac73ec14acef453784447cc12
  docile (1.4.0) sha256=5f1734bde23721245c20c3d723e76c104208e1aa01277a69901ce770f0ebb8d3
  json (2.7.1) sha256=187ea312fb58420ff0c40f40af1862651d4295c8675267c6a1c353f1a0ac3265
  language_server-protocol (3.17.0.3) sha256=3d5c58c02f44a20d972957a9febe386d7e7468ab3900ce6bd2b563dd910c6b3f
  mini_portile2 (2.8.5) sha256=7a37db8ae758086c3c3ac3a59c036704d331e965d5e106635e4a42d6e66089ce
  minitest (5.20.0) sha256=a3faf26a757ced073aaae0bd10481340f53e221a4f50d8a6033591555374752e
  minitest-parallel_fork (2.0.0) sha256=9d5889fc32bdef78ff92051a81ba82317ec0bee07a8594e0dec14c38d3710a7d
  parallel (1.23.0) sha256=27154713ad6ef32fa3dcb7788a721d6c07bca77e72443b4c6080a14145288c49
  parser (3.2.2.4) sha256=edbe6751f85599c8152173ccadbd708f444b7214de2a1d4969441a68e06ac964
  psych (5.1.1.1) sha256=44b0d1823629ac815f1f470af642dc7261489d67feb622a3f5573aa9f5cc5f72
  racc (1.7.3) sha256=b785ab8a30ec43bce073c51dbbe791fd27000f68d1c996c95da98bf685316905
  rainbow (3.1.1) sha256=039491aa3a89f42efa1d6dec2fc4e62ede96eb6acd95e52f1ad581182b79bc6a
  rake (13.1.0) sha256=be6a3e1aa7f66e6c65fa57555234eb75ce4cf4ada077658449207205474199c6
  rake-compiler (1.2.5) sha256=ab9b03a6db5fce88fb8dcff23c0cae3bccd7166f036c7fc8b2199f53ee38b2ac

For more information on this new feature, start with rubygems/rubygems#6374

HTH

from rules_ruby.

copy/place the generated Gemfile.lock file

We should require that the user has already done this using the typical developer tooling before invoking bazel for the first time. Is there a bundler command to lock the dependencies without actually installing them?

from rules_ruby.

Bazel ruby projects required the lock file to be in the ruby project before Bazel can be used to build it

Yes, this is how it works in other languages. You must run the resolver from the package manager in order to create the lockfile that our repository rule begins from.

from rules_ruby.

I was just reading the blog post on Tinder Engineering about migrating their iOS app to Bazel. They mentioned the same exact issue we are dealing with Bundler. Here is what they said:

from rules_ruby.

bundler accepts a CLI argument that points to a Gemfile if it's not called that or if it's in a different folder.

The environment variable BUNDLE_GEMFILE can also be used instead of the bundler CLI argument --gemfile=GEMFILE. Would that be easier to support?

I don't know if it's easier but the point is that some repos, in particular gems, might have several Gemfiles aimed at different parts of the source tree.

So we can't just assume that it's always called Gemfile and Gemfile.lock.

A repo with alternative files will have some macro (a makefile target, a shell script),that invokes commands by overriding the default Gemfile name either via the environment variable or via the flag to bundle. If we are going to convert some of those commands to Bazel we will have to preserve that and be able to parse the lock files named whatever.

In other words it might be a good idea to create a Bazel Struct or some other representation of both the Gemfile and Gemfile.lock that can be parametrized with a file name. Then any further bundle commands would have to depend on that target.

from rules_ruby.

Also, I thought Gemfile.lock does have a platform section?

It does, but it only informs about the platforms of the whole Gemfile, not a specific relationship between a gem and its platforms.

from rules_ruby.

@sushain97 could you please elaborate on __package.rb file?

Completely internal system. I don't think there's anything published externally about it but it's effectively a way for a set of Ruby files to package itself and export/import specific constants within that package's namespace.

edit: It does look similar to packwerk but I believe our system predates it.

from rules_ruby.

Is this a public thread? Can anyone see it?

I just reached out to @tenderlove and @indirect to weigh in.

from rules_ruby.

I agree it would be ideal, though I wonder how hard is it going to implement this. There are few questions for me at the moment:

1. Do we have example how to download from Git repositories? It's a common practice to have the following in gemfile lock:

GIT
  remote: [email protected]:p0deje/watirsome.git
  revision: f893d526333af7adcd8dd4e2f0f59d1204345664
  specs:
    watirsome (0.3.0)
      watir (>= 6.0)

2. Is there any way to propagate GitHub credentials to Bazel downloader? It's essential to install gems from private Git repositories.
3. How to keep Gemfile.lock updated? We still need to run bundle install to regenerate the lockfile.
4. Once downloaded, we can try to lay everything on disk similar to bundle fetch and then let bundle install to install it from this cache-like structure. I haven't played with it yet.

One alternative I planned to do was to try using bundle fetch + bundle install to at least avoid installation of gems in WORKSPACE. In this setup, bundle fetch would download all gems into a cache folder, while bundle install would be run as a BUILD target, installing gems from the cache.

from rules_ruby.

So let's not forget the main goal of bundler: it's to resolve the versions of the gems assuming lock file does not exist.

In Ruby we generally git checking Gemfile + Gemfile.lock for apps (rails apps, cli apps etc).

However, for gems we never git commit the Gemfile.lock because we want the gem to be pulled into a 3rd party app with its own versions.

from rules_ruby.

Bundler has the following common tasks:

• Bundle install
• Bundle update [gem1,...]
• Bundle update --bundler
• Bundle cache
• bundle config
• bundle check

from rules_ruby.

However, for gems we never git commit the Gemfile.lock because we want the gem to be pulled into a 3rd party app with its own versions.

This used to be the case, but currently Bundler recommends checking in Gemfile.lock for gems as well - https://bundler.io/guides/faq.html#using-gemfiles-inside-gems:

Q: Should I commit my Gemfile.lock when writing a gem?
A: Yes, you should commit it. The presence of a Gemfile.lock in a gem’s repository ensures that a fresh checkout of the repository uses the exact same set of dependencies every time. We believe this makes repositories more friendly towards new and existing contributors. Ideally, anyone should be able to clone the repo, run bundle install, and have passing tests. If you don’t check in your Gemfile.lock, new contributors can get different versions of your dependencies, and run into failing tests that they don’t know how to fix.

from rules_ruby.

Yup, another way to explain it: Bundler should still be used to read the user description on dependencies and run its resolver sand constraint solver, then the user must check in the result. We just need to parse it in starlark and try to download exactly the same files bundler would.

from rules_ruby.

from rules_ruby.

I suppose we need to re-implement https://github.com/rubygems/bundler/blob/master/lib/bundler/lockfile_parser.rb in Starlark first, then iterate over all the gems and download then. Ruby code relies a lot on usage of regular expressions, I am not sure it's possible in Starlark - at least there are no regexps there as far as I know.

@alexeagle Can we instead do this using Ruby script that would produce a list of files to download and then use this list in a repository rule?

from rules_ruby.

Getting a ruby interpreter in the repository rule context is very difficult and doing work in repository rules is a bad practice since the result isn't cached. We can write a parser in starlark, we've already had to do YAML (for pnpm) and TOML (for python pdm).

from rules_ruby.

We can write a parser in starlark, we've already had to do YAML (for pnpm) and TOML (for python pdm).

Oh, that sounds like fun :D Can you share some links here to see how it was done?

Is there a bundler command to lock the dependencies without actually installing them?

Yes, there is bundle lock.

from rules_ruby.

We can write a parser in starlark, we've already had to do YAML (for pnpm) and TOML (for python pdm).

I suppose it's in https://github.com/aspect-build/rules_js/blob/fdb95cffe366e2e3b62de41b2921c906621ec4aa/js/private/yaml.bzl, can't find TOML parser though.

from rules_ruby.

So let's not forget what bundler does as it's perhaps primary task. I believe most other features are extraneous.

The syntax that the Gemfile supports allows defining gems together with an optional version specification notation:

gem 'rails', '7.1.0'
gem 'rack',  '>=1.0'
gem 'puma',  '~>6'
gem 'puma-daemon'

Version specifiers are defined by the rubygems.

When you run bundle lock or install the first thing that bundler has to do is to traverse the version trees and their dependencies as defined on rubygems.org, and then find the combinations of the gem versions that satisfy the Gemfile specifications.

This is far from a trivial problem, and the algorithm itself went through multiple iterations. The latest update completely revamped it.

In other words, I think that if you can skip reimplementing any of this in Starlark, you absolutely should skip it.

TLDR

I think it might make most sense that if a Bazel ruby project has a Gemfile, but lacks the lock file, then maybe one of the following can happen:

1. If it's possible to run some shell rule that executes bundle lock to generate the lock file in the current folder, then we should do that.

2. Alternatively, I don't think it would be the end of the world if Bazel ruby projects required the lock file to be in the ruby project before Bazel can be used to build it.

Caveat

Another complication is that some ruby projects might have more than one Gemfile. While it's not common or recommended, bundler accepts a CLI argument that points to a Gemfile if it's not called that or if it's in a different folder.

Thoughts?

from rules_ruby.

bundler accepts a CLI argument that points to a Gemfile if it's not called that or if it's in a different folder.

The environment variable BUNDLE_GEMFILE can also be used instead of the bundler CLI argument --gemfile=GEMFILE. Would that be easier to support?

from rules_ruby.

I think that makes everything a lot easier.

An Idea about Bundler

What if there was a ruby-Bazel integration gem (written in ruby) that could do the following:

• Drive bundler in Bazel friendly way: generate the lock file and dump the gem sources into vendor/bundle

• Be able to quickly Bazelify a ruby project by creating BUILD.Bazel in the current folder (it can automate a lot of build targets by inspecting common files such as Procfile, spec folder, rake tasks, and files under bin/.

By vendoring gems with the project and providing Gemfile.lock we can make bundle a build rule instead of a workspace rule.

A Question about shared cache

My next question is about how does a giant ruby monorepo benefit from the ruby projects build by Bazel.

With compiled languages I get the point of a shared cache a 100%. But what are we sharing in a distributed cache for a large team of ruby developers? Especially if the lock files are already resolved and gems are vendored? Do we just get the benefit of faster build times because of a small number of gems with native extensions?

from rules_ruby.

@kigster This is interesting, can you share the link? If there is already a ruleset that implemented a parser for Podfile.lock, we can re-use and adapt it for Gemfile.lock.

from rules_ruby.

The link is above, just click on "Tinder Engineering" and it will take you to the blog. I don't know where their source are.

from rules_ruby.

https://medium.com/tinder/bazel-hermetic-toolchain-and-tooling-migration-c244dc0d3ae

from rules_ruby.

The link is above, just click on "Tinder Engineering" and it will take you to the blog. I don't know where their source are.

Thank you, my bad!

from rules_ruby.

Yesterday I met the engineers who own the Ruby rules at Stripe, and confirmed the design @p0deje and I have discussed is pretty much what they do. @sushain97 can help answer questions about the details.

from rules_ruby.

@p0deje here's my simplistic toml parser for a Python package manager
aspect-build/rules_py@main...pdm#diff-12a61de0f4ed1a9986e13381453751bdbe46fb754a094a6e3adf2d7fb0b6eb2e
like I said in our chat, in practice you're only obligated to parse the small subset of the language that the tool actually produces and that Bazel needs to consume, so it's not difficult to have pretty comprehensive test cases.

from rules_ruby.

👋 I'm happy to share code if it would help. I'd just need to redact anything sensitive. We have a tools/ruby/gemfile_parser.bzl and a tools/ruby/fake_git.sh that solve some of the issues mentioned above.

from rules_ruby.

@sushain97 That would be amazing, I'm still trying to figure out a proper way to tell bundle install from downloaded gems, if you have a working solution, I'd love to take a look at it too 🙇

from rules_ruby.

All of this is wonderful news.

I would like to get your feedback on the idea of bazel-rb ruby gem and whether that's useful, needed, or totally unnecessary.

The main use case is this: let's say we have an existing Rails App that is being pulled into a mono repo using the new/now-standard ruby rules.

Rails projects strive on following conventions. That's what makes Rails so great for starting web apps. Many repos will have a Procfile at the top that's used by Foreman gem to start multiple processes at once.

There may or may not be a Makefile, but there is nearly always a Rakefile that loads definitions of tasks people run via a rake command.

My main question has always been this: what is the process of "Bazelifying" such a repo? I.e. which of the commands/rake tasks/tests do we want to be able to invoke via Bazel?

And in the end we should be able to clearly formulate what is the biggest selling point of migrating a large Ruby or Rails app to Bazel.

Is it only driven by the desire to consolidate the code into a monorepo or are there additional advantages for ruby developers too?

Shared remote build cache is one of the "killer" features in Bazel that makes development of huge compiled codebases possible. But does it also help Ruby projects? Will Ruby developers working on the repo see Bazel integration as a boost to their productivity or hindrance?

I remembered seeing that some languages had a "bridge" CLI tooling that auto generated BUILD files by scanning the project, across many folders, making the process of migrating the repo much much easier.

Is this a valid pattern for Bazel integrations? Are there real world examples? Would it make sense to develop such a ruby gem in parallel with the rules themselves?

Sorry for the long post. TBH these are the questions I've been struggling from the day one of learning about Bazel and being tasked with creating Ruby rules.

from rules_ruby.

Sorry for the late reply. First, some answers:

My main question has always been this: what is the process of "Bazelifying" such a repo? I.e. which of the commands/rake tasks/tests do we want to be able to invoke via Bazel?

We're generally trying to avoid introducing Rake into any tasks executed by Bazel. Rake has its own dependency graph, unlike e.g. scripts in package.json. Having a dependency graph in Rake and in Bazel means that you will end up with monolithic targets that aren't well parallelizable.

We have a ruby_genrule and a ruby_binary as join points between Bazel and Ruby scripts.

Is it only driven by the desire to consolidate the code into a monorepo or are there additional advantages for ruby developers too?

A non-exhaustive listing of our reasons:

• Avoid implementing bespoke caching solutions
• Remote execution & caching
• Interoperability with other languages
• Unified toolset across stacks

But does it also help Ruby projects? Will Ruby developers working on the repo see Bazel integration as a boost to their productivity or hindrance?

For Ruby, we generally don't expose Bazel to our developers yet. An exception is for defining deployable container images.

I remembered seeing that some languages had a "bridge" CLI tooling that auto generated BUILD files by scanning the project, across many folders, making the process of migrating the repo much much easier.

Our approach is in many ways informed by having very bespoke tooling around basically all our application Ruby so I don't have a super useful perspective here.

Our BUILD.bazel files have a ruby_package() macro within them. It doesn't really take any arguments. The dependencies between Ruby packages (which is a concept that Stripe has added) are defined in a bespoke __package.rb. A step that runs prior to Bazel creates external repositories which define the dependencies between ruby_package()s in terms of SCCs (since they're not cycle free) based on the contents of all the __package.rb files.

In other words it might be a good idea to create a Bazel Struct or some other representation of both the Gemfile and Gemfile.lock that can be parametrized with a file name. Then any further bundle commands would have to depend on that target.

Yes, we have a number of Gemfiles. Each is declared as an independent external repository with a bundle_repository() call.

I've excerpted some of the code that we use here and uploaded it to https://github.com/sushain97/rules_ruby. The repository is private because the code itself is of varying quality and usefulness. I'm happy to discuss specifics in Slack. I've scrubbed the repository of anything sensitive but in case I missed anything, I'd request that we leave it private. The contents are licensed under MIT so relevant bits can be copied as appropriate.

from rules_ruby.

Here is a quick update on the progress I have so far. You can refer to https://github.com/bazel-contrib/rules_ruby/tree/bundle-fetch-attempt-304 to see WIP code and there are even some builds on CI passing (https://github.com/bazel-contrib/rules_ruby/actions/runs/7175240646), but there are multiple issues I'm running into.

First of all, thanks to @sushain97 I managed to have gemfle_lock_parser.bzl which reads Gemfile.lock and downloads all the gems using Bazel. These gems are later installed using toolchains gem command. However, Gemfile.lock cannot be used as the only source of truth without extra information such as the platform it's being used on. For example, the following is an excerpt from Gemfile.lock:

    jar-dependencies (0.4.1)
    psych (5.1.1.1)
      stringio
    psych (5.1.1.1-java)
      jar-dependencies (>= 0.1.7)
    stringio (3.0.9)

When resolving this part, we should install psych-5.1.1.1 + stringio-3.0.9 on MRI, and psych-5.1.1.1-java + jar-dependencies-0.4.1 on JRuby. We cannot attempt to install it all on all platforms - they are not compatible and fail.

Moving on, some gems are restrictable in Gemfile, not Gemfiel.lock:

# Gemfile
gem 'debug', '>= 1.0.0', platforms: %i[mri mswin64]

# Gemfile.lock
debug (1.6.3)

Bundle will not attempt to install debug-1.6.3 on JRuby, yet we cannot possibly know that by just reading Gemfile.lock.

I'm going to try using bundle install over downloaded gems rather than per-gem gem install. This would be slower in RBE because it's going to use a single target, but at least it should resolve platforms/dependencies correctly.

Another approach I'm thinking about is to generate Gemfile.Bazel.lock with all the missing information (platforms, checksums, etc) that should be committed to the source code repository. This would be somewhat similar to what rules_jvm_external and rules_rust do.

from rules_ruby.

Most Ruby developers would prefer that

bundle install

Be used to install gems because of potential future changes to bundler will require constant rewrite of the Bazel parsing rules.

Also, I thought Gemfile.lock does have a platform section?

from rules_ruby.

But I want to commend you for doing excellent work on pushing these rules forward and I wish I had more time to help.

from rules_ruby.

@sushain97 could you please elaborate on __package.rb file?

Are you referring to packwerk or something else?

from rules_ruby.

Personally I like where this is going with packwerk.

from rules_ruby.

As always, there are (partial) answers in other interpreted languages. @jvolkman has been working in python where we have the same problem: interpreting packages to be installed from the lockfile is platform- and interpreter-dependent.

Some thoughts:

1. how commonly do users have these constraints in the gemfile.lock? If it's quite rare, we could punt on that feature, or regress to "bundler does the downloads" when detecting these in the lockfile. My guess is it's common enough to reject this idea
2. would it help if Bazel were to "over-download", fetching all gems that might be needed and then a later process "installs" only a subset of them
3. how hard is it to port the logic from bundler that parses these, and have enough information available to the repository rule?
4. Sadly having the install depend on the interpreter means that a repo with multiple interpreter versions probably needs a complete parallel gem install for each one, I think it's unavoidable and same situation as python though

from rules_ruby.

There is another option.

I happen to know the maintainer of Bundler. If we can propose a change to the lock file that would retain all the information needed on each platform; perhaps running bundle on jRuby could augment the file in way that adds information to the previously resolved dependencies when run via the MRI ruby's bundle.

Maybe other languages already resolved this?

from rules_ruby.

1. how commonly do users have these constraints in the gemfile.lock?

They are not too common on enterprise rails apps, but more common within libraries eg gems themselves.

Also, as we discussed earlier versions of ruby gems advocated against including the lock file with the gem. Now they do. So more gems will have both now.

2. would it help if Bazel were to "over-download", fetching all gems that might be needed and then a later process "installs" only a subset of them

Personally I don't think it's a good idea. The algorithm for computing the versions that satisfy all constraints in the Gemfile is not ruby specific and is well documented. It might in fact be already within Basel. All you need to know is the list of available versions and platforms for each mentioned gem.

3. how hard is it to port the logic from bundler that parses these, and have enough information available to the repository rule?

I can't answer that, but here is a blog post detailing the last time bundlers algorithm has changed to speed up resolution by an order of magnitude.

4. Sadly having the install depend on the interpreter means that a repo with multiple interpreter versions probably needs a complete parallel gem install for each one, I think it's unavoidable and same situation as python though

I think that's right.

from rules_ruby.

I had the wrong link for the resolver algorithm, but it's now been updated to this:

https://bundler.io/blog/2023/01/31/bundler-v2-4.html

from rules_ruby.

how commonly do users have these constraints in the gemfile.lock? If it's quite rare, we could punt on that feature, or regress to "bundler does the downloads" when detecting these in the lockfile. My guess is it's common enough to reject this idea

It's not so common, but we already have this in our example because debug is the only debugger supporting remote debugging, which makes it ideal for bazel test --run_under. Sadly, this gem only works on MRI:

would it help if Bazel were to "over-download", fetching all gems that might be needed and then a later process "installs" only a subset of them

My implementation is already doing that, however we need to know which gems to install on which platforms. This information is only present in Gemfile.

how hard is it to port the logic from bundler that parses these, and have enough information available to the repository rule?

To reiterate, the platform information is not present in Gemfile.lock, only in Gemfile which cannot be parsed without an interpreter since it's a Ruby code. People write full-blown Ruby code there.

Sadly having the install depend on the interpreter means that a repo with multiple interpreter versions probably needs a complete parallel gem install for each one, I think it's unavoidable and same situation as python though

Yes, but the problem here is that a rule needs to make requests to rubygems.org to fetch information, which means that this action can only run with requires-network execution requirement. This won't work on RBE as far as I know.

Not to mention that running gem install as a separate action for each gem is much faster and more cache-efficient.

I happen to know the maintainer of Bundler. If we can propose a change to the lock file that would retain all the information needed on each platform; perhaps running bundle on jRuby could augment the file in way that adds information to the previously resolved dependencies when run via the MRI ruby's bundle.

That would be fantastic if Gemfile.lock had:

• platform to install for every gem
• checksum of every gem

So given we have this Gemfile.lock:

    jar-dependencies (0.4.1)
    psych (5.1.1.1)
      stringio
    psych (5.1.1.1-java)
      jar-dependencies (>= 0.1.7)
    stringio (3.0.9)

In a perfect world we would have the following:

    jar-dependencies (0.4.1) (jruby) (b2df2f1ecbff15334ce20ea7fdd5b8d8161faab67761ff72c7647d728e40d387)
    psych (5.1.1.1) (ruby, windows) (...checksum...)
      stringio
    psych (5.1.1.1-java) (jruby) (...checksum...)
      jar-dependencies (>= 0.1.7)
    stringio (3.0.9) (ruby, windows) (...checksum...)

from rules_ruby.

With respect to checksums, bundler 2.5.0.dev (which will be in the upcoming Ruby 3.3 release) does write checksums to the lock file, in a new section named CHECKSUMS. It looks like this:

Thank you for this info! It seems the time is right to ask Bundler maintainers for what Bazel needs.

@p0deje Perhaps we need to spec your proposal on the rubygems issues page?

from rules_ruby.

I finally reached a point where CI passes - it's been a trial-and-error to make it green given the variety of interpreter/OS combinations. What proved to work fine on MRI in Linux/macOS caused problems on Windows, likewise making JRuby/TruffleRuby work took a lot of effort. Here is a general overview of the current implementation

1. There is a new rb_bundle_fetch repository rule.
2. The rule would parse Gemfile.lock and download all the gems as *.gem files to vendor/cache directory in the repository.
3. The rule would also install bundler of the version specified in Gemfile.lock to a separate directory in the repository.
4. The rule would generate shims for gem executables (e.g. @bundle//bin:rake) that would automatically run the next step.
5. There is a new rb_bundle_install rule (not a repository one) that is defined by rb_bundle_fetch automatically.
6. The rule would run bundle install --local (installed in step 3) with cached gems (downloaded in step 2). Adding --local ensures there are no RubyGems.org calls made by the bundler (requires Bundler 2.2.19+). Bundler is expected to install to a separate directory can other targets can depend on.

The implementation has several limitations at the moment:

• bzlmod doesn't work (fixed).
• Gem dependencies are not installed automatically on Windows (I'm working to fix it).
• Gemfile.lock parser supports only RubyGems remotes (I'm working to fix it).
• Gemfile.lock parser doesn't work with Git repositories (@sushain97 has a fix in his repo, I'm yet to try and make it work for all OS-es). I propose we add it later because the changeset is already massive.
• Gemfile.lock parser doesn't read checksums. I propose we add it later.

from rules_ruby.

#48 is ready for initial review and I would appreciate if anyone takes a look.

from rules_ruby.