bazel-contrib / rules_bazel_integration_test Goto Github PK

bazel version 4.2.2
windows version: Windows 11 Enterprise Evaluation
I've created a small repo to demonstrate: https://github.com/k1nkreet/rules_bazel_integration_test_win_example

running bazel.exe test //:workspace_test throws this

ERROR: C:/users/user/rules_bazel_integration_test_win_example/BUILD.bazel:15:23: in select_file rule //:workspace_test_bazel_binary:
Traceback (most recent call last):
        File "C:/users/user/_bazel_user/d7burz7z/external/bazel_skylib/rules/select_file.bzl", line 36, column 13, in _impl
                fail("Can not find specified file in [%s]" % files_str)
Error in fail: Can not find specified file in [external/build_bazel_bazel_4_1_0/bazel.exe]
ERROR: Analysis of target '//:workspace_test' failed; build aborted: Analysis of target '//:workspace_test_bazel_binary' failed
INFO: Elapsed time: 29.947s

rules_bazel_integration_test might be used in a scenario where there's already scaffolding in place to run tests under different versions of Bazel. For example, with Bazel's own CI this might be configured as a test matrix where different rows of the matrix use a specific Bazel version. This could be used to execute an entire test suite, containing both unit and integration tests, with the same Bazel version.

Because this scaffolding would exist at a level higher than rules_bazel_integration_test, it would be desirable to configure rules_bazel_integration_test to simply use the same bazel binary that's executing the top-level bazel test command, rather than configuring an explicit binary via bazel_version or bazel_binary.

Is this something that could be supported?

When parsing the Bazel version from a .bazelversion file, rules_bazel_integration_test assumes that the file contains nothing but the version and perhaps surrounding whitespace. However, this does not match bazelisk's behavior, which is to read the first line of the file (using a Scanner).

This issue is easily encountered because the rules-template contains a .bazelversion file with trailing lines.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency build_bazel_rules_swift to v1.17.0
• chore(deps): update dependency cgrindel_bazel_starlib to v0.20.2
• chore(deps): update dependency rules_swift to v1.17.0
• chore(deps): update dependency rules_swift_package_manager to v0.29.1
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

When adding new Bazel packages to a workspace, it would be very useful to have the all_files filegroup added to the new packages and have the filegroup at the root updated with those targets.

Also, there could be a performance benefit to not use glob() and list the files.

The xmllint utility is available on MacOS by default. However, it is not installed on Linux. The tests will be updated to only run on MacOS for now.

If I depend on rules_bazel_integration_test via WORKSPACE my integration tests work fine. If I depend using bzlmod the same tests (I didn't change anything in BUILD file) fails to build with this error message:

ERROR: /home/andrzej/code/bazel/bazel-bsp/e2e/BUILD:27:24: no such package '@[unknown repo 'build_bazel_bazel_6_3_2' requested from @]//': The repository '@[unknown repo 'build_bazel_bazel_6_3_2' requested from @]' could not be resolved: No repository visible as '@build_bazel_bazel_6_3_2' from main repository and referenced by '//e2e:remote_jdk_test_bazel_6_3_2'

In both cases I used version 0.18

I'm trying to set up rules_bazel_integration test, following the instructions in the project README. When I try to run update_deleted_packages, it fails with this error output:

$ bazel run @rules_bazel_integration_test//tools:update_deleted_packages
INFO: Analyzed target @rules_bazel_integration_test//tools:update_deleted_packages (131 packages loaded, 544 targets configured).
INFO: Found 1 target...
Target @rules_bazel_integration_test//tools:update_deleted_packages up-to-date:
  bazel-bin/external/rules_bazel_integration_test/tools/update_deleted_packages
INFO: Elapsed time: 28.774s, Critical Path: 0.10s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
dirname: missing operand
Try 'dirname --help' for more information.

From what I can tell, one way to hit this error is to attempt to pass a blank/whitespace argument to dirname. If I were to guess, somewhere in one of the helpers, a path is constructed to pass to dirname, but accidentally ends up blank/empty.

Any advice on how to debug this further would be greatly appreciated. Thank you in advance.

Any test that uses runfiles and the Python integration test framework will see errors because the environment variables aren't set up properly.

Tasks

• Preparation
  • #123
  • #124
• #126
• #130
• #125
  • #128
  • #132
• #135
• #136
• Once cgrindel/rules_swift_package_manager#276 is fixed.
  • #137
• Once bazelbuild/bazel#14140 is fixed.
  • #138

This example stopped working on Xcode 13.3. Right now, the Xcode version for the example is set to 13.2.1.

The basic use case is to allow easily using the same bazel that the user uses to run their builds. The main use case I have in mind for this is when you're using a custom build of bazel specified using USE_BAZEL_VERSION. e.g. if one does:

USE_BAZEL_VERSION=/tmp/bazel-dev bazel //my:test

Then the version of bazel at /tmp/bazel-dev will be used, without having to go into the MODULE.bazel config (or elsewhere) to reconfigure this. Having to do that is a hassle and changes the name of generated targets.

I think the bazel_binaries.local() tag class can be easily adopted for this purpose:

• Add an attribute, shell_command, which takes a single word, e.g. "bazel", "mybazel", "bazel-dev", etc.
• In _local_bazel_binary_repo, call repository_ctx.which(shell_command) to get the actual path
• Add USE_BAZEL_VERSION and PATH to the repo rule's environment variables that affect it.

The end result looks something like:

bazel_binaries.local(shell_command="bazel", name="self")

I was able to emulate this by using a local path to a script that just re-execs using the command "bazel", but it'd be nicer if I didn't have to plumb through a wrapper to achieve that.

Prerequisites

• cgrindel/rules_swift_package_manager#276

Apparently, gazelle supports bzl_library targets.

Related discussion

Related to #128.

Sounds like the steps in "Integration Tests That Depend Upon The Parent Workspace" can be scripted with buildozer.
Maybe worthwhile having a utility to do that?

As part of #181, Bazelisk will be used to download the Bazel used for tests.

With the legacy WORKSPACE support, the Bazelisk version can be configured in the bazel_binaries call:

bazel_binaries(
    bazelisk_version = "1.17.0",
    versions = [
        "//:.bazelversion",
    ],
)

This is not currently possible with the bzlmod support, and should be added.

Possible syntax (this does not work currently):

bazel_binaries = use_extension(
    "@rules_bazel_integration_test//:extensions.bzl",
    "bazel_binaries",
    dev_dependency = True,
)
bazel_binaries.download(version_file = "//:.bazelversion")
bazel_binaries.use_bazelisk("1.17.0")
use_repo(bazel_binaries, "bazel_binaries")

Destination: https://github.com/bazelbuild/bazel-integration-testing

Slack Thread: https://bazelbuild.slack.com/archives/CDCMRLS23/p1638830205024500?thread_ts=1638550059.021400&cid=CDCMRLS23

Chuck Grindel  28 days ago
@Ittai Zeidman Do you know who is the primary maintainer of bazelbuild/bazel-integration_testing? I agree that it could be confusing to have both. However, if Google has internal dependencies on the repository, it might be painful to migrate to the new rule.

GitHubGitHub
GitHub - bazelbuild/bazel-integration-testing: Framework for integration tests that call Bazel
Framework for integration tests that call Bazel. Contribute to bazelbuild/bazel-integration-testing development by creating an account on GitHub. (54 kB)
https://github.com/bazelbuild/bazel-integration-testing


Alex Eagle  28 days ago
I'm 95% sure that google-internal doesn't use anything from that repo, and @Ittai Zeidman indicated in the past that they're okay with new maintainers there. If there isn't a technical limitation, it seems like the simplest answer to me

Ittai Zeidman  28 days ago
I’m 99% sure no google internal things use it.
I think there’s an IJ OSS test that uses it. Might be a good exercise moving from one API to another to see how hard the new repo is to use.
I’m the main maintainer (or neglecter :disappointed: :man-facepalming: ) and I’d be happy to add more maintainers.

Chuck Grindel  27 days ago
Then, bazebuild/bazel-integration-testing sounds like a good home to me. I would be happy to become a maintainer for the repository, as well.
Here are a few questions:
Would you be willing to help with the code reviews?
Are you generally happy with the structure that I have in my repository?
As an initial PR, would you prefer to see a big PR with changes that roughly recreates the repository that I have?

Alex Eagle  26 days ago
@Paul Johnston you had interest in the go_bazel_test rule and seeing that the goodness there is preserved in the canonical upstream thing, what's required for you to help out here? If we can connect some funding to the SIG so it can be billable time?

Alex Eagle  26 days ago
@Ittai Zeidman I think we should ask the Bazel admins to move the bazel-integration-testing repo from bazelbuild to bazel-contrib. If the SIG funds the development, we should also control the repo. Any concern?
(github does a great job hosting all the redirects so it's non-breaking)
:+1:
1


Ittai Zeidman  26 days ago
no concerns

Chuck Grindel  26 days ago
So, if we do move the repo and redirect, we can change the name of the repo. Correct? If that is true, I think that the rules_bazel_integration_test name fits the recommended guidance.

Alex Eagle  26 days ago
yup, asked Phillip and Sven to approve the move, they are working on it

Ittai Zeidman  26 days ago
Added both of you to the repo as maintainers

Chuck Grindel  25 days ago
Great. Any thoughts on my comment about the name change?

Alex Eagle  25 days ago
we'll change the name as part of the move. need a googler to do bazelbuild/bazel_integration_testing -> bazel-contrib/rules_bazel_integration_test

It is possible to specify that an integration test uses the Bazel version specified in a .bazelversion file. It would be useful to support Bazelisk's <FORK>/<VERSION>.

Tasks

• Transfer the repository
• Update references in workspace snippet and doc to new URL:
  • #48
  • #51
• Set up the release process
• #57
• Generate a release. (v0.6.2)
• Update references in other repos
  • cgrindel/bazel-starlib: cgrindel/bazel-starlib#139
  • keith/buildifier-prebuilt: keith/buildifier-prebuilt#30
  • buildbuddy-io/rules_xcodeproj: MobileNativeFoundation/rules_xcodeproj#302
  • cgrindel/rules_swiftformat: cgrindel/rules_swiftformat#55
  • cgrindel/rules_spm: cgrindel/rules_spm#137
• Send announcement on Slack and ???

I want to create my own macro around using the rules_bazel_integration_test and I created a custom test runner in python that uses pytest. I want to be able to specify a default set of "tests" required to pass on each target type, eg.: "bazel info" and "bazel query //..."

I would also like to have each caller of the custom macro specify unique tests. My macro is essentially this

def __get_label(label):
    """
    Return a Label object representation.

    Args:
        label: a string label"""
    if label.startswith("@") or label.startswith("//"):
        return label
    else:
        return Label("//{}:{}".format(native.package_name(), label.lstrip(":")))

def __bazel_py_intetration_test(
        name,
        srcs,
        workspace_path,
        run_files,
        main = None,
        tags = None,
        deps = [],
        env = {},
        target_compatible_with = None):
    test_scenario_name = "{}_scenario".format(name)
    tags = tags or []
    srcs = [__get_label(src) for src in srcs] + [
        "@//bazel/integration_tests:bit_python_runner.py",  ## The test runner py_library
        "@//bazel/integration_tests/py:common_py_bit_tests",  # common tests
    ]
    python.binary(
        name = test_scenario_name,
        main = ":bit_python_runner.py",
        srcs = srcs,
        args = [
            "--fixtures-per-test",
            "-rpa",
            "-vvvv",
        ] + ["$(locations {})".format(src) for src in srcs],
        deps = deps+[
           # any dependencies required for :bit_python_runner.py
        ],
        testonly = True,
    )

    workspace_files = integration_test_utils.glob_workspace_files(workspace_path)

    workspace_files = [
        "@//:local_repository_files",  # all files in the main workspace
    ] + workspace_files + run_files

    # Declare the integration tests.
    bazel_integration_test(
        name = name,
        bazel_binary = "@//tools:bazel",
        test_runner = ":{}".format(test_scenario_name),
        env = env,
        tags = ["exclusive", "bit"] + tags,
        workspace_files = workspace_files,
        workspace_path = workspace_path,
        target_compatible_with = target_compatible_with,
    )

The args value of the python.binary does not get called when invoking the bazel_integration_test.

My end goal is to be able to define a test once, and have said test execute against all "bazel_integration_test" targets. One option is to allow passing arguments to the test runner

Excerpt from Slack conversation:

Ittai Zeidman
Well done! One thought- I skimmed through the docs and I’m not sure how easy it is to use the built in language runners (e.g. junit in java).
I think there’s value in using the built in rules and runners.
Maybe by having a file based protocol (e.g. tests that use the macro can assume bazel path will be in a file named bazel.path and likewise) or env properties based protocol

Chuck Grindel
I think that is feasible. We could add an option to bazel_integration_test that creates a symlink for Bazel and the workspace with a predictable name. Would that work? Also, if we added that option, I presume that we would not attempt to pass the --bazel and --workspace flags to the test runner.

Ittai Zeidman
Sounds relevant. I have a vague recollection that also in bazel-integration-testing we needed to pass some args but maybe they can be avoided using symlinks and conventions. Not sure.

We do want to support testing from 6.0.0 forward, at a minimum.

This allows implementors that want to write their own test runner in a specific language to do so. Consider wrapping the test runner in an execute_binary declaration so that the arguments are captured.

This would allow several different flavors of download to be accepted. Also, it would be possible to pass along a sha256 for the download.

Example Syntax

bazel_binaries(
    downloads = [
       # Bazel version and sha256
        bazel_download("4.2.2", sha256 = "..."),
        # Just Bazel version
        bazel_download("5.0.0-pre.20211011.2"),
        # URL
        bazel_download(url = "https://...", sha256 = "...")
    ],
)

Prerequisites

• bazelbuild/bazel#14140

It would be cool to be able to select the bazel version from the .bazelversion file.

I followed the directions in the Quickstart guide. The project (rules_rust) has a .bazelrc file and I've added the following to it:

# To update these lines, execute 
# `bazel run @contrib_rules_bazel_integration_test//tools:update_deleted_packages`
build --deleted_packages=
query --deleted_packages=

When I run bazel run @contrib_rules_bazel_integration_test//tools:update_deleted_packages, I get the following output:

INFO: Analyzed target @contrib_rules_bazel_integration_test//tools:update_deleted_packages (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target @contrib_rules_bazel_integration_test//tools:update_deleted_packages up-to-date:
  bazel-bin/external/contrib_rules_bazel_integration_test/tools/update_deleted_packages
INFO: Elapsed time: 0.110s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/external/contrib_rules_bazel_integration_test/tools/update_deleted_packages
The bazelrc was not found. ./.bazelrc

And .bazelrc is unchanged.

hi, recently we bumped the version of the rules (0.15.1 -> 0.16.0) and our tests started failing on mac os

I think #181 (comment) might be relevant

even ProcessBuilder().command(bazelBinary, "info") returns exit code 1

how to reproduce:

1. clone https://github.com/JetBrains/bazel-bsp
2. run bazel test //e2e:sample_repo_test_bazel_5_3_2
3. on linux is should pass, on mac should fail

Currently, the rules rely on passing workspace files to the child workspace using runfiles. However, a known Bazel issue prevents files with spaces in their names from working properly. Ideally, this will be fixed in Bazel. However, for now, we may want to allow a client to specify whether to use an alternative mechanism.

Possible solutions:

1. Add the files to an archive file and pass that through runfiles
2. Create a mechanism that creates symlinks from a more robust file format as compared to the current, space-delimited format.

Context and Ideas

• Slack thread
• bazelbuild/bazel-gazelle#1423
• bazelbuild/bazel-gazelle#1404

Related to #120.

New workspace maker files:

• REPO.bazel
• MODULE.bazel

The list needs to be confirmed with the community.

Avoid future GitHub checksum debacles.

Today, the script assumes that examples are where the child workspaces will be located.

bazel_integration_test sets CC=clang in test environment on linux which makes it fails in shell without clang installed in PATH. I've failed to find a reason behind it, so creating this issue to raise a question :) (or remove it if it really is unnecessary)

rules_swift_package_manager and bazel-starlib work properly without listing the Bazel binary repositories that are declared by rules_bazel_integration_test extension. However, a JetBrains repo does require that they be listed.

Related to #233.

@cgrindel_bazel_starlib//shlib/lib:messages is a dependency.

Related to #233.