atlassian / bazel-tools Goto Github PK

Is there any way to provide the external dependencies or generated code to varcheck?

wryun/rjsone#12

The golangci-lint rules do not depend on the Go binary from the SDK. Without the dependency Bazel include the binary in the execution directory, resulting in the following error:

level=warning msg="Failed to discover go env: failed to run 'go env': exec: \"go\": executable file not found in $PATH"
level=error msg="Running error: context loading failed: failed to load program with go/packages: couldn't exec 'go [list -e -json -compiled -test=true -export=false -deps=true -find=false -- ./cmd/... ./internal/...]': exec: \"go\": executable file not found in $PATH *exec.Error"

By adding depset([sdk.go]) to the transitive dependencies list the Go binary is added to execution directory and everything is happy.

I have a repo with the structure:

/WORKSPACE
/src/go.mod
/src/a/main.go
/src/a/.golangci.yml
/src/b/main.go
/src/b/.golangci.yml

As /src/a and /src/b has separate linter configurations, it would be nice to run golangci-lint inside /src/a and /src/b separate.

As the current rule symlink the workspace root /, it would be nice if you could specify it to change directory into src/a before running golangci-lint.

It looks like this line causes issues with newer versions of Bazel (I'm using 0.24.1):

When running e.g. bazel build //... I get:

ERROR: infinite symlink expansion detected
[start of symlink chain]
<WORKSPACE ROOT>/bazel-<WORKSPACE DIRECTORY>/bazel-<WORKSPACE DIRECTORY>
/private/var/tmp/_bazel_<USERNAM>/<HASH>/execroot/__main__/bazel-<WORKSPACE DIRECTORY>
<WORKSPACE ROOT>bazel-<WORKSPACE DIRECTORY>
[end of symlink chain]

@obweger
Hi Hannes!
Could you help me re-enable the Buildkite integration for this repo please?
The webhook is configured correctly and builds are triggered, but they don't have permissions to report back. I've got this in the email from Buildkite:

Buildkite has automatically turned off GitHub commit status updates for the atlassian/bazel-tools pipeline. We kept getting 404's when trying to make API requests.

This can happen if someone has left your Buildkite or GitHub organization, or you've recently enabled third-party access restrictions:

https://help.github.com/articles/enabling-third-party-application-restrictions-for-your-organization/

If that's the case, you'll need to "Grant or Request Access to Buildkite" here:

https://github.com/settings/connections/applications/3c5a9a8d50182605e362

Another thing to check is that your GitHub user account is connected to Buildkite, and has write access to the repositories:

https://buildkite.com/user/connected-apps

Once you've followed the above steps, check "Update commit statuses" here and save:

https://buildkite.com/bazel/atlassian-bazel-tools/settings/repository

If everything is set up correctly, GitHub commit status updates should start working again. If you keep seeing this issue, reply to this email and we'll dig further into what might be going wrong.

Status reporting stopped working when I left the company and I cannot seem to fix it even though I have access to this repository now (thanks, btw!).

Builds are here and you can see there are daily builds but also builds for branches/PRs (i.e. the webhook works) https://buildkite.com/bazel/atlassian-bazel-tools

I don't know what exactly is broken or needs to be done to fix the issue, but I also cannot troubleshoot because I'm no longer an org owner. Please help :)

Hi,

I am trying to use gometalinter using bazel build system with reference the example code shared in this repo. Layout of codebase

WORKSPACE
BUILD
--> uses third party golang github packages

WORKSPACE
load("@com_github_atlassian_bazel_tools//gometalinter:deps.bzl", "gometalinter_dependencies")
gometalinter_dependencies()

BUILD
load("@com_github_atlassian_bazel_tools//gometalinter:def.bzl", "gometalinter")
gometalinter(
name = "gometalinter",
config = "//:gometalinter.json",
paths = [
"/...",
],
prefix = "<repo_path>",
)

On executing "bazel run //:gometalinter", bazel is having issues importing third party github repos internally used by code in dir1 folder.

Can you please let me know any solution for this?
The same is easily achievable using deps for go_binary/cc_binary, etc

Thanks
Kailash

Hi! I've been trying to use golangcilint on a bazel model repo I'm working on and one of the checks is failing to find generated imports:

service/go-client/main.go:45:19: NewUserServiceClient not declared by package users (typecheck)
	users := pbUsers.NewUserServiceClient(cc)

users is a generated protobuf package.

Full log here: https://app.circleci.com/pipelines/github/johanbrandhorst/bazel-mono/18/workflows/da9da62e-24a3-4417-bc09-1761b51abe75/jobs/128/steps

It's reproducible locally.

The golangci-lint rule does not depend on the SDK libraries. This results in a warning from govet (and possibly other linters):

WARN [runner] Can't run linter govet: can't typecheck package: can't init std types: import of type fmt.Stringer failed: can't find import: fmt

By adding depset(sdk.libs) to the transitive dependencies list the compiled libraries are added to execution directory and everything is happy.

See how it is implemented in rjsone rule as an example.

my current workaround is to add this at the end of my runcommand

<some starting command>; kill $(ps | grep '[n]ame-of-the-multirun-target' | awk '{print $1}')

golangci-lint rule runs the linter via a bash command. However, the issue is that even if the source files are not changed, the linter is invoked.

It is really two steps：
[run linter] + [cache linter-found issues]

If source code is not changed, then just output [cache linter-found issues] skipping [run linter] step.

I suggest to make the actual version and SHA256 of the binaries (i.e. golangci-lint) to be customizable for downstream repo to source from this repo directly.

Otherwise downstream repos will have to copy paste the code you have here and customize the version themselves.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

We're trying to use multirun to do multiple container_push's but can't run the multirun target even for simple binary targets.

Building of the multirun target succeeds but when running it we get

/private/var/tmp/_bazel_dimitrovd/a9507c16f9ff4a741b31cd8f280aebd6/execroot/__main__/bazel-out/darwin-fastbuild/bin/run_all.bash: line 18: /private/var/tmp/_bazel_dimitrovd/a9507c16f9ff4a741b31cd8f280aebd6/external/bazel_tools/tools/bash/runfiles/runfiles.bash: No such file or directory

Although I can see the file exists

lrwxr-xr-x  1 dimitrovd  wheel   121B  2 May 13:26 bazel-out/darwin-fastbuild/bin/run_all.bash.runfiles/bazel_tools/tools/bash/runfiles/runfiles.bash -> /private/var/tmp/_bazel_dimitrovd/a9507c16f9ff4a741b31cd8f280aebd6/external/bazel_tools/tools/bash/runfiles/runfiles.bash

bazel version

Build label: 0.24.1
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Apr 2 16:32:47 2019 (1554222767)
Build timestamp: 1554222767
Build timestamp as int: 1554222767

Issue is present when using 20cbdb1 of bazel-tools

A workaround was to downgrade to c7e0fd9

I'm posting below some of the relevant files

/BUILD:

load("@com_github_atlassian_bazel_tools//:multirun/def.bzl", "multirun")

multirun(
    name = "run_all",
    commands = [
        "//go/cmd/app:binary"
    ]
)

/go/cmd/app/BUILD:

...
load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")

go_library(
    name = "go_default_library",
    srcs = ["main.go"],
    importpath = "gitlab.local/team/cmd/app",
    visibility = ["//visibility:private"],
)

go_binary(
    name = "binary",
    embed = [":go_default_library"],
    visibility = ["//visibility:public"],
)
...

/WORKSPACE:

...
skylib_version = "0.8.0"

http_archive(
    name = "bazel_skylib",
    type = "tar.gz",
    url = "https://github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel-skylib.{}.tar.gz".format(skylib_version, skylib_version),
    sha256 = "2ef429f5d7ce7111263289644d233707dba35e39696377ebab8b0bc701f7818e",
)

http_archive(
    name = "com_github_atlassian_bazel_tools",
    strip_prefix = "bazel-tools-20cbdb188d18c5470697783258cd2ec2b531b710",
    urls = ["https://github.com/atlassian/bazel-tools/archive/20cbdb188d18c5470697783258cd2ec2b531b710.zip"],
)

load("@com_github_atlassian_bazel_tools//:multirun/deps.bzl", "multirun_dependencies")

multirun_dependencies()

load("@com_github_atlassian_bazel_tools//golangcilint:deps.bzl", "golangcilint_dependencies")

golangcilint_dependencies()
...

/bazel-out/darwin-fastbuild/bin/run_all.bash:

#!/usr/bin/env bash

# --- begin runfiles.bash initialization ---
# Copy-pasted from Bazel's Bash runfiles library (tools/bash/runfiles/runfiles.bash).
set -euo pipefail
if [[ ! -d "${RUNFILES_DIR:-/dev/null}" && ! -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" ]]; then
  if [[ -f "$0.runfiles_manifest" ]]; then
    export RUNFILES_MANIFEST_FILE="$0.runfiles_manifest"
  elif [[ -f "$0.runfiles/MANIFEST" ]]; then
    export RUNFILES_MANIFEST_FILE="$0.runfiles/MANIFEST"
  elif [[ -f "$0.runfiles/bazel_tools/tools/bash/runfiles/runfiles.bash" ]]; then
    export RUNFILES_DIR="$0.runfiles"
  fi
fi
if [[ -f "${RUNFILES_DIR:-/dev/null}/bazel_tools/tools/bash/runfiles/runfiles.bash" ]]; then
  source "${RUNFILES_DIR}/bazel_tools/tools/bash/runfiles/runfiles.bash"
elif [[ -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" ]]; then
  source "$(grep -m1 "^bazel_tools/tools/bash/runfiles/runfiles.bash "             "$RUNFILES_MANIFEST_FILE" | cut -d ' ' -f 2-)"
else
  echo >&2 "ERROR: cannot find @bazel_tools//tools/bash/runfiles:runfiles.bash"
  exit 1
fi
# --- end runfiles.bash initialization ---

# Export RUNFILES_* envvars (and a couple more) for subprocesses.
runfiles_export_envvars

echo Running '//go/cmd/app:binary'
./'go/cmd/app/darwin_amd64_stripped/binary' $@

Let me know if you need anything else shared.

I am trying to use ibazel with multirun, but having an issue with tag propagation: One of the two commands has the tags "ibazel_notify_changes" and "ibazel_live_reload" configured, in order to respond to ibazel reload commands. If I don't add these tags to the multirun call, then no reloading happens. If I do add them, the process that doesn't support ibazel reloading gets built, but never gets restarted.

The documentation for these various tools are very clear about how go modules and generated code are not supported, with a suggestion to look at nogo. Is there any plan to build those nogo integrations here or elsewhere?

In some of our use cases we run some things in parallel, and then some serially later. To do this we have 2 different multirun targets we run back to back. In the parallel case when it's successful there is no output, which makes the output from the serial case a bit surprising. Would it be reasonable to have a option on multirun that silences the running [target] output when running serially?

I sometimes see this in the parallel mode without interrupting the process

[multirun://production/local/docker/mongo-cluster:mongo] Failed to send SIGTERM, sending SIGKILL
[multirun://production/local/docker/mongo-cluster:mongo] Failed to send SIGKILL

I added a bit of debug output here 2bec8e8

and the error is

Failed to send SIGTERM os: process already finished, sending SIGKILL

So it seems there are 2 issues

1. when interrupt signal is sent the child process might be almost finished. So sending SIGTERM to it may result in Error os: process already finished. I think it is a legit situation. We need to handle this error as a success case. (PR)

2. sometimes the same situation happens even without sending an interrupt signal to bazel. So it seems to be a race condition somewhere. It might be not critical but nice to fix.

Hello,

I'd like to use this convenience buildozer rule, but wish it were a little less opinionated. Currently it seems to require Buildozer commands be fed in via a file while I would prefer to pass them in as an argument, for example bazel run //:buildozer -- somecommand.

The context here is that for historical reasons my Bazel monorepo has a lot of Go vendor directories. We're also using an older Go vendoring tool that helpfully generates go_test rules for said vendor directories. This means our BUILD file generation strategy looks something like:

$ bazel run //:gazelle -- fix services/myservice
$ buildozer delete //services/myservice/vendor/...:%go_test

I'm not a Bazel expert and could be wrong, but I don't believe there's any way to specify a label like //.../vendor/...:%go_test - if there were I'd be happy to throw such a command in //:buildozer_commands.txt and stick with the existing pattern.

When setting -local on goimports like this:

goimports(
    name = "goimports",
    display_diffs = True,
    local = [ "kubevirt.io" ],
    prefix = "kubevirt.io/kubevirt",
    write = True,
)

local has no effect. The rest of the config is picked up correctly.

Sometimes {{ and }} and needed inside of the template so allowing to use different delimiters can be quite useful. See https://golang.org/pkg/text/template/#Template.Delims

If a config attribute isn't passed in to the golangcilint, the flag --no-config is appended to golangci-lint run.

Is there any need for this?

If it would be dropped, an existing .golangci.yml file would be picked up if present instead. That would make it easy to write generic Bazel rules without having to specify the config.

Bazel version: 0.28.0
Bazel-tools: commit 7c3b97b8e8bf4699353fcdf7437c195e2cdcef3b - Jul 2019

Context: I'm able to run and test the actual project with bazel, but linter seems to not use the external deps defined on WORKSPACE

WORKSPACE:

http_archive(
    name = "com_github_atlassian_bazel_tools",
    strip_prefix = "bazel-tools-6fbc36c639a8f376182bb0057dd557eb2440d4ed",
    urls = ["https://github.com/atlassian/bazel-tools/archive/6fbc36c639a8f376182bb0057dd557eb2440d4ed.zip"],
    sha256 = "544495c6e9b509191c616694ec734a1b7101c85726bd5916a8713225658e33ab"
)

load("@com_github_atlassian_bazel_tools//golangcilint:deps.bzl", "golangcilint_dependencies")
golangcilint_dependencies()

go_repository(
    name = "com_google_firebase_go",
    importpath = "firebase.google.com/go",
    sum = "h1:A/KMJdcsLq1/miil8tONEbLBmUgOWpdVCLAxMHq0Xbw=",
    version = "v3.8.1+incompatible",
)

go_repository(
    name = "in_gopkg_check_v1",
    importpath = "gopkg.in/check.v1",
    sum = "h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=",
    version = "v0.0.0-20161208181325-20d25e280405",
)

go_repository(
    name = "in_gopkg_fsnotify_v1",
    importpath = "gopkg.in/fsnotify.v1",
    sum = "h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4=",
    version = "v1.4.7",
)

go_repository(
    name = "in_gopkg_tomb_v1",
    importpath = "gopkg.in/tomb.v1",
    sum = "h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=",
    version = "v1.0.0-20141024135613-dd632973f1e7",
)

go_repository(
    name = "in_gopkg_yaml_v2",
    importpath = "gopkg.in/yaml.v2",
    sum = "h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE=",
    version = "v2.2.1",
)

go_repository(
    name = "io_opencensus_go",
    importpath = "go.opencensus.io",
    sum = "h1:mU6zScU4U1YAFPHEHYk+3JC4SY7JxgkqS10ZOSyksNg=",
    version = "v0.21.0",
)

go_repository(
    name = "io_upper_db_v3",
    importpath = "upper.io/db.v3",
    sum = "h1:3MJSnJQ+NMxBxuNwO+gOKFiugwv+f61LbyuZYSPzoi4=",
    version = "v3.5.7+incompatible",
)
...

BUILD.bazel

golangcilint(
    name = "golangcilint",
    paths = [
        "packages/api/pkg/...",
    ],
    prefix = "github.com/my-private-repo/some-app",
)

error:

INFO: Analyzed target //:golangcilint (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:golangcilint up-to-date:
  bazel-bin/golangcilint.bash
INFO: Elapsed time: 0.150s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
ERRO Running error: context loading failed: failed to load program with go/packages: go [list -e -json -compiled -test=true -export=false -deps=true -find=false -- ./packages/api/pkg/...]: exit status 1: go build github.com/golang/protobuf/ptypes/timestamp: no Go files in
go build github.com/golang/protobuf/ptypes/empty: no Go files in
go build github.com/golang/protobuf/proto: no Go files in
go build google.golang.org/grpc: no Go files in
go build google.golang.org/grpc/codes: no Go files in
go build google.golang.org/grpc/status: no Go files in
go build github.com/mishudark/errors: no Go files in
go build github.com/go-redis/redis: no Go files in
go build firebase.google.com/go: no Go files in
go build firebase.google.com/go/messaging: no Go files in
go build github.com/go-ozzo/ozzo-validation: no Go files in
go build upper.io/db.v3/lib/sqlbuilder: no Go files in
go build google.golang.org/api/option: no Go files in
go build github.com/imdario/mergo: no Go files in
go build upper.io/db.v3: no Go files in
go build github.com/lestrrat-go/jwx/jwk: no Go files in
go build github.com/lestrrat-go/jwx/jws: no Go files in
go build google.golang.org/grpc/metadata: no Go files in

https://github.com/koalaman/shellcheck/wiki/SC2068

Hi, I've hacked together a small patch that allows running golangci-lint on the generated go code. It relies on @rules_go/go_path to assemble a source tree of all the dependencies and uses that as GOPATH for the golangci-lint invocation.

I've got it to work on my small repo that uses generated code for protobuf libraries and it works. The minimal patch to accept a go_path() output and wire it through GOPATH is at parabolala@3fd012a

It can be used as:

load("@io_bazel_rules_go//go:def.bzl", "go_path")
load("@com_github_atlassian_bazel_tools//golangcilint:def.bzl", "golangcilint")

go_path(
    name = "go_root",
    deps = [
      # Code entry points (go_binary, go_library) to lint. 
      # All their transitive deps are pulled in and are filtered by "prefix" below.
      "//service/cmd/server",
    ],
    include_pkg = True,
)

golangcilint(
    name = "golangcilint",
    go_path = ":go_root",
    prefix = "<prefix matching the code>",
)

Am I missing something essential? Does this look feasible? If yes, I'll be happy to flesh it out into a PR.

The bazel labels take up a lot of room in the console, for our use case console output is pretty important.
Could we add some config that would allow for specifying some shorter name for the label.
For example with the label //apps/my-app:bin we'd like to use MA instead

I'd like to have a config in the command rule, how hard would it be to add a data dep like specified here: https://docs.bazel.build/versions/master/be/common-definitions.html#common-attributes

I think it'd be nice to allow for multirun to support environment variables, right now I have a wrapper script to resolve them