Incompatible flag --incompatible_load_python_rules_from_bzl will break rules_jsonnet once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

• [:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
• [:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
• [:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

If you try to use the "imports" attribute of the jsonnet_to_json rule in a package without a name, the -J parameter formed for each entry winds up mistakenly specifying an absolute path starting at the root directory, instead of a relative path as intended.

Consider a project that has the following structure:

top
├── BUILD
├── WORKSPACE
├── service.jsonnet
└── lib
    ├── BUILD
    └── stuff.libsonnet

If in the top/BUILD file we write the following, Bazel will compose the -J argument as "/lib," instead of "lib" or "./lib":

load("@io_bazel_rules_jsonnet//jsonnet:jsonnet.bzl", "jsonnet_to_json")

jsonnet_to_json(
    name = "service",
    src  = "service.jsonnet",
    deps = ["//lib:stuff"],
    imports = ["lib"],
    outs = [
        "svc-1",
        "svc-2",
    ],
)

The problem is in function _jsonnet_to_json_impl, where it expects that ctx.label.package will be nonempty:

["-J %s/%s" % (ctx.label.package, im) for im in ctx.attr.imports] +

In this case, since we're at the root of the workspace, that attribute is empty. Instead, that should be something more like this:

["-J %s%s" % (ctx.label.package + '/' if len(ctx.label.package) > 0 else "", im) for im in ctx.attr.imports] +

That is, only include the separating slash if there's a preceding package path to use as the parent path for the import path.

While I'm here, I'll note that it would be nice to not need to specify that "imports" attribute, as it looks like it could be gleaned from the paths to the "deps" entries.

Now there is:

http_archive(
    name = "io_bazel_rules_jsonnet",
    urls = [
        "http://mirror.bazel.build/github.com/bazelbuild/rules_jsonnet/archive/0.0.2.tar.gz",
        "https://github.com/bazelbuild/rules_jsonnet/archive/0.0.2.tar.gz",
    ],
    sha256 = "5f788c7719a02ed2483641365f194e9e5340fbe54963d6d6caa09f91454d38b8",
    strip_prefix = "rules_jsonnet-0.0.2",
)
load("@io_bazel_rules_jsonnet//jsonnet:jsonnet.bzl", "jsonnet_repositories")

jsonnet_repositories()

And it should be:

http_archive(
    name = "io_bazel_rules_jsonnet",
    urls = [
        "http://mirror.bazel.build/github.com/bazelbuild/rules_jsonnet/archive/0.0.3.tar.gz",
        "https://github.com/bazelbuild/rules_jsonnet/archive/0.0.3.tar.gz",
    ],
    sha256 = "e0f42a31f5b2feeeb5496d40cb0b7c5d3872bd7e0bdb44b04543cce96de321f8",
    strip_prefix = "rules_jsonnet-0.0.3",
)
load("@io_bazel_rules_jsonnet//jsonnet:jsonnet.bzl", "jsonnet_repositories")

jsonnet_repositories()

$ cat prow/monitoring/mixins/prometheus/BUILD.bazel 
load("@io_bazel_rules_jsonnet//jsonnet:jsonnet.bzl", "jsonnet_library", "jsonnet_to_json")

jsonnet_library(
    name = "prom_lib",
    srcs = [
        "prometheus.libsonnet",
        "ci_absent_alerts.libsonnet",
        "prow_monitoring_absent_alerts.libsonnet",
    ],
)

jsonnet_to_json(
    name = "prow_prometheusrule",
    src = "prow_prometheusrule.jsonnet",
    outs = ["prow_prometheusrule.yaml"],
    deps = [":prom_lib"],
    #without this line, json output works fine.
    yaml_stream = True,
    visibility = ["//visibility:public"],
)

$ bazel build //prow/monitoring/mixins/prometheus:prow_prometheusrule --verbose_failures
DEBUG: /home/hongkliu/.cache/bazel/_bazel_hongkliu/ef5e8e918e6947b01dd429bce37b0e43/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:256:5: rbe_default is using checked-in configs 'struct(config_repos = [], create_cc_configs = True, create_java_configs = True, env = {"ABI_LIBC_VERSION": "glibc_2.19", "ABI_VERSION": "clang", "BAZEL_COMPILER": "clang", "BAZEL_HOST_SYSTEM": "i686-unknown-linux-gnu", "BAZEL_TARGET_CPU": "k8", "BAZEL_TARGET_LIBC": "glibc_2.19", "BAZEL_TARGET_SYSTEM": "x86_64-unknown-linux-gnu", "CC": "clang", "CC_TOOLCHAIN_NAME": "linux_gnu_x86"}, java_home = "/usr/lib/jvm/java-8-openjdk-amd64", name = "9.0.0")'
INFO: Analyzed target //prow/monitoring/mixins/prometheus:prow_prometheusrule (1 packages loaded, 6 targets configured).
INFO: Found 1 target...
ERROR: /home/hongkliu/go/src/k8s.io/test-infra/prow/monitoring/mixins/prometheus/BUILD.bazel:12:1: error executing shell command: '/bin/bash -c set -e; bazel-out/host/bin/external/jsonnet/cmd/jsonnet -J . -J bazel-out/k8-fastbuild/bin -J bazel-out/k8-fastbuild/bin -y prow/monitoring/mixins/prometheus/prow_prometheusrule.jsonne...' failed (Exit 1) bash failed: error executing command 
  (cd /home/hongkliu/.cache/bazel/_bazel_hongkliu/ef5e8e918e6947b01dd429bce37b0e43/sandbox/linux-sandbox/18/execroot/io_k8s_test_infra && \
  exec env - \
    PATH=/home/hongkliu/tool/node/bin:/home/hongkliu/go/bin:/home/hongkliu/go/src/k8s.io/test-infra/node_modules/.bin:/home/hongkliu/tool/go1.12.5/bin:/home/hongkliu/tool/node/bin:/home/hongkliu/go/bin:/home/hongkliu/.local/bin:/home/hongkliu/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/hongkliu/tool/go/bin:/home/hongkliu/go/bin:/home/hongkliu/tool/go/bin \
  /bin/bash -c 'set -e; bazel-out/host/bin/external/jsonnet/cmd/jsonnet -J . -J bazel-out/k8-fastbuild/bin -J bazel-out/k8-fastbuild/bin -y prow/monitoring/mixins/prometheus/prow_prometheusrule.jsonnet -o bazel-out/k8-fastbuild/bin/prow/monitoring/mixins/prometheus/prow_prometheusrule.yaml')
Execution platform: @bazel_tools//platforms:host_platform

Use --sandbox_debug to see verbose messages from the sandbox: bash failed: error executing command 
  (cd /home/hongkliu/.cache/bazel/_bazel_hongkliu/ef5e8e918e6947b01dd429bce37b0e43/sandbox/linux-sandbox/18/execroot/io_k8s_test_infra && \
  exec env - \
    PATH=/home/hongkliu/tool/node/bin:/home/hongkliu/go/bin:/home/hongkliu/go/src/k8s.io/test-infra/node_modules/.bin:/home/hongkliu/tool/go1.12.5/bin:/home/hongkliu/tool/node/bin:/home/hongkliu/go/bin:/home/hongkliu/.local/bin:/home/hongkliu/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/hongkliu/tool/go/bin:/home/hongkliu/go/bin:/home/hongkliu/tool/go/bin \
  /bin/bash -c 'set -e; bazel-out/host/bin/external/jsonnet/cmd/jsonnet -J . -J bazel-out/k8-fastbuild/bin -J bazel-out/k8-fastbuild/bin -y prow/monitoring/mixins/prometheus/prow_prometheusrule.jsonnet -o bazel-out/k8-fastbuild/bin/prow/monitoring/mixins/prometheus/prow_prometheusrule.yaml')
Execution platform: @bazel_tools//platforms:host_platform

Use --sandbox_debug to see verbose messages from the sandbox
RUNTIME ERROR: stream mode: top-level object was a object, should be an array whose elements hold the JSON for each document in the stream.
        During manifestation    
Target //prow/monitoring/mixins/prometheus:prow_prometheusrule failed to build
INFO: Elapsed time: 0.279s, Critical Path: 0.08s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

What did I miss? Thanks.

Add a data attribute to Jsonnet rules to support the importstr statement, which can be used to import contents from arbitrary files not limited to .jsonnet files.

+cc @sparkprime

I have a use case where I want to take values fetched from bazel BUILD files and pipe them into my jsonnet_library targets. More notably I want to pipe in the BUILD_USER into a value in my library, then use that library elsewhere. This is similar to the "data" param common in other bazel rules, but this one routes variables in BUILD files rather than the contents of files in.

The jsonnet command-line tool can emit JSON to multiple files when invoked with the -m flag. It interprets the keys of the top-level map as file paths to which to write. Those paths can include not just a file name, but a directory path as well. Though the jsonnet tool does not create these output directories, if you first ensure that they do exist, it will write the output files to the proper directories. That is, when facing manifested JSON with the following top level structure

{
  "dir1/file1.json": {},
  "dir2/file2.json": {}
}

the jsonnet -m parent command will write to

• parent/dir1/file1.json
• parent/dir2/file2.json

as intended.

However, the jsonnet_to_json rule precludes this behavior. Instead, it infers the value of -m flag to be the directory path of the first output file. In the example above, jsonnet_to_json sees that the first output path has dir1 as its directory, and decides to pass -m dir1 to jsonnet, causing it to try to write to the following paths:

• dir1/dir1/file1.json
• dir1/dir2/file2.json

Even if we had taken care to create dir1 and dir2 beforehand, jsonnet will still fail to write those files, as neither directory dir1/dir1 nor dir1/dir2 are likely to exist.

Better would be to accept an optional attribute to designate the root output directory, with a default value of '.' (the current directory), and not make any assumptions about the first output path being indicative of the others.

Incompatible flag --incompatible_load_cc_rules_from_bzl will break rules_jsonnet once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

• [:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
• [:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
• [:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

ext_code and ext_str take a map of variable names and values, but ext_code_files and ext_str_files use parallel arrays. In my opinion, the map is a much better experience since it is possible for the user to accidentally transpose values in a larger list without noticing.

Any reason to keep the parallel arrays or would it be ok to switch everything to a map?

I was getting started on a PR to #98 but this style issue needs to be resolved prior to introducing
more inconsistency.

Currently importing libraries that are not under the current folder seems to require references that walk up and down the directory tree (../../../). If paths from the workspace root is available please document how to do this.

Incompatible flag --incompatible_disable_target_provider_fields will break rules_jsonnet once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

• [:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
• [:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
• [:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

See the logs: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/246#18005cc9-c2f3-4b2c-90a5-cc024ea05ff1

The error message is:

ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/451a80333a3ab92c117af7ae3c3ad420/external/examples/BUILD:75:1: Couldn't build file external/examples/files.json: error executing shell command: 'set -e; bazel-out/host/bin/external/jsonnet/cmd/jsonnet -J . -J bazel-out/k8-fastbuild/genfiles -J bazel-out/k8-fastbuild/bin --ext-str 'MYTEST' --ext-code 'MYJSONNET' --ext-str-file test=external/...' failed (Exit 1)
  | ERROR: environment variable MYTEST was undefined.

This fails with the Bazel nightly, and the recent 0.13.0 release.

I suspect the problem is that with sandboxing, your rule can't access the environment variables it needs.

I have a jsonnet_library in an external workspace that uses importstr to load a text file included in the data attribute. This only works if I add an external/... prefix to the import paths, which then doesn't work when developing that imported project separately. Does anyone have any idea of how to solve this?

Hello!
Jsonnet v0.14 is out for pretty long time and i think we should update?
Current rules release still use v0.13.

Incompatible flag --incompatible_use_platforms_repo_for_constraints will break rules_jsonnet once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

• [:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
• [:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
• [:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

I expect that I can test if a Jsonnet file can be evaluated without error by omitting golden attribute in jsonnet_to_json_test rule because README.md says that golden is optional.

However actually I got an error while I am building such a jsonnet_to_json_test rule.

How to reproduce

Put the following files into a directory

WORKSPACE:

git_repository(
    name = "io_bazel_rules_jsonnet",
    commit = "d93423f6a964bac6d56e413fdafec407f7b0b93d",
    remote = "https://github.com/bazelbuild/rules_jsonnet.git",
)

load("@io_bazel_rules_jsonnet//jsonnet:jsonnet.bzl", "jsonnet_repositories")

jsonnet_repositories()

BUILD:

load("@io_bazel_rules_jsonnet//jsonnet:jsonnet.bzl", "jsonnet_to_json_test")

jsonnet_to_json_test(
    name = "example_test",
    src = "example.jsonnet",
)

example.jsonnet:

{}

Then, run the following command in the directory

$ bazel test :example_test

Expected result

Bazel successfully build the test, it runs the test and the test succeeds.

Actual result

$ bazel test :example_test
..
WARNING: /private/var/tmp/_bazel_yugui/0cb3c1307b36763856ca742e5d952976/external/jsonnet/WORKSPACE:1: Workspace name in /private/var/tmp/_bazel_yugui/0cb3c1307b36763856ca742e5d952976/external/jsonnet/WORKSPACE (@__main__) does not match the name given in the repository's definition (@jsonnet); this will cause a build error in future versions.
ERROR: /Users/yugui/tmp/ss/BUILD:3:1: in jsonnet_to_json_test rule //:example_test: 
Traceback (most recent call last):
        File "/Users/yugui/tmp/ss/BUILD", line 3
                jsonnet_to_json_test(name = 'example_test')
        File "/private/var/tmp/_bazel_yugui/0cb3c1307b36763856ca742e5d952976/external/io_bazel_rules_jsonnet/jsonnet/jsonnet.bzl", line 169, in _jsonnet_to_json_test_impl
                "
".join(["#!/bin/bash", jsonnet_command,...])
        File "/private/var/tmp/_bazel_yugui/0cb3c1307b36763856ca742e5d952976/external/io_bazel_rules_jsonnet/jsonnet/jsonnet.bzl", line 173, in "
".join
                diff_command
name 'diff_command' is not defined.
ERROR: Analysis of target '//:example_test' failed; build aborted.
INFO: Elapsed time: 6.088s
ERROR: Couldn't start the build. Unable to run tests.

$ echo $?
1

Hello!
It seems that current rules does not support jsonnet -S

-S / --string Expect a string, manifest as plain text

I'm new to starlark and this repo, could you please point me out to what exactly should i do to contribute this feature as PR?

I'm trying to inject make variables into the template expansion to pass variables in from the bazel command line. This is useful when there are external environmental values, such as host names, in the context of Kubernetes templating.

For example:
bazel build :k8s --define fqdn=example.com

jsonnet_to_json(
    name = "k8s",
    src = "k8s.jsonnet",
    outs = ["k8s.json"],
    vars = {"fqdn": "$(fqdn)"},
)

The stamp_keys section seems to throw the MD to HTML off. Please clean it up so the output looks nice instead of making the rest of the documentation into HTML blob in the description for stamp_keys

Would it be possible to get multiple file outputs without having to list them in the outs of the jsonnet_to_json rule? Currently I have files with dynamically generated filenames for the outputs, similar to this:

{
    [name + "-file.json"]: someStructure(),
}

Though it would be possible it becomes quite a hassle to have to declare every generated file once complexity increases, if using this pattern.

I'm not that familiar with bazel so I don't know if this is possible, or if it requires static knowledge of the output files.

We are relying quite heavily on this rule set in production now and based on some comments it seems quite a few other people seem to also. Plus there have been open PRs for weeks now, including very small ones like just updating the jsonnet version. So I am curios if it is possible to become an official contributor/project member for these rules to help with the backlog and also for our own benefit to not have to rely so much on our custom fork. I have contributed here a bit now, as well as to some other rules, like rules_docker and rules_nodejs, so would say I have a fair grasp of bazel by now and even more so of this rule set.

I am happy to get started in any way that makes sense.

Change 40a1ab9 removed "vars" and "code_vars" arguments without notice. This breaks a lot of existing rules.

CC @sparkprime

Because we are generating documentation with Skydoc, the skylark_doc target is visible to projects when they add rules_jsonnet as an external repository. As a result, we should move examples and docs into sub-workspaces, similar to what is set up for other rules repos, such as rules_rust.

My jsonnet config needs the "--string" flag to ouput straight yaml instead of quoted strings. I could not find support for this flag, or to passing extra_args to the jsonnet build commnd. Does that exist?

hi. I have repository A which has a bunch of bazel rules. Then I have repository B which uses local_repository to import repo A.

Everything seems to work when I'm working exclusively in repository B. But when I go to repository A and import from B, this rule doesn't work anymore:

  jsonnet_to_json(
      name = "something_json",
      src = "//bazel/templates:deployment.jsonnet",
      deps = ["//bazel:ksonnet-lib"],
      outs = [
          "something.json",
      ],
  )

ERROR: /Users/yves/.yourbase/github.com/yourbase/yourbase/BUILD:18:1: no such package 'bazel': BUILD file not found on package path and referenced by '//:something_json'
ERROR: Analysis of target '//:higher-level-macro' failed; build aborted: no such package 'bazel': BUILD file not found on package path

I suspect that when the macro above is being evaluated in an external context, the path //bazel is not being resolved relative to the imported repository, but to the outter repository.

Do you think this could be the case? I want to help fix it if that's the case, but I don't know how to express "import relative to the inner repo". Any help would be much appreciated.

From version 0.20 onwards, bazel by default will no longer support the native http and git repositories. Please switch to the Starlark versions of those rules.

For more background, see

• the original announcement https://groups.google.com/d/topic/bazel-discuss/dO2MHQLwJF0/discussion

• and the documentation of the incompatible changes https://docs.bazel.build/versions/master/skylark/backward-compatibility.html#remove-native-git-repository

Once migrated, please also file an issue against http://github.com/bazelbuild/continuous-integration to reenable testing of your project as a bazel downstream project.

Thanks,
Klaus

+@sparkprime

Just realized that with the changes in #16, jsonnet_to_json_tests for negative tests (i.e. error set to non-zero) are failing because we run jsonnet on the golden file, which in this case contains the expected error output.

The fix for this is to have the generated shell script cat the golden file if ctx.attr.error != 0 and use the default behavior of using jsonnet to canonicalize the golden file otherwise.

I'm trying to import ksonnet into a jsonnet file using bazel and I'm not sure how to do that.

Right now I've got the following:

WORKSPACE

# KSonnet
ksonnet_build_file = '''
filegroup(
    name = "ksonnet_files",
    srcs = glob(["ksonnet.beta.2/*.libsonnet"]),
    visibility = ["//visibility:public"],
)
'''

new_git_repository(
    name = "com_github_ksonnet_lib",
    remote = "https://github.com/ksonnet/ksonnet-lib.git",
    tag = "beta.2",
    build_file_content = ksonnet_build_file,
)

BUILD

jsonnet_library(
    name = "ksonnet-lib",
    srcs = [
        "@com_github_ksonnet_lib//:ksonnet_files",
    ],
)

jsonnet_to_json(
    name = "transform-receiver-test",
    src = "receiver.jsonnet",
    outs = ["receiver-test.json"],
    deps = ["ksonnet-lib"]
)

receiver.jsonnet

local k = import "external/com_github_ksonnet_lib/ksonnet.beta.2/k.libsonnet";

The import statement is not working. It's not clear to me what ksonnet-lib is supposed to do with the input files it's given. Does it create a new importable file?

Actually, this is almost working. Is jsonnet_library equivalent to a filegroup? Should I be adding an import call to the jsonnet_to_json call?

If you add the following to examples/BUILD:

genrule(
    name = "generated",
    outs = ["generated.jsonnet"],
    cmd = "echo \"(import 'import_me.libsonnet')\" > $@"
)

jsonnet_to_json(
    name = "import",
    src = "generated.jsonnet",
    imports = ["examples"],
    deps = [":import_lib"],
    outs = ["import.json"],
)

jsonnet_library(
    name = "import_lib",
    srcs = ["import_me.libsonnet"],
)

Results in an error:

RUNTIME ERROR: Couldn't open import "import_me.libsonnet": No match locally or in the Jsonnet library paths.

The command emitted by the rule is /bin/bash -c 'set -e; bazel-out/host/bin/external/jsonnet/cmd/jsonnet -J . -J bazel-out/darwin-fastbuild/genfiles -J bazel-out/darwin-fastbuild/bin bazel-out/darwin-fastbuild/genfiles/external/examples/generated.jsonnet -o bazel-out/darwin-fastbuild/bin/external/examples/import.json'

However, if src = some identical file that lives in the examples directory along with the import_me.libsonnet, then everything works.

In my real-world project I am able to workaround by adding imports = ["."], but that does not work in the examples directory, maybe because it is a workspace root and -J /. is appended to the command. Not sure how to make the scenario pass in the examples.

Is this expected behavior?

Based on the documentation, I expect to be able to pass in literal key-value pairs in the ext_strs dict:

Map of strings to pass to jsonnet as external variables via --ext-str key=value.

and not names of configuration variables.

However, at jsonnet.bzl, line 118, the value of the external string is taken using _quote(ctx.var[val]). Is there a reason _quote(val) was not used instead?

I often find it useful to diff the output of jsonnet_to_json when testing compatibility between new and old libsonnet code (e.g. for manifestation of ECS task and Service definitions).

I've written my own rule that does this, including normalization based on jq scripts.

1. It would be excellent to be able to supply a normalization script to be run post-manifestJson in jsonnet_to_json. For example:

# Apply f to composite entities recursively, and to atoms
def walk(f):
  . as $in
  | if type == "object" then
      reduce keys[] as $key
        ( {}; . + { ($key):  ($in[$key] | walk(f)) } ) | f
  elif type == "array" then map( walk(f) ) | f
  else f
  end;

def normalize: walk(if type == "array" then sort else . end) | del(.roleArn, .events, .schedulingStrategy, .deployments, .deploymentConfiguration, .status, .pendingCount, .createdAt, .runningCount, .serviceArn, .clusterArn);

2. It would be great to create a json _vs_json_diff bazel run target that depends on the outputs of jsonnet_to_json rules and presents the user with a jsondiffpatch of the new and original json. I have implemented this and have been using for a while, but have concerns regarding dependencies (like jsondiffpatch and jq).

It looks like sort of like this.

# rough implementation example
def _json_diff(ctx):
  # service json as generated via current jsonnet pipeline.
  original_json_arg = ctx.files.original_json[0].short_path
  original_json_arg = ctx.expand_make_variables("originalJSON", original_json_arg, {})
  new_json_arg = ctx.files.new_json[0].short_path
  new_json_arg = ctx.expand_make_variables("newJSON", new_json_arg, {})
  diff_arg = ctx.files._diff[0].short_path
  diff_arg = ctx.expand_make_variables("diff", diff_arg, {})

  substitutions = {
      "%{originalJSON}": original_json_arg,
      "%{newJSON}": new_json_arg,
      "%{diff}": diff_arg,
  }

  ctx.actions.expand_template(
      template = ctx.file._template,
      substitutions = substitutions,
      output = ctx.outputs.executable,
  )

  return [DefaultInfo(
      runfiles=ctx.runfiles(files=ctx.files.original_json + ctx.files.new_json + ctx.files._diff),
  )]

json_diff = rule(
    attrs = {
            "_template": attr.label(
                default = Label("//:json_diff.sh.tpl"),
                single_file = True,
                allow_files = True,
            ),
            "_diff": _diff_attr(),
            "new_json": attr.label(allow_single_file=True),
            "original_json": attr.label(allow_single_file=True),
      },
    executable = True,
    implementation = _json_diff,
)

template

#!/bin/bash
set -euo pipefail
DIFF_UTIL="${PWD}/%{diff}"

ORIGINAL_JSON_FILE="${PWD}/%{originalJSON}"
NEW_JSON_FILE="${PWD}/%{newJSON}"
cat $ORIGINAL_JSON_FILE | jq -S -f $JQ_SCRIPT > "${ORIGINAL_JSON_FILE}.clean"
cat $NEW_JSON_FILE | jq -S -f $JQ_SCRIPT > "${NEW_JSON_FILE}.clean"

$DIFF_UTIL "${ORIGINAL_JSON_FILE}.clean" "${NEW_JSON_FILE}.clean"

And the diff script

#!/usr/bin/env node

var jsondiffpatch = require('jsondiffpatch');
var differ = require('jsondiffpatch').create({
    textDiff: {
        minLength: Number.MAX_VALUE,
    },
});

const fs = require('fs');

// "original" JSON
const fileLeft = process.argv[2];
// "new" JSON or original + modifications
const fileRight = process.argv[3];

if (!fileLeft || !fileRight) {
  console.log('\n  USAGE: jsondiffpatch left.json right.json');
  return;
}

var left = JSON.parse(fs.readFileSync(fileLeft));
var right = JSON.parse(fs.readFileSync(fileRight));
var diff = differ.diff(left, right);
console.log(jsondiffpatch.formatters.console.format(diff));

Incompatible flag --incompatible_no_implicit_file_export will be enabled by default in a future Bazel release [1], thus breaking rules_jsonnet.

The flag is documented here: bazelbuild/bazel#10225

Please check the following CI builds for build and test results:

• Ubuntu 16.04, OpenJDK 8
• macOS, OpenJDK 8
• Ubuntu 18.04, OpenJDK 11

Never heard of incompatible flags before? We have documentation that explains everything.

If you don't want to receive any future issues for rules_jsonnet or if you have any questions,
please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

[1] The target release hasn't been determined yet. Our tool will update the issue title once the flag flip has been scheduled.

Not sure what facilities exist at the jsonnet level for injecting this kind of thing, but especially in the context of Kubernetes templating it would be fantastic if we could optionally inject build context.

The particular scenario I am envisioning would be environment modeling, where I might want to inject {BUILD_USER} into the template expansion (e.g. as a Docker Image tag). Or for integration testing on PRs I might want an environment per PR based on the PR number, which might come through --workspace_status_command.

No example exists that uses ext_strs.
There seems to be the intuitive way to use this param, and what seems to be coded, which is broken

You can fix this by changing this line here
https://github.com/bazelbuild/rules_jsonnet/blob/master/jsonnet/jsonnet.bzl#L251
to the following

   % (_quote(key), _quote(val)) for key, val in jsonnet_ext_strs.items()] +

Needing to look up the value in ctx.var was unnecessary and broken. the vars param is not legal for jsonnet_to_json and even if it was I don't think this line is accessing it correctly, and even if it was, this is a horrible way for the user to specify strings.

By needing to go through the vars parameter the user must specify 2 maps. One for listing what name the jsonnet can access sthe string, and another map to help your code access the value to pass as the value.

jsonnet_to_json(
name = "kube-service",
src = "kube-service.jsonnet",
outs = ["frontend_deployment.json", "service_deployment.json"],
multiple_outputs = 1,
vars = {"actual_image": "my value", "actual_port": "50001"},
ext_strs = {"image":"actual_image", "port": "actual_port"},
)

Instead just use the value in the ext_strs map

jsonnet_to_json(
name = "kube-service",
src = "kube-service.jsonnet",
outs = ["frontend_deployment.json", "service_deployment.json"],
multiple_outputs = 1,
ext_strs = {"image":"my value", "port": "50001"},
)

Some of the .bzl files use legacy starlark constructs that have been deprecated. The flag --all_incompatible_changes surfaces a few instances as such.

For example:

INFO: Invocation ID: d32772b6-fc44-4f2e-9ff4-26780574b35d
ERROR: /home/zhongming/git/LogiOcean/examples/jsonnet/BUILD:186:1: in jsonnet_to_json_test rule //examples/jsonnet:yaml_stream_test: 
Traceback (most recent call last):
        File "/home/zhongming/git/LogiOcean/examples/jsonnet/BUILD", line 186
                jsonnet_to_json_test(name = 'yaml_stream_test')
        File "/home/zhongming/.cache/bazel/_bazel_zhongming/2aea414f4d422ad78870bb37da8563ec/external/io_bazel_rules_jsonnet/jsonnet/jsonnet.bzl", line 355, in _jsonnet_to_json_test_impl
                ctx.file_action(output = ctx.outputs.executable, c...), ...)
Use ctx.actions.write instead of ctx.file_action. 
Use --incompatible_new_actions_api=false to temporarily disable this check.

Would like these incompatible changes to get cleaned up.

Incompatible flag --incompatible_no_implicit_file_export will break rules_jsonnet once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

• [:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
• [:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
• [:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

In this example:

genrule(
  name = "generated",
  outs = ["g.jsonnet"],
  cmd = "echo '{}' > $@"
)

jsonnet_to_json_test(
  name = "test",
  src = ":generated",
)

jsonnet_to_json(
  name = "build",
  src = ":generated",
  outs = ["j.json"]
)

While bazel build :build works as expected, bazel test :test fails with

Output: Opening input file: bazel-out/darwin-fastbuild/genfiles/g.jsonnet: No such file or directory

When running commands like npx bazel query --output=graph 'deps(//libs/bazel-rules/kube_app:app_lib)' where //libs/bazel-rules/kube_app:app_lib is a jsonnet_library you get the error:

ERROR: /home/f.wiles/.cache/bazel/_bazel_f.wiles/4e95cbad76bfb4ed000904165db39268/external/jsonnet_go/cmd/jsonnet/BUILD.bazel:3:1: no such package '@com_github_fatih_color//': The repository '@com_github_fatih_color' could not be resolved and referenced by '@jsonnet_go//cmd/jsonnet:go_default_library'

On occasion it is useful to refactor jsonnet across many jsonnet files. Depending on code-review practices and user abuse of jsonnet, it can more than just using go-ast to make the fixes and dump the AST, but rather tedius manual intervention.

It would be ideal if this could be handled via jsonnet itself by allowing jsonnet code to be evaluated and modified, then dumped jsonnet fmt style to jsonnet code.

The desired rule would consume a jsonnet file which could itself import a jsonnet file, modify it, and manifest it to jsonnet.

See google/go-jsonnet#193 (comment) and some example code below to get an idea of what I'm looking for.

local codefile = 'code.jsonnet';
local codefilestr = importsrt 'code.jsonnet';


local codefileModified = std.fmt(std.manifestJsonnet(codefile + {memory: std.native('ecsMemoryAlloc')(codefile.name, codefile.cluster, ...);
local codefileOriginal = std.fmt(codefileStr);
local normalize = function(json, func)
                                       func(json)     
local diffStr := std.diffJson(normalize(codefileModified, function(i) i)), normalize(codefileOriginal, function(i) i));
          
std.trace(diffStr, null)
 
// multiple outputs for a prettier diff function executed via bazel run                    
{
    // manifestJsonnet acts on snippets and returns the result of evaluation as a string 
    // with sugar, hidden fields, all preserved (like the output of importstr)
    "codefile-modified.jsonnet": 
    "codefile-original.jsonnet": std.fmt(codefilestr),
}

I get the following error when trying to run a bazel rule that depends on the google cloud go api if my workspace also loads jsonnet_repositories.

ERROR: .../external/com_github_googleapis_gax_go/BUILD.bazel:5:1: no such package '@org_golang_google_grpc//status': BUILD file not found on package path and referenced by '@com_github_googleapis_gax_go//:go_default_library'.
ERROR: Analysis of target '//deploy:deploy-to-minikube' failed; build aborted.

I am very puzzled by this, I don't see why jsonnet_repositories could trample an unrelated package.

It would be helpful for the jsonnet_to_json rule to have an argument triggering verification that the outs enumerated in the rule matches all the output files generated from the inputs.

I was recently bit by a case where I added several output files but forgot to update my BUILD rule.

Last release, 0.0.3, is 2 years old. It no longer works with current version of bazel. Sample code on readme points to it and makes adoption harder. A new release and documentation would be really nice.

jsonnet has cut a new release v0.11.2. Please update the dependency.

google/jsonnet#542

Any plans to update to jsonnet 0.9.0?

It would be handy to support top level arguments as a way to parameterize your jsonnet.

It would be useful for jsonnet_library targets to be able to read data from non-Jsonnet files.

It could probably also use rules to make consuming this straightforward.