facebookincubator / antlir Goto Github PK

antlir/bzl/shape.bzl implements a type checker in starlark (the limited flavor of python that is used by buck). As part of this goal, it also includes a python code generator to enable type-checking serialized shape instances at runtime when they are loaded by python scripts.

Unfortunately, the code generator today does not play very nicely with Python type checkers (specifically we have been using Pyre).

If a shape type is nested inside of another shape type, we are not able to annotate the nested type with Pyre.

As an example, if you have:

type_a = shape.shape(...)

type_y = shape.shape(

   ...

   z = shape.list(type_a)

)

Then you export an instance of type_y into a Python file, you cannot annotate type_a, as it's a member of the generated class for type_y with a hidden type name.

Options here are to explore Pyre to find something that works, or change the shape codegenning in a way that makes readable class names for all nested types as well such that they could be imported.

Testing changes

The existing unit tests for shape.bzl are pretty good, and you can run them with buck test //antlir/bzl/tests/shapes/.... The generated code can be found in buck-out, at something like these paths buck-out/gen/antlir/bzl/tests/shapes/test-shape#binary,link-tree/antlir/bzl/tests/shapes/*.py. The pyre docs should have good instructions for getting set up, including instructions for getting it setup with buck integration

• Antlir now supports building GPT images, vm infra should support booting these full disk images without specifying a kernel (+cmdline+initrd)
• Most of the 1 week timeline should be writing tests

• Build QEMU in an Antlir container (like we do for systemd internally) so that we can use more modern versions and be consistent across internal + oss usage
• This should be done first to have a baseline level of VM support in opensource for GitHub Actions and common ground across users

Have you considered extracting the rpm feature as their own library?
https://github.com/facebookincubator/fs_image/tree/master/fs_image/rpm
these would be quite useful of their own, even outside Buck/Bazel

• vmtest currently only supports a single vm with a host only network using hardcoded IPv6 addresses
• VMs should be able to connect to services running on the host or network with other VMs
  • There is some support for on-host services introduced in b99ddf9, but it is very rudimentary

buck test has provisions for running with a custom test runner (see https://buck.build/files-and-dirs/buckconfig.html#test.external_runner)

Benefits

A custom test runner would give us a lot of control over how tests are run

• failing tests could be retried
• consistently-failing tests can be marked as disabled (can be stored in a text file in the repo)
• test parallelism can be controlled (eg 1 process per test case, up to $(nproc) at a time)

Implementation details

• The buck config docs linked above describe the input that buck itself provides.
• The test runner would need to parse the buck-provided JSON file, and then have a switch based on the type to determine how to execute the test.
• The test runner should support pretty html output like we have today (e8bf770)
  • this can be done by either implementing an output format understood by the workflow we use now https://github.com/dorny/test-reporter#supported-formats (jest-junit is probably the best documented)
  • alternatively, you can just re-implement the functionality of that test reporter to create a status check with the github api and render some html however you want
• Automatic disabler (the test state machine) can be implemented as a periodically scheduled GitHub Action that looks at the past N days of test workflow runs and marks tests that have failed 3 times in a row (or some other threshold) as disabled (it can either submit a PR to a text file in the repo, or use a DB in AWS to track state)

Desired features

Features can be added in stages, the bare minimum for the first revision is a test runner that implements Python unit tests, and outputs a single pass/fail for the test target based on the return code obtained from just running the pex.

1. Parse buck test json, run each target and produce an output entry for the entire test target based on the returncode
2. Parse the individual test target output to produce output entries for each individual test case
3. List tests out of the test target binaries before running and run them in parallel
4. cli flag to retry failed tests (this can be passed to the runner through buck test as described in the docs)
5. Runner should not execute disabled test cases unless given a special --run-disabled flag
6. GitHub Actions workflow to read old test result artifacts and submit a PR / write to an AWS DB to mark consistently failing tests as disabled

This dir is non-standard and not really useful outside of the FB infra. Let's not have it implemented by default for all containers here.

• QEMU is the only supported VMM at this point in time, but Firecracker could make some significant performance improvements in the vm infra
• It might not work for vmtest (in dev mode anyway) due to a lack of host filesystem sharing, but can be used for generic vm support
• This can also be used on-host in MetalOS

• Our qemu setup only supports booting from disk
• Add support for a DHCP+TFTP server in antlir/vm (ideally fbtftp & dhcplb)
• Write integration tests
• This may also require being able to have vmtest tests survive reboots of the underlying VM

The installation section of the documentation is missing few important details which will help a new contributor to get started quickly with the setup. These include:

• Having rpm2cpio, zstd as dependencies
• Not running buck as root user
• As image builds themselves run through sudo, this requires adding the user to the sudoers file, so that password is not required on every invocation, else the build will fail.

I am interested to work on adding these details to the documentation.

Disregard- clicked the wrong project when creating this one.

The documentation on image.genrule_layer provides the following example:

image.layer(
    name = '_setup_foo',
    parent_layer = '...',
    features = [
        # `genrule_layer` runs as `nobody` by default
        image.ensure_subdirs_exist('/', 'output', user='nobody'),
        image.install(':foo', '/output/_temp_foo'),
    ],
)

image.genrule_layer(
    name = '_translate_foo',
    parent_layer = ':setup',
    cmd = ['/bin/sh', '-c', 'tr a-z A-Z < /output/_temp_foo > /output/FOO'],
)

image.layer(
    name = 'foo',
    parent_layer = ':_translate_foo',
    # Clean up temporary state
    features = [image.remove_path('/output/_temp_foo')],
)

However, this is broken in more than one way:

• The genrule_layer rule is missing two required arguments: rule_type and antlir_rule
• remove_path should be just remove
• The parent layer used by genrule should be :_setup_foo instead of :setup
• As it is currently written, there is a circular dependency on the target :foo, since it is installed in _setup_foo. I assume this would be some file in the host system that is installed in the image and used as the input to the genrule_layer.

Let me know if I got something wrong. Will be creating a PR to fix this soon.

Use the branch oss_snapshot (where a working build appliance exists). The target fails with the following stack trace:

Building: finished in 1.7 sec (100%) 56/56 jobs, 1 updated
  Total time: 3.0 sec
Command failed with exit code 1.

command: [/bin/bash, -e, /home/ls/work/fs_image/buck-out/tmp/genrule-4304085449718242770.sh]

stderr: + binary_path=(/usr/bin/python3 /home/ls/work/fs_image/buck-out/gen/fs_image/subvolume-version.pex)
++ /usr/bin/python3 /home/ls/work/fs_image/buck-out/gen/fs_image/subvolume-version.pex
+ subvolume_ver=JQ6eCm4.H80.-oc
+ subvolume_wrapper_dir=create_ops-from-dir:JQ6eCm4.H80.-oc
+ mkdir /home/ls/work/fs_image/buck-out/bin/fs_image/compiler/test_images/create_ops-from-dir__/out
+ echo '{}'
+ /usr/bin/python3 /home/ls/work/fs_image/buck-out/gen/fs_image/layer-mount-config.pex //fs_image/compiler/test_images:create_ops-from-dir
+ layer_json=/home/ls/work/fs_image/buck-out/bin/fs_image/compiler/test_images/create_ops-from-dir__/out/layer.json
++ readlink -f /home/ls/work/fs_image/buck-out/gen/../../buck-out/.volume-refcount-hardlinks/
+ refcounts_dir=/home/ls/work/fs_image/buck-out/.volume-refcount-hardlinks
+ /usr/bin/python3 /home/ls/work/fs_image/buck-out/gen/fs_image/subvolume-garbage-collector.pex --refcounts-dir /home/ls/work/fs_image/buck-out/.volume-refcount-hardlinks --subvolumes-dir /home/ls/work/fs_image/buck-image-out/volume/targets --new-subvolume-wrapper-dir create_ops-from-dir:JQ6eCm4.H80.-oc --new-subvolume-json /home/ls/work/fs_image/buck-out/bin/fs_image/compiler/test_images/create_ops-from-dir__/out/layer.json
Deleting create_ops-from-dir:JQ6dx8Q.GnY.fY7U since its refcount has 1 links
Delete subvolume (no-commit): '/home/ls/work/fs_image/buck-image-out/volume/targets/create_ops-from-dir:JQ6dx8Q.GnY.fY7U/create_ops'
+ /usr/bin/python3 -Es /home/ls/work/fs_image/buck-out/gen/fs_image/compiler.pex --artifacts-may-require-repo --subvolumes-dir /home/ls/work/fs_image/buck-image-out/volume/targets --subvolume-rel-path create_ops-from-dir:JQ6eCm4.H80.-oc/create_ops --build-appliance-json /home/ls/work/fs_image/buck-out/gen/appliance/build_appliance/layer/layer.json --rpm-installer dnf --allowed-host-mount-target=//appliance:host_build_appliance --child-layer-target //fs_image/compiler/test_images:create_ops-from-dir --child-feature-json /dev/fd/62 --child-dependencies //fs_image/compiler/test_images:create_ops-dir /home/ls/work/fs_image/buck-out/gen/fs_image/compiler/test_images/create_ops-dir/create_opscreate_ops-dir
++ echo '{"features":[{"receive_sendstreams":[{"source":{"content_hash":null,"generator":null,"generator_args":[],"layer":null,"path":"sendstream","source":{"__BUCK_TARGET":"//fs_image/compiler/test_images:create_ops-dir"}}}],"target":"//fs_image/compiler/test_images:create_ops-from-dir"}],"target":"//fs_image/compiler/test_images:create_ops-from-dir"}'
At subvol create_ops
ERROR: lsetxattr hello security.selinux=user_u:object_r:base_t failed: Invalid argument
Traceback (most recent call last):
  File "<string>", line 37, in <module>
  File "<string>", line 35, in __run
  File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ls/work/fs_image/buck-out/gen/fs_image/compiler#link-tree/fs_image/compiler/compiler.py", line 197, in <module>
    build_image(args).to_json_file(sys.stdout)
  File "/home/ls/work/fs_image/buck-out/gen/fs_image/compiler#link-tree/fs_image/compiler/compiler.py", line 167, in build_image
    builder(subvol)
  File "/home/ls/work/fs_image/buck-out/gen/fs_image/compiler#link-tree/fs_image/compiler/items/make_subvol.py", line 85, in builder
    pass
  File "/usr/lib64/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/home/ls/work/fs_image/buck-out/gen/fs_image/compiler#link-tree/fs_image/subvol_utils.py", line 715, in receive
    yield
  File "/usr/lib64/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/home/ls/work/fs_image/buck-out/gen/fs_image/compiler#link-tree/fs_image/subvol_utils.py", line 222, in popen_as_root
    check_popen_returncode(pr)
  File "/home/ls/work/fs_image/buck-out/gen/fs_image/compiler#link-tree/fs_image/common.py", line 73, in check_popen_returncode
    returncode=proc.returncode, cmd=proc.args,
subprocess.CalledProcessError: Command '['sudo', 'TMP=', '--', 'btrfs', 'receive', b'/proc/8151/fd/6']' returned non-zero exit status 1.

    When running <genrule>.
    When building rule //fs_image/compiler/test_images:create_ops-from-dir.