firesim / firemarshal Goto Github PK

Software workload management tool for RISC-V based SoC research. This is the default workload management tool for Chipyard and FireSim.

Home Page: https://docs.fires.im/en/latest/Advanced-Usage/Workloads/index.html

License: Other

Shell 7.04% HTML 0.05% Makefile 0.85% Python 51.11% Assembly 2.58% C 38.36%

buildroot chipyard distro fedora firesim linux qemu spike workload

firemarshal's Introduction

FireSim: Fast and Effortless FPGA-accelerated Hardware Simulation with On-Prem and Cloud Flexibility

We held the First FireSim and Chipyard User/Developer Workshop at ASPLOS 2023 on March 26, 2023! This workshop featured a full-day of talks from users and developers in the FireSim and Chipyard community. YouTube videos of the talks are available on the 2023 Workshop Page!

Using FireSim
What is FireSim?
What can I simulate with FireSim?
Need help?
Contributing
Publications

Using FireSim

To get started with FireSim, you can find our extensive documentation and getting-started guide at docs.fires.im. The FireSim codebase is open-source at github.com/firesim/firesim and we welcome pull requests and issues. You can also get help from the FireSim user community on our User Forum. Additionally, we frequently run tutorials at various conferences and events; for overview purposes, you can find the most recent slide decks at fires.im/tutorial-recent (you should still follow docs.fires.im for the most up to date getting-started guide).

Another good overview from a recent event (in video format) can be found on YouTube.

What is FireSim?

FireSim is an open-source FPGA-accelerated full-system hardware simulation platform that makes it easy to validate, profile, and debug RTL hardware implementations at 10s to 100s of MHz. FireSim simplifies co-simulating ASIC RTL with cycle-accurate hardware and software models for other system components (e.g. I/Os). FireSim can productively scale from individual SoC simulations hosted on on-prem FPGAs (e.g., a single Xilinx Alveo board attached to a desktop) to massive datacenter-scale simulations harnessing hundreds of cloud FPGAs (e.g., on Amazon EC2 F1).

Who's using and developing FireSim? FireSim users across academia and industry (at 20+ institutions) have published over 40 papers using FireSim in many areas, including computer architecture, systems, networking, security, scientific computing, circuits, design automation, and more (see the Publications page). FireSim has also been used in the development of shipping commercial silicon. FireSim was originally developed in the Electrical Engineering and Computer Sciences Department at the University of California, Berkeley, but now has industrial and academic contributors from all over the world.

You can learn more about FireSim in the following places:

FireSim website: https://fires.im
FireSim ISCA 2018 Paper: Paper PDF | IEEE Xplore | ACM DL | BibTeX | Selected as one of IEEE Micro’s “Top Picks from Computer Architecture Conferences, 2018”.
FireSim ISCA-50 Retrospective: Paper PDF
FireSim documentation: https://docs.fires.im
Scala API Documentation: https://fires.im/firesim/latest/api/
LatchUp 2023 Update Talk: YouTube
FireSim (+Chipyard) Tutorial: https://fires.im/tutorial-recent/
ASPLOS 2023 Tutorial Videos: YouTube Playlist
Updates/News: Changelog | FireSim Blog | Twitter

What can I simulate with FireSim?

FireSim can simulate arbitrary hardware designs written in Chisel or Verilog. With FireSim, users can write their own RTL (processors, accelerators, etc.) and run it at near-FPGA-prototype speeds on cloud or on-prem FPGAs, while obtaining performance results that match an ASIC implementation of the same design. Depending on the hardware design and the simulation scale, FireSim simulations run at 10s to 100s of MHz. Users can also integrate custom software models for components that they don't need or want to write as RTL. To help construct a closed and deterministic simulated environment around a design, FireSim includes validated and high-performance HW/SW models for I/Os like DRAM, Ethernet, Disks, UART, and more. The User Publications page links to a selection of papers written by FireSim users.

FireSim was originally developed to simulate datacenters by combining open RTL for RISC-V processors with a custom cycle-accurate network simulation. By default, FireSim provides all the RTL and models necessary to cycle-exactly simulate from one to thousands of multi-core compute nodes, derived directly from silicon-proven and open target-RTL (RISC-V Rocket Chip, BOOM, and Chipyard), with an optional cycle-accurate network simulation tying them together. FireSim also provides a Linux distribution that is compatible with the RISC-V systems it simulates and automates the process of including new workloads into this Linux distribution. These simulations run fast enough to interact with Linux on the simulated system at the command line, like a real computer. Users can even SSH into simulated systems in FireSim and access the Internet from within them.

Head to the FireSim Website to learn more.

Need help?

Join the FireSim Mailing list: https://groups.google.com/forum/#!forum/firesim
Post an issue on this repo
Follow on Twitter for project updates: @firesimproject

Contributing

See CONTRIBUTING.md

Publications

ISCA 2018: FireSim: FPGA-Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud

You can learn more about FireSim in our ISCA 2018 paper, which covers the overall FireSim infrastructure and large distributed simulations of networked clusters. This paper was selected as one of IEEE Micro’s "Top Picks from Computer Architecture Conferences, 2018" and for "ISCA@50 25-year Retrospective 1996-2020".

Sagar Karandikar, Howard Mao, Donggyu Kim, David Biancolin, Alon Amid, Dayeol Lee, Nathan Pemberton, Emmanuel Amaro, Colin Schmidt, Aditya Chopra, Qijing Huang, Kyle Kovacs, Borivoje Nikolic, Randy Katz, Jonathan Bachrach, and Krste Asanović. FireSim: FPGA-Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud. In proceedings of the 45th International Symposium on Computer Architecture (ISCA’18), Los Angeles, CA, June 2018.

Paper PDF | IEEE Xplore | ACM DL | BibTeX

FPGA 2019: FASED: FPGA-Accelerated Simulation and Evaluation of DRAM

Our paper from FPGA 2019 details the DRAM model used in FireSim:

David Biancolin, Sagar Karandikar, Donggyu Kim, Jack Koenig, Andrew Waterman, Jonathan Bachrach, Krste Asanović, FASED: FPGA-Accelerated Simulation and Evaluation of DRAM, In proceedings of the 27th ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, Seaside, CA, February 2018.

Paper PDF | ACM DL | BibTeX

IEEE Micro Top Picks of 2018: FireSim: FPGA-Accelerated, Cycle-Accurate Scale-Out System Simulation in the Public Cloud

This article discusses several updates since the FireSim ISCA 2018 paper:

Sagar Karandikar, Howard Mao, Donggyu Kim, David Biancolin, Alon Amid, Dayeol Lee, Nathan Pemberton, Emmanuel Amaro, Colin Schmidt, Aditya Chopra, Qijing Huang, Kyle Kovacs, Borivoje Nikolic, Randy Katz, Jonathan Bachrach, and Krste Asanović. FireSim: FPGA-Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud. IEEE Micro, vol. 39, no. 3, pp. 56-65, (Micro Top Picks 2018 Issue). May-June 2019.

Article PDF

ICCAD 2019: Golden Gate: Bridging The Resource-Efficiency Gap Between ASICs and FPGA Prototypes

Our paper describing FireSim's Compiler, Golden Gate:

Albert Magyar, David T. Biancolin, Jack Koenig, Sanjit Seshia, Jonathan Bachrach, Krste Asanović, Golden Gate: Bridging The Resource-Efficiency Gap Between ASICs and FPGA Prototypes, In proceedings of the 39th International Conference on Computer-Aided Design (ICCAD '19), Westminster, CO, November 2019.

Paper PDF

ASPLOS 2020: FirePerf: FPGA-Accelerated Full-System Hardware/Software Performance Profiling and Co-Design

Our paper that discusses system-level profiling features in FireSim:

Sagar Karandikar, Albert Ou, Alon Amid, Howard Mao, Randy Katz, Borivoje Nikolić, and Krste Asanović, FirePerf: FPGA-Accelerated Full-System Hardware/Software Performance Profiling and Co-Design, In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2020), Lausanne, Switzerland, March 2020.

Paper PDF

IEEE MICRO 2021: Accessible, FPGA Resource-Optimized Simulation of Multi-Clock Systems in FireSim

In this special issue, we describe the automated instance-multithreading optimization and support for multiple clock domains in the simulated target.

David Biancolin, Albert Magyar, Sagar Karandikar, Alon Amid, Borivoje Nikolić, Jonathan Bachrach, Krste Asanović. Accessible, FPGA Resource-Optimized Simulation of Multi-Clock Systems in FireSim. In IEEE Micro Volume: 41, Issue: 4, July-Aug. 1 2021

Article PDF

ISCA@50 Retrospective: 1996-2020: FireSim: FPGA-Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud

This retrospective paper, included in the "ISCA@50 Retrospective: 1996-2020" collection, provides an update and retrospective on FireSim's development and evolution since the original ISCA 2018 paper.

Sagar Karandikar, Howard Mao, Donggyu Kim, David Biancolin, Alon Amid, Dayeol Lee, Nathan Pemberton, Emmanuel Amaro, Colin Schmidt, Aditya Chopra, Qijing Huang, Kyle Kovacs, Borivoje Nikolic, Randy Katz, Jonathan Bachrach, and Krste Asanović. FireSim: FPGA-Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud. In ISCA@50 Retrospective: 1996-2020, Edited by José F. Martínez and Lizy K. John, June 2023.

Retrospective PDF

You can find other publications, including publications that use FireSim on the FireSim Website.

firemarshal's People

Contributors

Stargazers

Watchers

firemarshal's Issues

spike doesn't run with jobs

Running a job with the '--spike' flag actually runs the job in Qemu

Workload Launch Arguments

It would be nice to be able to pass launch-time arguments to the workload. This would allow users to build a single large workload with many tests, and then just tell it which test to run at launch time instead of building a bunch of independent workloads (and their associated disk cost).

This might be possible with the kernel command line, or a quick modification of the base image right before launch. Whatever we do should work in FireSim.

Drivers don't build with FTRACE enabled

Initially brought by [email protected]

We can't seem to build a workload with ftrace enabled.

To reproduce: create a workload with the following in its linux-config kfrag:

CONFIG_FUNCTION_TRACER=y 
CONFIG_FUNCTION_GRAPH_TRACER=y 
# CONFIG_PREEMPTIRQ_EVENTS is not set 
# CONFIG_IRQSOFF_TRACER is not set 
# CONFIG_SCHED_TRACER is not set 
# CONFIG_HWLAT_TRACER is not set 
CONFIG_FTRACE_SYSCALLS=y 
# CONFIG_TRACER_SNAPSHOT is not set 
CONFIG_BRANCH_PROFILE_NONE=y 
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set 
# CONFIG_PROFILE_ALL_BRANCHES is not set 
CONFIG_STACK_TRACER=y 
# CONFIG_BLK_DEV_IO_TRACE is not set 
CONFIG_UPROBE_EVENTS=y 
CONFIG_DYNAMIC_EVENTS=y 
CONFIG_PROBE_EVENTS=y 
CONFIG_DYNAMIC_FTRACE=y 
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y

Attempt to build the workload once to generate the kconfig:
$ ./marshal build test-workload.json

It will fail. To get a clearer error, try to build a driver directly:

$ cd boards/firechip/drivers/iceblk-driver
$ make LINUXSRC=../../../../riscv-linux/

make -C ../../../../riscv-linux/ ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- M=/home/centos/FireMarshal/boards/firechip/drivers/iceblk-driver
make[1]: Entering directory '/home/centos/FireMarshal/riscv-linux'
  Building modules, stage 2.
  MODPOST 1 modules
ERROR: "kmem_cache_alloc_trace" [/home/centos/FireMarshal/boards/firechip/drivers/iceblk-driver/iceblk.ko] undefined!
ERROR: "_mcount" [/home/centos/FireMarshal/boards/firechip/drivers/iceblk-driver/iceblk.ko] undefined!
make[2]: *** [scripts/Makefile.modpost:103: modules-modpost] Error 1
make[1]: *** [Makefile:1628: modules] Error 2
make[1]: Leaving directory '/home/centos/FireMarshal/riscv-linux'
make: *** [Makefile:13: iceblk.ko] Error 2

Switch buildroot submodule to their github mirror

I've found the remote we're using currently to be unreliable and we're already depending on github heavily anyway.

Changes to board drivers not detected

Marshal does not seem to register the board drivers as dependencies for workloads, so updates to the driver don't trigger a rebuild.

Optionally re-build from changed dependencies

In an abundance of caution, marshal currently does not inherit changes from dependencies if you don't clean first. This is to avoid wiping out ephemeral data on your rootfs accidentally. It's not a big deal to just "marshal clean work.json && marshal build work.json" if you want the new dependencies, but a "marshal build --force" might be more clear. This would re-inherit dependencies all the way down the chain if needed, but will remove any ephemeral data you had laying around in your rootfs.

FireMarshal does not support being run out of a NFS-mounted directory

Default runs of FireMarshal in a NFS-mounted directory will result in

Running: "guestmount --pid-file guestmount.pid -a /nscratch/jerryz/chipyard-proj/chipyard4/software/firemarshal/images/br-base.img -m /dev/sda /nscratch/jerryz/chipyard-proj/chipyard4/software/firemarshal/disk-mount" in /nscratch/jerryz/chipyard-proj/chipyard4/software/firemarshal
fusermount: failed to open mountpoint for reading: Permission denied
libguestfs: error: fuse_mount: /nscratch/jerryz/chipyard-proj/chipyard4/software/firemarshal/disk-mount: Operation not permitted
TaskError - taskid:/nscratch/jerryz/chipyard-proj/chipyard4/software/firemarshal/images/br-base.img

I'm not familiar with the details here, but there seems to be some issue with FUSE and NFS.

The workaround I am using is to set mnt-dir in wlutil.py to a local directory.

Kernel configuration settings of child workloads don't stack on parent kernel configuration

To reproduce, create a child workload of br-child.json with the following contents

{
  "name" : "br-child",
  "base" : " br-base.json",
  "linux-config" : "br-child-kfrag"
}

Create the br-child-kfrag file as well, but leave it blank. Build it with marshal and check riscv-linux/.config afterwards. You'll notice that things set in br-base/linux-config like CONFIG_CMDLINE are not set.

Workdir arg error

centos@ip-192-168-0-102 firesim-software]$ ./marshal -v --workdir /home/centos/some-dir build some-workload.json
Traceback (most recent call last):
  File "./marshal", line 201, in <module>
    main()
  File "./marshal", line 98, in main
    cfgs = wlutil.ConfigManager(workdirs.keys())
  File "/home/centos/chipyard-ee290/sims/firesim/sw/firesim-software/wlutil/config.py", line 356, in __init__
    for cfgFile in d.glob('*.json'):
AttributeError: 'str' object has no attribute 'glob'

Potential solution:

diff --git a/wlutil/config.py b/wlutil/config.py
index 21ea519..9b44678 100644
--- a/wlutil/config.py
+++ b/wlutil/config.py
@@ -353,7 +353,7 @@ class ConfigManager(collections.MutableMapping):
         if dirs != None:
             for d in dirs: 
-                for cfgFile in d.glob('*.json'):
+                for cfgFile in pathlib.Path(d).glob('*.json'):
                     cfgPaths.append(cfgFile)
         # First, load the base-configs specially. Note that these are indexed

doit.db location

Doit uses an internal database for tracking changes and dependencies. By default this goes to PWD. This is potentially a problem when you start calling marshal from different places because it leaves all these .doit.db files everywhere. We should probably figure out how to put them in specific places.

guestmount breaking Fedora

On some setups (Ubuntu 18.04 for example), Fedora is not booting correctly. Specifically, some systemd services (including login) fail to startup. The problem seems to come from guestmount, although the exact mechanism is not clear. On Centos 7 (similar to the FireSim ec2 setup), guestmount works properly. Systemd uses symlinks all over so that may be part of the problem.

In general, guestmount has been a source of constant problems (see #137 for example). It also prevents easy CI integration (can't run any FUSE in a container apparently). We really need a more robust solution to getting data on/off images. A solution needs the following features:

Copy files and directories out of the image and into a fixed location on the host
Copy files and directories to the guest either to a specific location or as an overlay
Package the entire image into a CPIO archive (for the nodisk builds)

PK not re-configured on toolchain change

Marshal tries to rebuild things when the toolchain changes, but it doesn't always reconfigure them. This is at least a problem for PK (and maybe other stuff?). We should wipe out the build/ directory in PK on a toolchain change.

@zhemao

iceblk-driver and icenet-driver may need to be updated

I'm currently working on the running some customized risc-v linux kernel on FireSim. I realized that there's a structural changes of the folders and some previous code should be updated accordingly.
Currently, in the Makefile's of the iceblk-driver and icenet-driver under firesim/sw/firesim-software/boards/firechip/drivers/, it says:
"LINUXSRC=../../../../riscv-linux"
However, it should be
"LINUXSRC= ../../linux "
in order to make the program compile inside its own folder.

Note: if I use the FireMarshal flow, it provides updated path and everything works well. This is just a trivial reminder to keep codes more consistent.

RISCV tools missing error

I have riscv tools installed and is added to path. But when i run marshal i get an error complaining riscv tools missing.

$./marshal
No riscv toolchain detected. Please install riscv-tools.

Generated Overlays

If the contents of an overlay are generated by a host init, they will not be detected. This is because Marshal determines the list of files at the time of config loading (early on). Since overlays are globbed, we won't see the files yet. This is not an issue for 'files' since they are explicitly listed and marshal won't even look for them until after host-init.

A simple-ish fix is to do the glob in the image build task (that runs after host-init), but this would not add the generated files to the list of dependencies (marshal would not rebuild if they change). While this probably isn't an issue most of the time (they were generated anyway) it could lead to some very confusing issues if you do modify them.

A more robust solution would be to add the dependencies after host-init, but this could be challenging in doit (I'm not even sure if it's possible to dynamically edit the dependency tree during execution).

[dev] Bad workdir relative path for symlink jsons

When running ./marshal build example-workloads/sha3-linux.json in dev I get the following error:

ValueError: host-init script /home/centos/general/firemarshal/example-workloads/sha3/marshal-configs/sha3/build.sh not found.

sha3-linux.json is a symlink to a json in marshal-configs within the workdir which seems to indicate that the workdir is being specified from the .json location not where the symlink is.

multiple workloads per invocation

Building multiple workloads on the same invocation seems to have stopped working. E.g.:

(fs) [xarc@xarc0 firemarshal]$ marshal --no-disk build test/command.json test/bbl.json
To check on progress, either call marshal with '-v' or see the live output at:
/data/repos/cy2/software/firemarshal/logs/command-build-2020-07-20--20-43-57-HLVWAMRS79E9J0MX.log
. /data/repos/cy2/software/firemarshal/boards/firechip/base-workloads/br-base/host-init.sh
Applying host-init: /data/repos/cy2/software/firemarshal/boards/firechip/base-workloads/br-base/host-init.sh
-- BuildBusybox
-- /data/repos/cy2/software/firemarshal/wlutil/br/buildroot/output/images/rootfs.ext2
. calc_br-base_dep
-- /data/repos/cy2/software/firemarshal/images/br-base.img
. calc_command_dep
-- /data/repos/cy2/software/firemarshal/images/command.img
-- /data/repos/cy2/software/firemarshal/images/command-bin-nodisk
To check on progress, either call marshal with '-v' or see the live output at:
/data/repos/cy2/software/firemarshal/logs/bbl-build-2020-07-20--20-43-59-U1Q0Z3CQ2J3W2RQL.log
ERROR: Invalid parameter: "/data/repos/cy2/software/firemarshal/images/bbl-bin-nodisk". Must be a command, task, or a target.
Type "marshal help" to see available commands.
Type "marshal list" to see available tasks.

Failed to build workload bbl.json
Log available at: /data/repos/cy2/software/firemarshal/logs/bbl-build-2020-07-20--20-43-59-U1Q0Z3CQ2J3W2RQL.log

This used to work and still should.

Metadata not used for uptodate

For file dependencies (e.g. overlay), marshal doesn't detect metadata-only changes. For example, changing the executable bit on a script will not cause the image to be rebuilt.

workdir search paths

The search paths for bases are somewhat messy and static right now. There is the --workdir option (which has a confusing name btw) and some builtin defaults (e.g. boards/firechip/default-workloads), but these are disparate mechanisms. Now that we have a proper configuration system, we should add a FIREMARSHAL_PATH option (or whatever we call it) that's like PATH in your shell but for workloads. If you set it, you can simply run 'marshal build workload.json' and if workload.json is in any of those paths, it will build. We could still override/append with --workdir or config files.

simArgs test determinism

The simArgs test started failing because the amount of memory reported by the simulator changed from 3958 to 3956 (the workload requests 4GiB). I'm not sure exactly where that discrepancy comes from (spike and qemu always differed by a few KB).

Ideally, we'd check to make sure memory settings work, but we might have to stick to just checking CPU count.

This is a problem with the test, not the feature. Simargs work correctly.

More Robust Filesystem Manipulation

Mounting target images is fraught. It's the source of many bugs (e.g. #168 #137). It also prevents CI for marshal (since Docker doesn't support mount or FUSE). We should really deal with these issues at the root rather than patching each issue with hacks.

I think the most promising method is to use e2fsprogs to manually copy files in and out of the filesystem. It would require some changes to the build process (that uses mounts to simplify data transfers), but it could be done.

FireMarshal br-build crashes

Trying to run ./marshal -v -d build br-base.json still.
I'm not sure what the issue is, but I'm guessing it has something to do with RAM since the errors seem resource and fork related. Is 16 gigs insufficient for this? Is there any workaround for this?

2021-08-24 17:09:47,914 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'fs/nfs/nfs4namespace.o' failed
2021-08-24 17:09:47,914 [run         ] [DEBUG]  make[2]: *** [fs/nfs/nfs4namespace.o] Error 1
2021-08-24 17:09:47,914 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'drivers/gpu/drm/radeon/rs690.o' failed
2021-08-24 17:09:47,914 [run         ] [DEBUG]  make[4]: *** [drivers/gpu/drm/radeon/rs690.o] Error 1
2021-08-24 17:09:47,914 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'drivers/gpu/drm/radeon/radeon_legacy_tv.o' failed
2021-08-24 17:09:47,914 [run         ] [DEBUG]  make[4]: *** [drivers/gpu/drm/radeon/radeon_legacy_tv.o] Error 1
2021-08-24 17:09:47,914 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'lib/earlycpio.o' failed
2021-08-24 17:09:47,914 [run         ] [DEBUG]  make[1]: *** [lib/earlycpio.o] Error 1
2021-08-24 17:09:47,914 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:47,914 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'fs/nfs/nfs4namespace.o' failed
2021-08-24 17:09:47,914 [run         ] [DEBUG]  make[2]: *** [fs/nfs/nfs4namespace.o] Error 1
2021-08-24 17:09:47,914 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'drivers/gpu/drm/radeon/rs690.o' failed
2021-08-24 17:09:47,914 [run         ] [DEBUG]  make[4]: *** [drivers/gpu/drm/radeon/rs690.o] Error 1
2021-08-24 17:09:47,914 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'drivers/gpu/drm/radeon/radeon_legacy_tv.o' failed
2021-08-24 17:09:47,914 [run         ] [DEBUG]  make[4]: *** [drivers/gpu/drm/radeon/radeon_legacy_tv.o] Error 1
2021-08-24 17:09:47,914 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'lib/earlycpio.o' failed
2021-08-24 17:09:47,914 [run         ] [DEBUG]  make[1]: *** [lib/earlycpio.o] Error 1
2021-08-24 17:09:47,914 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:47,914 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'fs/nfs/nfs4namespace.o' failed
2021-08-24 17:09:47,914 [run         ] [DEBUG]  make[2]: *** [fs/nfs/nfs4namespace.o] Error 1
2021-08-24 17:09:47,914 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'drivers/gpu/drm/radeon/rs690.o' failed
2021-08-24 17:09:47,914 [run         ] [DEBUG]  make[4]: *** [drivers/gpu/drm/radeon/rs690.o] Error 1
2021-08-24 17:09:47,914 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'drivers/gpu/drm/radeon/radeon_legacy_tv.o' failed
2021-08-24 17:09:47,914 [run         ] [DEBUG]  make[4]: *** [drivers/gpu/drm/radeon/radeon_legacy_tv.o] Error 1
2021-08-24 17:09:47,914 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'lib/earlycpio.o' failed
2021-08-24 17:09:47,914 [run         ] [DEBUG]  make[1]: *** [lib/earlycpio.o] Error 1
2021-08-24 17:09:47,914 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:18,784 [run         ] [DEBUG]  /bin/sh: fork: retry: No child processes
2021-08-24 17:09:18,784 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:18,785 [run         ] [DEBUG]  compilation terminated.
2021-08-24 17:09:18,785 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:18,785 [run         ] [DEBUG]  /bin/sh: fork: retry: No child processes
2021-08-24 17:09:18,785 [run         ] [DEBUG]  /bin/sh: fork: retry: No child processes
2021-08-24 17:09:18,785 [run         ] [DEBUG]  compilation terminated.
2021-08-24 17:09:18,785 [run         ] [DEBUG]  /bin/sh: fork: retry: No child processes
2021-08-24 17:09:18,785 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:18,786 [run         ] [DEBUG]  compilation terminated.
2021-08-24 17:09:18,786 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:18,786 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:18,786 [run         ] [DEBUG]  /bin/sh: fork: retry: No child processes
2021-08-24 17:09:18,786 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:18,786 [run         ] [DEBUG]  CC      lib/irq_regs.o
2021-08-24 17:09:18,787 [run         ] [DEBUG]  /bin/sh: fork: retry: No child processes
2021-08-24 17:09:18,787 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:18,787 [run         ] [DEBUG]  compilation terminated.
2021-08-24 17:09:18,787 [run         ] [DEBUG]  /bin/sh: fork: Resource temporarily unavailable
2021-08-24 17:09:18,787 [run         ] [DEBUG]  CC      drivers/gpu/drm/radeon/evergreen_blit_shaders.o
2021-08-24 17:09:18,787 [run         ] [DEBUG]  /bin/sh: fork: retry: No child processes
2021-08-24 17:09:18,788 [run         ] [DEBUG]  compilation terminated.
2021-08-24 17:09:18,788 [run         ] [DEBUG]  /bin/sh: fork: retry: No child processes
2021-08-24 17:09:18,796 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:18,796 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:18,796 [run         ] [DEBUG]  riscv64-unknown-linux-gnu-gcc: fatal error: cannot execute '/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install/libexec/gcc/riscv64-unknown-linux-gnu/9.2.0/cc1': vfork: Resource temporarily unavailable
2021-08-24 17:09:18,796 [run         ] [DEBUG]  compilation terminated.
2021-08-24 17:09:18,796 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'drivers/gpu/drm/radeon/r600.o' failed
2021-08-24 17:09:18,796 [run         ] [DEBUG]  make[4]: *** [drivers/gpu/drm/radeon/r600.o] Error 1
2021-08-24 17:09:18,796 [run         ] [DEBUG]  scripts/Makefile.build:266: recipe for target 'lib/extable.o' failed
2021-08-24 17:09:18,797 [run         ] [DEBUG]  make[1]: *** [lib/extable.o] Error 1
2021-08-24 17:09:18,797 [run         ] [DEBUG]  make[1]: *** Waiting for unfinished jobs....

changing buildroot config not propagating?

It seems that if I change wlutil/br/buildroot-config to add a package and re-run ./marshal -v build workloads/br-base.json, the rootfs is not rebuilt

Chipyard FireMarshal br-base workload on CentOS7 failing

All dependencies installed. Following chipyard's recommendations for building the workload.
Ran
./init-submodules.sh
Ran python and centos installations.
Added board-dir : 'boards/prototype' to my marshal-config.yaml
Ran ./marshal -v -d build br-base.json # here the -d indicates --nodisk or initramfs which failed with this output:

DEBUG: You seem to have the current working directory in your
DEBUG: LD_LIBRARY_PATH environment variable. This doesn't work.
DEBUG: support/dependencies/dependencies.mk:27: recipe for target 'dependencies' failed
DEBUG: make[1]: *** [dependencies] Error 1
DEBUG: Makefile:84: recipe for target '_all' failed
DEBUG: make: *** [_all] Error 2
TaskError - taskid:/mnt/Vivado_part/chipyard/chipyard/software/firemarshal/images/br.bc88.img
PythonAction Error
Traceback (most recent call last):
  File "/home/_eda/.local/lib/python3.6/site-packages/doit/action.py", line 437, in execute
    returned_value = self.py_callable(*self.args, **kwargs)
  File "/mnt/Vivado_part/chipyard/chipyard/software/firemarshal/boards/prototype/distros/br/br.py", line 190, in buildBaseImage
    wlutil.run(['make'], cwd=br_dir / "buildroot", env=env)
  File "/mnt/Vivado_part/chipyard/chipyard/software/firemarshal/wlutil/wlutil.py", line 498, in run
    raise sp.CalledProcessError(p.returncode, prettyCmd)
subprocess.CalledProcessError: Command 'make' returned non-zero exit status 2.

I've sourced env.sh from Chipyard which sources env-riscv-tools.sh. This seems to be what sets LD_LIBRARY_PATH. Not sure why this wouldn't work.

export CHIPYARD_TOOLCHAIN_SOURCED=1
export RISCV=/mnt/Vivado_part/chipyard/chipyard/riscv-tools-install
export PATH=${RISCV}/bin:${PATH}
export LD_LIBRARY_PATH=${RISCV}/lib${LD_LIBRARY_PATH:+":${LD_LIBRARY_PATH}"}

Does br-base for Chipyard prototype wait for mouse input after `random: crng init done`?

Kind of cross-posting my issue from ucb-bar/chipyard#957, but since I'm still figuring out how to put files into the hfs part of the partition, I figured I'd check someone saying that "At boot, the kernel waits for mouse movements to initialize the random number generator." Maybe that's why after random: crng init done is printed the uart output hangs?

No module named 'doit'

When I tried to run marshal to build sha3 Bare-Metal test with:
./marshal build example-workloads/sha3/marshal-configs/sha3-bare-rocc.json

Errors occurred like below, and I could not find out any dependencies related to 'doit' to install.
Traceback (most recent call last): File "./marshal", line 6, in <module> import wlutil File "/home/frederick/workspace/chipyard/sims/firesim/sw/firesim-software/wlutil/__init__.py", line 6, in <module> from .build import buildWorkload File "/home/frederick/workspace/chipyard/sims/firesim/sw/firesim-software/wlutil/build.py", line 1, in <module> import doit ModuleNotFoundError: No module named 'doit'

Thanks in advance!

qemu-system-riscv64: cannot set up guest memory 'riscv_virt_board.ram': Cannot allocate memory

Hello,
I am beginner here and was following the documentation to boot Linux kernal using the buildroot-based linux distribution provided in the repo. I could build the workload cleanly but I am facing the below issue when I try to run it in qemu.
Command used to run: ./marshal -v launch workloads/br-base.json

Running: qemu-system-riscv64 -nographic -bios none -smp 4 -machine virt -m 16384 -kernel //chipyard/sims/firesim/sw/firesim-software/images/br-base-bin -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-device,rng=rng0 -device virtio-net-device,netdev=usernet -netdev user,id=usernet,hostfwd=tcp::48179-:22 -device virtio-blk-device,drive=hd0 -drive file=//chipyard/sims/firesim/sw/firesim-software/images/br-base.img,format=raw,id=hd0
qemu-system-riscv64: cannot set up guest memory 'riscv_virt_board.ram': Cannot allocate memory

Can anyone help me with this? Thanks!

Adding default GW breaks firesim-cluster-manager

For whatever reason #4 seems to break all tasks the cluster manager attempts to run on FPGAs as hosts.

Under the hood paramiko complains: SSHException: Error reading SSH protocol banner

But i can still SSH into them manually.

building unneeded binaries

workloads that don't have a guest-init don't need to build a binary (especially if they are a parent of something else). We can remove the binary dependency for rootfs's for workloads that don't have a guest-init.

Can't generators images

Type the command ./marshal -d build ./boards/firechip/base-workloads/br-base.json
Return ERROR: Failed to build workload br-base.json

Open ./boards/default/distros/br/buildroot/output/build/ncurses-6.1/config.log

After executing the command with the change soft link, it is still the same.

Are these warnings of concern for building br-base?

WARNING: Unrecognized Option: version
WARNING: Unrecognized Option: sphinx
WARNING: Skipping /mnt/Vivado_part/chipyard/chipyard/software/firemarshal/.readthedocs.yaml:
WARNING: 	Missing required option 'name'
DEBUG: Loading /mnt/Vivado_part/chipyard/chipyard/software/firemarshal/marshal-config.yaml
WARNING: Unrecognized Option: board-dir
WARNING: Unrecognized Option: firesim-dir
WARNING: Skipping /mnt/Vivado_part/chipyard/chipyard/software/firemarshal/marshal-config.yaml:
WARNING: 	Missing required option 'name'

I'm making this a new issue since we identified where the makefile was hanging before.

I'm running ./marshal -v -d build br-base.json # here the -d indicates --nodisk or initramfs after setting up FireMarshal and getting these warnings at the start of the build.

[dev] install.py needs to use ctx['key']

For the root_dir and wlutil_dir values, these need to be ctx['root-dir'] and ctx['wlutil-dir'] respectively. Otherwise, install will error since root_dir does not exist.

host-init not running for dependencies

If a workload depends on some base that requires a host-init script, the script may not always run. This seems to be an issue for workloads that depend on a bare-metal workload where the binary is generated in the host-init script (e.g. running spike-jobs.json before spike.json won't generate the spike.json binary).

Support qcow2

FireSim now supports qcow2. It would be good to add this to FireMarshal.

jobs don't work on firesim

jobs aren't loading drivers for our peripherals

Fedora Repos Down

The default fedora repos have gone down. They are also not super secure (non https, can't access through normal browser, etc.). 'yum install' is currently not working.

firemarshal build linux image for the FPGA prototype failed

when execute
~/chipyard/software/firemarshal$ ./marshal -v -d build br-base.json
i got error

DEBUG: >>> gdb 10.1 Configuring
DEBUG: (cd /home/zsf/chipyard/software/firemarshal/boards/default/distros/br/buildroot/output/build/gdb-10.1/gdb/gdbserver && rm -rf config.cache && ...
DEBUG: bin/bash: line 0: cd: /home/zsf/chipyard/software/firemarshal/boards/default/distros/br/buildroot/output/build/gdb-10.1/gdb/gdbserver: No such file or directory
DEBUG: package/pkg-generic.mk:249: recipe for target '/home/zsf/chipyard/software/firemarshal/boards/default/distros/br/buildroot/output/build/gdb-10.1/.stamp_configured' failed

i thought because the GDB 10.1 has move the gdbserver at the top-level, so i edit the
~/chipayrd/software/firemarshal/boards/prototype/distros/br/buildroot/package/gdb/gdb.mk
change "GDB_SUBDIR = gdb/gdbserver to GDB_SUBDIR = gdbserver"


GDB_VERSION = $(call qstrip,$(BR2_GDB_VERSION))
GDB_SITE = $(BR2_GNU_MIRROR)/gdb
GDB_SOURCE = gdb-$(GDB_VERSION).tar.xz

# recent gdb versions (>= 10) have gdbserver moved at the top-level,
# which requires a different build logic.
ifeq ($(BR2_GDB_VERSION_10),y)
GDB_GDBSERVER_TOPLEVEL = y
endif

ifeq ($(BR2_arc),y)
GDB_SITE = $(call github,foss-for-synopsys-dwc-arc-processors,binutils-gdb,$(GDB_VERSION))
GDB_SOURCE = gdb-$(GDB_VERSION).tar.gz
GDB_FROM_GIT = y
# recent gdb versions (>= 10) have gdbserver moved at the top-level,
# which requires a different build logic.
GDB_GDBSERVER_TOPLEVEL = y
endif

ifeq ($(BR2_csky),y)
GDB_SITE = $(call github,c-sky,binutils-gdb,$(GDB_VERSION))
GDB_SOURCE = gdb-$(GDB_VERSION).tar.gz
GDB_FROM_GIT = y
endif

GDB_LICENSE = GPL-2.0+, LGPL-2.0+, GPL-3.0+, LGPL-3.0+
GDB_LICENSE_FILES = COPYING COPYING.LIB COPYING3 COPYING3.LIB
GDB_CPE_ID_VENDOR = gnu

# On gdb < 10, if you want to build only gdbserver, you need to
# configure only gdb/gdbserver.
ifeq ($(BR2_PACKAGE_GDB_DEBUGGER)$(GDB_GDBSERVER_TOPLEVEL),)
GDB_SUBDIR = gdb/gdbserver # change to gdbserver

after execute
~/chipyard/software/firemarshal$ ./marshal -v -d build br-base.json
i still got

EBUG: configure: WARNING: unrecognized options: --disable-gtk-doc, --disable-gtk-doc-html, --disable-doc, --disable-docs, --disable-documentation, --with-xmlto, --with-fop, --disable-dependency-tracking, --enable-ipv6, --disable-nls, --disable-static, --enable-shared, --without-uiout, --disable-gdbtk, --without-x, --disable-sim, --disable-binutils, --disable-install-libbfd, --disable-ld, --disable-gas, --disable-gprof, --without-included-gettext, --enable-static, --without-mpfr, --disable-gdb, --without-curses, --enable-gdbserver, --disable-tui, --without-python, --without-expat, --without-lzma, --without-zlib
DEBUG: >>> gdb 10.1 Building
DEBUG: PATH="/home/zsf/chipyard/software/firemarshal/boards/default/distros/br/buildroot/output/host/bin:/home/zsf/chipyard/software/firemarshal/boards/default/distros/br/buildroot/output/host/sbin:/home/zsf/chipyard/riscv-tools-install/bin:/home/zsf/chipyard/software/firemarshal:/home/zsf/chipyard/bin:/home/zsf/chipyard/software/firemarshal:/home/zsf/chipyard/bin:/home/zsf/chipyard/software/firemarshal:/home/zsf/chipyard/bin:/home/zsf/bin:/home/zsf/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cmake/bin" gl_cv_func_gettimeofday_clobber=no gl_cv_func_working_strerror=yes gl_cv_func_strerror_0_works=yes gdb_cv_prfpregset_t_broken=no /usr/bin/make -j2 MAKEINFO=true -C /home/zsf/chipyard/software/firemarshal/boards/default/distros/br/buildroot/output/build/gdb-10.1/gdbserver
DEBUG: make[2]: *** No rule to make target '../gnulib/import/libgnu.a', needed by 'gdbserver'.  Stop.

How to fix this?

full_test.sh broken

A few things are wrong with the full_test.sh script:

test/spike.json is failing because a previous commit removed part of it accidently
failures are not being reported correctly because we pipe test results to 'tee' but check the exit status (it's checking tee's exit status instead of marshal's)
the 'clean' command can remove the symlinks for bare-metal tests

Need sudo for build?

Is there any way to build without sudo?

outputs and clean tests failing due to pathlib

Posting here for posterity:

The outputs feature is failing in marshal on newer versions of firesim. Specifically, newer versions of python3 (marshal was tested on 3.49). The issues has to do with pathlib interoperability. Check back soon for a fix.

Error message
TypeError: all arguments must be str, not: PosixPath('/home/centos/firesim/sw/firesim-software/test/outputs/refOutput/outputs/runOutput')

Difference in runs between `guest-init` on BR vs Fedora

Figured I would note this here even though I don't have a working solution for this.

Seems like the functionality of guest-init in Fedora vs BR is different. For example, take the following marshal config:

https://github.com/abejgonzalez/tpch-monetdb-workload/blob/br-base/marshal-configs/monetdb-br-base-full-auto.yaml

Within the guest-init script, it is supposed to load a DB (spawning a DB server, load it, then shutdown). However, it is unable to spawn the DB server in the background if I switch the base to Fedora. This isn't an issue in buildroot.

Rootfs resizing

If you specify a lot of data in an overlay, you can run out of filesystem space in the rootfs. Current images are also much bigger than needed to make this less likely.

Feature request: Dynamically resize the rootfs based on the workload, leave a more modest margin for manual data. This will need an option in the workload config to increase that margin (e.g. if tests produce lots of data).

subprocess.CalledProcessError: Command 'make -j2' returned non-zero exit status 2

When I tried to run marshal to build sha3 Bare-Metal test with:
./marshal build example-workloads/sha3/marshal-configs/sha3-bare-rocc.json

Errors occurred as below:
Traceback (most recent call last): File "./marshal", line 170, in <module> main() File "./marshal", line 67, in main wlutil.initialize() File "/home/frederick/workspace/chipyard/sims/firesim/sw/firesim-software/wlutil/wlutil.py", line 85, in initialize run(['make', jlevel], cwd=busybox_dir) File "/home/frederick/workspace/chipyard/sims/firesim/sw/firesim-software/wlutil/wlutil.py", line 180, in run raise sp.CalledProcessError(p.returncode, prettyCmd) subprocess.CalledProcessError: Command 'make -j2' returned non-zero exit status 2.

Selective Workload Loading

Working with a workload in a directory with lots of other workloads is much slower than it should be. This is because we're doing more build dependency checking than we really have to:

Every workload is checked for uptodate, even if it's not involved in the current command at all. This didn't used to be a big time-sync, but we now include git version checking which is significantly slower.
There is no dedup of dependency checking. In particular, almost every workload uses the built in Linux, but we check Linux for every unique workload.
We load every workload in the workload search paths every time marshal runs.

There were practical reasons for this:

it wasn't clear how to get doit to defer checking the git versions until runtime. It took some doing to get it to check at all.
Dedup would be tedious and rely on some shared state that might be confusing and lead to problems down the road.
Workloads depend on a recursive chain of 'base' options, it can be tricky to walk that dynamically while loading. It's much easier to just load everything and let doit sort it out.

The load times are the primary concern, but there are other issues as well. For example, bugs in unrelated workloads will cause a job to fail.

The solution is probably to rework the configuration loading scheme to manually trace the chain of 'base' arguments and only load workloads that actually matter. This will require a refactoring of the config loading system which might be fairly involved.

If we still want faster loads after that, we can work on deduping the dependency checking (probably by making each config_changed style criteria to its own task and performing the dedup at that point.

Corrupt file system when using OpenSBI with non-default initramfs

I'm summarizing the bug investigation and proposal solution by @a0u :

By default, FW_PAYLOAD_FDT_ADDR in OpenSBI is hardcoded to 0x82200000.
(https://github.com/riscv/opensbi/blob/v0.8/platform/generic/config.mk#L31).

This must not overlap with the kernel image, otherwise the data section will be overwritten with a copy of the FDT that OpenSBI passes to the kernel.
Therefore, when we have an initramfs that's bigger than ~35 MB, there is a chance for file system corruption.
This happens quite often with a non-default initramfs.

Temporary hacky fixes include just increasing that offset to something like 0x20000000 (i.e, 512 MiB) to give the payload a more generous margin.
The better solution is to have FireMarshal override FW_PAYLOAD_FDT_ADDR based on the payload size.

Firesim not exiting on buildroot 'poweroff'

FireSim is not exiting the simulation when buildroot calls 'poweroff'. It is necessary to call 'poweroff -f'. This is almost certainly an issue with buildroot. See #58 for a temporary workaround.

no-disk builds failing over a certain size

--no-disk builds over about 128mb have begun failing. This used to work so it's a mystery as to why it's happening again.

Issue originally reported here: ucb-bar/chipyard#950

Reproduction instructions:
use a workload like tests/overlay.yaml and dd a large file (>100MB) into the overlay. Building without no-disk should work, with --no-disk will hang early in the boot process. See the original bug report on chipyard for details.

guestmount failing on ubuntu

Marshal may fail with a somewhat cryptic error on ubuntu systems. This is a known issue and it's being documented here for posterity:

Reproduce:
./marshal build test/command.json

Symptom (e.g.):

subprocess.CalledProcessError: Command 'guestmount -a /scratch/nathanp/firesim/sw/firesim-software/images/br-base.img -m /dev/sda /scratch/nathanp/firesim/sw/firesim-software/disk-mount' returned non-zero exit status 1

In the detailed logs you'll see:

libguestfs: error: /usr/bin/supermin exited with error status 1.

This is an issue with guestmount and ubuntu (marshal uses guestmount to manipulate target filesystems without requiring sudo). The bugs are documented here and here. The solution is to add read privileges to the kernel (e.g. sudo chmod +r /boot/vmlinuz-*).

Warn on undefined option

Currently, if you have a typo in a config, marshal won't notice. It just ignores undefined options in workload configs. This is obviously not ideal since errors can go unnoticed. It's an easy fix.

Add deep clean option

"marshal clean" currently only cleans the top-level workload. It does not affect any dependencies. So if you have a deep chain of workloads that are based off eachother, you have to manually clean all of them (or just go nuclear and "rm images/*").

It would be nice if 'clean' had a '--deep' option that cleaned dependencies all the way down.