warewulf / warewulf Goto Github PK

Warewulf is a stateless and diskless container operating system provisioning system for large clusters of bare metal and/or virtual systems.

Home Page: https://warewulf.org

License: Other

Makefile 2.05% Go 93.23% Shell 3.90% Python 0.22% Dockerfile 0.60%

warewulf containers hpc clusters provisioning stateless

warewulf's Introduction

warewulf v4

Quick Links:

About Warewulf

History

For over two decades, Warewulf has powered HPC systems around the world. From simple “under the desk” clusters to large institutional systems at HPC centers as well as enterprises who rely on performance critical computing.

Through the evolution of Warewulf, we have seen various iterations provisioning models starting from CDROM / ISO images to Etherboot (predecessor to PXE), then PXE, and more recently iPXE, but even during these different bootloaders, Warewulf in it’s heart, has always been first and foremost a stateless provisioning system (e.g. the operating system node image is not written to any persistent storage and rather it boots from the network directly into a runtime system).

Warewulf v3 has been in production for over 6 years now as it has stabilized into a very solid and full featured solution. But over the last few years, there have been many innovations in Enterprise technologies which can (and should) be leveraged as part of Warewulf. Additionally, some of the lessons learned from Warewulf v3 architecture should be rolled into an updated architecture for provisioning management.

Warewulf v4

Leveraging this legacy of provisioning and cluster management brings us to where we are today. The next generation of Warewulf. Warewulf v4 is a complete rewrite in GoLang, taking in the legacy of what we've come to expect with Warewulf, bringing it into the present, and looking out into the future.

Warewulf v4 combines ultra scalability, flexibility, and simplicity with being light weight, non-intrusive, and a great tool for scientists and seasoned system administrators alike. Warewulf empowers you to scalably and easily manage thousands of compute resources.

Architecture

One of the design tenants of Warewulf is how to scalably administrate many thousands of compute nodes. Generally speaking, operating system state introduce a surface for potential discrepancies and version creep between nodes and thus Warewulf has always gone with the "single system image" approach to clustered operating system management. This means that you can have a single management node "image". For the last 20 years, we've been calling this a Virtual Node File System (VNFS), but in hindsight, this is a container.

At its core, Warewulf v4 focuses on what has made Warewulf so widely loved: simplicity, ultra scalable, lightweight, and an easy to manage solution built for both scientists and seasons system administrators to be able to design a highly functional yet easy to maintain cluster no matter how big or small or customized it needs to be.

iPXE

Warewulf uses iPXE for network boot. Typically iPXE is provided by the operating system; but the iPXE binaries can be built with scripts/build-ipxe.sh. This script accepts command-line arguments that are passed to the underlying make process. e.g.,

echo "#!ipxe
echo Tagging with vlan 1000
vcreate --tag 1000 net0 autoboot || shell" >vlan-1000.ipxe

sh scripts/build-ipxe.sh EMBED=$(readlink -f vlan-1000.ipxe)

By default, build-ipxe.sh will attempt to write iPXE builds to /usr/local/share/ipxe/. This path can be specified using the DESTDIR environment variable. Other supported environment variables include IPXE_BRANCH and TARGETS.

IPXE_BRANCH=master TARGETS=bin-arm64-efi/snponly.efi DESTDIR=. sh scripts/build-ipxe.sh

Update warewulf.conf to use locally-built iPXE.

tftp:
  enabled: true
  systemd name: tftp
  ipxe:
    00:09: /usr/local/share/ipxe/bin-x86_64-efi-snponly.efi
    00:00: /usr/local/share/ipxe/bin-x86_64-pcbios-undionly.kpxe
    00:0B: /usr/local/share/ipxe/bin-arm64-efi-snponly.efi
    00:07: /usr/local/share/ipxe/bin-x86_64-efi-snponly.efi

warewulf's People

Contributors

Stargazers

Watchers

warewulf's Issues

What's the role of docker

So one difference from the previous version that I noticed is that at least the vnfs are downloaded from docker. My question is how reliant is the new warewulf infrastructure on docker? Can we ignore it all together on the nodes? Is it now recommended that we install the applications on the nodes via docker? Is it just the VNFS that are provided by docker or are other services as well?

wwclient update on HUP

wwclient should properly catch the HUP signal and pull/update files on that signal.

--kernelargs or -A sets the kernel, not the args

Ouch! Trying to set the kernel arguments sets the kernel image.

wwctl profile set default --kernelargs="noquiet init=/init crashkernel=no rootfstype=tmpfs"

# wwctl profile list default
################################################################################
PROFILE NAME         FIELD              VALUE
default              Id                 default
default              Comment            This profile is automatically included for each node
default              ClusterName        --
default              ContainerName      centos
default              KernelVersion      noquiet init=/init crashkernel=no rootfstype=tmpfs
default              KernelArgs         --
default              Init               --
default              Root               --
default              RuntimeOverlay     --
default              SystemOverlay      --
default              Ipxe               --
default              IpmiIpaddr         --
default              IpmiNetmask        --
default              IpmiGateway        --
default              IpmiUserName       --

Docker/OCI Registry Support for Credentials

Warewulf doesn't currently support credentials for non-public Docker and OCI registries. Best implementation here should probably be to read environment variables.

--debug and --verbose are mutually exclusive

While playing with container import, It seems --debug and --verbose are mutually exclusive:

[root@localhost ~]# wwctl container import docker://warewulf/centos-8 centos-8 --debug
[DEBUG]    12:18:58.271320 Set log level to: 5
[WARN]     12:18:58.271588 VNFS name has illegal characters: centos-8
[ERROR]           12:18:58.271596 VNFS name contains illegal characters: centos-8
[root@localhost ~]# wwctl container import docker://warewulf/centos-8 centos-8 --debug --verbose
[WARN]     VNFS name has illegal characters: centos-8
[ERROR]           VNFS name contains illegal characters: centos-8

I don't believe they need to be from a tooling and UI perspective.

It appears to be the code in ./wwwctl/root.go that makes that decision.

Daemon indexing of HW addresses

To achieve greater scale, the Warewulf daemon should index the node configurations by HW address into a map/hash array.

Don't bind mount /etc/resolv.conf into container on exec

see title

copy and move commands for overlays

It would be nice to have commands like
wwctl overlay cp -s default myFancyProfile
to duplicate overlays directly with wwctl, likewise a move command might be useful.

Make Install overwrites existing conf file, no backup

re-running make install should probably at least back up the original conf, or if it exists create a warewulf.conf.sample file instead.

profile add [flags] doesn't appear to work.

The documentation (in-line and on-line: https://docs.ctrliq.com/warewulf/docs/wwctl/profile ) suggests you can use flags when doing profile add. However, the flags do not appear to work:

[root@com-1-1 tools]# wwctl profile add -K 4.18.0-240.15.1.el8_3.aarch64 -C centos8aarch64 centos8_profile
Error: unknown shorthand flag: 'K' in -K
Usage:
  wwctl profile add [flags] <profile name>

Flags:
  -h, --help   help for add

Global Flags:
  -d, --debug     Run with debugging messages enabled.
  -v, --verbose   Run with increased verbosity.

[root@com-1-1 tools]# wwctl profile add --kernel 4.18.0-240.15.1.el8_3.aarch64 -C centos8aarch64 centos8_profile                                                            
Error: unknown flag: --kernel
Usage:
  wwctl profile add [flags] <profile name>

Flags:
  -h, --help   help for add

Global Flags:
  -d, --debug     Run with debugging messages enabled.
  -v, --verbose   Run with increased verbosity.

Add non-interactive support to all configuration commands to support automated build and testing

I'm using Vagrant to provision a small ww4 test cluster simply to learn the interface/features of warewulf.

My config steps follow an early version of the EL8 QuickStart Guide. I'm not sure if the behaviors have changed or I didn't notice in early provisioning attempts but wwctl commands to setup and/or modify profiles seem to prefer/require interactive response when making changes even when the --force option is used.

Here is a brief excerpt from my Vagrant provisioning script which generates the output below...

    14
    15	# Pull and build the VNFS container and kernel
    16	sudo wwctl container import docker://warewulf/centos-8 centos8 --setdefault
    17	sudo wwctl kernel import "$(uname -r)" --setdefault
    18
    19	# Setup the default node profile
    20	#sudo wwctl profile set default -K $(uname -r) -C centos-8
    21	sudo wwctl profile set --force default --netdev eth1 -M 255.255.255.0 -G 10.0.2.1
    22	sudo wwctl profile set --force default --netdev eth0 -M 255.255.255.0 -G 192.168.15.15
    23	sudo wwctl profile set --force default --netdev enp0s3 -M 255.255.255.0 -G 10.0.2.1
    24	sudo wwctl profile set --force default --netdev enp0s8 -M 255.255.255.0 -G 192.168.15.15
    25
    26	sudo wwctl profile list
    27

Here is a brief excerpt from the Vagrant provisioning log to demonstrate...

  1570      mgr: + sudo wwctl container import docker://warewulf/centos-8 centos8 --setdefault
  1571      mgr: Getting image source signatures
  1572      mgr: Copying blob sha256:33e4c05dc1b693c06eceb4dc9b5fb8484c27f9b099cf07be9fda16a07d75ffd4
  1573      mgr: Copying config sha256:f2997960e742caf41ef7be6e288b09b1f97a4308881d7df642ccea1d2e6c90dd
  1574      mgr: Writing manifest to image destination
  1575      mgr: Storing signatures
  1576      mgr: [LOG]       info unpack layer: sha256:33e4c05dc1b693c06eceb4dc9b5fb8484c27f9b099cf07be9fda16a07d75ffd4
  1577      mgr: Building container: centos8
  1578      mgr: Set default profile to container: centos8
  1579      mgr: [INFO]     Warewulf daemon process not running (/tmp/warewulfd.pid)
  1580      mgr: ++ uname -r
  1581      mgr: + sudo wwctl kernel import 4.18.0-240.15.1.el8_3.x86_64 --setdefault
  1582      mgr: 4.18.0-240.15.1.el8_3.x86_64: Done
  1583      mgr: Set default kernel version to: 4.18.0-240.15.1.el8_3.x86_64
  1584      mgr: [INFO]     Warewulf daemon process not running (/tmp/warewulfd.pid)
  1585      mgr: + sudo wwctl profile set --force default --netdev eth1 -M 255.255.255.0 -G 10.0.2.1
  1586      mgr: ^M? Are you sure you want to modify 1 profile(s)? [y/N] █
  1587      mgr: ^M? Are you sure you want to modify 1 profile(s)? [y/N] e█
  1588      mgr: ^M? Are you sure you want to modify 1 profile(s)? [y/N] ex█
  1589      mgr: ^M? Are you sure you want to modify 1 profile(s)? [y/N] exi█
  1590      mgr: ^M? Are you sure you want to modify 1 profile(s)? [y/N] exit█
  1591      mgr: ^M? Are you sure you want to modify 1 profile(s)? [y/N] exit█
  1592      mgr: ^M✗ Are you sure you want to modify 1 profile(s):
  1593      mgr:
  1594      mgr: + sudo wwctl profile set --force default --netdev eth0 -M 255.255.255.0 -G 192.168.15.15
  1595      mgr: ^M? Are you sure you want to modify 1 profile(s)? [y/N] █
  1596      mgr:
  1597      mgr:
  1598      mgr: + sudo wwctl profile set --force default --netdev enp0s3 -M 255.255.255.0 -G 10.0.2.1
  1599      mgr: ^M? Are you sure you want to modify 1 profile(s)? [y/N] █
  1600      mgr:
  1601      mgr:
  1602      mgr: + sudo wwctl profile set --force default --netdev enp0s8 -M 255.255.255.0 -G 192.168.15.15
  1603      mgr: ^M? Are you sure you want to modify 1 profile(s)? [y/N] █
  1604      mgr:
  1605      mgr:
  1606      mgr: + sudo wwctl profile list

I think the wwctl profile set ... command may actually be working (magically because Vagrant shell provisioner appears to be providing inputs to the prompts?) but I haven't dug in more.

It seems all configuration commands could use an option similar to wwctl node add --yes ... to support automated configuration in Vagrant and/or in CI/CD systems.

warewulfd.service won't start

[root@com-1-1 ipxe]# systemctl status warewulfd.service
● warewulfd.service - Warewulf cluster management daemon
   Loaded: loaded (/usr/lib/systemd/system/warewulfd.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2021-03-02 14:36:50 GMT; 4s ago
     Docs: https://warewulf.readthedocs.io
  Process: 16191 ExecStart=/usr/bin/wwctl server start --foreground (code=exited, status=216/GROUP)
 Main PID: 16191 (code=exited, status=216/GROUP)

Mar 02 14:36:50 com-1-1 systemd[1]: Unit warewulfd.service entered failed state.
Mar 02 14:36:50 com-1-1 systemd[1]: warewulfd.service failed.
Mar 02 14:36:50 com-1-1 systemd[1]: warewulfd.service holdoff time over, scheduling restart.
Mar 02 14:36:50 com-1-1 systemd[1]: Stopped Warewulf cluster management daemon.
Mar 02 14:36:50 com-1-1 systemd[1]: start request repeated too quickly for warewulfd.service
Mar 02 14:36:50 com-1-1 systemd[1]: Failed to start Warewulf cluster management daemon.
Mar 02 14:36:50 com-1-1 systemd[1]: Unit warewulfd.service entered failed state.
Mar 02 14:36:50 com-1-1 systemd[1]: warewulfd.service failed.

[root@com-1-1 ipxe]# journalctl -xe
-- Subject: Unit warewulfd.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit warewulfd.service has finished starting up.
--
-- The start-up result is done.
Mar 02 14:36:50 com-1-1 systemd[16191]: Failed at step GROUP spawning /usr/bin/wwctl: No such process
-- Subject: Process /usr/bin/wwctl could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The process /usr/bin/wwctl could not be executed and failed.
--
-- The error number returned by this process is 3.
Mar 02 14:36:50 com-1-1 systemd[1]: warewulfd.service: main process exited, code=exited, status=216/GROUP
Mar 02 14:36:50 com-1-1 systemd[1]: Unit warewulfd.service entered failed state.
Mar 02 14:36:50 com-1-1 systemd[1]: warewulfd.service failed.
Mar 02 14:36:50 com-1-1 systemd[1]: warewulfd.service holdoff time over, scheduling restart.
Mar 02 14:36:50 com-1-1 systemd[1]: Stopped Warewulf cluster management daemon.
-- Subject: Unit warewulfd.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit warewulfd.service has finished shutting down.
Mar 02 14:36:50 com-1-1 systemd[1]: start request repeated too quickly for warewulfd.service
Mar 02 14:36:50 com-1-1 systemd[1]: Failed to start Warewulf cluster management daemon.
-- Subject: Unit warewulfd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit warewulfd.service has failed.
--
-- The result is failed.
Mar 02 14:36:50 com-1-1 systemd[1]: Unit warewulfd.service entered failed state.
Mar 02 14:36:50 com-1-1 systemd[1]: warewulfd.service failed.
lines 2504-2539/2539 (END)

Request For Comment: Introducing CLA

Warewulf needs a CLA for the project. Please review and add comments on:
https://docs.google.com/document/d/1uaREO0Fke9rND6abjZtMf4ASou0Zaejtv4Yix7I3O8c/edit

Include vendor packages in distribution release/build

A build from the dist tarball should not require any additional retrieval of dependencies.

Canary testing for images, overlays, and kernel

Simplify the process of A/B testing with updates and testing (e.g. copying container images (not possible right now).

How do you keep jobs from landing on testing environments? (possibly use a profile with a different "clustername" which will be interpreted as a domain suffix)

interface for ipmi should be configureable

All power and sensor commands use per default the hardcoded lan interface for communication. It should be able set this interface in the host config.

AArch64 seems to requre system overlay image to be compressed.

after: wwctl overlay build -a

# file /var/warewulf/provision/overlays/system/node1.img
node1.img: ASCII cpio archive (SVR4 with no CRC)

on boot gives:

[   27.467888] Initramfs unpacking failed: junk in compressed archive
...

if I bzip2 /var/warewulf/provision/overlays/system/node1.img and give the compressed img the name: node1.img everything works as expected.

wwclient, no route to host

I lately run into the issue that the wwclient can't communicate with the master.

root@n4003:~# tail /var/log/wwclient.log 
[LOG]      2021/04/29 11:12:03 Updating live file system LIVE, cancel now if this is in error
[LOG]      2021/04/29 11:12:10 Get http://129.129.224.32:9873/overlay-runtime: dial tcp :987->129.129.224.32:9873: connect: no route to host
[LOG]      2021/04/29 11:12:12 Updating system
[LOG]      2021/04/29 11:12:12 ERROR: Failed running CPIO: exit status 2

I do have a route to the host. It can hardly be more specific of a route as this

root@n4003:~# ip route
default via 129.129.144.1 dev eno2 
default via 172.23.205.1 dev eth0 metric 30 onlink 
129.129.144.0/24 dev eno2 proto kernel scope link src 129.129.144.102 
129.129.224.32 via 172.23.205.1 dev eth0 
172.23.205.0/24 dev eth0 proto kernel scope link src 172.23.205.102 metric 30

I can also ping the master

root@n4003:~# ping 129.129.224.32
PING 129.129.224.32 (129.129.224.32) 56(84) bytes of data.
64 bytes from 129.129.224.32: icmp_seq=1 ttl=57 time=0.691 ms

is there an easy way to diagnose such issues?

runtime overlays are broken

The runtime overlay provision seems to be broken. I see on the nodes in /var/log/wwclient.log with following error:

[LOG]      2021/09/10 14:30:07 Not updating runtime overlay, got status code: 404 (edited)

and on the server

[LOG]      UNKNOWN: 192.168.100.101: /overlay-runtime

Cleanup failed container import

When running the container import command, remove residual chroot directory if the import fails.
/var/warewulf/chroot/xxxx

If there are debug reasons for leaving it, add it as a command option.

log should have timestamps

The log file /var/log/warewulfd.log has no timestamps. This makes debugging more complicated than it could be.

no documented method to change a node's profile.

I've created a new profile with an updated OS. I want to change an existing node to now inherit my new profile, but the documentation does not include an obvious method to do so:

https://docs.ctrliq.com/warewulf/docs/wwctl/profile

compile fails

note: make clean does not clean up completely - the go cache files are left behind in ~/go/mod/cache/ which makes it look unclean, but it's OK I think.

[greg@controller7 warewulf]# make all
get "golang.org/x/text": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at //golang.org/x/text?go-get=1
get "google.golang.org/protobuf": found meta tag get.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf?go-get=1
go: github.com/containers/image/[email protected] requires
github.com/containers/[email protected] requires
github.com/Microsoft/[email protected] requires
github.com/opencontainers/[email protected]: invalid version: git fetch --unshallow -f origin in /root/go/pkg/mod/cache/vcs/552be1eeae05d101756f44ffae429e48bc88b11bff177e7eb912ebdd03383c62: exit status 128:
fatal: git fetch-pack: expected shallow list
make: *** [vendor] Error 1

wwctld kernel has no mean to change the default kernel

wwctl kernel
should have an option to set the default kernel, at the moment this can be only be done by reimporting the kernel

missing snp.efi on AArch64?

I have set up a test environment.
I have had success adding an x86 machine in with something like:
virt-install --pxe --network network=nodhcp --name x86_pxe --memory 2048 --disk size=10 --nographics --boot menu=on,useserial=on However, when I add an AArch64 machine (using QEMU emulation) to the cluster with:

virt-install \
    --arch aarch64 \
    --pxe \
    --network network=nodhcp \
    --name aarch64_pxe \
    --memory 2048 \
    --disk size=10 \
    --nographics \
    --boot menu=on

I get:

...
>>Start PXE over IPv4..                                                                                                                                                     
  Station IP address is 192.168.123.180                                                                                                                                     
                                                                                                                                                                            
  Server IP address is 192.168.123.10                                                                                                                                       
  NBP filename is /warewulf/ipxe/bin-arm64-efi/snp.efi                                                                                                                      
  NBP filesize is 0 BytesPXE-E23: Client received TFTP error from server.

and on my Warewulf primary:

# ll /var/lib/tftpboot/warewulf/ipxe/
total 0

I believe there is a missing bin-arm64-efi/snp.efi?

Can't boot compressed kernel on Aarch64

iPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP iSCSI TFTP AoE EFI Menu

net1: f4:03:43:fe:90:01 using NII on NII-0000:17:00.1 (open)
  [Link:down, TX:0 TXE:0 RX:0 RXE:0]
  [Link status: Unknown (http://ipxe.org/1a086194)]
Configuring (net1 f4:03:43:fe:90:01)...... ok
net1: 10.1.232.195/255.255.255.0
Next server: 10.1.232.194
Filename: http://10.1.232.194:9873/ipxe/f4-03-43-fe-90-01
http://10.1.232.194:9873/ipxe/f4-03-43-fe-90-01... ok
f4-03-43-fe-90-01 : 824 bytes [script]

================================================================================
Warewulf v4 now booting: com-1-2

Container: centos
Kernel: 4.14.0-49.el7a.aarch64
KernelArgs: quiet crashkernel=no vga=791 rootfstype=tmpfs

http://10.1.232.194:9873/kernel/f4-03-43-fe-90-01... ok
Could not select: Exec format error (http://ipxe.org/2e008081)

But if I manually decompress the kernel image /var/warewulf/provision/kernel/4.14.0-49.el7a.aarch64/vmlinuz:

iPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP iSCSI TFTP AoE EFI Menu

net1: f4:03:43:fe:90:01 using NII on NII-0000:17:00.1 (open)
  [Link:down, TX:0 TXE:0 RX:0 RXE:0]
  [Link status: Unknown (http://ipxe.org/1a086194)]
Configuring (net1 f4:03:43:fe:90:01)...... ok
net1: 10.1.232.195/255.255.255.0
Next server: 10.1.232.194
Filename: http://10.1.232.194:9873/ipxe/f4-03-43-fe-90-01
http://10.1.232.194:9873/ipxe/f4-03-43-fe-90-01... ok
f4-03-43-fe-90-01 : 824 bytes [script]

================================================================================
Warewulf v4 now booting: com-1-2

Container: centos
Kernel: 4.14.0-49.el7a.aarch64
KernelArgs: quiet crashkernel=no vga=791 rootfstype=tmpfs

http://10.1.232.194:9873/kernel/f4-03-43-fe-90-01... ok
http://10.1.232.194:9873/container/f4-03-43-fe-90-01... ok
http://10.1.232.194:9873/kmods/f4-03-43-fe-90-01... ok
http://10.1.232.194:9873/overlay-system/f4-03-43-fe-90-01... ok
[    0.000000] ACPI: SPCR: Unexpected SPCR Access Width.  Defaulting to byte size
[    7.420100] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[    7.434255] CPU: 3 PID: 1 Comm: swapper/0 Not tainted 4.14.0-49.el7a.aarch64 #1
[    7.441550] Hardware name: HPE ASSY,ARx4z SystemBoard/Comanche_2S_CN99X_ARM , BIOS L50_5.13_1.0.6 07/10/2018
[    7.451362] Call trace:
[    7.453801] [<ffff000008088e6c>] dump_backtrace+0x0/0x23c
[    7.459187] [<ffff0000080890cc>] show_stack+0x24/0x2c
[    7.464228] [<ffff0000087f607c>] dump_stack+0x84/0xa8
[    7.469268] [<ffff0000080d2794>] panic+0x138/0x2a0
[    7.474047] [<ffff00000880a3b4>] kernel_] Memory Limit: none
[    7.498499] ---[ end Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.

So at least it's booting, though it can't find the init script.

Is there any problem with GZIP decompression in x86.efi?

I discovered that a container image that can boot machine A, but another machine B cannot boot with the error "Initramfs unpacking failed: junk in compressed archive".
Machine A will boot with i386.kpxe and machine B will boot with x86.efi.

Then, I changed the defined(*) compression level to "gzip -c --fast" and created a container image. With that image, machine B booted successfully.
(*) https://github.com/ctrliq/warewulf/blob/main/internal/pkg/container/build.go#L44

I don't have the ability to parse i386.kpxe or x86.efi, so that's all I was able to investigate.
Is there any problem with GZIP decompression in x86.efi?

Details:

/var/warewulf/provision/container/c_centos7_02.1_ganglia.img.gz = my test container image
du -sh /var/warewulf/chroots/c_centos7_02.1_ganglia/ -> 1.1GB
machine_A: Intel(R) Xeon(R) CPU E5-2698 v4, 128GB memory, boot with /warewulf/i386.kpxe
machine_B: Intel(R) Xeon(R) CPU D-1541, 64GB memory, boot with /warewulf/x86.efi

[Case-1] gzip -c (same as original)

$ gzip -dc _original/c_centos7_02.1_ganglia.img.gz | md5sum
e7b9ad6a0ff2ca1ef54ae614dde54092  -
$ file _original/c_centos7_02.1_ganglia.img.gz
_original/c_centos7_02.1_ganglia.img.gz: gzip compressed data, from Unix, last modified: Fri Apr  2 17:14:10 2021

machine_A (/warewulf/i386.kpxe) -> boot OK
machine_B (/warewulf/x86.efi) -> boot FAILED with this message below
[ 14.094709] Initramfs unpacking failed: junk in compressed archive

[Case-2] gzip -c --best

$ gzip -dc _best/c_centos7_02.1_ganglia.img.gz | md5sum
e7b9ad6a0ff2ca1ef54ae614dde54092  -
$ file _best/c_centos7_02.1_ganglia.img.gz
_best/c_centos7_02.1_ganglia.img.gz: gzip compressed data, from Unix, last modified: Mon Apr  5 12:45:27 2021, max compression

machine_A (/warewulf/i386.kpxe) -> boot OK
machine_B (/warewulf/x86.efi) -> boot FAILED with this message below
[ 14.000888] Initramfs unpacking failed: junk in compressed archive

[Case-3] gzip -c --fast

$ gzip -dc _fast/c_centos7_02.1_ganglia.img.gz | md5sum
e7b9ad6a0ff2ca1ef54ae614dde54092  -
$ file _fast/c_centos7_02.1_ganglia.img.gz
_fast/c_centos7_02.1_ganglia.img.gz: gzip compressed data, from Unix, last modified: Mon Apr  5 12:57:51 2021, max speed

machine_A (/warewulf/i386.kpxe) -> boot OK
machine_B (/warewulf/x86.efi) -> boot OK

wwctl overlay list, generic error

improvement

wwctl overlay list, creates a generic error
ERROR: requires at least 1 arg(s), only received 0

To improve user experience, a more meaningful error should be returned.
ERROR: overlay kind must be specified. Use -h or --help for additional information.

Service Startup docs

This syntax appears to be obsolete...
"sudo ./wwctl service -a"

It may be replaced by...
"sudo ./wwctl server start"

Please update the Readme/Docs

e.g.
[greg@controller7 warewulf]$ sudo ./wwctl server status
[INFO] Warewulf daemon process not running (/tmp/warewulfd.pid)
[greg@controller7 warewulf]$ sudo ./wwctl server start
Warewulf daemon is running at PID: 14962
[greg@controller7 warewulf]$ sudo ./wwctl server status
Warewulf daemon is running at PID: 14962

'wwctl server' and systemd service are disconnected

The systemd startup works great for me:

# systemctl status warewulfd.service 
● warewulfd.service - Warewulf cluster management daemon
   Loaded: loaded (/usr/lib/systemd/system/warewulfd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-03-04 16:41:04 EST; 23s ago
     Docs: https://docs.ctrliq.com/warewulf
 Main PID: 2120 (wwctl)
    Tasks: 7 (limit: 5979)
   Memory: 8.1M
   CGroup: /system.slice/warewulfd.service
           └─2120 /usr/bin/wwctl server start --foreground

However, now it is started with systemd, the wwctl system commands seem unable to interact with the service:

# wwctl server status
[INFO]     Warewulf daemon process not running (/tmp/warewulfd.pid)

wwctl profile set --force should not require user confirmation

When automating the deployment of a warewulf v4 cluster (for example with Vagrant) attempts to specify configuration for node profiles during provisioning require input from the user. This prevents the use of automation tools (such as Vagrant) to deploy a warewulf v4 cluster.

The --force option (or perhaps another flag) should bypass the requirement for user input.

profile before modification

[vagrant@mgr ~]$ sudo wwctl profile list
################################################################################
PROFILE NAME         FIELD              VALUE
default              Id                 default
default              Comment            This profile is automatically included for each node
default              ClusterName        --
default              ContainerName      centos-8
default              KernelVersion      4.18.0-240.1.1.el8_3.x86_64
default              KernelArgs         --
default              Init               --
default              Root               --
default              RuntimeOverlay     --
default              SystemOverlay      --
default              Ipxe               --
default              IpmiIpaddr         --
default              IpmiNetmask        --
default              IpmiGateway        --
default              IpmiUserName       --

profile modification without --force AND 'N' user confirmation

[vagrant@mgr ~]$ sudo wwctl profile set default --netdev eth0 -M 255.255.255.0 -G 192.168.15.15
? Are you sure you want to modify 1 profile(s)? [y/N] N

profile modification with --force AND 'N' user confirmation

[vagrant@mgr ~]$ sudo wwctl profile set --force default --netdev eth0 -M 255.255.255.0 -G 192.168.15.15
? Are you sure you want to modify 1 profile(s)? [y/N] N

profile modification with --force AND user confirmation

[vagrant@mgr ~]$ sudo wwctl profile set --force default --netdev eth0 -M 255.255.255.0 -G 192.168.15.15
Are you sure you want to modify 1 profile(s): y
[INFO]     Warewulf daemon process not running (/tmp/warewulfd.pid)

profile after modification

[vagrant@mgr ~]$ sudo wwctl profile list
################################################################################
PROFILE NAME         FIELD              VALUE
default              Id                 default
default              Comment            This profile is automatically included for each node
default              ClusterName        --
default              ContainerName      centos-8
default              KernelVersion      4.18.0-240.1.1.el8_3.x86_64
default              KernelArgs         --
default              Init               --
default              Root               --
default              RuntimeOverlay     --
default              SystemOverlay      --
default              Ipxe               --
default              IpmiIpaddr         --
default              IpmiNetmask        --
default              IpmiGateway        --
default              IpmiUserName       --
default              eth0:IPADDR        --
default              eth0:NETMASK       255.255.255.0
default              eth0:GATEWAY       192.168.15.15
default              eth0:HWADDR        --
default              eth0:TYPE          --
default              eth0:DEFAULT       false

Update repository documentation links to github.com/hpcng

Minor markdown files should reflect new repository location at github.com/hpcng/warewulf

[BUG] wwctl overlay build with flags (-a|--all) no longer works

Describe Bug

Recent changes to overlay commands break some pre-existing behavior.

For example, the -a flag for wwctl overlay build... command no longer works.

To Reproduce

Steps to reproduce the behavior:

Modify runtime overlay
Attempt to build all overlays
See error

Help text

[vagrant@mgr ~]$ sudo wwctl overlay build --help
This command will build a system or runtime overlay.

Usage:
  wwctl overlay build [flags] <overlay kind> <overlay name>

Flags:
  -a, --all    Build all overlays (runtime and system)
  -h, --help   help for build

Global Flags:
  -d, --debug     Run with debugging messages enabled.
  -v, --verbose   Run with increased verbosity.

Previous (working) execution

[vagrant@mgr ~]$ sudo wwctl overlay build -a
ERROR: accepts 2 arg(s), received 0

Attempt to build default system and runtime overlays

[vagrant@mgr ~]$ wwctl overlay build -a default
ERROR: accepts 2 arg(s), received 1

Working commands

[vagrant@mgr ~]$ sudo wwctl overlay build runtime default
[INFO]     Updating Runtime Overlays...
[INFO]     cn0                                : Done
[INFO]     cn1                                : Done
[vagrant@mgr ~]$ sudo wwctl overlay build system default
[INFO]     Updating System Overlays...
[INFO]     cn0                                : Done
[INFO]     cn1                                : Done

Expected behavior

Do one of...

re-implement the behavior currently provided by the -a|--all option to wwctl overlay build
remove the -a|--all option from the wwctl overlay build ... CLI option (and help) completely,

System (please complete the following information):

OS: Rocky Linux 8.4
Warewulf Version: v4.1.0 (git hash acf2990)

Missing support for AArch64 containers

I'm testing Warewulf on an AArch64 cluster. I am most interested in CentOS8. The aarch64 container is missing from the Warewulf docker hub here: https://hub.docker.com/r/warewulf/centos-8

In addition, I can't find the Dockerfile to build this container myself.

I prefer to use a locally build container and I singularity seems the ideal tool for this job. However, there isn't a centos-8 def file. I have crudely modified the Centos-7 file - which gives a working container - but the boot hangs after multiuser.target and doesn't get to the login screen. I'm working on this problem, but I think a known working centos-8 def file or the centos-8 DockerFile would help me.

Finding and verifying a working AArch64 container is a prerequisite for adding support for AArch64 containers.

arm support

I have tried to add boot an aarch64 node form x86 host running warewulf v4. For this I added an aarch64 container as an docker image. I also added the kernel from this container (see #120). After a restart of tftp did only boot to ipxe and showed following error

iPXE 1.21.1+ (g56f7d) -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP iSCSI TFTP AoE EFI Menu

net0: 52:54:00:0f:75:a0 using SNP on SNP-0000:01:00.0 (Ethernet) [open]
  [Link:up, TX:0 TXE:1 RX:0 RXE:0]
  [TXE: 1 x "Network unreachable (http://ipxe.org/28086090)"]
Configuring (net0 52:54:00:0f:75:a0)...... ok
net0: 192.168.100.103/255.255.255.0
net0: fe80::5054:ff:fe0f:75a0/64
Next server: 192.168.100.250
Filename: http://192.168.100.250:9873/ipxe/52-54-00-0f-75-a0
http://192.168.100.250:9873/ipxe/52-54-00-0f-75-a0... ok
[Security] 3rd party image[0] can be loaded after EndOfDxe: .
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 43A5AEC40
Unloading driver at 0x00000000000
52-54-00-0f-75-a0 : 784 bytes [script]

================================================================================
Warewulf v4 now booting: arm01

Container: leap-15.3-arm-hpc:latest
Kernel: 5.3.18-59.19-default.gz
KernelArgs: crashkernel=no

http://192.168.100.250:9873/kernel/52-54-00-0f-75-a0... ok
[Security] 3rd party image[0] can be loaded after EndOfDxe: .
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 43A5AE1C0
Unloading driver at 0x00000000000
Could not select: Exec format error (http://ipxe.org/2e008081)

There was an error, rebooting in 15s...

Even this error is fixed, I would need a executable of wwclient for arch64. I can achieve this by creating a default-aarch64 system profile and adding a aarch54 executable to this profile. Is this the best practice way to crossboot in warewulf, or should there be an architectur flag added?

Please add sudo to centos-8 Docker container

When automating the deployment of a Warewulf v4 cluster (for example with Vagrant) certain vagrant commands attempt to use sudo inside the machine.

For example, the Vagrant command, vagrant up <machine> will attempt to ssh into container and use sudo to perform any provisioning steps specified for the machine using sudo. These always fail because sudo is not present.

For example...

~/boxes/warewulf-vagrant on main !6 ?10 ❯ vagrant up cn0
Bringing machine 'cn0' up with 'virtualbox' provider...
==> cn0: Importing base box 'onyxpoint/empty'...
==> cn0: Matching MAC address for NAT networking...
==> cn0: Checking if box 'onyxpoint/empty' version '1.0.0' is up to date...
==> cn0: Setting the name of the VM: warewulf-vagrant_cn0_1618014203007_2463
==> cn0: Fixed port collision for 22 => 2222. Now on port 2200.
==> cn0: Clearing any previously set network interfaces...
==> cn0: Preparing network interfaces based on configuration...
    cn0: Adapter 1: nat
    cn0: Adapter 2: hostonly
==> cn0: Forwarding ports...
    cn0: 22 (guest) => 2200 (host) (adapter 1)
==> cn0: Running 'pre-boot' VM customizations...
==> cn0: Booting VM...
==> cn0: Waiting for machine to boot. This may take a few minutes...
    cn0: SSH address: 127.0.0.1:2200
    cn0: SSH username: vagrant
    cn0: SSH auth method: private key
    cn0: Warning: Authentication failure. Retrying...
==> cn0: Machine booted and ready!
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

sed -i '/#VAGRANT-BEGIN/,/#VAGRANT-END/d' /etc/fstab

Stdout from the command:

Stderr from the command:

Running vagrant with the --debug option allows you to examine the cause of the failure and it is clearly caused by sudo not being present.

Here is an excerpt from the debug log of another command requiring sudo. The vagrant reload command will detect the OS inside the machine and execute the appropriate command to reboot it. For CentOS this amounts to sudo shutdown -h now (followed later by appropriate provider power on semanitcs)...

~/boxes/warewulf-vagrant on main !6 ?10 ❯ vagrant reload --debug cn0 2>&1 | grep "Detected: centos" -B5 -A30
  2819	DEBUG guest: Trying: centos
  2820	DEBUG ssh: Re-using SSH connection.
  2821	 INFO ssh: Execute: cat /etc/centos-release (sudo=false)
  2822	DEBUG ssh: stderr: 41e57d38-b4f7-4e46-9c38-13873d338b86-vagrant-ssh
  2823	DEBUG ssh: Exit status: 0
  2824	 INFO guest: Detected: centos!
  2825	DEBUG guest: Searching for cap: halt
  2826	DEBUG guest: Checking in: centos
  2827	DEBUG guest: Checking in: redhat
  2828	DEBUG guest: Checking in: linux
  2829	DEBUG guest: Found cap: halt in linux
  2830	 INFO guest: Execute capability: halt [#<Vagrant::Machine: cn0 (VagrantPlugins::ProviderVirtualBox::Provider)>] (centos)
  2831	DEBUG ssh: Re-using SSH connection.
  2832	 INFO ssh: Execute: ps -o comm= 1 | grep systemd (sudo=true)
  2833	DEBUG ssh: Exit status: 127
  2834	DEBUG ssh: stderr: bash: sudo: command not found
  2835
  2836	DEBUG ssh: Re-using SSH connection.
  2837	 INFO ssh: Execute: shutdown -h now (sudo=true)
  2838	DEBUG ssh: Exit status: 127
  2839	DEBUG ssh: stderr: bash: sudo: command not found
  2840
  2841	ERROR warden: Error occurred: The following SSH command responded with a non-zero exit status.
  2842	Vagrant assumes that this means the command failed!
  2843
  2844	shutdown -h now
  2845
  2846	Stdout from the command:
  2847
  2848
  2849
  2850	Stderr from the command:
  2851
  2852
  2853	 INFO warden: Beginning recovery process...
  2854	 INFO warden: Recovery complete.

A solution to these problems with Vagrant is to install sudo into the Docker container that is pulled following the instructions in the Warewulf Quickstart Guide for EL8.

FYI: selinux enabled: tftp

I've been working with selinux enabled, in Enforcing mode. It all seems to work ok, with the exception of tftp server. For tftp server I need to give that process permissive mode: semanage permissive -a tftpd_t.

I don't think this is a warewulf issue - as tftp is pulled from repo in my case (RHEL8, AArch64), but I thought I would record it here in case others are interested in keeping selinux enabled.

Cannot install kernels that are not in /boot

In my hybrid x86 and aarch64 cluster my host node (WW primary) needs to to have two kernel entries for WW.

It looks like all kernels must be in /boot/ and modules in /lib/modules/ : https://github.com/ctrliq/warewulf/blob/ec3345760d2db88ad393909f6abecda75d61b7ef/internal/pkg/kernel/kernel.go#L75

This is easily satisfied if you want to use the kernel from the machine that is already installed.

In my case, I want to use a kernel that is of a different architecture (and actually a newer distro), and I don't really want to 'yum install' the kernel as it feels a bit messy and can cause problems with post-install scripts doing their thing.

What would be great is to have the 'wwctl kernel' have sufficient flexibility to be de-coupled from the host machine kernel.

Any thoughts?

Move documentation hosting to hpcng

Currently the docs live at docs.ctrliq.com/warewulf and should exist at an hpcng controlled domain

warewulfd port in config has no effect

the port setting in the config file does not have any effect on the daemon.

Please add an `unset` subcommand

e.g. wwctl profile unset default --kernelargs

Not able to PXE Boot the Compute Node

Hi WW4 Team,
I followed the steps given in the link: https://warewulf.hpcng.org/docs/quickstart/el8
However my compute node is not PXE booting. I get following error:

There is not much troubleshooting help available for Warewulf4. Can you help me in troubleshooting this issue?

documentation is missing a method to toggle node 'Disabled' status

Looking at this documentation:
https://github.com/ctrliq/warewulf/blob/main/docs/docs/wwctl/node.md
it appears to be missing a mechanism to toggle the node Disabled status.

outdated documentation links

The links to the documentation are outdated and should be updated.

It's also a good idea to play through the quickstart guides to figure wrong/outdated commands before the RC is issued.
I know for sure that the wwctl configure {hosts,dhcp, ...} commands need --persist to take effect.

init script not found on Aarch64

The init script at /init in the system overlay isn't being found by the kernel on Aarch64:

PXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP iSCSI TFTP AoE EFI Menu

net1: f4:03:43:fe:90:01 using NII on NII-0000:17:00.1 (open)
  [Link:down, TX:0 TXE:0 RX:0 RXE:0]
  [Link status: Unknown (http://ipxe.org/1a086194)]
Configuring (net1 f4:03:43:fe:90:01)...... ok
net1: 10.1.232.195/255.255.255.0
Next server: 10.1.232.194
Filename: http://10.1.232.194:9873/ipxe/f4-03-43-fe-90-01
http://10.1.232.194:9873/ipxe/f4-03-43-fe-90-01... ok
f4-03-43-fe-90-01 : 824 bytes [script]

================================================================================
Warewulf v4 now booting: com-1-2

Container: centos
Kernel: 4.14.0-49.el7a.aarch64
KernelArgs: quiet crashkernel=no vga=791 rootfstype=tmpfs

http://10.1.232.194:9873/kernel/f4-03-43-fe-90-01... ok
http://10.1.232.194:9873/container/f4-03-43-fe-90-01... ok
http://10.1.232.194:9873/kmods/f4-03-43-fe-90-01... ok
http://10.1.232.194:9873/overlay-system/f4-03-43-fe-90-01... ok
[    0.000000] ACPI: SPCR: Unexpected SPCR Access Width.  Defaulting to byte size
[    7.420100] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[    7.434255] CPU: 3 PID: 1 Comm: swapper/0 Not tainted 4.14.0-49.el7a.aarch64 #1
[    7.441550] Hardware name: HPE ASSY,ARx4z SystemBoard/Comanche_2S_CN99X_ARM , BIOS L50_5.13_1.0.6 07/10/2018
[    7.451362] Call trace:
[    7.453801] [<ffff000008088e6c>] dump_backtrace+0x0/0x23c
[    7.459187] [<ffff0000080890cc>] show_stack+0x24/0x2c
[    7.464228] [<ffff0000087f607c>] dump_stack+0x84/0xa8
[    7.469268] [<ffff0000080d2794>] panic+0x138/0x2a0
[    7.474047] [<ffff00000880a3b4>] kernel_] Memory Limit: none
[    7.498499] ---[ end Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.

On the controller:

[root@com-1-1 x]# curl http://10.1.232.194:9873/overlay-system/f4-03-43-fe-90-01 | cpio -idmv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0.
etc
etc/fstab
etc/hostname
etc/ssh
etc/ssh/ssh_host_dsa_key.pub
etc/ssh/ssh_host_dsa_key
etc/ssh/ssh_host_ecdsa_key.pub
etc/ssh/ssh_host_ecdsa_key
etc/ssh/ssh_host_ed25519_key.pub
etc/ssh/ssh_host_ed25519_key
etc/ssh/ssh_host_rsa_key.pub
etc/ssh/ssh_host_rsa_key
etc/sysconfig
etc/sysconfig/network-scripts
etc/sysconfig/network-scripts/ifcfg-eth0
etc/sysconfig/network
etc/warewulf
etc/warewulf/warewulf.conf
init
warewulf
warewulf/bin
100 7282k  100 7282k    0     0   196M      0 --:--:-- --:--:-- --:--:--  197M
warewulf/bin/wwclient
warewulf/config
warewulf/init.d
warewulf/init.d/20-loopback
warewulf/init.d/50-ipmi
warewulf/init.d/75-vnfs_fixes
warewulf/init.d/80-wwclient
warewulf/init.d/90-selinux
warewulf/wwinit
14564 blocks
[root@com-1-1 x]#

I've tried moving /init to /sbin/init and I've tried passing the init= kernel flag, but neither approach worked.

wwctl container file operation

There is no proper way to copy files inside a existing container. It can be achieved by copy files directly to /var/warewulf/container/chroots/FOO/rootfs, so this is not pending issue.
Perhaps this could be possible by having a bind mount option wwctl container exec --bind BAAR FOO /bin/bash

Container import is missing tag or digest support.

I'm trying to import a container for AArch64 on my x86 primary (where WW is running). It seems that importing an image and specifying a tag or digest isn't supported:

 wwctl container import docker://centos:7@sha256:43964203bf5d7fe38c6fca6166ac89e4c095e2b0c0a28f6c7c678a1348ddc7fa aarch64officialcentos7
[ERROR]           Could not import image: unable to parse uri: Docker references with both a tag and digest are currently not supported

As a consequence, I can't import some images into the system. This is important for me as I can use the digest to specify the platform build I would like. It may also be important for others who which so specify individual images precisely.

It would be great if WW could support digest. (It would also be great if WW could support --platform during import!)

wwctl serves empty files instead of error messages

If the system overlay isn't built, the client gets an empty file from the controller. It would be helpful to get an error message instead that suggests rebuilding the system overlay via wwctl overlay build -s -a

iPXE> imgfetch --name overlay-system  ${base}/overlay-system/f4-03-43-fe-90-01
http://10.1.232.194:9873/overlay-system/f4-03-43-fe-90-01... ok
iPXE> imgstat
f4-03-43-fe-90-01 : 14791168 bytes [EFI] [SELECTED] "rootfstype=tmpfs"
container : 72456645 bytes
kmods : 84094976 bytes
overlay-system : 0 bytes
iPXE>

Can't build warewulf on a vanilla CentOS 7.9.2009

Following the Quickstart guide, warewulf fails to build.

$ git clone https://github.com/hpcng/warewulf.git
Cloning into 'warewulf'...
remote: Enumerating objects: 5002, done.
remote: Counting objects: 100% (1219/1219), done.
remote: Compressing objects: 100% (823/823), done.
remote: Total 5002 (delta 429), reused 1072 (delta 351), pack-reused 3783
Receiving objects: 100% (5002/5002), 3.41 MiB | 3.23 MiB/s, done.
Resolving deltas: 100% (2218/2218), done.
$ cd warewulf
$ make all
golangci/golangci-lint info checking GitHub for tag 'v1.31.0'
golangci/golangci-lint info found version: 1.31.0 for v1.31.0/linux/amd64
golangci/golangci-lint info installed .tools/bin/golangci-lint
get "google.golang.org/protobuf": found meta tag     get.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf?go-get=1
go: github.com/containers/image/[email protected] requires
	github.com/containers/[email protected] requires.
	github.com/Microsoft/[email protected] requires
	github.com/opencontainers/[email protected]: invalid version: git fetch --unshallow -f origin in     /home/anon/go/pkg/mod/cache/vcs/552be1eeae05d101756f44ffae429e48bc88b11bff177e7eb912ebdd03383c62: exit status 128:
fatal: git fetch-pack: expected shallow list
make: *** [vendor] Error 1
$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.9.2009 (Core)
Release:	7.9.2009
Codename:	Core
$

'container' is clobbered by 'kmods'

During my AArch64 testing, I have observed that because the 'container' initrd preceeds the 'kmods' initrd in this link:
https://github.com/hpcng/warewulf/blob/9b5ff497186ff55fe9850d10994efa3daaa3bbc8/etc/ipxe/default.ipxe#L19
kmods clobbers the container.

This is undesirable in my case. However, because I haven't see any other comments on this I am wondering if it is AArch64 specific. I have looked around for evidence and I can't find any, so I'm opening this ticket for reference.