Debugging VM that doesn't start about cbsd HOT 9 CLOSED

You can attach disk from broken VM (e.g. bad1) to another (good) VM (e.g. good1). The easiest way:

1. create second disk to vm1 ( you won't use it, we just need the register second disk ):

cbsd bhyve-dsk mode=attach jname=good1 dsk_controller=virtio-blk dsk_size=1g imgtype=md

(or via cbsd bconfig)

2. delete ~cbsd/jails-data/good1/dsk2.vhd ( newly created disk ):

rm -f ~cbsd/jails-data/good1/dsk2.vhd

3. Use a symlink so that the second drive points to ZVOL from broken VM ( you can use readlink to get the full path ):

readlink ~cbsd/jails-data/bad1/dsk1.vhd

ln -s /dev/zvol/XXX/bad1/dsk1.vhd ~cbsd/jails-data/good1/dsk2.vhd

After restarting the good1 VM, you will be able to mount disk from bad1 VM.

After you copy the file, just detach second disk via cbsd bconfig -> good1 -> bhyvedsk -> dsk2 -> remove

from cbsd.

Interestingly, when I start another VM first, bstart does actually print error information when trying to start the VM that doesn't want to start:

➜  ~ sudo cbsd bstart ubuntu
VRDP is enabled. VNC bind/port: 127.0.0.1:5902
For attach VM console, use: vncviewer 127.0.0.1:5902
Resolution: 1920x1080.
VNC pass: cbsd
bhyve renice: 1
Waiting for PID..........
PID: 0
Mon Nov  6 17:20:59 CET 2023
cmd: env LIB9P_LOGGING=/zroot/cbsd/jails-system/ubuntu/cbsd_lib9p.log /usr/bin/nice -n 1 /usr/sbin/bhyve  -c cpus=8,sockets=8,cores=1,threads=1 -m 17179869184 -H -A -U 10db560c-f896-11ed-abf2-000acd2d4844 -s 0,hostbridge   -s 4,virtio-blk,/zroot/cbsd/vm/ubuntu/dsk1.vhd,sectorsize=512/4096    -s 5,virtio-net,tap3,mtu=1500,mac=00:a0:98:de:04:3f  -s 6,virtio-rnd  -s 8,fbuf,tcp=127.0.0.1:5902,w=1920,h=1080,password=cbsd -s 30,xhci,tablet  -s 31,lpc -l com1,stdio -l bootrom,/usr/local/cbsd/upgrade/patch/efi.fd,/zroot/cbsd/jails-system/ubuntu/BHYVE_UEFI_VARS.fd   ubuntu
-----
Failed to emulate instruction sequence [ 41f646040874064c017d90eb144c89 ] at 0xbfbb45e3
fbuf frame buffer base: 0x1469e4600000 [sz 16777216]
Please use for debug: /usr/local/cbsd/share/bhyverun.sh -c /zroot/cbsd/jails-system/ubuntu/bhyve.conf
bstart done in 12 seconds

from cbsd.

Please use for debug: /usr/local/cbsd/share/bhyverun.sh -c /zroot/cbsd/jails-system/ubuntu/bhyve.conf

;)

CBSD is just a generator of arguments to bhyve (+jail,xen).
For debug, you can completely repeat the bhyve command line that the CBSD tries to execute:

ifconfig tap3 create

/usr/sbin/bhyve  -c cpus=8,sockets=8,cores=1,threads=1 -m 17179869184 -H -A -U 10db560c-f896-11ed-abf2-000acd2d4844 -s 0,hostbridge   -s 4,virtio-blk,/zroot/cbsd/vm/ubuntu/dsk1.vhd,sectorsize=512/4096    -s 5,virtio-net,tap3,mtu=1500,mac=00:a0:98:de:04:3f  -s 6,virtio-rnd  -s 8,fbuf,tcp=127.0.0.1:5902,w=1920,h=1080,password=cbsd -s 30,xhci,tablet  -s 31,lpc -l com1,stdio -l bootrom,/usr/local/cbsd/upgrade/patch/efi.fd,/zroot/cbsd/jails-system/ubuntu/BHYVE_UEFI_VARS.fd   ubuntu

But if bhyve doesn't work (without CBSD command), it's most likely an upstream problem ( to ML: freebsd-virtualization@ or https://bugs.freebsd.org )

The only thing that can depend on the CBSD - it uses a custom bhyve-firmware (and/or reFind), but it's easy to change and test (you need to change /usr/local/cbsd/upgrade/patch/efi.fd to /usr/local/share/edk2-bhyve/BHYVE_UEFI_CODE.fd in bhyve args. Via CBSD:

pkg install -y bhyve-firmware
cbsd bset efi_firmware=/usr/local/share/edk2-bhyve/BHYVE_UEFI_CODE.fd jname=VMNAME
// or `cbsd bconfig' -> bhyve_options -> efi_firmware //

from cbsd.

-c cpus=8,sockets=8,cores=1,threads=1

How many virtual machines has this host already started/running? There may be some limit to CPU overcommitis ?
If this host is already serving virtual machines, will there be a difference if you temporarily turn off some other VM and reduce the number of virtual cores (e.g. cbsd bset vm_cpus=1 )?

from cbsd.

The problem occurs with no other VMs running and with one other VM running.
The host has 12 CPU cores total, the VM has 8 assigned. Even if I set it to one CPU as per your suggestion the problem remains (same error message).

from cbsd.

Most likely, this behavior does not depend on the CBSD. I can only recommend using the bhyve launch line directly (and without using CBSD UEFI firmware). With this check, we will eliminate any suspicion that the CBSD somehow influences this behavior and it makes sense to report the problem to upstream (FreeBSD/bhyve developers)

from cbsd.

Hmm, yeah - seems like a problem unrelated to CBSD at first glance.

Given that other VMs boot properly, is there a way that I can attach the disk of the non-bootable VM to a bootable-VM using CBSD? Or can I mount the *.vhd disk on my host to access files from it?
I don't care much about the VM itself but I'd like to retrieve a file on that VM disk.

from cbsd.

BTW, It seems that I once saw a similar error on AMD processors+bhyve (however, I did not see it on Intel) and this happened when I tried to launch another VM while all the cores were occupied (owned) by other virtual machines.
Of course, we should count on an overcommit in such a situation. But perhaps there is some kind of error in the bhyve hypervisor (or restrictions/limits that I don't know about)

from cbsd.

Attaching the disk to a different VM as per your instructions worked - thank you!

This is on an Intel host (i7-8086). I really have no other problems with it. All other VMs are working. The VM doesn't boot anymore no matter whether other VMs are running or not. The host can be entirely in idle, completely busy or anything inbetween.

The one thing I noticed is that the mountpoint of the VM's dataset is different. It's ~cbsd/vm/bad_vm whereas all other VM datasets are mounted to ~cbsd/jails-data/.

from cbsd.