no pcie hotplug support for q35/ovmf about edk2 HOT 26 CLOSED

It works on my system (see below for details).

Which version (or git hash) of qemu are you using?
Are you trying to use pci-assign, or the vfio-pci?
Which build/branch of OVMF?
How are you setting up the pci card on the host for passthrough?
What parameters are you using to run qemu?

My config details for reference

qemu 2.4.50, OVMF built from UDK2014.SP1 branch, qemu invocation:

qemu-system-x86_64 -name uefi -M q35 -m size=16384 -cpu host \
  -drive unit=0,if=pflash,format=raw,file=/home/bcran/workspace/edk2/Build/OvmfX64/RELEASE_GCC49/FV/OVMF.fd \
  -device ahci,id=ahci0 -drive file=/home/bcran/uefi.img,if=ide,id=drive-sata0-0-0,index=0,media=disk,format=raw \
  -nodefaults -realtime mlock=off \
  -enable-kvm -s -S \
  -rtc base=utc -monitor stdio \
  -vga cirrus -vnc :0 \
  -chardev pty,id=debugcon -device isa-debugcon,chardev=debugcon,iobase=0x402 \
  -device vfio-pci,host=01:00.0,id=mypcicard,romfile=/home/bcran/workspace/edk2/MyPciCard/stage/DEBUG_GCC49/X64/my_pci_card.rom

From the UEFI Shell in the guest:

Shell> pci
Seg Bus Dev Func
--- --- --- ----    
...
00   00  03  00 ==> Mass Storage Controller - Other mass storage controller
        Vendor 0x1234 Device 0x1000 Prog Interface 0

from edk2.

I do not mean pci passthrough support. I mean pci express hotplugging support, when you add a virtual device after the machine boots, and the system detects it automatically. It probably requires a root port to be added. It works properly on seabios, does not work on ovmf.

from edk2.

Hi,
Indeed, you cannot hot-plug PCIe devices into a Root Complex, you can do it on a root port or a downstream switch port. This is because of the SHPC controllers that are present only on ports. And it would not be possible also in the real world, as far as I know, since inside Root Ports we have only Integrated End Points.

I think this issue can be closed.

Thanks,
Marcel

from edk2.

from edk2.

Hi,
I am lost. MMCONFIG table is not related to the PCI express native hotplug. The OS detects the hotplug by the means of interrupt(s) sent by the SHPC controller following the hot-plug sequence:
(the way I know it, at least )

• The operator installs the card and secures the MRL
• The MRL sensor(if present), causes a system interrupt to be sent to the
  Root Complex signaling the SHPC driver that the latch is closed
• User initiates hot add using Attention Button or UI
• The SHPC driver causes the slot's Power Indicator to blink for 5 seconds,
  this being the window to cancel the operation and also the time for the
  software to validate the request.
• The SHPC driver issues a request to the hot plug controller to power on the slot
• Once link training is complete, the Hot plug driver causes re-enumeration of the slot bus;
  The hot added device is found, configured and driver is loaded.

What am I missing?

Thanks,
Marcel

from edk2.

from edk2.

I understand. Lets put aside the mmconfig table and look at the missing bus rescan issue.
I tried it on my machine with latest QEMU/OVMF and a fedora guest and is working just fine.
Can you please share:

• QEMU version
• QEMU command line
• Guest OS
• OVMF version (latest?, special build commands?)
• how you hot-add a device

I'll try to reproduce the problem.

Thanks,
Marcel

from edk2.

from edk2.

Sure, please do that and let us know.
If the problem still exists, please provide all the information above and I'll try to reproduce.

Thanks,
Marcel

from edk2.

linux kernel version on host: 4.4, linux kernel version on guest: 4.2
(distribution is archlinux, arch livecd, but older than latest).
Qemu version: 2.5, ovmf version, newest git, kvm is enabled.
Qemu command line:
qemu-system-x86_64 -machine q35,accel=kvm -cdrom Pobrane/arch.iso
-nographic -pflash OVMF.fd -device ioh3420,port=1,chassis=1,id=port1
-device ioh3420,port=2,chassis=2,id=port2 -m 512
You can try
device_add rtl8139,bus=port1
and it will work, but os will not auto pick the device up until manual
bus rescan.

from edk2.

Thanks. And exactly the same configuration with SeaBIOS it will work? I'll try it and get back to you.

from edk2.

from edk2.

I succeeded to reproduce the problem and (I think) I found the root cause. Funny enough, it is because the missing MMCONFIG even if it shouldn't be, so you were right. Thank you for your patience to report the issue!

Let's dive into the details.

• Why it should work? All the registers required to control the "hot-plugging" are available through PCI Express Capability which resides in the legacy PCI configuration space (See PCi Express spec 7.8.10, 7.8.11) , not in the Extended Configuration space which is available only through MMConfig.
• Why it doesn't work? Linux Kernel requires all [PCI_SEGMENT, PCI_EXT_CONFIG, ASPM, MSI] features to be available in order to request OS control from ACPI for native (PCIe) hotplug. (see "negotiate_os_control" function in drivers/acpi/pci_root.c).
  Since OVMF does not init Q35_HOST_BRIDGE_PCIEXBAR (and reserve the memory space ?) -> QEMU will build a dummy ACPI table "QEMU" instead of MCFG" -> guest Linux will not enable PCI_EXT_CONFIG -> will not even ask for native PCIe hotplug. Q35 does not support legacy ACPI hotplug, so no fall back is available.

I'll see if I can implement MMConfig for Q35 that will resolve the problem.
I am still trying to remember what configuration I used that it did work, maybe a guest with an older kernel ...

Thanks,
Marcel

from edk2.

from edk2.

Yes, it should be enough, however in OVMF implementations tend to be more complicated than in the straight forward SeaBIOS :)

from edk2.

@marcel-apf: thanks for the awesome analysis!

I think this functionality overlaps with the current PciHostBridgeLib work. Namely, OVMF uses 0xCF8/0xCFC to access PCI config space even on Q35. What I have in mind is to create a PciLib instance for OVMF that can dynamically select between ioport and MMCFG config space access.

In practice this would be a library that unifies the internals of the following two libraries:

• MdePkg/Library/BasePciLibCf8/BasePciLibCf8.inf
• MdePkg/Library/BasePciLibPciExpress/BasePciLibPciExpress.inf

The type of the QEMU machine (i440fx vs. Q35) is already recognized and stored in a dynamic PCD -- grep OvmfPkg for PcdOvmfHostBridgePciDevId and OVMF_HOSTBRIDGE_DID. In the library constructor function, the PCD should be interrogated, stored in a static variable, and that variable should be used for all future branching.

Now, I think this new unified / dynamic library instance would not be appropriate for all phases of the firmware. It should only be used in module types that come strictly after PEI. This will require some customization in the various LibraryClasses sections in the *.dsc files under OvmfPkg.

Why I say this overlaps with the current PciHostBridgeLib work: because that driver actually depends on the PciLib instance we specify (not directly, but indirectly, via PciSegmentLib). Also, the core PciHostBridgeDxe driver must be told about the size of the PCI config space, and we'll have to keep that in sync with the PciLib backend that will be active.

... Can you help me understand how the host bridge exbar is supposed to be programmed? (This obviously has to occur through 0xCF8 / 0xCFC first.) I think that this programming and the associated memory reservation should happen in PlatformPei. In OvmfPkg/PlatformPei/Platform.c. The MiscInitialization() function sets PcdOvmfHostBridgePciDevId. Around those parts we should enable the exbar, and also set PcdPciExpressBaseAddress.

MemMapInitialization() should be kept in sync.

I'd be happy to prototype this in a small series (but it will only make sense on top of the current PciHostBridgeLib work). Can you educate me how the exbar is supposed to be programmed? And, can we set it at a fixed address perhaps? (That would be best.) As far as I can see, the QEMU-default [2816MB, 3072MB) range would be optimal. OVMF already takes care, on Q35, to start the 32-bit PCI MMIO window at 3GB.

Thanks!

from edk2.

Sure,
It is a fixed address, so: from SeaBIOS, take it like a pseudo-code.
All you have to do is to configure the BAR for QEMU, indeed with the legacy IO registers.

#define Q35_HOST_BRIDGE_PCIEXBAR_ADDR 0xb0000000
#define Q35_HOST_BRIDGE_PCIEXBAR 0x60
#define Q35_HOST_BRIDGE_PCIEXBAR_SIZE (256 * 1024 * 1024)

u64 addr = Q35_HOST_BRIDGE_PCIEXBAR_ADDR;
u32 size = Q35_HOST_BRIDGE_PCIEXBAR_SIZE;

/* setup mmconfig */
u32 upper = addr >> 32;
u32 lower = (addr & 0xffffffff) | Q35_HOST_BRIDGE_PCIEXBAREN;
pci_config_writel(bdf, Q35_HOST_BRIDGE_PCIEXBAR, 0);
pci_config_writel(bdf, Q35_HOST_BRIDGE_PCIEXBAR + 4, upper);
pci_config_writel(bdf, Q35_HOST_BRIDGE_PCIEXBAR, lower);

e820_add(addr, size, E820_RESERVED);

Nothing more.
Is the above enough?

Thanks,
Marcel

from edk2.

The bdf is of the device: PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_Q35_MCH

from edk2.

This is perfect. Thanks.

from edk2.

Posted patches:

http://thread.gmane.org/gmane.comp.bios.edk2.devel/8707

from edk2.

Posted v2 series:

http://thread.gmane.org/gmane.comp.bios.edk2.devel/8871

from edk2.

@webczat Please consider testing the v2 series as well (using the same method as for v1); I think @jljusten might delay his review of v2 until you report back. Thanks.

from edk2.

from edk2.

@webczat sorry for pushing you, but I have another series on top of this work (for #59), plus a large feature in planning (virtio 1.0 support) that depends on getting forward with these PCI bits. Please try to test the v2 series today, if you can. Thank you.

from edk2.

The series at 7e869ee..7daf240 fixes this issue. (The directly related commits have been automatically listed by github above.) Closing.

from edk2.

This (closed) item has been manually migrated to
https://tianocore.acgmultimedia.com/show_bug.cgi?id=75

from edk2.