bareflank / hypervisor Goto Github PK

When running CPU-Z, Windows freezes on real hardware. This does not seem to occur on VMWare

Most distributions are moving toward deprecation of LSB to use native systemd interfaces. We should update our setup scripts to reflect these changes.

We need to change

if ((pending_debug_exceptions & 0xFFFFFFFFFFFF2FF0) != 0)

to

if ((pending_debug_exceptions & 0xFFFFFFFFFFFEAFF0) != 0)

Section 26.3.1.5 (first item under Pending debug exceptions) states the reserved bits are 63:17, 15, 13, and 11:4.

According to SL.io.50 in the C++ Core Guidelines the use of std::endl should be replaced with '\n' for performance reasons...

Therefore bfendl should be updated...

The ELF error values defined in elf_loader.h are defined as const integer types, and subsequently used within the elf loader (specifically switch-case statements). Relying on non-standard defined behavior is going to hurt portability.

See:
http://stackoverflow.com/a/25597551

Same as issue #215 but for the guest version.

With regard to this issue "C.128: Should destructors be marked "override"?"

And the comment from @gdr-at-ms

hypervisor's repo should be updated... One such instance can be found here

There is an issue with the doxygen script. Once you use it, it removes all of the documentation.

Hi,
I do as the readme.md, but when I make load, it report this error.
Now I found something, does this must pre-install the docker daemon?

best regards
from Xuguo Wang

The first bullet of section 26.3.1.1 of the Intel manual says that bits 0 and 31 of cr0 aren't checked if the "unrestricted guest" VM-execution control is 1. But check_guest_cr0_for_unsupported_bits checks those two bits without checking "unrestricted guest" first. This may cause the check to fail when it really shouldn't. Should check_guest_cr0_for_unsupported_bits be changed to check "unrestricted guest" first?

The reserved bits for the interruptibility state fields in the guest vmcs state are 31:5 (see section 26.3.1.5 of the Intel manual). So the mask in vmcs_intel_x64::check_guest_interruptibility_state_reserved should be 0xFFFFFFE0 rather than 0xFFFFFFFFFFFFFFF0.

From section 26.2.1.1 of the Intel manual:

If the "enable PML" VM-execution control is 1, the "enable EPT" VM-execution control must be 1.

This check, along with the check that the PML address is a 4KB-aligned physical address, is absent in vmcs_intel_x64_check_controls.cpp

The ELF loader does not compile under Windows, and the resulting binary does not execute correctly under Windows.

The compilation errors include:

g++ test.cpp -o .build/test.o -c -std=c++0x  -MT .build/test.o -MMD -MP -MF .build/test.Td -I../include/ -I./ -I../../include/
In file included from ./test.h:25:0,
                 from test.cpp:22:
../../include/unittest.h: In member function ‘virtual int unittest::run()’:
../../include/unittest.h:53:20: error: ‘EXIT_FAILURE’ was not declared in this scope
             return EXIT_FAILURE;
                    ^
../../include/unittest.h:58:20: error: ‘EXIT_FAILURE’ was not declared in this scope
             return EXIT_FAILURE;
                    ^
../../include/unittest.h:66:24: error: ‘EXIT_FAILURE’ was not declared in this scope
                 return EXIT_FAILURE;
                        ^
../../include/unittest.h:76:20: error: ‘EXIT_FAILURE’ was not declared in this scope
             return EXIT_FAILURE;
                    ^
../../include/unittest.h:80:20: error: ‘EXIT_FAILURE’ was not declared in this scope
             return EXIT_FAILURE;
                    ^
../../include/unittest.h:82:16: error: ‘EXIT_SUCCESS’ was not declared in this scope
         return EXIT_SUCCESS;
                ^
../../common/common_target.mk:119: recipe for target '.build/test.o' failed
make[2]: *** [.build/test.o] Error 1
../common/common_subdir.mk:38: recipe for target 'all' failed
make[1]: *** [all] Error 2
common/common_subdir.mk:38: recipe for target 'all' failed
make: *** [all] Error 2

From section 26.2.1.3 from the Intel manual:

If the interruption type is software interrupt, software exception, or privileged software exception, the
VM-entry instruction-length field is in the range 0–15. A VM-entry instruction length of 0 is allowed only if IA32_VMX_MISC[30] is read as 1; see Appendix A.6.

Should we read the MISC msr to make sure that a length of 0 isn't allowed? Currently check_control_event_injection_instr_length_checks throws a logic_error if the length is 0. I noticed that IA32_VMX_MISC_MSR isn't in intrinsics_intel_x64.h so it would need to be added.

The ELF loader is not compiling because of the name change to use "class" as this is a keyword.

../include/elf_loader.h:328:31: error: declaration of anonymous class must be a definition
ei_class_to_str(unsigned char class);

We need parentheses around the first two operands in each assignment statement to pat0 through pat7 in check_host_verify_load_ia32_pat. This will produce the intended behavior in the check.

Hello,

I've seen that VMM-Isolation is on the complete notes of this project but i'm not sure I understood what it means.

Does it means that the VMM is self protected? meaning that if i'm installing it on windows system no other code can interfer with the hypervisor? (etc take it off, writing to it's memory - DMA, editing the VMCS and more..)

Is such behavior even possible on Host-Only Hypervisors? (given the required support - VT-d for example)

Thank you for your help!

Guys,

I'm trying to get things rolling on openSUSE Tumbleweed. So far so good, but during build I'm getting this error:

/home/harry/dev/hypervisor/bfelf_loader/test/test.cpp:263:20: error: could not convert ‘false’ from ‘bool’ to ‘std::shared_ptr’

Could you help out?

We should use SEGMENT_ACCESS_RIGHTS_RESERVED rather than the hardcoded 0x0F00 in vmcs_intel_x64::check_guest_ldtr_access_rights_reserved_must_be_0().

In vmcs_intel_x64::check_guest_ldtr_access_rights_remaining_reserved_bit_0(), the "remaining" reserved bits are 31:17 so we should have

if ((ldtr_access & 0xFFFE0000) != 0)
    throw std::logic_error("guest ldtr access rights bits 31:17 must be 0 if ldtr is usable");

rather than

if ((ldtr_access & SEGMENT_ACCESS_RIGHTS_RESERVED) != 0)
    throw std::logic_error("ldtr access rights reserved bits must be 0 if ldtr is usable");

In section 26.3.1.5, the third dash under the "Pending debug exceptions" bullet describes a check that occurs if RTM (bit 16 of pending debug exceptions) is 1. Do we need to implement this check?

• The VMCS_HOST_IA32_SYSENTER_EIP field is read rather than VMCS_HOST_IA32_SYSENTER_ESP in check_host_ia32_sysenter_esp_canonical_address.
• The VMCS_HOST_FS_BASE field is read rather than VMCS_HOST_TR_BASE in
  check_host_tr_canonical_address.

After successfully building the cross-compiler with 'setup-debian.sh', I tried to build the hypervisor with make, but the compilation failed.

Would appreciate any help in fixing this compilation problem. Thanks!

========= [LOGS]

/Research/hypervisor > uname -a
Linux desktop 3.19.0-32-generic #3714.04.1-Ubuntu SMP Thu Oct 22 09:41:40 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

~/Research/hypervisor > gcc --version
gcc (GCC) 5.3.0

~/Research/hypervisor> ./tools/scripts/setup-debian.sh
... (successful... truncated) ...
cp -Rf /home/jamint/opt/cross/x86_64-elf/lib/libc++.so.1.0 /home/jamint/Research/hypervisor/bfvmm/bin/cross/libc++.so
touch completed_install_libcxx
set +x

~/Research/hypervisor> make
... (truncated) ...
/home/jamint/Research/hypervisor/bfvmm/src/vcpu/test
g++ .build/native/test.o .build/native/test_vcpu.o .build/native/test_vcpu_factory.o -o ../bin/native/test -L../bin/native -L../../vmxon/bin/native -L../../vmcs/bin/native -L../../debug_ring/bin/native -L../../intrinsics/bin/native -L../../exit_handler/bin/native -lvcpu -lvmxon -lvmcs -ldebug_ring -lintrinsics -lexit_handler
../../vmxon/bin/native/libvmxon.so: undefined reference to memory_manager::instance()' ../bin/native/libvcpu.so: undefined reference toserial_port_x86::~serial_port_x86()'
../bin/native/libvcpu.so: undefined reference to serial_port_x86::serial_port_x86(unsigned char, unsigned int, unsigned char, PARITY_MODE, unsigned char)' ../bin/native/libvcpu.so: undefined reference toserial_port_x86::write(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
collect2: error: ld returned 1 exit status
make[4]: *** [../bin/native/test] Error 1
make[3]: *** [test] Error 2
make[2]: *** [vcpu] Error 2
make[1]: *** [src] Error 2
make: *** [bfvmm] Error 2

When compiling on Ubuntu LTS server 12.04, you get the following warnings:

gcc elf_loader.c -o .build/elf_loader.o -c  -fpic -DENABLE_ELF_DEBUGGING -MT .build/elf_loader.o -MMD -MP -MF .build/elf_loader.Td -I./ -I../include/
elf_loader.c: In function ‘elf_file_print_header’:
elf_loader.c:693:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_addr’ [-Wformat]
elf_loader.c:694:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_off’ [-Wformat]
elf_loader.c:694:5: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 4 has type ‘elf64_off’ [-Wformat]
elf_loader.c:695:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_off’ [-Wformat]
elf_loader.c:695:5: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 4 has type ‘elf64_off’ [-Wformat]
elf_loader.c: In function ‘elf_print_section_header’:
elf_loader.c:880:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_addr’ [-Wformat]
elf_loader.c:881:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_off’ [-Wformat]
elf_loader.c:882:5: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_xword’ [-Wformat]
elf_loader.c:885:5: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_xword’ [-Wformat]
elf_loader.c:886:5: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_xword’ [-Wformat]
elf_loader.c: In function ‘elf_print_sym’:
elf_loader.c:1314:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_addr’ [-Wformat]
elf_loader.c: In function ‘elf_print_relocation’:
elf_loader.c:1572:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_addr’ [-Wformat]
elf_loader.c:1572:5: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 4 has type ‘elf64_xword’ [-Wformat]
elf_loader.c: In function ‘elf_print_relocation_addend’:
elf_loader.c:1607:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_addr’ [-Wformat]
elf_loader.c:1607:5: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 4 has type ‘elf64_xword’ [-Wformat]
elf_loader.c:1607:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 5 has type ‘elf64_sxword’ [-Wformat]
elf_loader.c: In function ‘elf_print_program_header’:
elf_loader.c:1954:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_off’ [-Wformat]
elf_loader.c:1955:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_addr’ [-Wformat]
elf_loader.c:1956:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_addr’ [-Wformat]
elf_loader.c:1957:5: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_xword’ [-Wformat]
elf_loader.c:1958:5: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_xword’ [-Wformat]
elf_loader.c:1959:5: warning: format ‘%llX’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘elf64_xword’ [-Wformat]
gcc -o ../bin/libelf_loader.so .build/elf_loader.o  -shared

Section 26.2.1.1, Volume 3C of the Intel manual states in the eighth bullet down,

If the "use TPR shadow" VM-execution control is 1 and the "virtual-interrupt delivery" VM-execution control is 0, bits 31:4 of the TPR threshold VM-execution control field must be 0.

The next bullet states

The following check is performed if the "use TPR shadow" VM-execution control is 1 and the "virtualize APIC accesses" and "virtual-interrupt delivery" VM-execution controls are both 0: the value of bits 3:0 of the TPR threshold VM-execution control field should not be greater than the value of bits 7:4 of VTPR (see Section 29.1.1).

The function that implements the check is vmcs_intel_x64::check_control_tpr_shadow_and_virtual_apic. It currently checks bits 31:4 of the TPR threshold only if TPR shadow is 1 and virtual-interrupt delivery is 1. Shouldn't the check occur when TPR shadow is 1 and virtual-interrupt delivery is 0?

Likewise bits 3:0 of TPR threshold are checked against bits 7:4 of VTPR only if TPR shadow, virtual-interrupt delivery and virtualize APIC accesses are all 1 when the manual states the latter two should be 0.

Same issue as in #227..the code

if (bs == 1 && tf == 0 && btf == 1)
    throw std::logic_error("pending debug exception bs must be 0 if "
                           "rflags tf is 0 and debugctl btf is 1");

should be

if (bs != 0 && tf == 0 && btf != 0)
    throw std::logic_error("pending debug exception bs must be 0 if "
                           "rflags tf is 0 and debugctl btf is 1");

Unless you have GCC >= 4.7, it doesn't recognize -std=c++11. Instead, you need to use -std=c++0x

Applications like DbgView and OneDrive are crashing as soon as you launch the hypervisor. When they crash, Windows reports that the application could not access a memory location at 0x00000000FFFFFFFF. Any attempt to run a 32bit application while the hypervisor is running also results in a crash

The last two checks (bullet points) in section 26.2.1.3 of the Intel manual are

If the processor is not in SMM, the “entry to SMM” and “deactivate dual-monitor treatment” VM-entry controls must be 0.

and

The “entry to SMM” and “deactivate dual-monitor treatment” VM-entry controls cannot both be 1.

seem to be missing from vmcs_intel_x64_check_controls.cpp. Should these be implemented as well?

There is a bug in the VCPU manager that deadlocks the hypervisor if you attempt to print from a constructor or destructor.

Would you like to replace more defines for constant values by enumerations to stress their relationships?

• ENTRY_ERROR_…
• VMM_…

The emulation of the new operator to throw bad alloc and provide aligned memory for the VMCS is causing issues with Coverity. To be honest, Clang-Tidy and some of the -W warnings were also having issues with the code, so I think it makes sense to see if there is a different approach that these checks aren't as finicky about.

The functions from vmcs_intel_x64::check_guest_cs_access_rights_type to vmcs_intel_x64::check_guest_gs_access_rights_remaining_reserved_bit_0 need to check if virtual-8086 mode is disabled before the check occurs. See section 26.3.1.2 (second bullet of the seventh bullet)

It appears that #include <driver_entry_interface.h> is missing from debug.h, and thus the linux driver is not compiling.

In the next couple of days we will be implementing a VMCall interface that can be used by users of Bareflank to quickly and easily talk to the hypervisor. Our goal here is to keep this as simple as possible so that it can be extended in the future without causing issues with API changes

Since we intended to work with ARM in the future, we cannot VMCall from userspace as this is not supported on ARM (sad story). Instead, we will provide an IOCTL interface so that userspace applications like BFM can VMCall to the hypervisor.

One issue with supporting both ARM and Intel is the total number of registers. To support this in a sane way, I propose we use the ABIs are our guide. On Intel, the System V ABI defines the order of each register. We can define a struct that has 32 registers (i.e. r0 -> r31) and the ABI will tell use which register in our struct belongs to which physical register (i.e. r0 -> rax, r1 -> rdx, etc...). From here, the driver can use this information to produce a VMCall, and the only "arch" specific code will be the register decoding. This should produce a simple VMCall interface that can be used generically.

In addition to register decoding, Bareflank will need to reserve a register for defining the type of VMCall that is occurring. At the moment, I envision three different types of VMCalls, but this could be expanded in the future:

• raw register VMCall (minus reserved registers)
• data VMCall
• unit test VMCall

The raw register VMCall would do nothing more than send raw register values to the hypervisor. This is likely to be the most used VMCall. The data VMCall would send in / out pages to the hypervisor to handle. The unit test VMCall can be used to unit test code inside the hypervisor (e.g. libcxx). To ensure future compatibility, I would like to reserve r0 and r1 (rax / rdx on Intel) for Bareflank, and extensions can use all of the registers for whatever they want.

Thoughts?

To start, I would like to add raw register and unit test support to BFM, and the data VMCall can be added later (its a little more complicated as it has to map / unmap memory, and we need a clean way to send binary / string data via BFM).

Way back when we first started the project, we put together some cross compiler documentation that was to be used by developers to hand create the cross compilers. This process is scripted now, and would be insane to document now that we also support docker, and will in the next coming weeks support Clang//LLVM.

So the question is... does anyone have an issue with me deleting this documentation? It will always be available in the version 1.0 tag (even though it is out-of-date in that tag too) just in case people want to refer back to it.

Extra null pointer checks are not needed in the function "driver_entry_ut::fini".

The use of "auto" is not supported with GCC <= 5.0 as glibc does not implement the move constructor needed to support this keyword. This results in the following error:

test.cpp:33:74: error: use of deleted function ‘std::basic_ifstream::basic_ifstream(const std::basic_ifstream&)’
     auto dummy1_ifs = std::ifstream(c_dummy1_filename, std::ifstream::ate);

The check

if ((eptp & 0xFFFF000000000000) != 0 || (eptp & 0x0000000000000F80) != 0)
       throw std::logic_error("bits 11:7 and 63:48 of the eptp must be 0");

in vmcs_intel_x64::check_control_enable_ept_checks essentially hard-codes the physical address width. Would it be better to call is_physical_address_valid(eptp) instead?

cd ~/
git https://github.com/bareflank/hypervisor.git
cd ~/hypervisor
git checkout -b v1.0.0

./tools/scripts/setup_.sh

Here the git command should be:
git clone https://github.com/bareflank/hypervisor.git

BTW how could I send the patch to this project?

In section 26.3.1.1 of the Intel manual, the last bullet describes a check if "load IA32_BNDCFGS" is set. Do we need to implement this check?

The Windows driver does not unload properly. If you load the Windows driver, start Bareflank, and then unload the Windows Driver, the system is left in an unrecoverable state because Bareflank is not being shutdown.

if (!is_ldtr_usable())
    return;

needs to be added at the beginning of vmcs_intel_x64::check_guest_ldtr_granularity() per section 26.3.1.2 of the Intel manual.

Windows appears to be broken in a couple of different ways:

• Cygwin just did an update which causes more of the GSL code to complain during compilation. We actually contributed strnlen functionality to the GSL specifically for cygwin and it appears that we will need to do the same thing for wcsnlen
• Somehow the IOCTL for write got reversed in our Clang Tidy PR, and as a result, BFM it unable to provide the driver with module information, resulting in BFM failing to load the hypervisor. This is a simple fix that hopefully will not happen again
• The TLS work appears to be killing Windows (BSOD). This doesn't seem to happen as much on real hardware, which allowed this to sneak though during testing as it's a race condition. Specifically, the us of FS seems to make Windows very mad if an interrupt fires while the hypervisor is being loaded. We are investigating different approaches to solve this which include: don't use FS, which would break support for thread_storage in the future, or disable interrupts while loading the hypervisor which might be the better approach (if it works).

In either case, we hope to have Windows support fixed in the very near future.

The Linux driver does not compile on older kernels because cr4_init_shadow does not exist. It appears that this optimization was added in Linux 4.0

TL;DR

I would like to add a modified version of the following JSON library to Bareflank:

https://github.com/nlohmann/json

Description

Specifically, we have a couple of reasons why we need some way to format text.

• BFM currently has it's own, custom module file format. This format will need to be modified to something more complicated as we will need a way to specify the default amount of RAM to provide the hypervisor. Ultimately this would mean creating our own version of JSON just to support this. In the future this will only get more complicated as we add support for defining which vCPUs to start, etc... JSON will make easy work of this task
• The extended APIs will support JSON VMCalls to communicate with the hypervisor. This will also be used by the HyperKernel repo for sending commands for starting / stoping guest VMs including defining things like the amount of RAM, the total number of CPUs and which image file to load. This interface will be similar to QMP. Providing base support in Bareflank will provide basic support for this same feature for others, as well as lay the foundation for the Extended APIs and the HyperKernel
• A file format will be needed for our "XSM" wanna-be implementation. JSON is fast, and provides a simple means for defining tree structures that would certainly be needed to support this in the VMM. It's a logical format for denying access to hypercalls.

The only modification that would be made to the library is the removal of double/float (signed and unsigned integers are already supported). I already tested this approach and it works great (even has compile time checks).

Also, this library would not be the first library to be added to Bareflank as we already have support for Hippomocks and the Guidelines Support Library.

Just to stress the point... the BFM file format will have to change to something else to support defining more than just filenames for the modules. Since we already plan to support JSON for other things in extension repos, it makes no sense to me, IMO, to use anything other than JSON for the file format for BFM.

Thoughts?

• is_ds_usable should be changed to is_ss_usable in check_guest_ss_base_upper_dword_0
• need to add a throw in check_guest_valid_activity_state and check_guest_activity_state_not_hlt_when_dpl_not_0.

The doxygen comments are currently in the .cpp, while clutters the implementation, while leaving the header relatively useless. Instead, the code should be documented in the header files.

vmcs_intel_x64::check_guest_valid_activity_state

and

vmcs_intel_x64::check_guest_activity_state_not_hlt_when_dpl_not_0

are called twice in vmcs_intel_x64::checks_on_guest_non_register_state.

We need to check

if (sti && mov_ss)
    throw ...

rather than

if (sti == 1 && mov_ss == 1)
    throw ...

since for that check we just care about whether the individual bits are set or not.

The VMM is not starting with an error of:

ERROR: ----------------------------------------
ERROR: - General Exception Caught             
ERROR: ----------------------------------------
vmxon capabilities not supported:
    - vmx_basic_msr: 0xd8100000000001
    - 55: 0x37
    - func: check_vmx_capabilities_msr
    - line: 108