Skip to content

Add support for qemu virtual machines using KVM#423

Draft
IkerGalardi wants to merge 7 commits intoseL4:mainfrom
IkerGalardi:kvm-support
Draft

Add support for qemu virtual machines using KVM#423
IkerGalardi wants to merge 7 commits intoseL4:mainfrom
IkerGalardi:kvm-support

Conversation

@IkerGalardi
Copy link

@IkerGalardi IkerGalardi commented Feb 26, 2026

This PR adds support for running microkit based operating systems using qemu with KVM enabled.

The main issue with the qemu platform with KVM enabled is that it drops the kernel in EL1 instead of the expected EL2. This PR builds a custom kernel for this platform with hypervisor support disabled.

The loader seems to work fine, but jumping to the kernel causes a instruction abort exception. The next is the log:

LDR|ERROR: loader trapped exception: Synchronous (Current Exception level with SP_ELx)
    esr_el1: 0x0000000086000004
    ec: 0x00000021 (Instruction Abort taken without a change in Exception level)
    il: 0x0000000000000001
    iss: 0x0000000000000004
    far: 0xffffff8040000000
    reg: 0x00000000: 0x0000000040041000
    reg: 0x00000001: 0x0000000000211274
    reg: 0x00000002: 0x0000000000000000
    reg: 0x00000003: 0x0000000000000000
    reg: 0x00000004: 0xffffff8040000000
    reg: 0x00000005: 0xffffffffffffffff
    reg: 0x00000006: 0x0000000000c5183d
    reg: 0x00000007: 0x0000000000001124
    reg: 0x00000008: 0x0000001485100510
    reg: 0x00000009: 0x0000000000000000
    reg: 0x0000000a: 0x0000000000007830
    reg: 0x0000000b: 0x000000000000000d
    reg: 0x0000000c: 0x0000000000000030
    reg: 0x0000000d: 0x0000000000000030
    reg: 0x0000000e: 0xffffff8040000000
    reg: 0x0000000f: 0x0000000000000030
    reg: 0x00000010: 0x0000000000000030
    reg: 0x00000011: 0x0000000070003298
    reg: 0x00000012: 0x0000000000000000
    reg: 0x00000013: 0x0000000070004000
    reg: 0x00000014: 0x00000000700032f0
    reg: 0x00000015: 0x0000000070003550
    reg: 0x00000016: 0x0000000000000100
    reg: 0x00000017: 0x000000007000c008
    reg: 0x00000018: 0x0000000070003510
    reg: 0x00000019: 0x00000000700034f8
    reg: 0x0000001a: 0x000000007000c008
    reg: 0x0000001b: 0x0000000070005f70
    reg: 0x0000001c: 0x0000000000000000
    reg: 0x0000001d: 0x0000000000000000
    reg: 0x0000001e: 0x0000000000000000
    reg: 0x0000001f: 0x0000000000000000

This board differs from the standard qemu_virt_aarch64 board in that
the entry point is in EL2 instead of EL1. This is necessary due to the
lack of nested virtualization support on the KVM subsystem.

Signed-off-by: IkerGalardi <contacto.ikergalardi@gmail.com>
Previously the defaults where applied ALWAYS, meaning that if some
board tried to specialize a kernel building parameter that the default
configuration did, it would get overwriten by the default
configuration. This patch applies the specialization to the default
configuration instead of doing it the other way around.

Signed-off-by: IkerGalardi <contacto.ikergalardi@gmail.com>
Signed-off-by: IkerGalardi <contacto.ikergalardi@gmail.com>
EL1 software can not access interrupt group registers on the GIC
distributor.

Signed-off-by: IkerGalardi <contacto.ikergalardi@gmail.com>
@IkerGalardi IkerGalardi force-pushed the kvm-support branch 2 times, most recently from a6c4c36 to fe30316 Compare March 9, 2026 14:31
The el1_mmu_disable function at the start pushed both x29 and x30
(frame pointer and link register) into the stack. But when ending the
function, before the RET instruction, it poped 4 values instead of
just the pushed 2, eating the stack frame of the parent scope.

Probably copy-pasted from the el2_mmu_disable, which pushes x27, x28,
x29 and x30 and pops all of them. Both functions are identical, but
el1 version for some reason does not push x27 and x28.

Signed-off-by: IkerGalardi <contacto.ikergalardi@gmail.com>
There was a missing ldp instruction poping the x27 and x28 values from
the stack.

Signed-off-by: IkerGalardi <contacto.ikergalardi@gmail.com>
@Ivan-Velickovic
Copy link
Collaborator

Ivan-Velickovic commented Mar 23, 2026

@IkerGalardi you should include the full logs for the loader, it's not clear to me that the loader is working properly. There is most likely an issue with the initial virtual address space that is setup by the loader.

I also would imagine that the smc calls in the loader would need to be replaced with hvc calls when using KVM, as I had to do something similar in the past.

@IkerGalardi
Copy link
Author

IkerGalardi commented Mar 23, 2026

Here are the full logs:

LDR|INFO: disabling MMU (if it was enabled)
LDR|INFO: PSCI version is 1.1
LDR|INFO: altloader for seL4 starting
LDR|INFO: flags:
LDR|INFO: kernel:      entry:   0xffffff8040000000
LDR|INFO: root server: physmem: 0x0000000040241000 -- 0x00000000406cc000
LDR|INFO:              virtmem: 0x0000000000200000 -- 0x000000000068b000
LDR|INFO:              entry  : 0x0000000000211274
LDR|INFO: region: 0x00000000   addr: 0x0000000040000000   size: 0x0000000000241000   offset: 0x0000000000000000   type: 0x0000000000000001
LDR|INFO: region: 0x00000001   addr: 0x0000000040241000   size: 0x00000000000086e8   offset: 0x0000000000241000   type: 0x0000000000000001
LDR|INFO: region: 0x00000002   addr: 0x000000004024a6e8   size: 0x0000000000015428   offset: 0x00000000002496e8   type: 0x0000000000000001
LDR|INFO: region: 0x00000003   addr: 0x0000000040260b10   size: 0x00000000000100b0   offset: 0x000000000025eb10   type: 0x0000000000000001
LDR|INFO: region: 0x00000004   addr: 0x0000000040271000   size: 0x000000000005b3cc   offset: 0x000000000026ebc0   type: 0x0000000000000001
LDR|INFO: region: 0x00000005   addr: 0x00000000402cd000   size: 0x00000000003ff000   offset: 0x00000000002c9f8c   type: 0x0000000000000001
LDR|INFO: copying region 0x00000000
LDR|INFO: copying region 0x00000001
LDR|INFO: copying region 0x00000002
LDR|INFO: copying region 0x00000003
LDR|INFO: copying region 0x00000004
LDR|INFO: copying region 0x00000005
LDR|INFO|CPU0: active CPUs to start: 0x00000001
LDR|INFO|CPU0: enabling MMU
LDR|INFO|CPU0: CurrentEL=EL1
LDR|INFO|CPU0: enabling MMU
LDR|INFO|CPU0: jumping to kernel

LDR|ERROR: loader trapped exception: Synchronous (Current Exception level with SP_ELx)
    esr_el1: 0x0000000086000004
    ec: 0x00000021 (Instruction Abort taken without a change in Exception level)
    il: 0x0000000000000001
    iss: 0x0000000000000004
    far: 0xffffff8040000000
    reg: 0x00000000: 0x0000000040041000
    reg: 0x00000001: 0x0000000000211274
    reg: 0x00000002: 0x0000000000000000
    reg: 0x00000003: 0x0000000000000000
    reg: 0x00000004: 0xffffff8040000000
    reg: 0x00000005: 0xffffffffffffffff
    reg: 0x00000006: 0x0000000000c5183d
    reg: 0x00000007: 0x0000000000001124
    reg: 0x00000008: 0x0000001485100510
    reg: 0x00000009: 0x0000000000000000
    reg: 0x0000000a: 0x0000000000007830
    reg: 0x0000000b: 0x000000000000000d
    reg: 0x0000000c: 0x0000000000000030
    reg: 0x0000000d: 0x0000000000000030
    reg: 0x0000000e: 0xffffff8040000000
    reg: 0x0000000f: 0x0000000000000030
    reg: 0x00000010: 0x0000000000000030
    reg: 0x00000011: 0x0000000070003298
    reg: 0x00000012: 0x0000000000000000
    reg: 0x00000013: 0x0000000070004000
    reg: 0x00000014: 0x00000000700032f0
    reg: 0x00000015: 0x0000000070003550
    reg: 0x00000016: 0x0000000000000100
    reg: 0x00000017: 0x000000007000c008
    reg: 0x00000018: 0x0000000070003510
    reg: 0x00000019: 0x00000000700034f8
    reg: 0x0000001a: 0x000000007000c008
    reg: 0x0000001b: 0x0000000070005f70
    reg: 0x0000001c: 0x0000000000000000
    reg: 0x0000001d: 0x0000000000000000
    reg: 0x0000001e: 0x0000000000000000
    reg: 0x0000001f: 0x0000000000000000

The issue could be the page tables, but still, the kernel entry being at 0xffffff8040000000 feels kinda strange.

About the smc instruction, secure monitor calling convention states that hypervisors catch and emulate those smc calls, so it should be fine™.

@Indanz
Copy link

Indanz commented Mar 23, 2026

The issue could be the page tables, but still, the kernel entry being at 0xffffff8040000000 feels kinda strange.

On non-HYP, the kernel uses the upper address range for its page table configured by TTBR1_EL1. It uses TTBR0_EL1 to configure the user space tables. For HYP, when running in EL2, there is no corresponding TTBR1_EL2, only a TTBR0_EL2. However, it is not shared with user space like TTBR0_EL1 and TTBR1_EL1 are.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants