vcpu_steal

measures actual vcpu steal time from a host that's running as a virtual guest inside of kvm, like a vps from most common providers. we do this by busy-looping a cycle counter on a singular pinned vcpu, trying to find gaps between any iteration that count as a preemption by the hypervisor.

usually the actual steal time is not exposed on cheap vps providers, and even more expensive providers, and the value in /proc/stat which should represent it is often just empty. since we fully pin a single cpu and run a busy loop, the only events that can interrupt us are guest-kernel preemption (timer ticks, irqs) and the host stealing our vcpu. the guest-kernel events are short and usually sub-microsecond or just a little higher, so our 5us threshold for found gaps filters them out as noise, leaving the host preemption as the primary thing producing the longer gaps we measure. this results in an error of +1-2% at the absolute most.

reports the cycle counter gap value, alongside the kernel wall vs. cpu time for comparison. note that the guest kernel doesn't really know what's going on and thinks that any time stolen from a task by the hypervisor belongs to the running task, thus this is only here to contrast the tested value we compute.

building

make

./vcpu_steal

requirements: any c compiler, make

supported platforms

this utility only works on linux! we use sched_setaffinity and cpu_set_t for cpu pinning, both linux-specific, hence the linux requirement.

both x86_64 (uses rdtscp) and aarch64 (uses cntvct_el0) are supported.

usage

./vcpu_steal [seconds] [cpu]

we run for 10 seconds, on cpu 0 by default if arguments not given.

output

a 5-second run on vcpu 0 of a severely oversold vps:

ran at 2.795 GHz, for 117653562 iterations!

kernel says:   3.108% stolen (wall=5000.04ms, cpu=4844.63ms, diff=155.41ms)
we say:        24.093% stolen (gaps=11607, total=1204.67ms, max=78.164ms)

histogram:
    <10us     =   2369
    10-50us   =   7572
    50-100us  =    570
    100-500us =    891
    500us-1ms =     95
    1-5ms     =     73
    5-10ms    =     19
    >=10ms    =     18

the kernel reports only 3% stolen due to it not having access to the paravirtualized steal-time model-specific register.

our detected value reports 24% stolen, with one gap managing to last for a whole 78ms! pretty bad!