Skip to content

Instantly share code, notes, and snippets.

@yskzalloc

yskzalloc/Makefile

Last active August 27, 2026 23:31
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save yskzalloc/5acfdef88e6354dc047604c9883faa8b to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save yskzalloc/5acfdef88e6354dc047604c9883faa8b to your computer and use it in GitHub Desktop.
Download ZIP
cacheinfo-oob-repro
Raw
README.md

Userspace reproducer: slab-out-of-bounds write in populate_cache_leaves()

A write(2) to /sys/devices/system/cpu/cpuN/online makes the kernel perform an 8-byte out-of-bounds write in arch/x86/kernel/cpu/cacheinfo.c:__cache_cpumap_setup().

This directory contains two programs:

program needs root what it does
cacheinfo-oob-predict no Reads CPUID on every online CPU, replays the kernel's own sibling test, and says whether this machine can hit the out-of-bounds write. Writes nothing.
cacheinfo-oob-trigger yes Brings a CPU online, so the buggy path runs, and captures the KASAN report from /dev/kmsg. Prints the predicted offsets before triggering, then compares them with what KASAN reports. 
make            # or: make static, for dropping into a guest image
./cacheinfo-oob-predict
sudo ./cacheinfo-oob-trigger --all

Exit codes: cacheinfo-oob-predict returns 0 affected, 1 not affected, 2 indeterminate (some CPU is offline, so its leaf count cannot be read yet). cacheinfo-oob-trigger returns 0 reproduced, 1 not reproduced, 2 nothing left to try, 3 usage error.

The bug

__cache_cpumap_setup() addresses a sibling's cacheinfo array with this CPU's leaf index:

	for_each_online_cpu(i)
		if (cpu_data(i).topo.apicid >> index_msb == c->topo.apicid >> index_msb) {
			struct cpu_cacheinfo *sib_cpu_ci = get_cpu_cacheinfo(i);

			/* Skip if itself or no cacheinfo */
			if (i == cpu || !sib_cpu_ci->info_list)
				continue;

			sibling_ci = sib_cpu_ci->info_list + index;		/* <-- */
			cpumask_set_cpu(i, &ci->shared_cpu_map);
			cpumask_set_cpu(cpu, &sibling_ci->shared_cpu_map);	/* <-- OOB */
		}

The only guard is "does the sibling have an array at all". Nothing checks index < sib_cpu_ci->num_leaves. Since commit 9677be09e5e4 ("x86/cacheinfo: Delete global num_cache_leaves") the leaf count is per-CPU, so a CPU selected by the APIC-ID test can have a shorter array than the CPU indexing it, and the cpumask_set_cpu() writes past its end.

The generic implementation was fixed for exactly this in drivers/base/cacheinfo.c:cache_shared_cpu_map_setup() (CVE-2023-53254): it walks the sibling's own leaf count and matches on level and type. The x86 implementation was not updated.

Why a userspace program can reach it

populate_cache_leaves() is only called from cacheinfo_cpu_online(), the CPUHP_AP_BASE_CACHEINFO_ONLINE callback. That callback runs on every CPU that comes online during boot, and equally when userspace writes 1 to /sys/devices/system/cpu/cpuN/online. So the path is a plain syscall away.

Two properties of the path shape the reproducer:

  1. It is a first-online path. Offlining and re-onlining a CPU does not re-enter it: free_cache_attributes() only clears the shared maps and never frees info_list, so last_level_cache_is_valid() stays true and detect_cache_attributes() skips populate_cache_leaves() and goes straight to the generic (fixed) cache_shared_cpu_map_setup(). The buggy x86 code therefore runs once per CPU per boot.

  2. Direction matters. The fault needs the CPU with more leaves to come up while a CPU with fewer leaves is already online, so that its loop reaches an index the shorter array does not have. The reverse order is harmless.

Hold a CPU back from boot with maxcpus=, and userspace controls that order. That turns a boot-time race into a deterministic, on-demand reproducer:

boot with maxcpus=1
echo 1 > /sys/devices/system/cpu/cpu1/online     <- the out-of-bounds write

Reproducing it

The precondition is two CPUs that enumerate different numbers of CPUID leaf 4 subleaves while the APIC-ID test still treats them as cache siblings. Run cacheinfo-oob-predict to find out whether a given machine has it; it prints the leaf counts, APIC IDs, num_threads_sharing, index_msb and the resulting sibling relation.

Under crosvm on a hybrid host

crosvm builds CPUID per vCPU and its leaf-4 arm executes the host CPUID instruction inline, and in each vCPU thread set_vcpu_thread_scheduling() (which applies --cpu-affinity) runs before configure_vcpu() → setup_cpuid(). So each vCPU samples leaf 4 from the host CPU it is pinned to, and --cpu-affinity selects the orientation directly: no waiting for a favourable boot.

run-crosvm-repro.sh does this. On the host used here (Intel Core Ultra 7 268V: P-cores cpu0-3 enumerate 4 leaves, E-cores cpu4-7 enumerate 3):

./run-crosvm-repro.sh                  # unpatched kernel: KASAN report
./run-crosvm-repro.sh -k <patched>     # patched kernel: same setup, no report
./run-qemu-control.sh                  # QEMU: no leaf mismatch arises at all

-e 4 pins vcpu0 to an E-core (3 leaves) and -p 0 pins vcpu1 to a P-core (4 leaves), so cpu0 is populated during boot with the shorter array and cpu1 onlined later, by the program is the one that overruns it.

On bare metal

The same code is one APIC-ID assignment away from faulting without any VMM. On the Lunar Lake host used here it does not fault, and cacheinfo-oob-predict shows why:

  cpu0   apicid=0   num_leaves=4
         index3 L3 Unified  threads_sharing=64  index_msb=6  cache_id=0  shared_cpu_list=0-3
  cpu4   apicid=64  num_leaves=3
         index2 L2 Unified  threads_sharing=8   index_msb=3  cache_id=8   shared_cpu_list=4-7

The P-cores' L3 leaf reports num_threads_sharing=64, giving index_msb=6, so the sibling window is apicid >> 6. P-core APIC IDs are 0, 8, 16, 24 (window 0) and E-core APIC IDs are 64, 66, 68, 70 (window 1), so the 3-leaf E-cores fall outside the 4-leaf P-cores' window and are never indexed. Nothing but that numbering prevents the write: a part whose hybrid cores land in the same apicid >> index_msb window, or firmware that numbers them more densely, hits it on bare metal, in which case the reproducer above works unchanged with maxcpus=1 and --order chosen so a 4-leaf CPU comes up after a 3-leaf one.

Run cacheinfo-oob-predict on any hybrid part to check that machine.

Results on this host

Host: Dell Pro 14 Premium PA14250, Intel Core Ultra 7 268V (Lunar Lake). Guest kernel: Debian 7.2~rc7-1~exp1 amd64 with CONFIG_KASAN=y, CONFIG_NR_CPUS=8192. Guest booted with maxcpus=1, two vCPUs, --cpu-affinity 0=4:1=0.

VMM kernel cpu0 / cpu1 leaves boot after echo 1 > cpu1/online trigger exit
crosvm unpatched 3 / 4 clean KASAN slab-out-of-bounds write (6/6 runs) 0
crosvm patched 3 / 4 clean clean (5/5 runs) 1
QEMU unpatched 4 / 4 clean clean, no mismatch to begin with 1

The boot itself is clean in every case, so the KASAN report is attributable to the write(2) and nothing else. The patched run is the meaningful control: the leaf-count mismatch is present and the reproducer confirms it (cpu0 3 leaves, cpu1 4 leaves, out-of-bounds write predicted), and the report does not happen.

The prediction is computed from CPUID before the trigger runs, and matches the report exactly:

  cpu1 (leaf index 3, 4 leaves) -> cpu0 (3 leaves)
    sibling test: num_threads_sharing=64 index_msb=6, apicid 1>>6 == 0>>6 == 0
    write:        8 bytes at info_list(cpu0) + 3*1088 + 32 = +3296,
                  32 bytes past the end of the 3264-byte allocation
    KASAN should report: "Write of size 8" ... "located 32 bytes to the right of
                         allocated 3264-byte region"
...
  prediction vs report:
    allocated region : predicted 3264, reported 3264  MATCH
    bytes to the right: predicted 32, reported 32  MATCH

with the kernel reporting:

BUG: KASAN: slab-out-of-bounds in populate_cache_leaves+0x9d0/0x16d0
Write of size 8 at addr ffff8880038f2ce0 by task cpuhp/1/112
...
 populate_cache_leaves+0x9d0/0x16d0
 detect_cache_attributes+0x323/0x11a0
 cacheinfo_cpu_online+0x29/0xb30
 cpuhp_invoke_callback+0x3f6/0x1530
...
The buggy address is located 32 bytes to the right of
 allocated 3264-byte region [ffff8880038f2000, ffff8880038f2cc0)

3264 = 3 × sizeof(struct cacheinfo) (1088 at CONFIG_NR_CPUS=8192) and 32 = offsetof(struct cacheinfo, shared_cpu_map), i.e. the write lands exactly on info_list[3].shared_cpu_map of a CPU that allocated only three leaves. This is the same signature as the original crash found by boot churn under syzkaller.

Notes

  • Without CONFIG_KASAN=y the write still happens; at these sizes it lands in the slack of the 4096-byte kmalloc object, so nothing reports it. The tools print, per prediction, whether the write stays in slack or leaves the object (which depends on CONFIG_NR_CPUS and the leaf count) when it leaves, it corrupts the next object. slub_debug=Z also catches the in-slack case.
  • A KASAN-free footprint is checked too: if sysfs lists a CPU in a cache's shared_cpu_list while that CPU has no cache of the same level and type, that cross-link can only come from this write. Both tools report it.
  • Only the Intel/fallback __cache_cpumap_setup() path is modelled. __cache_amd_cpumap_setup() has the same unbounded info_list + index pattern for index == 3 and for X86_FEATURE_TOPOEXT, also with only a NULL check, and is not covered by these tools.

Author: Yunseong Kim yunseong.kim@est.tech

Raw
cacheinfo-oob-predict.c
// SPDX-License-Identifier: GPL-2.0
/*
* cacheinfo-oob-predict - decide, from userspace and without touching
* anything, whether this machine can hit the slab-out-of-bounds write in
* arch/x86/kernel/cpu/cacheinfo.c:__cache_cpumap_setup().
*
* The kernel writes a sibling CPU's cacheinfo array using *this* CPU's leaf
* index:
*
* sibling_ci = sib_cpu_ci->info_list + index;
* cpumask_set_cpu(cpu, &sibling_ci->shared_cpu_map);
*
* guarded only by "does the sibling have an info_list at all". Every input to
* that decision - the per-CPU leaf count, the APIC ID, and
* num_threads_sharing - is readable from userspace with CPUID, so the outcome
* can be computed here exactly, before anything is written.
*
* Read-only. No root required. Exit status:
* 0 this machine can hit the out-of-bounds write (details printed)
* 1 it cannot: no sibling pair with a short array
* 2 cannot tell: CPUs are offline and their leaf counts are unknown
* 3 usage or environment error
*
* Author: Yunseong Kim <yunseong.kim@est.tech>
*/
#include "cacheinfo_oob.h"
static void usage(const char *me)
{
printf("usage: %s [--nr-cpus N] [--quiet]\n"
" --nr-cpus N assume CONFIG_NR_CPUS=N when computing the KASAN\n"
" offsets (default: read from the running kernel's config)\n"
" --quiet verdict only\n", me);
}
int main(int argc, char **argv)
{
unsigned int nr_cpus_override = 0;
bool quiet = false;
struct machine m;
int a, v, i;
int pairs = 0, offline = 0, crosslinks;
for (i = 1; i < argc; i++) {
if (!strcmp(argv[i], "--nr-cpus") && i + 1 < argc)
nr_cpus_override = (unsigned int)atoi(argv[++i]);
else if (!strcmp(argv[i], "--quiet"))
quiet = true;
else {
usage(argv[0]);
return 3;
}
}
machine_scan(&m, nr_cpus_override);
if (!quiet)
print_machine(&m);
for (i = 0; i < m.nr_cpus; i++)
if (m.cpu[i].present && !m.cpu[i].online)
offline++;
if (!m.intel_path)
printf("note: vendor is %s, so leaves handled by "
"__cache_amd_cpumap_setup() take a different (also unbounded) path;\n"
" this tool only models the Intel/fallback "
"__cache_cpumap_setup() path.\n\n", m.vendor);
printf("out-of-bounds write predictions (kernel logic replayed on the measured CPUID data)\n");
for (a = 0; a < m.nr_cpus; a++) {
for (v = 0; v < m.nr_cpus; v++) {
struct oob_hit hit;
if (!find_oob(&m, a, v, &hit))
continue;
pairs++;
print_oob_prediction(&m, &hit);
printf(" requires: cpu%d already online when cpu%d runs "
"populate_cache_leaves()\n"
" (cpu%d must be brought online *after* cpu%d)\n",
v, a, a, v);
}
}
if (!pairs)
printf(" none among the %d online CPUs\n", m.nr_online);
printf("\n");
printf("sysfs cross-link check (footprint the write leaves behind, visible without KASAN)\n");
crosslinks = find_sysfs_crosslinks(&m, true);
if (!crosslinks)
printf(" none: every shared_cpu_list entry has a cache of the same level and type\n");
printf("\n");
if (pairs) {
printf("VERDICT: affected. %d ordered CPU pair(s) let the kernel index a "
"sibling's\n"
" cacheinfo array past its end. Reproduce it with "
"cacheinfo-oob-trigger.\n", pairs);
return 0;
}
if (offline) {
printf("VERDICT: indeterminate. %d present CPU(s) are offline, and a CPU's leaf\n"
" count cannot be read until it is online. Run "
"cacheinfo-oob-trigger,\n"
" which measures each CPU as it brings it up.\n", offline);
return 2;
}
printf("VERDICT: not affected in the current configuration.\n");
if (m.nr_online > 1)
printf(" Every CPU the APIC-ID test treats as a cache sibling "
"enumerates at\n least as many leaves as the CPU indexing "
"it.\n");
return 1;
}
Raw
cacheinfo-oob-trigger.c
// SPDX-License-Identifier: GPL-2.0
/*
* cacheinfo-oob-trigger - trigger the slab-out-of-bounds write in
* arch/x86/kernel/cpu/cacheinfo.c:__cache_cpumap_setup() from userspace, with
* a write(2) to sysfs, and capture the KASAN report it produces.
*
* Why this works from userspace
* -----------------------------
* populate_cache_leaves() is only ever called from cacheinfo_cpu_online(),
* the CPUHP_AP_BASE_CACHEINFO_ONLINE callback, so it runs whenever a CPU is
* brought online - during boot, and equally when userspace writes to
*
* /sys/devices/system/cpu/cpuN/online
*
* It is *not* re-run when a CPU that has already been populated is offlined
* and onlined again: free_cache_attributes() only clears the shared maps and
* never frees info_list, so last_level_cache_is_valid() stays true and
* detect_cache_attributes() skips straight to the generic (already fixed)
* cache_shared_cpu_map_setup(). The buggy x86 path therefore runs exactly
* once per CPU per boot: on that CPU's first online.
*
* That is enough for a deterministic reproducer, because the fault needs the
* CPU with *more* leaves to come up while a CPU with *fewer* leaves is
* already online - and with a CPU held back from boot (maxcpus=), userspace
* chooses that order:
*
* boot with maxcpus=1, then
* echo 1 > /sys/devices/system/cpu/cpuN/online <- the 8-byte OOB write
*
* Exit status:
* 0 reproduced: a KASAN report for populate_cache_leaves() was captured
* 1 not reproduced (the reason is printed)
* 2 nothing to try: no CPU is held offline, so no CPU's first online is left
* 3 usage or environment error
*
* Author: Yunseong Kim <yunseong.kim@est.tech>
*/
#include "cacheinfo_oob.h"
#include <poll.h>
#include <sys/time.h>
#define KMSG_CAP (256 * 1024)
struct kmsg_reader {
int fd;
char buf[KMSG_CAP];
size_t len;
};
static int kmsg_open(struct kmsg_reader *k)
{
k->len = 0;
k->buf[0] = '\0';
k->fd = open("/dev/kmsg", O_RDONLY | O_NONBLOCK);
if (k->fd < 0)
return -1;
/* Skip everything already in the buffer. */
lseek(k->fd, 0, SEEK_END);
return 0;
}
/* Collect records for @ms, extending the deadline while lines keep coming. */
static void kmsg_collect(struct kmsg_reader *k, int ms)
{
int idle = 0;
if (k->fd < 0)
return;
while (idle < ms) {
struct pollfd p = { .fd = k->fd, .events = POLLIN };
char rec[8192];
ssize_t n;
int r = poll(&p, 1, 100);
if (r <= 0) {
idle += 100;
continue;
}
idle = 0;
while ((n = read(k->fd, rec, sizeof(rec) - 1)) > 0) {
char *text;
size_t i;
rec[n] = '\0';
text = strchr(rec, ';'); /* skip prio,seq,ts,flag */
text = text ? text + 1 : rec;
for (i = 0; text[i] && k->len + 2 < KMSG_CAP; i++) {
/* /dev/kmsg escapes embedded newlines as \x0a */
if (!strncmp(text + i, "\\x0a", 4)) {
k->buf[k->len++] = '\n';
i += 3;
continue;
}
k->buf[k->len++] = text[i];
}
if (k->len && k->buf[k->len - 1] != '\n' && k->len + 1 < KMSG_CAP)
k->buf[k->len++] = '\n';
k->buf[k->len] = '\0';
}
}
}
static bool kmsg_has(const struct kmsg_reader *k, const char *needle)
{
return strstr(k->buf, needle) != NULL;
}
static void kmsg_print(const struct kmsg_reader *k, const char *prefix)
{
const char *p = k->buf;
while (*p) {
const char *nl = strchr(p, '\n');
int len = nl ? (int)(nl - p) : (int)strlen(p);
printf("%s%.*s\n", prefix, len, p);
if (!nl)
break;
p = nl + 1;
}
}
/*
* Pull "allocated N-byte region" / "is located N bytes to the right" out of a
* report. Every occurrence of @pre is tried, because the needles also appear
* inside unrelated text ("Allocated by task 21").
*/
static long kmsg_scan_long(const struct kmsg_reader *k, const char *pre, const char *post)
{
const char *p = k->buf;
while ((p = strstr(p, pre)) != NULL) {
const char *num = p + strlen(pre);
char *end;
long v = strtol(num, &end, 10);
p = num;
if (end == num)
continue;
if (post && strncmp(end, post, strlen(post)))
continue;
return v;
}
return -1;
}
static int cpu_set_online(int cpu, const char *val)
{
char path[128];
snprintf(path, sizeof(path), SYSFS_CPU "/cpu%d/online", cpu);
return write_file(path, val);
}
static void usage(const char *me)
{
printf("usage: %s [options]\n"
" --order LIST comma-separated CPUs to bring online, in that order\n"
" (default: every offline CPU, ascending)\n"
" --all keep going after the first KASAN report\n"
" --dry-run print the plan and the prediction, write nothing\n"
" --nr-cpus N assume CONFIG_NR_CPUS=N for the offset arithmetic\n"
" --timeout MS how long to wait for kernel output per CPU (default 3000)\n",
me);
}
int main(int argc, char **argv)
{
int order[MAX_CPUS], nr_order = 0;
unsigned int nr_cpus_override = 0;
bool dry_run = false, all = false;
int timeout_ms = 3000;
struct kmsg_reader kmsg;
struct machine m;
int i, reproduced = 0, attempted = 0, path_exercised = 0;
for (i = 1; i < argc; i++) {
if (!strcmp(argv[i], "--order") && i + 1 < argc) {
char *s = argv[++i], *tok;
for (tok = strtok(s, ","); tok; tok = strtok(NULL, ","))
if (nr_order < MAX_CPUS)
order[nr_order++] = atoi(tok);
} else if (!strcmp(argv[i], "--all")) {
all = true;
} else if (!strcmp(argv[i], "--dry-run")) {
dry_run = true;
} else if (!strcmp(argv[i], "--nr-cpus") && i + 1 < argc) {
nr_cpus_override = (unsigned int)atoi(argv[++i]);
} else if (!strcmp(argv[i], "--timeout") && i + 1 < argc) {
timeout_ms = atoi(argv[++i]);
} else {
usage(argv[0]);
return 3;
}
}
if (geteuid() != 0 && !dry_run)
die("must run as root: this writes to " SYSFS_CPU "/cpuN/online");
machine_scan(&m, nr_cpus_override);
print_machine(&m);
if (!m.intel_path)
printf("note: vendor %s - the leaves handled by "
"__cache_amd_cpumap_setup() are not modelled here.\n\n", m.vendor);
if (!nr_order) {
for (i = 0; i < m.nr_cpus; i++)
if (m.cpu[i].present && !m.cpu[i].online)
order[nr_order++] = i;
}
if (!nr_order) {
printf("nothing to do: every present CPU is already online.\n\n"
"populate_cache_leaves() runs only on a CPU's *first* online, so\n"
"offlining and re-onlining a CPU does not re-enter the buggy path:\n"
"free_cache_attributes() never frees info_list, so\n"
"last_level_cache_is_valid() stays true and detect_cache_attributes()\n"
"skips populate_cache_leaves() entirely.\n\n"
"Reboot with a CPU held back, then run this again:\n"
" maxcpus=1 on the kernel command line\n"
"and see README.md for the full recipe.\n");
return 2;
}
printf("plan: bring cpu");
for (i = 0; i < nr_order; i++)
printf("%s%d", i ? ",cpu" : "", order[i]);
printf(" online, one at a time, watching /dev/kmsg\n");
printf(" each of those is a first online, so each one runs "
"populate_cache_leaves()\n\n");
if (dry_run) {
printf("--dry-run: stopping here. Leaf counts of offline CPUs are unknown\n"
" until they are onlined, so the prediction for them cannot\n"
" be made without writing to sysfs.\n");
return 1;
}
if (kmsg_open(&kmsg) < 0)
printf("warning: cannot read /dev/kmsg (%s); check dmesg by hand\n\n",
strerror(errno));
for (i = 0; i < nr_order; i++) {
int cpu = order[i], v, hits = 0;
struct oob_hit predicted;
bool have_prediction = false;
long kasan_alloc, kasan_right;
if (cpu < 0 || cpu >= m.nr_cpus || !m.cpu[cpu].present) {
printf("cpu%d: not present, skipping\n", cpu);
continue;
}
if (m.cpu[cpu].online) {
printf("cpu%d: already online, skipping (its first online is spent)\n",
cpu);
continue;
}
printf("=== onlining cpu%d ===\n", cpu);
printf(" online CPUs before: ");
for (v = 0; v < m.nr_cpus; v++)
if (m.cpu[v].online)
printf("cpu%d(%d leaves) ", v, cpu_num_leaves(&m.cpu[v]));
printf("\n");
/* Drain anything already pending so the capture below is ours. */
kmsg_collect(&kmsg, 100);
kmsg.len = 0;
kmsg.buf[0] = '\0';
attempted++;
if (cpu_set_online(cpu, "1") < 0) {
printf(" write to " SYSFS_CPU "/cpu%d/online failed: %s\n",
cpu, strerror(errno));
continue;
}
path_exercised++;
machine_rescan_cpu(&m, cpu);
if (!m.cpu[cpu].online) {
printf(" cpu%d did not come online\n", cpu);
continue;
}
printf(" cpu%d online: apicid=%u num_leaves=%d\n",
cpu, m.cpu[cpu].apicid, cpu_num_leaves(&m.cpu[cpu]));
for (v = 0; v < m.nr_cpus; v++) {
struct oob_hit hit;
if (v == cpu || !m.cpu[v].online)
continue;
if (!find_oob(&m, cpu, v, &hit))
continue;
if (!hits) {
predicted = hit;
have_prediction = true;
}
hits++;
printf(" predicted out-of-bounds write:\n");
print_oob_prediction(&m, &hit);
}
if (!hits)
printf(" no out-of-bounds write predicted for this transition:\n"
" cpu%d enumerates %d leaves and every cache sibling of it "
"has at least that many\n", cpu, cpu_num_leaves(&m.cpu[cpu]));
kmsg_collect(&kmsg, timeout_ms);
if (kmsg_has(&kmsg, "KASAN") &&
(kmsg_has(&kmsg, "populate_cache_leaves") ||
kmsg_has(&kmsg, "cache_cpumap_setup"))) {
reproduced++;
printf("\n REPRODUCED: the write(2) above produced a KASAN report\n");
printf(" --- kernel output -------------------------------------------\n");
kmsg_print(&kmsg, " | ");
printf(" -------------------------------------------------------------\n");
kasan_alloc = kmsg_scan_long(&kmsg, "allocated ", "-byte region");
kasan_right = kmsg_scan_long(&kmsg, "is located ", " bytes to the right");
if (have_prediction && kasan_alloc > 0 && kasan_right >= 0) {
size_t want_right = predicted.write_off - predicted.alloc_size;
printf(" prediction vs report:\n");
printf(" allocated region : predicted %zu, reported %ld %s\n",
predicted.alloc_size, kasan_alloc,
(size_t)kasan_alloc == predicted.alloc_size ? "MATCH" : "differs");
printf(" bytes to the right: predicted %zu, reported %ld %s\n",
want_right, kasan_right,
(size_t)kasan_right == want_right ? "MATCH" : "differs");
}
printf("\n");
if (!all)
break;
} else if (kmsg.len) {
printf(" no KASAN report. Kernel said:\n");
kmsg_print(&kmsg, " | ");
} else {
printf(" no KASAN report and no kernel output.\n");
}
printf("\n");
}
printf("summary: %d CPU(s) onlined, %d first-online transition(s) exercised the "
"x86 populate_cache_leaves() path, %d KASAN report(s) captured\n",
attempted, path_exercised, reproduced);
if (reproduced)
return 0;
printf("\nnot reproduced. Checklist:\n");
if (path_exercised && reproduced == 0)
printf(" - was an out-of-bounds write predicted above? If it was and no report\n"
" followed, this kernel either carries the __cache_cpumap_setup()\n"
" bounds check or was not built with CONFIG_KASAN=y. The prediction\n"
" models the unpatched code.\n");
printf(" - is the kernel built with CONFIG_KASAN=y? Without it the write still\n"
" happens, it is just not reported (see the slack note above).\n");
printf(" - do two CPUs enumerate different leaf counts? The table above shows\n"
" the counts actually measured on the CPUs that were online.\n");
printf(" - did the CPU with *more* leaves come online after one with fewer?\n"
" Use --order to control that; onlining them the other way is harmless.\n");
i = find_sysfs_crosslinks(&m, false);
if (i)
printf(" - note: %d sysfs cross-link(s) to a CPU without that cache level "
"exist,\n which is the footprint of the write happening anyway.\n", i);
return 1;
}
Raw
cacheinfo_oob.h
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Shared helpers for the x86 cacheinfo slab-out-of-bounds reproducer.
*
* Everything here is userspace-only: CPUID reads on a pinned CPU, sysfs
* parsing, and a byte-for-byte re-implementation of the two pieces of kernel
* logic that decide whether the out-of-bounds write happens:
*
* arch/x86/kernel/cpu/cacheinfo.c:__cache_cpumap_setup() (the sibling test)
* arch/x86/kernel/cpu/cacheinfo.c:find_num_cache_leaves() (the array length)
*
* Author: Yunseong Kim <yunseong.kim@est.tech>
*/
#ifndef CACHEINFO_OOB_H
#define CACHEINFO_OOB_H
#define _GNU_SOURCE
#include <cpuid.h>
#include <ctype.h>
#include <errno.h>
#include <fcntl.h>
#include <sched.h>
#include <stdarg.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>
#define MAX_CPUS 1024
#define MAX_LEAVES 16
#define SYSFS_CPU "/sys/devices/system/cpu"
/* enum cache_type, mirrors include/linux/cacheinfo.h ordering of CPUID.4 types */
enum leaf_type {
LEAF_NULL = 0,
LEAF_DATA = 1,
LEAF_INST = 2,
LEAF_UNIFIED = 3,
};
static const char *leaf_type_name(unsigned int t)
{
switch (t) {
case LEAF_DATA: return "Data";
case LEAF_INST: return "Instruction";
case LEAF_UNIFIED: return "Unified";
default: return "Null";
}
}
struct leaf_info {
unsigned int level;
unsigned int type; /* enum leaf_type */
unsigned int num_threads_sharing;
unsigned int size; /* bytes, informational */
int index_msb; /* get_count_order(num_threads_sharing) */
unsigned int cache_id; /* apicid >> index_msb, as the kernel computes */
};
struct cpu_info {
int cpu;
bool present;
bool online;
bool probed; /* CPUID actually read on this CPU */
unsigned int apicid; /* x2APIC ID, what cpu_data(cpu).topo.apicid holds */
int cpuid_leaves; /* find_num_cache_leaves() equivalent */
int sysfs_leaves; /* number of cacheN/indexN dirs = kernel num_leaves */
struct leaf_info leaf[MAX_LEAVES];
/* sysfs view, only filled for online CPUs */
unsigned int sysfs_level[MAX_LEAVES];
unsigned int sysfs_type[MAX_LEAVES];
char sysfs_shared[MAX_LEAVES][128];
};
struct machine {
struct cpu_info cpu[MAX_CPUS];
int nr_cpus; /* highest present cpu + 1 */
int nr_online;
int nr_probed;
unsigned int nr_cpus_config; /* CONFIG_NR_CPUS of the running kernel */
bool nr_cpus_config_known;
size_t sizeof_cacheinfo;
size_t offsetof_shared_cpu_map;
char vendor[13];
char model_name[96];
bool intel_path; /* !AMD && !Hygon: the __cache_cpumap_setup() path */
};
/* ---------------------------------------------------------------- utilities */
static void die(const char *fmt, ...)
{
va_list ap;
va_start(ap, fmt);
fprintf(stderr, "error: ");
vfprintf(stderr, fmt, ap);
va_end(ap);
fprintf(stderr, "\n");
exit(3);
}
static int read_file(const char *path, char *buf, size_t len)
{
int fd = open(path, O_RDONLY);
ssize_t n;
if (fd < 0)
return -1;
n = read(fd, buf, len - 1);
close(fd);
if (n < 0)
return -1;
buf[n] = '\0';
while (n > 0 && (buf[n - 1] == '\n' || buf[n - 1] == '\r'))
buf[--n] = '\0';
return 0;
}
static inline int write_file(const char *path, const char *val)
{
int fd = open(path, O_WRONLY);
ssize_t n;
if (fd < 0)
return -1;
n = write(fd, val, strlen(val));
if (close(fd) < 0 && n >= 0)
return -1;
return n < 0 ? -1 : 0;
}
/* "0-3,7" -> set bits in mask[] */
static void parse_cpu_list(const char *s, bool *mask, int max)
{
while (*s) {
int a = 0, b;
while (*s && !isdigit((unsigned char)*s))
s++;
if (!*s)
return;
a = (int)strtol(s, (char **)&s, 10);
b = a;
if (*s == '-') {
s++;
b = (int)strtol(s, (char **)&s, 10);
}
for (; a <= b && a < max; a++)
if (a >= 0)
mask[a] = true;
}
}
/* include/linux/bitops.h:get_count_order() */
static int get_count_order(unsigned int count)
{
int fls = 0;
if (count == 0)
return -1;
count--;
while (count) { /* fls(count) */
fls++;
count >>= 1;
}
return fls;
}
/*
* struct cacheinfo layout on x86-64, derived from include/linux/cacheinfo.h:
*
* 8 x unsigned int -> 0..32
* cpumask_t shared_cpu_map -> 32..32+mask
* unsigned int attributes; void *fw_token; bool disable_sysfs; void *priv;
*
* which lands at sizeof == mask_bytes + 64 with shared_cpu_map at offset 32.
* For CONFIG_NR_CPUS=8192 that is 1088/32, matching pahole on a built vmlinux.
*/
static size_t cacheinfo_mask_bytes(unsigned int nr_cpus_config)
{
return ((nr_cpus_config + 63) / 64) * 8;
}
static size_t cacheinfo_size(unsigned int nr_cpus_config)
{
return cacheinfo_mask_bytes(nr_cpus_config) + 64;
}
/* kmalloc bucket that an allocation of @n bytes comes from (KMALLOC_NORMAL) */
static size_t kmalloc_bucket(size_t n)
{
static const size_t small[] = { 8, 16, 32, 64, 96, 128, 192, 256 };
size_t b;
unsigned int i;
for (i = 0; i < sizeof(small) / sizeof(small[0]); i++)
if (n <= small[i])
return small[i];
for (b = 512; b <= (size_t)1 << 30; b <<= 1)
if (n <= b)
return b;
return n;
}
/* ------------------------------------------------------------------- CPUID */
static bool pin_to_cpu(int cpu)
{
cpu_set_t set;
CPU_ZERO(&set);
CPU_SET(cpu, &set);
if (sched_setaffinity(0, sizeof(set), &set) != 0)
return false;
sched_yield();
return sched_getcpu() == cpu;
}
static unsigned int read_apicid(void)
{
unsigned int a, b, c, d, max_leaf;
__cpuid(0, max_leaf, b, c, d);
/* CPUID.1F:EDX and CPUID.0B:EDX both return the 32-bit x2APIC ID. */
if (max_leaf >= 0x1f) {
__cpuid_count(0x1f, 0, a, b, c, d);
if (b)
return d;
}
if (max_leaf >= 0x0b) {
__cpuid_count(0x0b, 0, a, b, c, d);
if (b)
return d;
}
/* Legacy initial APIC ID, CPUID.1:EBX[31:24]. */
__cpuid(1, a, b, c, d);
return b >> 24;
}
/*
* arch/x86/kernel/cpu/cacheinfo.c:find_num_cache_leaves() and the CPUID.4
* decoding in intel_fill_cpuid4_info(), for the CPU we are pinned to.
*/
static void probe_cpuid_leaves(struct cpu_info *ci, bool amd_like)
{
unsigned int op = amd_like ? 0x8000001d : 4;
int i;
ci->cpuid_leaves = 0;
for (i = 0; i < MAX_LEAVES; i++) {
unsigned int eax, ebx, ecx, edx;
struct leaf_info *l = &ci->leaf[i];
__cpuid_count(op, (unsigned int)i, eax, ebx, ecx, edx);
if ((eax & 0x1f) == LEAF_NULL)
break;
l->type = eax & 0x1f;
l->level = (eax >> 5) & 0x7;
l->num_threads_sharing = ((eax >> 14) & 0xfff) + 1;
l->index_msb = get_count_order(l->num_threads_sharing);
l->cache_id = ci->apicid >> l->index_msb;
l->size = (((ecx & 0xffffffff) + 1) *
((ebx & 0xfff) + 1) *
(((ebx >> 12) & 0x3ff) + 1) *
(((ebx >> 22) & 0x3ff) + 1));
ci->cpuid_leaves = i + 1;
}
}
/* ------------------------------------------------------------------- sysfs */
static int count_sysfs_leaves(int cpu)
{
int i;
for (i = 0; i < MAX_LEAVES; i++) {
char path[160];
struct stat st;
snprintf(path, sizeof(path), SYSFS_CPU "/cpu%d/cache/index%d", cpu, i);
if (stat(path, &st) != 0)
break;
}
return i;
}
static void read_sysfs_leaves(struct cpu_info *ci)
{
int i;
ci->sysfs_leaves = count_sysfs_leaves(ci->cpu);
for (i = 0; i < ci->sysfs_leaves && i < MAX_LEAVES; i++) {
char path[192], buf[128];
snprintf(path, sizeof(path), SYSFS_CPU "/cpu%d/cache/index%d/level",
ci->cpu, i);
ci->sysfs_level[i] = read_file(path, buf, sizeof(buf)) ? 0 :
(unsigned int)atoi(buf);
snprintf(path, sizeof(path), SYSFS_CPU "/cpu%d/cache/index%d/type",
ci->cpu, i);
ci->sysfs_type[i] = LEAF_NULL;
if (!read_file(path, buf, sizeof(buf))) {
if (!strcmp(buf, "Data"))
ci->sysfs_type[i] = LEAF_DATA;
else if (!strcmp(buf, "Instruction"))
ci->sysfs_type[i] = LEAF_INST;
else if (!strcmp(buf, "Unified"))
ci->sysfs_type[i] = LEAF_UNIFIED;
}
snprintf(path, sizeof(path),
SYSFS_CPU "/cpu%d/cache/index%d/shared_cpu_list", ci->cpu, i);
if (read_file(path, buf, sizeof(buf)))
buf[0] = '\0';
snprintf(ci->sysfs_shared[i], sizeof(ci->sysfs_shared[i]), "%s", buf);
}
}
static void read_vendor(struct machine *m)
{
unsigned int a, b, c, d;
FILE *f;
char line[256];
__cpuid(0, a, b, c, d);
memcpy(m->vendor + 0, &b, 4);
memcpy(m->vendor + 4, &d, 4);
memcpy(m->vendor + 8, &c, 4);
m->vendor[12] = '\0';
m->intel_path = strcmp(m->vendor, "AuthenticAMD") &&
strcmp(m->vendor, "HygonGenuine");
m->model_name[0] = '\0';
f = fopen("/proc/cpuinfo", "r");
if (!f)
return;
while (fgets(line, sizeof(line), f)) {
char *p = strstr(line, "model name");
if (!p)
continue;
p = strchr(line, ':');
if (!p)
continue;
p += 2;
p[strcspn(p, "\n")] = '\0';
snprintf(m->model_name, sizeof(m->model_name), "%s", p);
break;
}
fclose(f);
}
/*
* CONFIG_NR_CPUS decides sizeof(struct cacheinfo) and therefore the exact
* offsets a KASAN report will print. Take it from the running kernel's config
* when it is available, otherwise let the caller override it.
*/
static void read_nr_cpus_config(struct machine *m)
{
char path[320], line[256];
FILE *f = fopen("/proc/config.gz", "r");
m->nr_cpus_config_known = false;
if (f) {
fclose(f);
f = popen("zcat /proc/config.gz 2>/dev/null | grep -m1 '^CONFIG_NR_CPUS='", "r");
if (f) {
if (fgets(line, sizeof(line), f)) {
char *p = strchr(line, '=');
if (p) {
m->nr_cpus_config = (unsigned int)atoi(p + 1);
m->nr_cpus_config_known = m->nr_cpus_config > 0;
}
}
pclose(f);
}
}
if (!m->nr_cpus_config_known) {
char rel[128];
if (!read_file("/proc/sys/kernel/osrelease", rel, sizeof(rel))) {
snprintf(path, sizeof(path), "/boot/config-%s", rel);
f = fopen(path, "r");
if (f) {
while (fgets(line, sizeof(line), f)) {
if (!strncmp(line, "CONFIG_NR_CPUS=", 15)) {
m->nr_cpus_config = (unsigned int)atoi(line + 15);
m->nr_cpus_config_known = m->nr_cpus_config > 0;
break;
}
}
fclose(f);
}
}
}
if (!m->nr_cpus_config_known)
m->nr_cpus_config = 8192; /* Debian's value, see README */
}
/* ------------------------------------------------------- machine enumeration */
static void machine_scan(struct machine *m, unsigned int nr_cpus_override)
{
bool present[MAX_CPUS] = { false }, online[MAX_CPUS] = { false };
char buf[4096];
cpu_set_t saved;
int i;
memset(m, 0, sizeof(*m));
read_vendor(m);
read_nr_cpus_config(m);
if (nr_cpus_override) {
m->nr_cpus_config = nr_cpus_override;
m->nr_cpus_config_known = true;
}
m->sizeof_cacheinfo = cacheinfo_size(m->nr_cpus_config);
m->offsetof_shared_cpu_map = 32;
if (read_file(SYSFS_CPU "/present", buf, sizeof(buf)))
die("cannot read " SYSFS_CPU "/present: %s", strerror(errno));
parse_cpu_list(buf, present, MAX_CPUS);
if (read_file(SYSFS_CPU "/online", buf, sizeof(buf)))
die("cannot read " SYSFS_CPU "/online: %s", strerror(errno));
parse_cpu_list(buf, online, MAX_CPUS);
if (sched_getaffinity(0, sizeof(saved), &saved) != 0)
die("sched_getaffinity: %s", strerror(errno));
for (i = 0; i < MAX_CPUS; i++) {
struct cpu_info *ci = &m->cpu[i];
ci->cpu = i;
ci->present = present[i];
ci->online = online[i];
if (!ci->present)
continue;
m->nr_cpus = i + 1;
if (!ci->online)
continue;
m->nr_online++;
read_sysfs_leaves(ci);
if (pin_to_cpu(i)) {
ci->apicid = read_apicid();
probe_cpuid_leaves(ci, !m->intel_path);
ci->probed = true;
m->nr_probed++;
}
}
sched_setaffinity(0, sizeof(saved), &saved);
}
/*
* Refresh one CPU after a hotplug transition.
*/
static inline void machine_rescan_cpu(struct machine *m, int cpu)
{
struct cpu_info *ci = &m->cpu[cpu];
bool online[MAX_CPUS] = { false };
cpu_set_t saved;
char buf[4096];
if (!read_file(SYSFS_CPU "/online", buf, sizeof(buf)))
parse_cpu_list(buf, online, MAX_CPUS);
ci->online = online[cpu];
if (!ci->online)
return;
read_sysfs_leaves(ci);
if (sched_getaffinity(0, sizeof(saved), &saved) == 0) {
if (pin_to_cpu(cpu)) {
ci->apicid = read_apicid();
probe_cpuid_leaves(ci, !m->intel_path);
ci->probed = true;
}
sched_setaffinity(0, sizeof(saved), &saved);
}
}
/* Kernel num_leaves: sysfs is authoritative when online, CPUID otherwise. */
static int cpu_num_leaves(const struct cpu_info *ci)
{
if (ci->online && ci->sysfs_leaves > 0)
return ci->sysfs_leaves;
return ci->cpuid_leaves;
}
/* ------------------------------------------------- the kernel's sibling test */
struct oob_hit {
int attacker; /* CPU running populate_cache_leaves() */
int victim; /* sibling whose array is too short */
int index; /* leaf index being processed */
int victim_leaves;
unsigned int num_threads_sharing;
int index_msb;
size_t alloc_size; /* victim's kmalloc size */
size_t write_off; /* offset of the 8-byte write in that object */
size_t bucket; /* kmalloc bucket the victim's array came from */
};
/*
* Replicates, for a hypothetical "attacker comes online while victim is
* already online" transition:
*
* for (idx = 0; idx < this_cpu_ci->num_leaves; idx++) [:620]
* __cache_cpumap_setup(cpu, idx, &id4); [:631]
*
* index_msb = get_count_order(num_threads_sharing);
* for_each_online_cpu(i)
* if (cpu_data(i).topo.apicid >> index_msb ==
* c->topo.apicid >> index_msb) { [:570]
* if (i == cpu || !sib_cpu_ci->info_list)
* continue;
* sibling_ci = sib_cpu_ci->info_list + index; [:578]
* cpumask_set_cpu(cpu, &sibling_ci->shared_cpu_map); [:580]
*
* A hit means index >= num_leaves(victim), i.e. the write at :580 lands past
* the end of the victim's array.
*/
static int find_oob(const struct machine *m, int attacker, int victim,
struct oob_hit *hit)
{
const struct cpu_info *a = &m->cpu[attacker], *v = &m->cpu[victim];
int a_leaves = cpu_num_leaves(a), v_leaves = cpu_num_leaves(v);
int idx, hits = 0;
if (!m->intel_path || attacker == victim || !a->probed || !v->probed)
return 0;
for (idx = 0; idx < a_leaves && idx < MAX_LEAVES; idx++) {
const struct leaf_info *l = &a->leaf[idx];
if (l->num_threads_sharing == 1) /* returns before the loop */
continue;
if ((v->apicid >> l->index_msb) != (a->apicid >> l->index_msb))
continue;
if (idx < v_leaves)
continue;
if (hit && !hits) {
hit->attacker = attacker;
hit->victim = victim;
hit->index = idx;
hit->victim_leaves = v_leaves;
hit->num_threads_sharing = l->num_threads_sharing;
hit->index_msb = l->index_msb;
hit->alloc_size = (size_t)v_leaves * m->sizeof_cacheinfo;
hit->write_off = (size_t)idx * m->sizeof_cacheinfo +
m->offsetof_shared_cpu_map;
hit->bucket = kmalloc_bucket(hit->alloc_size);
}
hits++;
}
return hits;
}
static void print_oob_prediction(const struct machine *m, const struct oob_hit *h)
{
printf(" cpu%d (leaf index %d, %u leaves) -> cpu%d (%d leaves)\n",
h->attacker, h->index, cpu_num_leaves(&m->cpu[h->attacker]),
h->victim, h->victim_leaves);
printf(" sibling test: num_threads_sharing=%u index_msb=%d, "
"apicid %u>>%d == %u>>%d == %u\n",
h->num_threads_sharing, h->index_msb,
m->cpu[h->attacker].apicid, h->index_msb,
m->cpu[h->victim].apicid, h->index_msb,
m->cpu[h->victim].apicid >> h->index_msb);
printf(" write: 8 bytes at info_list(cpu%d) + %d*%zu + %zu = +%zu,\n",
h->victim, h->index, m->sizeof_cacheinfo,
m->offsetof_shared_cpu_map, h->write_off);
printf(" %zu bytes past the end of the %zu-byte allocation\n",
h->write_off - h->alloc_size, h->alloc_size);
printf(" KASAN should report: \"Write of size 8\" ... \"located %zu bytes to the "
"right of allocated %zu-byte region\"\n",
h->write_off - h->alloc_size, h->alloc_size);
if (h->write_off >= h->bucket)
printf(" without KASAN: the write leaves the %zu-byte kmalloc object "
"entirely and corrupts the next one\n", h->bucket);
else
printf(" without KASAN: the write stays inside the %zu-byte kmalloc "
"object's slack, so KASAN (or slub_debug=Z) is needed to see it\n",
h->bucket);
}
/*
* A KASAN-free footprint of the same bug: sysfs claims a CPU shares a cache
* with a CPU that has no cache of that level and type. That cross-link can
* only come from the out-of-bounds write.
*/
static int find_sysfs_crosslinks(const struct machine *m, bool verbose)
{
int cpu, idx, found = 0;
for (cpu = 0; cpu < m->nr_cpus; cpu++) {
const struct cpu_info *ci = &m->cpu[cpu];
if (!ci->online)
continue;
for (idx = 0; idx < ci->sysfs_leaves; idx++) {
bool shared[MAX_CPUS] = { false };
int sib;
parse_cpu_list(ci->sysfs_shared[idx], shared, MAX_CPUS);
for (sib = 0; sib < m->nr_cpus; sib++) {
const struct cpu_info *sc = &m->cpu[sib];
bool match = false;
int j;
if (sib == cpu || !shared[sib] || !sc->online)
continue;
for (j = 0; j < sc->sysfs_leaves; j++)
if (sc->sysfs_level[j] == ci->sysfs_level[idx] &&
sc->sysfs_type[j] == ci->sysfs_type[idx])
match = true;
if (match)
continue;
found++;
if (verbose)
printf(" cpu%d/cache/index%d (L%u %s) lists cpu%d as "
"sharing it, but cpu%d has no L%u %s cache\n",
cpu, idx, ci->sysfs_level[idx],
leaf_type_name(ci->sysfs_type[idx]), sib,
sib, ci->sysfs_level[idx],
leaf_type_name(ci->sysfs_type[idx]));
}
}
}
return found;
}
static void print_machine(const struct machine *m)
{
int cpu, i;
printf("machine\n");
printf(" model : %s\n", m->model_name[0] ? m->model_name : "(unknown)");
printf(" vendor : %s (%s shared_cpu_map path)\n", m->vendor,
m->intel_path ? "__cache_cpumap_setup" : "__cache_amd_cpumap_setup");
printf(" cpus present : %d, online: %d, probed: %d\n",
m->nr_cpus, m->nr_online, m->nr_probed);
printf(" CONFIG_NR_CPUS : %u%s\n", m->nr_cpus_config,
m->nr_cpus_config_known ? "" : " (assumed, kernel config not readable)");
printf(" sizeof(struct cacheinfo) = %zu, offsetof(shared_cpu_map) = %zu\n\n",
m->sizeof_cacheinfo, m->offsetof_shared_cpu_map);
printf("per-cpu cache leaves (CPUID.4 as the kernel reads it)\n");
for (cpu = 0; cpu < m->nr_cpus; cpu++) {
const struct cpu_info *ci = &m->cpu[cpu];
if (!ci->present)
continue;
if (!ci->online) {
printf(" cpu%-3d offline, leaf count unknown until it is onlined\n",
cpu);
continue;
}
printf(" cpu%-3d apicid=%-3u num_leaves=%d (sysfs %d)%s\n",
cpu, ci->apicid, ci->cpuid_leaves, ci->sysfs_leaves,
ci->cpuid_leaves != ci->sysfs_leaves ? " <-- MISMATCH" : "");
for (i = 0; i < ci->cpuid_leaves; i++) {
const struct leaf_info *l = &ci->leaf[i];
printf(" index%d L%u %-11s size=%-7u threads_sharing=%-3u "
"index_msb=%d cache_id=%u shared_cpu_list=%s\n",
i, l->level, leaf_type_name(l->type), l->size,
l->num_threads_sharing, l->index_msb, l->cache_id,
i < ci->sysfs_leaves ? ci->sysfs_shared[i] : "-");
}
}
printf("\n");
}
#endif /* CACHEINFO_OOB_H */
Raw
last-console-PATCHED.log
dev/nullne):/# mount -t proc proc /proc 2>/dev/null; mount -t sysfs sys /sys 2>/d
root@(none):/# mount -t devtmpfs dev /dev 2>/dev/null; ls /dev/vd*
/dev/vda /dev/vda1 /dev/vda14 /dev/vda15 /dev/vdb
mnt/reproe):/# mkdir -p /mnt/repro; mount -t ext4 /dev/vdb /mnt/repro 2>&1; ls /m
[ 34.498417] EXT4-fs (vdb): mounted filesystem 279cd03f-fe2e-4af9-b4ae-7d217d7a8572 r/w with ordered data mode. Quota mode: none.
cacheinfo-oob-predict cacheinfo-oob-trigger lost+found
root@(none):/# echo MARK_BOOTCLEAN; dmesg | grep -c "KASAN: slab-out-of-bounds"
MARK_BOOTCLEAN
0
root@(none):/# echo MARK_ONLINE; cat /sys/devices/system/cpu/online
MARK_ONLINE
0
PREDICT_EXIT=$?"cho MARK_PREDICT_BEGIN; /mnt/repro/cacheinfo-oob-predict; echo "P
MARK_PREDICT_BEGIN
machine
model : Intel(R) Core(TM) Ultra 7 268V
vendor : GenuineIntel (__cache_cpumap_setup shared_cpu_map path)
cpus present : 2, online: 1, probed: 1
CONFIG_NR_CPUS : 8192 (assumed, kernel config not readable)
sizeof(struct cacheinfo) = 1088, offsetof(shared_cpu_map) = 32
per-cpu cache leaves (CPUID.4 as the kernel reads it)
cpu0 apicid=0 num_leaves=3 (sysfs 3)
index0 L1 Data size=32768 threads_sharing=1 index_msb=0 cache_id=0 shared_cpu_list=0
index1 L1 Instruction size=65536 threads_sharing=1 index_msb=0 cache_id=0 shared_cpu_list=0
index2 L2 Unified size=4194304 threads_sharing=8 index_msb=3 cache_id=0 shared_cpu_list=0
cpu1 offline, leaf count unknown until it is onlined
out-of-bounds write predictions (kernel logic replayed on the measured CPUID data)
none among the 1 online CPUs
sysfs cross-link check (footprint the write leaves behind, visible without KASAN)
none: every shared_cpu_list entry has a cache of the same level and type
VERDICT: indeterminate. 1 present CPU(s) are offline, and a CPU's leaf
count cannot be read until it is online. Run cacheinfo-oob-trigger,
which measures each CPU as it brings it up.
PREDICT_EXIT=2
echo "TRIGGER_EXIT=$?"RK_TRIGGER_BEGIN; /mnt/repro/cacheinfo-oob-trigger --all; e
MARK_TRIGGER_BEGIN
machine
model : Intel(R) Core(TM) Ultra 7 268V
vendor : GenuineIntel (__cache_cpumap_setup shared_cpu_map path)
cpus present : 2, online: 1, probed: 1
CONFIG_NR_CPUS : 8192 (assumed, kernel config not readable)
sizeof(struct cacheinfo) = 1088, offsetof(shared_cpu_map) = 32
per-cpu cache leaves (CPUID.4 as the kernel reads it)
cpu0 apicid=0 num_leaves=3 (sysfs 3)
index0 L1 Data size=32768 threads_sharing=1 index_msb=0 cache_id=0 shared_cpu_list=0
index1 L1 Instruction size=65536 threads_sharing=1 index_msb=0 cache_id=0 shared_cpu_list=0
index2 L2 Unified size=4194304 threads_sharing=8 index_msb=3 cache_id=0 shared_cpu_list=0
cpu1 offline, leaf count unknown until it is onlined
plan: bring cpu1 online, one at a time, watching /dev/kmsg
each of those is a first online, so each one runs populate_cache_leaves()
=== onlining cpu1 ===
online CPUs before: cpu0(3 leaves)
[ 34.751565] smpboot: Booting Node 0 Processor 1 APIC 0x1
cpu1 online: apicid=1 num_leaves=4
predicted out-of-bounds write:
cpu1 (leaf index 3, 4 leaves) -> cpu0 (3 leaves)
sibling test: num_threads_sharing=64 index_msb=6, apicid 1>>6 == 0>>6 == 0
write: 8 bytes at info_list(cpu0) + 3*1088 + 32 = +3296,
32 bytes past the end of the 3264-byte allocation
KASAN should report: "Write of size 8" ... "located 32 bytes to the right of allocated 3264-byte region"
without KASAN: the write stays inside the 4096-byte kmalloc object's slack, so KASAN (or slub_debug=Z) is needed to see it
no KASAN report. Kernel said:
| smpboot: Booting Node 0 Processor 1 APIC 0x1
summary: 1 CPU(s) onlined, 1 first-online transition(s) exercised the x86 populate_cache_leaves() path, 0 KASAN report(s) captured
not reproduced. Checklist:
- was an out-of-bounds write predicted above? If it was and no report
followed, this kernel either carries the __cache_cpumap_setup()
bounds check or was not built with CONFIG_KASAN=y. The prediction
models the unpatched code.
- is the kernel built with CONFIG_KASAN=y? Without it the write still
happens, it is just not reported (see the slack note above).
- do two CPUs enumerate different leaf counts? The table above shows
the counts actually measured on the CPUs that were online.
- did the CPU with *more* leaves come online after one with fewer?
Use --order to control that; onlining them the other way is harmless.
TRIGGER_EXIT=1
root@(none):/# echo MARK_TOPO
MARK_TOPO
(cat $i/shared_cpu_list)"; done; doned 2>/dev/null) size=$(cat $i/size) shared=$(
cpu0/index0 L1 Data id=0 size=32K shared=0-1
cpu0/index1 L1 Instruction id=0 size=64K shared=0-1
cpu0/index2 L2 Unified id=0 size=4096K shared=0-1
cpu1/index0 L1 Data id=0 size=48K shared=0-1
cpu1/index1 L1 Instruction id=0 size=64K shared=0-1
cpu1/index2 L2 Unified id=0 size=2560K shared=0-1
cpu1/index3 L3 Unified id=0 size=12288K shared=1
"oot@(none):/# echo MARK_KASAN_TOTAL; dmesg | grep -c "KASAN: slab-out-of-bounds"
MARK_KASAN_TOTAL
0
Raw
last-console-UNPATCHED.log
dev/nullne):/# mount -t proc proc /proc 2>/dev/null; mount -t sysfs sys /sys 2>/d
root@(none):/# mount -t devtmpfs dev /dev 2>/dev/null; ls /dev/vd*
/dev/vda /dev/vda1 /dev/vda14 /dev/vda15 /dev/vdb
mnt/reproe):/# mkdir -p /mnt/repro; mount -t ext4 /dev/vdb /mnt/repro 2>&1; ls /m
[ 34.494540] EXT4-fs (vdb): mounted filesystem 91e3b941-4a27-4d9a-8c27-e5dabbca5c06 r/w with ordered data mode. Quota mode: none.
cacheinfo-oob-predict cacheinfo-oob-trigger lost+found
root@(none):/# echo MARK_BOOTCLEAN; dmesg | grep -c "KASAN: slab-out-of-bounds"
MARK_BOOTCLEAN
0
root@(none):/# echo MARK_ONLINE; cat /sys/devices/system/cpu/online
MARK_ONLINE
0
PREDICT_EXIT=$?"cho MARK_PREDICT_BEGIN; /mnt/repro/cacheinfo-oob-predict; echo "P
MARK_PREDICT_BEGIN
machine
model : Intel(R) Core(TM) Ultra 7 268V
vendor : GenuineIntel (__cache_cpumap_setup shared_cpu_map path)
cpus present : 2, online: 1, probed: 1
CONFIG_NR_CPUS : 8192
sizeof(struct cacheinfo) = 1088, offsetof(shared_cpu_map) = 32
per-cpu cache leaves (CPUID.4 as the kernel reads it)
cpu0 apicid=0 num_leaves=3 (sysfs 3)
index0 L1 Data size=32768 threads_sharing=1 index_msb=0 cache_id=0 shared_cpu_list=0
index1 L1 Instruction size=65536 threads_sharing=1 index_msb=0 cache_id=0 shared_cpu_list=0
index2 L2 Unified size=4194304 threads_sharing=8 index_msb=3 cache_id=0 shared_cpu_list=0
cpu1 offline, leaf count unknown until it is onlined
out-of-bounds write predictions (kernel logic replayed on the measured CPUID data)
none among the 1 online CPUs
sysfs cross-link check (footprint the write leaves behind, visible without KASAN)
none: every shared_cpu_list entry has a cache of the same level and type
VERDICT: indeterminate. 1 present CPU(s) are offline, and a CPU's leaf
count cannot be read until it is online. Run cacheinfo-oob-trigger,
which measures each CPU as it brings it up.
PREDICT_EXIT=2
echo "TRIGGER_EXIT=$?"RK_TRIGGER_BEGIN; /mnt/repro/cacheinfo-oob-trigger --all; e
MARK_TRIGGER_BEGIN
machine
model : Intel(R) Core(TM) Ultra 7 268V
vendor : GenuineIntel (__cache_cpumap_setup shared_cpu_map path)
cpus present : 2, online: 1, probed: 1
CONFIG_NR_CPUS : 8192
sizeof(struct cacheinfo) = 1088, offsetof(shared_cpu_map) = 32
per-cpu cache leaves (CPUID.4 as the kernel reads it)
cpu0 apicid=0 num_leaves=3 (sysfs 3)
index0 L1 Data size=32768 threads_sharing=1 index_msb=0 cache_id=0 shared_cpu_list=0
index1 L1 Instruction size=65536 threads_sharing=1 index_msb=0 cache_id=0 shared_cpu_list=0
index2 L2 Unified size=4194304 threads_sharing=8 index_msb=3 cache_id=0 shared_cpu_list=0
cpu1 offline, leaf count unknown until it is onlined
plan: bring cpu1 online, one at a time, watching /dev/kmsg
each of those is a first online, so each one runs populate_cache_leaves()
=== onlining cpu1 ===
online CPUs before: cpu0(3 leaves)
[ 34.747704] SMP alternatives: switching to SMP code
[ 34.763671] smpboot: Booting Node 0 Processor 1 APIC 0x1
[ 34.767051] ==================================================================
[ 34.767357] BUG: KASAN: slab-out-of-bounds in populate_cache_leaves+0x9d0/0x16d0
[ 34.767665] Write of size 8 at addr ffff888003908ce0 by task cpuhp/1/112
[ 34.767930]
[ 34.768000] CPU: 1 UID: 0 PID: 112 Comm: cpuhp/1 Not tainted 7.2-amd64 #1 PREEMPT(lazy) Debian 7.2~rc7-1~exp1
[ 34.768008] Hardware name: ChromiumOS crosvm, BIOS 0
[ 34.768011] Call Trace:
[ 34.768014] <TASK>
[ 34.768017] dump_stack_lvl+0xd5/0x130
[ 34.768024] print_report+0x14b/0x4b0
[ 34.768031] ? __virt_addr_valid+0x186/0x4e0
[ 34.768037] ? __virt_addr_valid+0x254/0x4e0
[ 34.768043] kasan_report+0x108/0x130
[ 34.768050] ? populate_cache_leaves+0x9d0/0x16d0
[ 34.768055] ? populate_cache_leaves+0x9d0/0x16d0
[ 34.768061] kasan_check_range+0x134/0x220
[ 34.768066] populate_cache_leaves+0x9d0/0x16d0
[ 34.768073] ? __pfx_populate_cache_leaves+0x10/0x10
[ 34.768079] ? __kmalloc_noprof+0x450/0x8c0
[ 34.768085] ? detect_cache_attributes+0x4b4/0x11a0
[ 34.768092] detect_cache_attributes+0x323/0x11a0
[ 34.768098] ? kfree+0x27d/0x700
[ 34.768102] ? __lock_acquire+0x3d5/0x28b0
[ 34.768108] cacheinfo_cpu_online+0x29/0xb30
[ 34.768114] ? trace_hardirqs_on+0x18/0x1a0
[ 34.768120] cpuhp_invoke_callback+0x3f6/0x1530
[ 34.768126] ? __pfx_cacheinfo_cpu_online+0x10/0x10
[ 34.768132] ? cpuhp_thread_fun+0x306/0x800
[ 34.768137] ? cpuhp_thread_fun+0xba/0x800
[ 34.768143] ? __pfx_cpuhp_thread_fun+0x10/0x10
[ 34.768149] cpuhp_thread_fun+0x3e6/0x800
[ 34.768155] smpboot_thread_fn+0x42a/0x9e0
[ 34.768160] ? kthread+0x1b3/0x4e0
[ 34.768165] ? __pfx_smpboot_thread_fn+0x10/0x10
[ 34.768169] kthread+0x3e1/0x4e0
[ 34.768173] ? __pfx_kthread+0x10/0x10
[ 34.768177] ret_from_fork+0x8f1/0xcb0
[ 34.768182] ? __pfx_ret_from_fork+0x10/0x10
[ 34.768186] ? native_load_tls+0x14/0x80
[ 34.768195] ? __switch_to+0x838/0x1040
[ 34.768201] ? __pfx_kthread+0x10/0x10
[ 34.768205] ret_from_fork_asm+0x1a/0x30
[ 34.768213] </TASK>
[ 34.768214]
[ 34.774822] Allocated by task 21:
[ 34.774952] kasan_save_stack+0x2f/0x50
[ 34.775101] kasan_save_track+0x14/0x30
[ 34.775250] __kasan_kmalloc+0x9a/0xb0
[ 34.775396] __kmalloc_noprof+0x300/0x8c0
[ 34.775550] detect_cache_attributes+0x4b4/0x11a0
[ 34.775730] cacheinfo_cpu_online+0x29/0xb30
[ 34.775895] cpuhp_invoke_callback+0x3f6/0x1530
[ 34.776070] cpuhp_thread_fun+0x3e6/0x800
[ 34.776225] smpboot_thread_fn+0x42a/0x9e0
[ 34.776382] kthread+0x3e1/0x4e0
[ 34.776510] ret_from_fork+0x8f1/0xcb0
[ 34.776654] ret_from_fork_asm+0x1a/0x30
[ 34.776805]
[ 34.776871] The buggy address belongs to the object at ffff888003908000
[ 34.776871] which belongs to the cache kmalloc-4k of size 4096
[ 34.777308] The buggy address is located 32 bytes to the right of
[ 34.777308] allocated 3264-byte region [ffff888003908000, ffff888003908cc0)
[ 34.777767]
[ 34.777834] The buggy address belongs to the physical page:
[ 34.778061] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x3908
[ 34.778373] head: order:3 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0
[ 34.778648] flags: 0xfffffc0000040(head|node=0|zone=1|lastcpupid=0x1fffff)
[ 34.778897] page_type: f5(slab)
[ 34.779022] raw: 000fffffc0000040 ffff888001042140 dead000000000122 0000000000000000
[ 34.779300] raw: 0000000000000000 0000000800040004 00000000f5000000 0000000000000000
[ 34.779587] head: 000fffffc0000040 ffff888001042140 dead000000000122 0000000000000000
[ 34.779890] head: 0000000000000000 0000000800040004 00000000f5000000 0000000000000000
[ 34.780210] head: 000fffffc0000003 fffffffffffffe01 00000000ffffffff 00000000ffffffff
[ 34.780505] head: ffffffffffffffff 0000000000000000 00000000ffffffff 0000000000000008
[ 34.780783] page dumped because: kasan: bad access detected
[ 34.780985]
[ 34.781051] Memory state around the buggy address:
[ 34.781228] ffff888003908b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 34.781488] ffff888003908c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 34.781748] >ffff888003908c80: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc
[ 34.782011] ^
[ 34.782280] ffff888003908d00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 34.782585] ffff888003908d80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 34.782868] ==================================================================
[ 34.783147] Disabling lock debugging due to kernel taint
cpu1 online: apicid=1 num_leaves=4
predicted out-of-bounds write:
cpu1 (leaf index 3, 4 leaves) -> cpu0 (3 leaves)
sibling test: num_threads_sharing=64 index_msb=6, apicid 1>>6 == 0>>6 == 0
write: 8 bytes at info_list(cpu0) + 3*1088 + 32 = +3296,
32 bytes past the end of the 3264-byte allocation
KASAN should report: "Write of size 8" ... "located 32 bytes to the right of allocated 3264-byte region"
without KASAN: the write stays inside the 4096-byte kmalloc object's slack, so KASAN (or slub_debug=Z) is needed to see it
REPRODUCED: the write(2) above produced a KASAN report
--- kernel output -------------------------------------------
| SMP alternatives: switching to SMP code
| smpboot: Booting Node 0 Processor 1 APIC 0x1
| ==================================================================
| BUG: KASAN: slab-out-of-bounds in populate_cache_leaves+0x9d0/0x16d0
| Write of size 8 at addr ffff888003908ce0 by task cpuhp/1/112
|
| CPU: 1 UID: 0 PID: 112 Comm: cpuhp/1 Not tainted 7.2-amd64 #1 PREEMPT(lazy) Debian 7.2~rc7-1~exp1
| Hardware name: ChromiumOS crosvm, BIOS 0
| Call Trace:
| <TASK>
| dump_stack_lvl+0xd5/0x130
| print_report+0x14b/0x4b0
| ? __virt_addr_valid+0x186/0x4e0
| ? __virt_addr_valid+0x254/0x4e0
| kasan_report+0x108/0x130
| ? populate_cache_leaves+0x9d0/0x16d0
| ? populate_cache_leaves+0x9d0/0x16d0
| kasan_check_range+0x134/0x220
| populate_cache_leaves+0x9d0/0x16d0
| ? __pfx_populate_cache_leaves+0x10/0x10
| ? __kmalloc_noprof+0x450/0x8c0
| ? detect_cache_attributes+0x4b4/0x11a0
| detect_cache_attributes+0x323/0x11a0
| ? kfree+0x27d/0x700
| ? __lock_acquire+0x3d5/0x28b0
| cacheinfo_cpu_online+0x29/0xb30
| ? trace_hardirqs_on+0x18/0x1a0
| cpuhp_invoke_callback+0x3f6/0x1530
| ? __pfx_cacheinfo_cpu_online+0x10/0x10
| ? cpuhp_thread_fun+0x306/0x800
| ? cpuhp_thread_fun+0xba/0x800
| ? __pfx_cpuhp_thread_fun+0x10/0x10
| cpuhp_thread_fun+0x3e6/0x800
| smpboot_thread_fn+0x42a/0x9e0
| ? kthread+0x1b3/0x4e0
| ? __pfx_smpboot_thread_fn+0x10/0x10
| kthread+0x3e1/0x4e0
| ? __pfx_kthread+0x10/0x10
| ret_from_fork+0x8f1/0xcb0
| ? __pfx_ret_from_fork+0x10/0x10
| ? native_load_tls+0x14/0x80
| ? __switch_to+0x838/0x1040
| ? __pfx_kthread+0x10/0x10
| ret_from_fork_asm+0x1a/0x30
| </TASK>
|
| Allocated by task 21:
| kasan_save_stack+0x2f/0x50
| kasan_save_track+0x14/0x30
| __kasan_kmalloc+0x9a/0xb0
| __kmalloc_noprof+0x300/0x8c0
| detect_cache_attributes+0x4b4/0x11a0
| cacheinfo_cpu_online+0x29/0xb30
| cpuhp_invoke_callback+0x3f6/0x1530
| cpuhp_thread_fun+0x3e6/0x800
| smpboot_thread_fn+0x42a/0x9e0
| kthread+0x3e1/0x4e0
| ret_from_fork+0x8f1/0xcb0
| ret_from_fork_asm+0x1a/0x30
|
| The buggy address belongs to the object at ffff888003908000
| which belongs to the cache kmalloc-4k of size 4096
| The buggy address is located 32 bytes to the right of
| allocated 3264-byte region [ffff888003908000, ffff888003908cc0)
|
| The buggy address belongs to the physical page:
| page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x3908
| head: order:3 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0
| flags: 0xfffffc0000040(head|node=0|zone=1|lastcpupid=0x1fffff)
| page_type: f5(slab)
| raw: 000fffffc0000040 ffff888001042140 dead000000000122 0000000000000000
| raw: 0000000000000000 0000000800040004 00000000f5000000 0000000000000000
| head: 000fffffc0000040 ffff888001042140 dead000000000122 0000000000000000
| head: 0000000000000000 0000000800040004 00000000f5000000 0000000000000000
| head: 000fffffc0000003 fffffffffffffe01 00000000ffffffff 00000000ffffffff
| head: ffffffffffffffff 0000000000000000 00000000ffffffff 0000000000000008
| page dumped because: kasan: bad access detected
|
| Memory state around the buggy address:
| ffff888003908b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
| ffff888003908c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
| >ffff888003908c80: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc
| ^
| ffff888003908d00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
| ffff888003908d80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
| ==================================================================
| Disabling lock debugging due to kernel taint
-------------------------------------------------------------
prediction vs report:
allocated region : predicted 3264, reported 3264 MATCH
bytes to the right: predicted 32, reported 32 MATCH
summary: 1 CPU(s) onlined, 1 first-online transition(s) exercised the x86 populate_cache_leaves() path, 1 KASAN report(s) captured
TRIGGER_EXIT=0
root@(none):/# echo MARK_TOPO
MARK_TOPO
(cat $i/shared_cpu_list)"; done; doned 2>/dev/null) size=$(cat $i/size) shared=$(
cpu0/index0 L1 Data id=0 size=32K shared=0-1
cpu0/index1 L1 Instruction id=0 size=64K shared=0-1
cpu0/index2 L2 Unified id=0 size=4096K shared=0-1
cpu1/index0 L1 Data id=0 size=48K shared=0-1
cpu1/index1 L1 Instruction id=0 size=64K shared=0-1
cpu1/index2 L2 Unified id=0 size=2560K shared=0-1
cpu1/index3 L3 Unified id=0 size=12288K shared=0-1
"oot@(none):/# echo MARK_KASAN_TOTAL; dmesg | grep -c "KASAN: slab-out-of-bounds"
MARK_KASAN_TOTAL
1
Raw
Makefile
# SPDX-License-Identifier: GPL-2.0
CC ?= gcc
CFLAGS ?= -O2 -Wall -Wextra -Wno-unused-parameter -std=gnu99
LDFLAGS ?=
BINS = cacheinfo-oob-predict cacheinfo-oob-trigger
all: $(BINS)
# Statically linked copies, for dropping into a guest image that may not have
# a matching libc.
static: CFLAGS += -static
static: LDFLAGS += -static
static: $(BINS)
cacheinfo-oob-predict: cacheinfo-oob-predict.c cacheinfo_oob.h
$(CC) $(CFLAGS) -o $@ $< $(LDFLAGS)
cacheinfo-oob-trigger: cacheinfo-oob-trigger.c cacheinfo_oob.h
$(CC) $(CFLAGS) -o $@ $< $(LDFLAGS)
clean:
rm -f $(BINS)
.PHONY: all static clean
Raw
run-crosvm-repro.sh
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0
#
# run-crosvm-repro.sh - drive the userspace reproducer inside a crosvm guest.
#
# The guest is booted with two vCPUs whose CPUID leaf 4 is deliberately sampled
# from two different host core types, and with maxcpus=1 so that cpu1 stays
# offline. Nothing has faulted yet at that point: the reproducer inside the
# guest then writes to /sys/devices/system/cpu/cpu1/online and captures what
# the kernel prints.
#
# The orientation is deterministic, not a lottery: crosvm's vCPU thread calls
# set_vcpu_thread_scheduling() (which applies --cpu-affinity) before
# configure_vcpu() -> setup_cpuid(), and its CPUID leaf 4 arm executes the host
# CPUID instruction inline. So vcpu N samples the host CPU it is pinned to.
#
# usage: run-crosvm-repro.sh [-k bzImage] [-e HOSTCPU] [-p HOSTCPU] [-c NCPU]
# -k kernel to boot (default: the unpatched 7.2-rc7 KASAN build)
# -e host CPU for vcpu0 (default 4, must be a 3-leaf E-core)
# -p host CPU for vcpu1 (default 0, must be a 4-leaf P-core)
#
# vcpu0 gets the *fewer* leaves and vcpu1 the *more*: cpu0 is populated during
# boot, then cpu1 is onlined by the reproducer and indexes cpu0's shorter array.
set -u
HERE=$(cd "$(dirname "$0")" && pwd)
CROSVM=${CROSVM:-/home/debian-sid/crosvm/target/release/crosvm}
IMG=${IMG:-/home/debian-sid/.cache/syz-crosvm/image.qcow2}
KERNEL=/home/debian-sid/.cache/syz-crosvm/unpatched-7.2rc7/bzImage
ECPU=4
PCPU=0
while getopts "k:e:p:" o; do
case $o in
k) KERNEL=$OPTARG ;;
e) ECPU=$OPTARG ;;
p) PCPU=$OPTARG ;;
*) exit 3 ;;
esac
done
WD=$(mktemp -d /tmp/cacheinfo-oob.XXXXXX)
trap 'rm -rf "$WD"' EXIT
echo "### kernel : $KERNEL"
echo "### vcpu0 -> host cpu$ECPU (expected: the CPU with fewer cache leaves)"
echo "### vcpu1 -> host cpu$PCPU (expected: the CPU with more cache leaves)"
# ---------------------------------------------------------------- build+stage
make -C "$HERE" static >/dev/null || { echo "build failed"; exit 3; }
mkdir -p "$WD/stage"
cp "$HERE/cacheinfo-oob-predict" "$HERE/cacheinfo-oob-trigger" "$WD/stage/"
# mke2fs -d populates the image without needing root.
/sbin/mke2fs -q -t ext4 -F -d "$WD/stage" -b 1024 "$WD/repro.img" 8192 >/dev/null 2>&1 \
|| { echo "mke2fs failed"; exit 3; }
"$CROSVM" create_qcow2 --backing-file "$IMG" "$WD/root.qcow2" >/dev/null 2>&1
mkfifo "$WD/in"
exec 9<>"$WD/in"
# --------------------------------------------------------------------- boot it
"$CROSVM" run --socket "$WD/c.sock" \
--cpus num-cores=2 --cpu-affinity "0=$ECPU:1=$PCPU" \
--mem size=2048 \
--serial type=stdout,hardware=serial,console=true,earlycon=true,stdin=true \
--no-usb --disable-sandbox \
--block path="$WD/root.qcow2" \
--block path="$WD/repro.img" \
--params "root=/dev/vda1 rw console=ttyS0 init=/bin/bash net.ifnames=0 maxcpus=1" \
"$KERNEL" < "$WD/in" > "$WD/console.log" 2>&1 &
CPID=$!
sleep 35
{
printf '\n'
printf 'mount -t proc proc /proc 2>/dev/null; mount -t sysfs sys /sys 2>/dev/null\n'
printf 'mount -t devtmpfs dev /dev 2>/dev/null; ls /dev/vd*\n'
printf 'mkdir -p /mnt/repro; mount -t ext4 /dev/vdb /mnt/repro 2>&1; ls /mnt/repro\n'
printf 'echo MARK_BOOTCLEAN; dmesg | grep -c "KASAN: slab-out-of-bounds"\n'
printf 'echo MARK_ONLINE; cat /sys/devices/system/cpu/online\n'
printf 'echo MARK_PREDICT_BEGIN; /mnt/repro/cacheinfo-oob-predict; echo "PREDICT_EXIT=$?"\n'
printf 'echo MARK_TRIGGER_BEGIN; /mnt/repro/cacheinfo-oob-trigger --all; echo "TRIGGER_EXIT=$?"\n'
printf 'echo MARK_TOPO\n'
printf 'for c in /sys/devices/system/cpu/cpu[0-9]*; do for i in $c/cache/index*; do [ -e "$i" ] || continue; echo "$(basename $c)/$(basename $i) L$(cat $i/level) $(cat $i/type) id=$(cat $i/id 2>/dev/null) size=$(cat $i/size) shared=$(cat $i/shared_cpu_list)"; done; done\n'
printf 'echo MARK_KASAN_TOTAL; dmesg | grep -c "KASAN: slab-out-of-bounds"\n'
printf 'echo MARK_DONE\n'
} >&9
sleep 45
echo "===== guest output ====="
sed -n '/MARK_BOOTCLEAN/,/MARK_DONE/p' "$WD/console.log" | sed 's/\r$//'
echo "===== end ====="
cp "$WD/console.log" "$HERE/last-console.log" 2>/dev/null
exec 9>&-
kill "$CPID" 2>/dev/null
wait "$CPID" 2>/dev/null
echo "(full console log copied to $HERE/last-console.log)"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment