Skip to content

Instantly share code, notes, and snippets.

@zooko
Created April 1, 2026 18:36
Show Gist options
  • Select an option

  • Save zooko/b15a15b9d8eff9ad085d6e8b8212bb87 to your computer and use it in GitHub Desktop.

Select an option

Save zooko/b15a15b9d8eff9ad085d6e8b8212bb87 to your computer and use it in GitHub Desktop.
--- log of AI interactions:
---- GPT 5.4:
prompt:
I'm writing a memory allocator. It is already extremely simple — far simpler than comparable memory allocators like mimalloc, snmalloc, or rpmalloc — and extremely fast. Now I'm thinking about "hardening" against exploitation. Hardening against exploitation is a complicated topic, and I find it difficult to assess which kinds of hardening actually provide the most "bang for the buck" in terms of stopping attacks effectively while adding a minimal cost in terms of complexity and runtime. How would you go about determining what "hardening" features provide the most real-world protection?
Here's the current source code:
{ copy of 0f4643535037a116abdb1a0c3801b0e09f748d84 }
response:
I’d do this as an exploit-primitive study, not as a feature checklist.
The strongest signal from production hardened allocators is not “add every mitigation.” It’s “protect allocator metadata first, then add a small amount of temporal and spatial friction, and be skeptical of expensive features unless measurements justify them.” Scudo keeps checksummed per-chunk metadata and emphasizes randomness, but its quarantine is explicitly described as fairly costly and is disabled by default. PartitionAlloc isolates most metadata out-of-line behind guard pages and hardens freelist pointers with encoding plus a differently encoded shadow copy. hardened_malloc’s lighter configuration keeps zero-on-free and canaries, but drops slot randomization, slab quarantines, and write-after-free checking; its metadata sealing feature is off by default because of significant performance cost. That combination of defaults is a pretty good prior for “bang for the buck.” llvm.org
For your allocator specifically, the obvious attack surface is visible in the code you pasted:
freed-slot links are stored inline in the freed slot itself, in a raw 32-bit word
most integrity checks aredebug_assert!, so they disappear in release
reuse is immediate, so temporal bugs stay highly deterministic
there is no spatial overflow absorber/detector for small allocations
That means I would evaluate features by how well they break these four exploit primitives:
freelist poisoning
invalid free / forged free
use-after-free with immediate same-size reuse
small linear overflow into adjacent state
The method I would use
Build a tiny corpus of allocator-focused exploit harnesses, not just correctness tests.
For each size class you care about, write tests that do:
overwrite the first 4 bytes of a freed slot with a forged next pointer
partially overwrite that word
double-free
free an interior pointer
free a forged pointer with valid-looking sizeclass bits
UAF write, then immediately allocate same size
UAF write, then allocate through a few intervening frees
1-byte, 2-byte, 8-byte, and cache-line-sized overflows past the requested size
underflows by a few bytes
cross-thread cases, because your slab failover is thread-influenced
Then score each hardening option on:
prevented before useful primitive exists
detected before reuse
detected only later
silent success
Run each test thousands of times with ASLR on, because randomness-based hardening is about success probability, not just pass/fail. Separately record p50/p99 alloc/free latency, total CPU, RSS/committed memory, and code size. That gives you a real “security gained per cost paid” number instead of intuition.
What I would try first
1. Always-on validation and aborts on the deallocation / freelist path
This is the first thing I’d do.
In a hardened build, your currentdebug_assert!-only checks should become real checks that abort on failure, at least on:
-dealloc pointer validity
decoded freelist entry validity
impossible sentinel / slot-number relationships
This is cheap, fits your current design, and turns many “heap corruption becomes allocator metadata corruption” cases into immediate process death instead.
Why first? Because all the major hardened allocators prioritize metadata integrity in some form: Scudo checks chunk headers, PartitionAlloc protects allocator metadata and validates freelist structure, and hardened_malloc isolates allocator metadata. llvm.org
2. Encode freelist entries, ideally with a shadow/check value
Your raw inline freelist word is the most obvious exploit target.
The PartitionAlloc pattern is very instructive here: freelist pointers are encoded, and a shadow copy encoded differently is stored alongside them to detect corruption. That is exactly the kind of feature that tends to have excellent security/complexity ratio, because it targets a primary exploitation primitive directly. chromium.googlesource.com
For your design, I would strongly prefer something that uses no new out-of-line per-slot metadata, because your smallest classes imply enormous slot counts. So the best fit is:
encode the next-entry word with a per-process or per-slab secret and the slot address
on classes with enough room, store a second check word with a different encoding
on pop, decode and validate before CAS succeeds
on failure, abort, not recover
That specifically raises the cost of freelist poisoning and partial overwrites.
3. A very small quarantine or delayed-reuse window
Your allocator currently makes UAF reuse very deterministic.
Scudo documents quarantine as effective against some UAFs but fairly costly, and disables it by default. hardened_malloc keeps quarantines in its stronger profile but removes them from its lighter one. My read is: quarantine works, but you should start tiny and size it only after measurement. llvm.org
So I would not jump to a big quarantine. I would prototype a tiny per-thread FIFO for small size classes only. Even 8–32 objects per class is enough to test whether you meaningfully reduce exploit determinism for your target workloads. If it helps a lot and costs little, grow it. If it hurts too much, kill it.
4. Zero-on-free, optionally with zero-check-on-alloc
hardened_malloc’s light profile keeps zero-on-free and explicitly treats it as one of the baseline features worth retaining, while dropping slot randomization and quarantine. It also has an optional write-after-free check that verifies zero filling remained intact on reallocation. That is a strong hint that zero-on-free is often a better baseline purchase than more elaborate randomization. github.com
For your allocator, zero-on-free is attractive because:
it mitigates stale-data disclosure
it weakens some UAF read/write patterns
it composes well with delayed reuse
it doesn’t require extra metadata
I would benchmark it only for smaller classes first, because the cost scales with size. If you want to be stricter, add “verify still zero on alloc” in hardened mode; that catches post-free writes, though at more cost. hardened_malloc documents exactly that tradeoff. github.com
5. Tail canaries, but only if they’re almost free in your design
I would rank canaries below freelist hardening and delayed reuse.
hardened_malloc’s docs are useful here: its canaries are mainly to absorb small fixed-size overflows and C-string overruns; checking them on free helps detect corruption, but the project explicitly says checking on free is often too late to prevent exploitation, so that is not their main purpose. github.com
The nice part for you is that Rust’sGlobalAlloc::dealloc andrealloc give you the originalLayout, so you already know the requested size at free time. That means you can place a canary atptr + requested_size and check it later without adding “requested size” metadata. I would exploit that.
The version I’d test first is:
only place a canary when there is already slack in the chosen size class
do not bump to a larger size class just to fit a canary, at least initially
That gives you most of the benefit on non-power-of-two sizes at nearly zero complexity cost.
What I would deprioritize
Slot randomization by itself
hardened_malloc explicitly says slot randomization has measurable cost and is not one of the important security features. I would believe that unless your exploit corpus shows otherwise. github.com
So I would not spend much effort on “pick a more random slot” before:
freelist integrity
delayed reuse
zero-on-free
Metadata sealing with MPK / similar mechanisms
This can be strong, but hardened_malloc keeps its metadata sealing option off by default because of significant performance cost, even though the allocator already isolates metadata in a dedicated region. That tells me it’s a later-stage feature, not your first hardening dollar. github.com
Heavyweight quarantine by default
Scudo says its quarantine is fairly costly and disables it by default. So if you do quarantine, make it small and prove it earns its keep. llvm.org
One thing I would absolutely add for measurement, not baseline mitigation
A sampled guard-page mode.
GWP-ASan samples a very small percentage of allocations and applies guard pages only to those sampled allocations, which LLVM documents as giving very low performance overhead. That is not a universal mitigation, but it is excellent for answering “what bugs do I actually have in the field?” llvm.org
So I would maintain two separate production-grade knobs:
hardening mode: the cheap always-on protections above
sampling mode: a GWP-ASan-style telemetry/debug path
Those solve different problems.
My tentative ranking for your allocator
If I were placing bets before benchmarking, my order would be:
Always-on dealloc/freelist validity checks + abort
Encoded freelist entries, with shadow/check where slot size allows
Tiny delayed-reuse window for small classes
Zero-on-free for small classes
Optional zero-check-on-alloc in hardened mode
Slack-only tail canaries
Later: larger quarantines, slot randomization, metadata sealing
That order follows both your code’s current weak points and the priorities visible in existing hardened allocators. llvm.org
The key mindset
Don’t ask:
“Which defenses sound good?”
Ask:
“Which exploit primitive does this break in my allocator?”
“Does it fail closed before the attacker gets a useful primitive?”
“How many nanoseconds and how much complexity did that cost me?”
If you want, I can turn this into a concrete hardening plan for your exact codebase, with a staged order like:
phase 1: no-layout-change hardening
phase 2: one-word-per-free-slot hardening
phase 3: temporal hardening
phase 4: optional debug/sampled modes
and I can keep it tightly constrained to your performance rules.
---- Claude 4.6 Opus
’d do this as an exploit-primitive study, not as a feature checklist.
The strongest signal from production hardened allocators is not “add every mitigation.” It’s “protect allocator metadata first, then add a small amount of temporal and spatial friction, and be skeptical of expensive features unless measurements justify them.” Scudo keeps checksummed per-chunk metadata and emphasizes randomness, but its quarantine is explicitly described as fairly costly and is disabled by default. PartitionAlloc isolates most metadata out-of-line behind guard pages and hardens freelist pointers with encoding plus a differently encoded shadow copy. hardened_malloc’s lighter configuration keeps zero-on-free and canaries, but drops slot randomization, slab quarantines, and write-after-free checking; its metadata sealing feature is off by default because of significant performance cost. That combination of defaults is a pretty good prior for “bang for the buck.” llvm.org
For your allocator specifically, the obvious attack surface is visible in the code you pasted:
freed-slot links are stored inline in the freed slot itself, in a raw 32-bit word
most integrity checks aredebug_assert!, so they disappear in release
reuse is immediate, so temporal bugs stay highly deterministic
there is no spatial overflow absorber/detector for small allocations
That means I would evaluate features by how well they break these four exploit primitives:
freelist poisoning
invalid free / forged free
use-after-free with immediate same-size reuse
small linear overflow into adjacent state
The method I would use
Build a tiny corpus of allocator-focused exploit harnesses, not just correctness tests.
For each size class you care about, write tests that do:
overwrite the first 4 bytes of a freed slot with a forged next pointer
partially overwrite that word
double-free
free an interior pointer
free a forged pointer with valid-looking sizeclass bits
UAF write, then immediately allocate same size
UAF write, then allocate through a few intervening frees
1-byte, 2-byte, 8-byte, and cache-line-sized overflows past the requested size
underflows by a few bytes
cross-thread cases, because your slab failover is thread-influenced
Then score each hardening option on:
prevented before useful primitive exists
detected before reuse
detected only later
silent success
Run each test thousands of times with ASLR on, because randomness-based hardening is about success probability, not just pass/fail. Separately record p50/p99 alloc/free latency, total CPU, RSS/committed memory, and code size. That gives you a real “security gained per cost paid” number instead of intuition.
What I would try first
1. Always-on validation and aborts on the deallocation / freelist path
This is the first thing I’d do.
In a hardened build, your currentdebug_assert!-only checks should become real checks that abort on failure, at least on:
-dealloc pointer validity
decoded freelist entry validity
impossible sentinel / slot-number relationships
This is cheap, fits your current design, and turns many “heap corruption becomes allocator metadata corruption” cases into immediate process death instead.
Why first? Because all the major hardened allocators prioritize metadata integrity in some form: Scudo checks chunk headers, PartitionAlloc protects allocator metadata and validates freelist structure, and hardened_malloc isolates allocator metadata. llvm.org
2. Encode freelist entries, ideally with a shadow/check value
Your raw inline freelist word is the most obvious exploit target.
The PartitionAlloc pattern is very instructive here: freelist pointers are encoded, and a shadow copy encoded differently is stored alongside them to detect corruption. That is exactly the kind of feature that tends to have excellent security/complexity ratio, because it targets a primary exploitation primitive directly. chromium.googlesource.com
For your design, I would strongly prefer something that uses no new out-of-line per-slot metadata, because your smallest classes imply enormous slot counts. So the best fit is:
encode the next-entry word with a per-process or per-slab secret and the slot address
on classes with enough room, store a second check word with a different encoding
on pop, decode and validate before CAS succeeds
on failure, abort, not recover
That specifically raises the cost of freelist poisoning and partial overwrites.
3. A very small quarantine or delayed-reuse window
Your allocator currently makes UAF reuse very deterministic.
Scudo documents quarantine as effective against some UAFs but fairly costly, and disables it by default. hardened_malloc keeps quarantines in its stronger profile but removes them from its lighter one. My read is: quarantine works, but you should start tiny and size it only after measurement. llvm.org
So I would not jump to a big quarantine. I would prototype a tiny per-thread FIFO for small size classes only. Even 8–32 objects per class is enough to test whether you meaningfully reduce exploit determinism for your target workloads. If it helps a lot and costs little, grow it. If it hurts too much, kill it.
4. Zero-on-free, optionally with zero-check-on-alloc
hardened_malloc’s light profile keeps zero-on-free and explicitly treats it as one of the baseline features worth retaining, while dropping slot randomization and quarantine. It also has an optional write-after-free check that verifies zero filling remained intact on reallocation. That is a strong hint that zero-on-free is often a better baseline purchase than more elaborate randomization. github.com
For your allocator, zero-on-free is attractive because:
it mitigates stale-data disclosure
it weakens some UAF read/write patterns
it composes well with delayed reuse
it doesn’t require extra metadata
I would benchmark it only for smaller classes first, because the cost scales with size. If you want to be stricter, add “verify still zero on alloc” in hardened mode; that catches post-free writes, though at more cost. hardened_malloc documents exactly that tradeoff. github.com
5. Tail canaries, but only if they’re almost free in your design
I would rank canaries below freelist hardening and delayed reuse.
hardened_malloc’s docs are useful here: its canaries are mainly to absorb small fixed-size overflows and C-string overruns; checking them on free helps detect corruption, but the project explicitly says checking on free is often too late to prevent exploitation, so that is not their main purpose. github.com
The nice part for you is that Rust’sGlobalAlloc::dealloc andrealloc give you the originalLayout, so you already know the requested size at free time. That means you can place a canary atptr + requested_size and check it later without adding “requested size” metadata. I would exploit that.
The version I’d test first is:
only place a canary when there is already slack in the chosen size class
do not bump to a larger size class just to fit a canary, at least initially
That gives you most of the benefit on non-power-of-two sizes at nearly zero complexity cost.
What I would deprioritize
Slot randomization by itself
hardened_malloc explicitly says slot randomization has measurable cost and is not one of the important security features. I would believe that unless your exploit corpus shows otherwise. github.com
So I would not spend much effort on “pick a more random slot” before:
freelist integrity
delayed reuse
zero-on-free
Metadata sealing with MPK / similar mechanisms
This can be strong, but hardened_malloc keeps its metadata sealing option off by default because of significant performance cost, even though the allocator already isolates metadata in a dedicated region. That tells me it’s a later-stage feature, not your first hardening dollar. github.com
Heavyweight quarantine by default
Scudo says its quarantine is fairly costly and disables it by default. So if you do quarantine, make it small and prove it earns its keep. llvm.org
One thing I would absolutely add for measurement, not baseline mitigation
A sampled guard-page mode.
GWP-ASan samples a very small percentage of allocations and applies guard pages only to those sampled allocations, which LLVM documents as giving very low performance overhead. That is not a universal mitigation, but it is excellent for answering “what bugs do I actually have in the field?” llvm.org
So I would maintain two separate production-grade knobs:
hardening mode: the cheap always-on protections above
sampling mode: a GWP-ASan-style telemetry/debug path
Those solve different problems.
My tentative ranking for your allocator
If I were placing bets before benchmarking, my order would be:
Always-on dealloc/freelist validity checks + abort
Encoded freelist entries, with shadow/check where slot size allows
Tiny delayed-reuse window for small classes
Zero-on-free for small classes
Optional zero-check-on-alloc in hardened mode
Slack-only tail canaries
Later: larger quarantines, slot randomization, metadata sealing
That order follows both your code’s current weak points and the priorities visible in existing hardened allocators. llvm.org
The key mindset
Don’t ask:
“Which defenses sound good?”
Ask:
“Which exploit primitive does this break in my allocator?”
“Does it fail closed before the attacker gets a useful primitive?”
“How many nanoseconds and how much complexity did that cost me?”
If you want, I can turn this into a concrete hardening plan for your exact codebase, with a staged order like:
phase 1: no-layout-change hardening
phase 2: one-word-per-free-slot hardening
phase 3: temporal hardening
phase 4: optional debug/sampled modes
and I can keep it tightly constrained to your performance rules.
---- Gemini 3.1 Pro (Preview)
Assessing "bang for the buck" in allocator hardening requires identifying which exploit primitives are most common and neutralizing them with the fewest CPU cycles. Your allocator already possesses a massive inherent security advantage: by deriving size classes and metadata from pointer addresses (block-addressing), you avoid inline chunk headers. This completely eliminates the classic glibcptmalloc2 vector where an out-of-bounds write corrupts the adjacent chunk's header metadata.
Based on modern hardened allocators like LLVM Scudo and GrapheneOS hardened_malloc, here is a breakdown of the specific mitigations that offer the highest real-world protection against the lowest performance cost.
High Protection, Zero or Nano-second Cost
1. Free-list Pointer Obfuscation
Your code currently writes the free-list link directly into the freed user payload:*(p_addr as *mut u32) = curfirstentry. If an attacker triggers a use-after-free or linear overflow into a freed chunk, they can overwrite this pointer to force the allocator to return an arbitrary memory address on the next allocation map.
Because you operate at the nanosecond scale, you cannot offload this free-list into out-of-band metadata without major cache-locality and complexity costs. Instead, you can mask the pointer using a cryptographically random thread-local or global secret mixed with the address of the pointer itself.
If the attacker does not know the secret and the heap layout, they cannot forge a valid inline free-list pointer.
// Generate a random 32-bit secret during initialization
static SECRET: AtomicU32 = AtomicU32::new(0);
// On dealloc, mask the true slotnum:
let addr_entropy = (p_addr >> 4) as u32;
let obfuscated = curfirstentry ^ addr_entropy ^ SECRET.load(Relaxed);
unsafe { *(p_addr as *mut u32) = obfuscated };
// On alloc, unmask the slotnum:
let obfuscated = unsafe { *(curfirstentry_p as *mut u32) };
let addr_entropy = (curfirstentry_p >> 4) as u32;
let revealed = obfuscated ^ addr_entropy ^ SECRET.load(Relaxed);
// If the revealed slotnum exceeds sc_to_sentinel_slotnum(sc), crash immediately.
assert!(revealed & ENTRY_SLOTNUM_MASK <= sentinel_slotnum);
2. Virtual Memory Guard Pages
GrapheneOS hardened_malloc relies heavily on guard slabs—unmapped pages between memory regions. Because you are allocating the maximum amount of virtual memory upfront and do not need to minimize virtual address space usage, guard regions cost you absolute zero in physical memory, zero in syscall overhead, and zero in computation.
Modify your address bit layout to leave intentional, unmapped gaps betweenTOTAL_VIRTUAL_MEMORY slabs or size classes. Any linear buffer overflow that breaches the boundaries of a slab will immediately hit an unmapped page and trigger a hardware page fault (segfault), terminating the process before it can compromise adjacent allocator structures.
3. Hardware Memory Tagging (MTE)
Both Scudo and GrapheneOS integrate ARM Memory Tagging Extension (MTE) rules. If your target architecture supports it, MTE provides memory safety with effectively zero software overhead. When a chunk is freed, its tag is invalidated. Any dangling pointer dereference (use-after-free) instantly faults. As Scudo documentation notes, this also allows you to zero-fill memory on allocation for "free" by leaning on the hardware retagging instruction.
Lower Protection or High Cost (To Avoid)
Some features commonly found in hardened allocators handle edge cases but will ruin the performance of an extremely fast allocator.
1. Delayed Free Lists (Quarantine)
Scudo implements a delayed freelist to mitigate use-after-free vulnerabilities. A chunk is not immediately placed back into circulation but sits in a quarantine queue. The LLVM documentation notes this feature is "fairly costly in terms of performance and memory footprint." Forsmalloc, a quarantine requires tracking timestamps or counter epochs, trashing your L1 cache efficiency and heavily bloating your atomic compare-and-exchange logic.
2. Explicit Zeroing on Free
GrapheneOS completely wipes the memory of a slot by zeroing it out on free. While this destroys sensitive data and prevents data leaks across allocations, clearing a cacheline costs dozens of CPU cycles.smalloc correctly offers analloc_zeroed for when the caller needs zeroed memory. Enforcing it globally ondealloc destroys benchmark performance.
3. Metadata Checksums
Scudo uses a checksumming algorithm to validate chunk headers before operating on them. Because your allocator design stores metadata out-of-band in theflh and derives properties mathematically viaSLABNUM_BITS_ADDR_MASK, you do not have inline headers to checksum. Adding headers just to checksum them would introduce unnecessary memory accesses.
Summary Comparison
Mitigation
Exploit
Runtime
Complexity
Guard Pages
Linear overflow
Zero
Low
Address Randomization
Heap spraying
Zero
Low
Pointer Obfuscation
Free list hijack
~1ns
Low
Hardware MTE
Use after free
Zero
Low
Zero on Free
Information leak
High
Low
Quarantine
Use after free
High
High
---- Grok 4.20 Beta Reasoning
Evaluating hardening features for a minimal, high-performance allocator
Determining the highest "bang for the buck" requires a structured threat-model-first approach rather than adding every feature from hardened_malloc or Scudo. The goal is to stop the most common real-world exploitation primitives while preserving your allocator's extreme simplicity and speed.
Step-by-step methodology
Map the dominant heap exploitation primitives
The most common classes (from years of browser, OS, and application CVEs) are:
Use-after-free (UAF): Freed memory is reused while still referenced. Very common in C/C++.
Heap buffer overflow into adjacent objects or metadata.
Double-free / invalid free leading to free-list corruption.
Metadata corruption turning a small bug into arbitrary read/write (classic ptmalloc unlink attacks).
Study proven hardened allocators
GrapheneOS hardened_malloc: Emphasizes guard slabs/regions, zeroing on free, canaries, and strong pointer validation. It accepts high virtual memory usage and many mappings for isolation. It is explicitly designed as a security-focused replacement with "substantial defenses against the most common classes of vulnerabilities (heap memory corruption)".
LLVM Scudo (Android's default since Android 11): Uses size-class separation, randomized mapping, header checksums, and an optional delayed freelist. It is pragmatic — security improvements without catastrophic performance regression.
glibc ptmalloc2 internals and papers on heap exploitation show how traditional chunk headers and free-list unlink operations were exploited for decades.
Score each candidate feature
For each idea, ask:
Which of the top 3-4 primitives does it stop or make unreliable?
What is the added code complexity in your design?
What is the runtime cost (cycles on hot alloc/dealloc paths, memory bandwidth)?
Does it break your current near-zero-metadata, address-encoded design?
Measure, don't guess
Add one feature at a time, then benchmark your existing micro-benchmarks plus real workloads. Use tools like GWP-ASan (available in Scudo) or AddressSanitizer to quantify how many bugs the feature would have caught.
High-value features for your specific design
Your allocator already has several excellent properties for hardening:
Almost no traditional per-chunk metadata (links live in user memory when freed).
Everything encoded in address bits (size class, slab, slot). This gives strong spatial separation.
64 slabs per size class with thread-local slab selection and failover on collision.
-is_smalloc_ptr already performs several validation checks.
Massive virtual address reservation (you explicitly allocate the maximum up front).
This address-encoding approach is similar in spirit to some of the partitioning techniques used in modern hardened allocators.
Strongly recommended (best bang-for-buck):
Zeroing on free (for larger allocations)
Reduces sensitive data lifetime and makes many UAFs less useful for leaking or controlling data. hardened_malloc does this. Youralloc_zeroed path already conditionally zeros touched memory. Extending this todealloc for slots above a certain size (e.g. ≥ 128 bytes) is relatively cheap on modern CPUs (especially withrep stosb optimizations). Avoid zeroing tiny slots on every free if you care about nanoseconds.
Strengthen pointer validation
Expandis_smalloc_ptr and call it on more paths. Your current checks (address range, size class bits, sentinel slot avoidance, untouched bit) are already good. Adding a simple canary or checksum in the free-list link word (the first 4 bytes) would catch many corruptions with very low cost. hardened_malloc and Scudo both protect metadata integrity.
Delayed/reuse-resistant freelist (quarantine lite)
Scudo's delayed freelist is one of the most effective UAF mitigations. You could keep a small per-slab quarantine ring (e.g. 4-8 entries) before actually linking into the main free list. This is higher cost but extremely effective against the most common UAF exploitation pattern.
Lower priority for your design:
Full guard pages between every allocation: Your massive VA space and slab separation already give you significant isolation. Adding guard slabs (like hardened_malloc) could be done by simply never allocating certain slab numbers, but the cost in virtual memory and complexity may not be worth it given your current layout.
Heavy randomization: You already use thread ID for slab selection. Adding more randomness to initial slab choice or free list shuffling helps against spraying but adds complexity.
Full MTE support: Valuable on ARM but requires hardware and changes the ABI.
Features to avoid initially:
Complex per-chunk headers (you deliberately avoided this).
Thread caches / tcache (hardened_malloc explicitly avoids these for security reasons — they create more attack surface and are less suitable for hardened designs).
Concrete next steps for smalloc
Read the GrapheneOS hardened_malloc README and the Synacktiv article on its internals (both appear in the search results). Pay special attention to their canary and zeroing strategy.
Read the Scudo documentation for its pragmatic checksum and delayed freelist approach.
Add zeroing on free (guarded by size) and a canary in the free-list link as the first two experiments. Both fit your current 4-byte link design with minimal code.
Measure the performance impact on your hottest paths.
Strengthendealloc validation so that double-frees and invalid frees are reliably detected (youris_smalloc_ptr is a great foundation).
Your allocator's extreme minimalism is a feature for security analysis — fewer places for bugs to hide. The address-encoding trick already puts you ahead of traditional dlmalloc-style designs on spatial safety. Focus on making UAF and free-list corruption unreliable first; those give the largest real-world protection per line of code added.
If you want concrete code suggestions for any of these (e.g. where exactly to put a canary in the FLH update or zeroing logic), provide a specific feature and I can show minimal changes that respect your "no redundancy, minimize wall-clock time, assertions instead of defensive code" constraints.
---- Zooko's notes
Okay, three of the AIs recommend some kind of free list obfuscation as the first additional defense and the fourth (GPT 5.4) recommended it as the second additional defense—the first one being turning on validation and abort on that path (i.e. make those validation tests `assert` instead of `debug_assert`). :-) So I guess I'll investigate this free list obfuscation thing. All four AIs described UAF as being the most important/common exploit path. Need to learn more about that…
Okay I asked each of the four AIs the following question and read all their answers: "Okay, tell me more about UAF. How exactly does a modern, practical exploit use UAF to proceed to gain further control of the process? Tell me step by step, so I can understand how the design of smalloc could interact with that exploit chain."
Based on their answers, it sounds to me like there are two sub-classes of UAF exploitation: allocator metadata corruption and type confusion. And it sounds like type confusion is the more powerful and common of the two.
By the way, of the four AIs, Gemini 3.1 Pro (Preview) explained the different sub-classes most clearly, which was very helpful to me, and Claude 4.6 Opus explained the type-confusion path in the most detail (although it completely skipped the allocator metadata corruption path), which was also helpful to me. Grok 4.20 Beta Reasoning's and GPT 5.4's answers were less helpful to me.
Anyway, at the moment it seems to me like there are two techniques we could apply to impede the type confusion path (the second and more important of the two mentioned above). Technique 1: MTE (on CPUs that support it), Technique 2:
"Hm, smalloc already has to write into the first 4 bytes of the slot on every free, to update that slot's next-slot pointer. It would be more or less free to in the same memory access overwrite the first cache line's worth (64 or 128 bytes depending on CPU architecture), right? What exact Rust code would do that, and what assembly code would result on amd64 and arm64, and would it take much longer than the 4-byte write does? And, my *guess* is that most of the "high value objects" that attackers want to achieve type-confusion with will be less than or equal to 64 bytes in size. What do you think?"
Grok and Gemini responded positively, effectively affirming my *confused!* implication that zeroing would prevent type confusion. GPT 5.4 pointed out to me that this wouldn't prevent type confusion. :-) Claude returned an error message. :-)
So basically I think MTE is the only practical technique to protect against type confusion attacks. Except for maybe the "linked lists become fifo queues instead of lifo stacks" approach (see the "Future Work" section of smalloc's README.md) but I suspect that would cause virtual memory thrash in at least some cases.
---
Okay I went another round with each of the 4 AIs:
```text
Hm, the thing is, this whole "quarantining freed allocations" thing…seems like the attacker would sometimes be able to work around that. Okay but maybe not. They are after all limited to using a "weird machine". Okay, what if we had an array of, say, I dunno, 64 or 256 or 1024 words (4 byte words) to hold the slotnumbers of recently-freed slots. Then if we had a counter showing which element in that array was next and we incremented the counter, it would be a fifo queue. I would prefer a randomized algorithm where a random element gets evicted out of the array when a new element is to be added, but there's no efficient source of randomness for this purpose. Remember: smalloc is minimal in code. Minimal, minimal, minimal! Here is the entire implementation ofdealloc!
#[inline(always)]
pub fn dealloc(&self, p_addr: usize) {
debug_assert!(self.is_smalloc_ptr(p_addr));
// Okay now we know that it is a pointer into smalloc's region.
// The sizeclass is encoded into these bits of the address:
let sc = ptr_to_sc(p_addr);
debug_assert!(sc >= NUM_UNUSED_SCS);
debug_assert!(sc < NUM_SCS);
let flhptr = self.smbp.load(Relaxed) | ptr_to_flhaddr(p_addr);
let flh = unsafe { AtomicU64::from_ptr(flhptr as *mut u64) };
let newslotnum = ptr_to_slotnum(p_addr);
let sentinel_slotnum = sc_to_sentinel_slotnum(sc);
debug_assert!(newslotnum < sentinel_slotnum);
loop {
// Load the value (current first entry slotnum and next-entry-touched bit) from the
// flh
let flhword = flh.load(Relaxed);
// The low-order 4-byte word is the slotnum and the touched-bit of the first entry
let curfirstentry = flhword as u32;
let curfirstentryslotnum = curfirstentry & ENTRY_SLOTNUM_MASK;
debug_assert!(newslotnum != curfirstentryslotnum);
// The curfirstentryslotnum can be the sentinel slotnum but not greater.
debug_assert!(curfirstentryslotnum <= sentinel_slotnum);
// Write it into the new slot's link
unsafe { *(p_addr as *mut u32) = curfirstentry };
// The high-order 4-byte word is the push counter. Increment it.
let push_counter = (flhword & FLHWORD_PUSH_COUNTER_MASK).wrapping_add(FLHWORD_PUSH_COUNTER_INCR);
// The new flh word is the push counter, the next-entry-touched-bit (set), and the
// next-entry slotnum.
let newflhword = push_counter | ENTRY_NEXT_TOUCHED_BIT as u64 | newslotnum as u64;
// Compare and exchange
if flh.compare_exchange_weak(flhword, newflhword, Release, Relaxed).is_ok() {
break;
}
}
}
```
GPT 5.4 gave me what seems like a some bad or at least less-helpful advice on quarantining, but also reminded me of a good point — that setting the link in a freed allocation was useful hardening itself because it clobbers the first 4 bytes. Oh! And actually now that I look back at it, GPT 5.4 pointed out an excellent point: that sharing the quarantine/cache across all size classes would allow the attacker to flush it by churning a different size class than their target. Great point!
Claude 4.6 Opus gave me a helpful idea — leverage the ASLR of the stack pointer for pseudo-randomization — plus what looks like reasonable source code implementating it.
Gemini 3.1 Pro (Preview) did likewise, recommending an 3-instruction XOR-Shift PRNG.
Grok 4.20 Beta Reasoning advised me to abandon the quarantine and focus on zeroing, which I don't think is good advice since UAF->TypeConfusion is the biggest threat, and quarantining helps against that and zeroing (mostly) doesn't.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment