~/posts/binary/nvidia-thread-state-oops-vmalloc-lifetime.md

Stale tree nodes after an Oops: NVIDIA driver UAF

Trace stack-backed thread state through an Oops, vmalloc reuse, and constrained red-black-tree writes, then compare the null checks and heap-backed lifetime in NVIDIA's release sources.

date[31:24]
read[23:16]
10 min
cat[15:8]
Binary
Contents
  1. 0x00Scope and evidence
  2. 0x01Why the global tree retains a stack address
  3. 0x02Freeing is not immediate reuse
  4. 0x03Connecting a user mapping to the tree
  5. 0x04From child pointers to address calibration
  6. 0x05A right rotation provides a constrained pointer store
  7. 0x06Stable distances do not remove the race
  8. 0x07Fix the entry condition and the lifetime
  9. 0x08References

In NVIDIA's 570.86.15 open Linux driver, a thread-state object allocated on a task's stack is registered in a global red-black tree. If a kernel fault interrupts cleanup, the task's stack may eventually be reclaimed while the tree still points into it. A null-pointer fault can therefore leave the conditions for a use-after-free.

The important connection is object lifetime across the fault, followed by virtual-address reuse and later tree operations. The historical environment was Ubuntu Noble, kernel 6.11.0-24-generic, and a T550 Laptop GPU. The analysis below separates that record from pinned source inspection and pure-data models; no driver was loaded or kernel privilege-escalation chain rerun on the current machine.

Scope and evidence

Keep the evidence classes separate4 rows
Material What it establishes Limit
570.86.15 and 570.195.03 release sources Registration, cleanup, and null-check changes Release snapshots, not individual fix commits
Upstream Linux v6.11 Stack caches, deferred freeing, and vmalloc pools Not the complete Ubuntu distribution build
Historical terminal recording Driver, GPU, and privilege result at that time Not a reproduction on a current version
Offline models Layouts, thresholds, pointer stores, and relative distances No measurement of kernel race reliability

NVIDIA's October 2025 bulletin identifies CVE-2025-23280 as a Linux driver UAF, rated 7.0. It also lists several null-pointer issues, including CVE-2025-23300 and CVE-2025-23330. The bulletin does not map each CVE to source functions, so none of those IDs alone identifies every step below. It credits Robin Bastide for reporting CVE-2025-23280.

Why the global tree retains a stack address

flowchart TD
  accTitle: Thread-state lifetime diverges at the fault
  accDescr: Normal cleanup removes the node before returning. A fault may bypass cleanup and leave a global reference into an eventually reclaimed task stack.
  A["Stack-local THREAD_STATE_NODE"] --> B["threadStateInit: insert into global tree"]
  B --> C["Perform memory operation"]
  C --> D["Normal exit: threadStateFree"]
  D --> E["Remove node before the frame ends"]
  C --> F["Oops interrupts normal cleanup"]
  F --> G["Task exits; stack eventually reclaimed"]
  G --> H["Tree retains the old virtual address"]

dupMemory declares a local THREAD_STATE_NODE, calls threadStateInit, and then acquires the GPU-operation locks. Registration inserts the node under the tree's own spinlock and releases that lock before returning. A GPU lock stranded by a later fault is a different lock. The normal exit calls threadStateFree; a fatal exception may bypass it.

The external mapping path reaches this function through nvUvmInterfaceDupMemory. NoDeviceMemory permits an ADDR_SYSMEM descriptor with pGpu == NULL, but the older IOMMU predicate does not exclude that object. If execution reaches the allocated-system-memory check in memdescMapIommu, the null GPU pointer flows into the DMA address-range query:

mem_desc.c:4415–4416c
OBJGPU *pGpu = pMemDesc->pGpu;
RmPhysAddr dmaWindowEndAddr = gpuGetDmaEndAddress_HAL(pGpu);

gpuGetDmaEndAddress_HAL proceeds to the physical-address-width query, whose dispatcher accesses a function pointer through pGpu. This locates a source-level null-pointer risk rather than guessing from a nearby call name: in this release, gpumgrCheckIndirectPeer_IMPL simply returns NV_FALSE on x86. Its remote-GPU field accesses belong to the PPC64LE branch.

Freeing is not immediate reuse

flowchart TD
  accTitle: From an unused stack to a reusable virtual range
  accDescr: Per-CPU caching, RCU, and deferred virtual-range reclamation affect when an old address can serve another allocation.
  A["Task stack no longer needed"] --> B{"Enter per-CPU stack cache?"}
  B -->|"Yes"| C["Retained for stack reuse"]
  B -->|"No"| D["RCU-delayed free"]
  D --> E{"Callback retries the cache"}
  E -->|"Success"| C
  E -->|"Miss"| F["vfree and lazy reclamation"]
  F --> G["Purge routes range to a size pool or global free space"]

The resource being reused is a vmalloc virtual-address range, not a slab object type or physically contiguous memory. Upstream v6.11's kernel/fork.c caches 2 stacks per CPU, and its delayed-free callback tries the cache again. Task exit, the actual vfree, and renewed allocation eligibility are distinct events.

mm/vmalloc.c adds another distinction: lazy reclamation and size-based pools. Whether a small request fits an earlier hole or a larger request skips it depends on free ranges, alignment, and pool state. A simple lowest-address-first drawing omits those conditions.

lazy-threshold-model.pypython
def lazy_page_threshold(online_cpus, page_size=4096):
    if online_cpus <= 0 or page_size <= 0:
        raise ValueError("positive parameters required")
    return online_cpus.bit_length() * (32 * 1024 * 1024 // page_size)

assert lazy_page_threshold(1) == 8192
assert lazy_page_threshold(8) == 32768

In upstream v6.11, the threshold is fls(online_cpus) × (32 MiB / PAGE_SIZE). With 4 KiB pages and 8 online CPUs, that is 32768 pages, or 128 MiB. This branch schedules cleanup only when the count is greater than the threshold. Reaching the threshold does not synchronously complete a purge, and other paths may also advance reclamation.

Connecting a user mapping to the tree

Video buffers matter because of their shared backing. In v6.11, videobuf2-vmalloc allocates with vmalloc_user and establishes the user mapping through remap_vmalloc_range. Kernel tree operations and user-space observations can then involve the same backing pages.

This requires more than the presence of a video device. The driver must select that memory backend, the process must have device access, and the buffer sizes, counts, and mapping mode must satisfy the device's constraints.

Layout goals, not guaranteed adjacency4 rows
Stage Retained objects and changes Required observation
Separate holes Keep task stacks allocated between free ranges Small and large requests face different available ranges
Mark positions Keep a marker task, buffers, and the eventual stale stack present Relative placement comes from observations, not the diagram
Advance reclamation Release candidate regions and wait for caching and purge Available driver paths may differ before and after the fault
Reoccupy the range A buffer covers the old node address The global tree pointer actually lands in the shared mapping

The retained stacks used to separate holes are live allocations, not the unmapped guard pages that VMAP_STACK uses to detect overflows. After the Oops, the historical chain also depends on the device-open path still performing tree operations; it does not assume that every ioctl remains usable.

From child pointers to address calibration

MapNode layout for a 64-bit non-checked buildx86-64 · LE
OffsetNameTypeSize
0x00keyNvU648
0x08pParentMapNode *8
0x10pLeftMapNode *8
0x18pRightMapNode *8
0x20bIsRedNvBool1
0x21padding7
sizeof(struct MapNode) = 0x28 (40 bytes) · padding 7

THREAD_STATE_NODE embeds this MapNode. NvBool occupies one byte. In this 64-bit, non-checked model, the parent pointer is at +8, the child pointers at +16 and +24, and the structure occupies 40 bytes including tail padding. A checked build may append another field. This is not the layout of the entire thread-state object.

If a shared buffer covers the stale node, insertion of a later stack-local node can briefly place that task's stack address in a child field. This is an observation of a container update, not yet the kernel image base. The interval between insertion and removal is short.

Calibration must then relate a candidate kernel address to the changed offset in the user mapping. A store at address A, observed at offset o, implies buffer base B = A - o. The pure arithmetic example 0x1000e000 - 0xe000 = 0x10000000 illustrates that identity; these are neither historical machine addresses nor portable offsets.

A right rotation provides a constrained pointer store

map.c:591–596c
MapNode *y = x->pLeft;
x->pLeft = y->pRight;

if (y->pRight)
    y->pRight->pParent = x;

This excerpt comes from _mapRotateRight. With valid nodes, it maintains parent-child links. The danger comes from a node whose lifetime has ended and whose contents have been replaced, not from the rotation algorithm itself.

Before rotation
flowchart TD
  accTitle: Before right rotation
  accDescr: X has left child Y; Y has children A and B.
  X["X"] --> Y["Y"]
  Y --> A["A"]
  Y --> B["B"]
After rotation
flowchart TD
  accTitle: After right rotation
  accDescr: Y becomes the local root, X its right child, and X becomes the parent of B.
  Y["Y"] --> A["A"]
  Y --> X["X"]
  X --> B["B"]
parent-write-model.pypython
def model_parent_update(memory, right_node_address, x_address):
    memory[right_node_address + 8] = x_address

mem = {}
model_parent_update(mem, 0x2000, 0x8000)
assert mem == {0x2008: 0x8000}

The model checks only the assignment's data flow. The destination is the selected node address plus the parent-field offset, and the stored value is the address of x. Choosing such a pointer is not the same as choosing any 64-bit constant. Real tree operations also access other links and impose ordering, color, and parent-child constraints.

Stable distances do not remove the race

Random stack displacement at syscall entry shifts the current call chain. For a fixed binary and path, the node and a saved slot may move together while retaining their relative distance. A different compiler, configuration, or call path requires that distance to be checked again.

The historical approach added work through repeated recoloring during insertion repair, separating the operation into address appearance, window extension, and rotation. That extends a race window rather than eliminating the race. Other GPU calls, task creation, and system load still affect timing.

Several boundaries remain before a file object becomes a capability4 rows
Layer Required condition
Stack slot The exact register-save location in the matching build
Replacement file object Layout, flags, and reference counts survive subsequent use
Address discrimination An operations-table comparison produces an observable result
Read/write capability Valid operation entry points match their calling arguments

Overwriting a saved file pointer is therefore not equivalent to obtaining full kernel read/write access. A file-type check involving f_op supplies an address clue only through its particular path and return behavior. Object control alone does not imply an unrestricted call primitive.

The historical terminal contains the following stage messages. All 105 frames were inspected. Host identifiers, unrelated group memberships, and later address details are omitted here.

Historical stage-output excerptlog
Triggering oops
Hopefully got in control of the UAF
Searching for UAF ...

The final id excerpt retains only its UID/GID fields; the following group list is omitted:

Historical privilege-result excerpt
$ id
uid=0(root) gid=0(root)

The recording shows a root result in that environment, not continued exposure in a current driver or a measured success rate. The current verification ran only layout and arithmetic models:

Local offline-model output
MapNode x64: key=0 parent=8 left=16 right=24 red=32 size=40
lifecycle=4 threshold=8 boundary=8 rotation=3 shared-shift=4 calibration=1: PASS
lazy_pages(1)=8192; lazy_pages(8)=32768; scheduling requires > threshold

Fix the entry condition and the lifetime

Comparing 570.86.15 with 570.195.03 shows two relevant changes in dupMemory: targeted null checks for deviceless objects, and allocation through threadStateAlloc. The following comparison isolates the lifetime-related calls and omits intervening operations. It is a summary, not an applicable patch.

Source comparison of thread-state lifetime+5 −3
dupMemory-lifetime
@@ -1,3 +1,5 @@
THREAD_STATE_NODE threadState;
threadStateInit(&threadState, THREAD_STATE_FLAGS_NONE);
threadStateFree(&threadState);
THREAD_STATE_NODE *pThreadState;
pThreadState = threadStateAlloc(THREAD_STATE_FLAGS_NONE);
if (!pThreadState)
return NV_ERR_NO_MEMORY;
threadStateFree(pThreadState);

The new implementation allocates a node from nonpaged heap storage, and threadStateFree recognizes and frees heap-backed objects. Even if a fault bypasses cleanup, task-stack reclamation no longer frees that node's storage. Leaked nodes, stranded locks, and other partially completed state still need separate handling. The legacy threadStateInit API remains present, so adding the new API does not establish that every caller was converted.

Linux display-driver update boundaries in the October 2025 bulletin3 rows
Branch Fixed version
R580 580.95.05
R570 570.195.03
R535 535.274.02

These are historical fix boundaries for the listed Linux display-driver products, not interchangeable Windows or vGPU-manager versions. Maintenance should select a fixed update supported by the hardware and distribution.

When reviewing similar faults, first identify globally registered objects, acquired locks, and cleanup ownership at the exception point. The invariant is simple: a global container must stop referencing an object before that object's lifetime ends. Repairing the crashing instruction alone does not establish that guarantee.

References

NORMAL~/posts/binary/nvidia-thread-state-oops-vmalloc-lifetime.md§--
0%en