MSVC XFG: from type encoding to call-site hashes

A 64-bit constant near an x64 indirect call need not be an address or a digest of machine code. XFG encodes type information at the call site and target. Explaining a match requires following type normalization, front-end serialization, back-end masking, and the runtime's treatment of marker bits.

The scope here is the C compilation path in Visual Studio 2019 16.8.0 Preview 2.1, with both c1.dll and c2.dll at 19.28.29213.0. These internal encodings describe that preview implementation, not a stable interface for later toolchains or a complete model of C++ types handled by c1xx.dll.

Separate the call-site constant from the entry marker

CFG's basic boundary is whether an indirect-call target is valid. The XFG path examined here adds a function-type comparison. The C prototype used to observe it is:

typedef float (*FPTR)(float, float);

float difference(float a, float b) {
    return b - a;
}

int main(void) {
    FPTR fn = difference;
    return fn(1.00001f, 2.00002f) > 0;
}

The case uses the preview toolchain's x64 developer command prompt and cl /Zi /guard:xfg example.c. This is a version-specific compilation input, not a promise about arbitrary current MSVC options. If optimization folds the indirect call into a direct call, first check whether the intended call site still exists.

Dispatch proceeds through __guard_xfg_dispatch_icall_fptr. The relevant comparison can be reduced to the two instructions below. They are not a complete, contiguous dispatcher listing; target-address checks also occur between them.

or  r10, 1
cmp r10, [rax-8]

The one-bit difference is therefore not a type conflict. Compare the forms the dispatcher actually uses, rather than testing two constants from the file for direct equality.

What the front end and back end each decide

XFGHasher::add_function_type appends the parameter count, each parameter's type digest, a variadic flag, and the calling convention. add_type then appends the return-type digest. A nested digest enters the next hash as bytes; little-endian interpretation becomes relevant when those bytes are read as a 64-bit integer.

XFGHelper__GetHashForType can reuse a cached result associated with Type_t. Caching changes the cost, not the encoded content. The object under analysis is a type representation, not a digest of the function body, variable names, or source file.

The field order in a function prototype

Let H(x) = SHA256(x)[:8]. An ordinary C prototype builds its front-end input in this order:

In this implementation, the default x64 convention's internal value 0x201 contributes low nibble 1; __vectorcall value 0x208 contributes 8. These are internal compiler encodings. Microsoft's x64 calling convention and __vectorcall documentation explain ABI semantics, not a contract for these enumeration values.

The count comes from internal representations such as RealNumberOfParameters(). Variadic and special-entry paths can adjust it, so a general encoder must use the normalized prototype rather than count commas in source text. The branch involving virtual information was not exercised by the C tests described here; its full meaning remains separately unverified.

Normalization precedes recursive type hashing

Parameter processing performs array or function decay and clears some top-level modifiers on the ordinary path, including const and volatile. It does not erase qualifiers at every nesting level.

In const void *, the qualifier belongs to the referenced void. In void * const, it belongs to the pointer. Identifying that level before recursive encoding is essential to reproducing the result.

A type sequence begins with a qualifier byte, a type-group byte, and group-specific data. Internal modifier bits 0x800 and 0x40 denote const and volatile; they become bits 0 and 1 of the qualifier byte. Neither qualifier produces 0; both produce 3.

Group selection uses internal type flags 0x100, 0x200, and 0x400 rather than direct C syntax-category numbers. A special generic path should not be continued as an ordinary primitive encoding. The tagged-type observations also mean that names, anonymous types, scope, and member layout must be evaluated through the actual encoder. Equal memory layouts alone do not establish equal hashes.

For the three explicit primitive examples, float has internal type value 0x26, void has 0x40, and the unsigned 64-bit size_t in this model has 0x4019. They map to the one-byte codes shown above. Thus void hashes input 00 01 0e, while const void hashes 01 01 0e; that difference propagates into the enclosing pointer digest.

Back-end masking does not preserve the digest unchanged

encoded = (frontend & 0xFFFDBFFF7EDFFB70) | 0x8000060010500070
entry = encoded | 1

frontend is the truncated digest interpreted as a little-endian integer. The AND fixes some bits; the OR forces others to one. A 64-bit storage width therefore does not mean that 64 variable digest bits survive.

For these specific masks, the bits that can still vary with input are AND_MASK & ~OR_MASK: 0x7ffdb9ff6e8ffb00, with 44 set bits. The model below verifies this bitwise result.

Validate the chain with two prototypes

The memcpy prototype covers ordinary pointers, a pointee qualifier, an unsigned integer, and a pointer return type:

void *memcpy(void *dest, const void *src, size_t count);

Its front-end input is 41 bytes long. The bytes below come from the following script's calculation. They are serialization-model output, not machine code extracted from an executable.

The script implements only the type combinations explicitly covered here. It does not parse C source or handle every special array, tag-scope rule, or C++ semantic case. A second assertion checks float(float, float) so that validation does not rest on one final constant alone.

import hashlib, struct

def h8(data):
    return hashlib.sha256(data).digest()[:8]

def primitive(code, qualifiers=0):
    return h8(bytes([qualifiers, 1, code]))

def pointer(referenced_hash, qualifiers=0):
    return h8(bytes([qualifiers, 3]) + referenced_hash + b"\x02")

def function_bytes(params, result, variadic=0, convention=1):
    return (struct.pack("<I", len(params)) + b"".join(params)
            + bytes([variadic]) + struct.pack("<I", convention & 0x0f)
            + result)

def callsite_hash(payload):
    front = int.from_bytes(h8(payload), "little")
    return (front & 0xFFFDBFFF7EDFFB70) | 0x8000060010500070

void_ptr = pointer(primitive(0x0e))
const_void_ptr = pointer(primitive(0x0e, qualifiers=1))
size_t_hash = primitive(0x88)
payload = function_bytes([void_ptr, const_void_ptr, size_t_hash], void_ptr)
front = int.from_bytes(h8(payload), "little")
callsite = callsite_hash(payload)
entry = callsite | 1
assert len(payload) == 41
assert front == 0x1da7d393d6b63a72
assert callsite == 0x9da5979356d63a70
f32 = primitive(0x0b)
assert callsite_hash(function_bytes([f32, f32], f32)) == 0x99743f3270d52870
assert void_ptr != const_void_ptr
free_mask = 0xFFFDBFFF7EDFFB70 & ~0x8000060010500070
assert free_mask.bit_count() == 44
print(f"serialized bytes: {len(payload)}")
print(f"frontend: 0x{front:016x}")
print(f"callsite: 0x{callsite:016x}")
print(f"entry:    0x{entry:016x}")
print("PASS: memcpy; float(float,float); pointee const; 44 variable mask bits")

$ python xfg_prototype_model.py
serialized bytes: 41
frontend: 0x1da7d393d6b63a72
callsite: 0x9da5979356d63a70
entry:    0x9da5979356d63a71
PASS: memcpy; float(float,float); pointee const; 44 variable mask bits

The calculated memcpy call-site value is 0x9da5979356d63a70, matching the case's R10 constant. The float prototype also produces 0x99743f3270d52870. These results check the model's byte order, type recursion, masks, and marker relationship. The executed test is the offline calculation above, not a fresh run of the historical Visual Studio preview toolchain.

Find the earliest point of disagreement

The important part of XFG type hashing is not SHA-256 itself but the type information retained before hashing. Normalization defines the equivalence classes being compared; truncation, masking, and runtime markers determine how that information reaches the check.

References

Related compiler type-hash research: Francisco Falcon. The official material below supplies CFG and calling-convention background, not a stability guarantee for preview compiler internals.

• Microsoft: Control Flow Guard
• Microsoft: x64 calling convention
• Microsoft: __vectorcall