x86 shellcode: hidden assumptions in a ROR13 resolver

Raw x86 shellcode has no conventional PE loader to populate its imports. Recovering its shared API resolver before following arguments and handles is more useful than translating every instruction. The same analysis also exposes assumptions the resolver makes about its host process.

The subject is a 324-byte TCP reverse-shell sample associated with Metasploit v6.0.30-dev, recorded with SHA-256 3792f355d1266459ed7c5615dac62c3a5aa63cf9e2c3c0f4ba036e6728763903. Offsets below are relative to the raw byte stream, not runtime virtual addresses. Offline hash checks neither execute the sample nor independently verify that recorded file digest.

Find the shared resolver through call / pop

These are noncontiguous instruction excerpts. The initial call 0x88 pushes return position 0x06, which Main retrieves with pop ebp. Later call ebp instructions return to the resolver entry. cld makes subsequent string instructions advance through increasing addresses.

Apart from the initial call, the sites at 0x99, 0xA9, 0xB8, 0xD2, 0xE2, 0x115, 0x123, 0x12F, and 0x142 share this entry. Their preceding constants differ. Module traversal, export-table access, and hash comparisons turn the shared-resolver hypothesis into a structural conclusion.

A list node is not the start of its container

EAX has already been cleared in this excerpt. fs:[0x30] provides the PEB, followed by Ldr and InMemoryOrderModuleList. EDX then points to the embedded InMemoryOrderLinks member rather than the start of the enclosing module record.

The actual field read is MaximumLength. Microsoft defines both it and Length in bytes: one describes buffer capacity, the other string length. When a terminator is present, Length excludes it. Hashing capacity means identical visible text need not produce an identical hash. UNICODE_STRING

The modules form a doubly linked list. When iteration returns to its head, that head is not a module record. Continuing to interpret it as LDR_DATA_TABLE_ENTRY can produce invalid reads rather than merely an endless loop. These offsets belong to the particular x86 layout, not x64.

A signed branch comes before ROR13

Module names are UTF-16LE, but lodsb reads individual bytes. Ordinary ASCII DLL names therefore contribute alternating character and zero bytes. A zero byte adds nothing, but still causes a rotation.

Crucially, jl is a signed comparison. After cmp al, 0x61, only bytes in 0x61…0x7F enter the subtract-0x20 branch; 0x80…0xFF are interpreted as negative and skip it. This is neither Unicode case conversion nor an ASCII conversion restricted to a…z: 0x7B becomes 0x5B.

The first K = 0x4B leaves a zero-initialized accumulator at 0x4B. Processing the following zero gives ROR13(0x4B) = 0x02580000. Each step retains 32 bits; the final module and export hashes are then added.

Export names, ordinals, and addresses are separate lookups

In a mapped PE image, module-relative +0x3C supplies e_lfanew. The PE32 export-directory RVA is at +0x78 from the resulting NT header. That offset and the frequently illustrated 0x60 refer to different bases: the latter is relative to the Optional Header.

The sample walks names backwards, preserves export-name case, and includes the final NUL in ROR13. It compares (module_hash + export_hash) mod 2^32. On a match, it reads the 16-bit ordinal entry, then the 32-bit function RVA, then adds the module base. A name index is not a function address, and an RVA is not a disk offset. Microsoft PE format

After restoring registers, the resolver removes the caller's return address and hash, pushes the return address back, and jumps through EAX. The API can return to the original call site because this jmp preserves the required stack shape. Follow stack effects alongside control flow.

An export entry can also identify a forwarder string. The displayed path does not resolve forwarders and is not a complete Windows loader. Rapid7's official resolver source credits Stephen Fewer. Its current branch differs from this sample in length selection and hash composition, so it is a comparison reference, not a replacement for the observed bytes.

Recover nine API constants with an offline model

MASK = 0xffffffff
def ror13(x):
    return ((x >> 13) | (x << 19)) & MASK
def hash_bytes(data, module=False):
    value = 0
    for byte in data:
        if module and 0x61 <= byte < 0x80:
            byte = (byte - 0x20) & 0xff
        value = (ror13(value) + byte) & MASK
    return value
def module_hash(name):
    return hash_bytes((name + "\0").encode("utf-16le"), True)
def api_hash(dll, name):
    return (module_hash(dll) + hash_bytes(name.encode("ascii") + b"\0")) & MASK

assert module_hash("KERNEL32.DLL") == 0x92af16da
assert api_hash("KERNEL32.DLL", "LoadLibraryA") == 0x0726774c
assert api_hash("WS2_32.DLL", "WSAStartup") == 0x006b8029
assert api_hash("WS2_32.DLL", "WSASocketA") == 0xe0df0fea

The module_hash convenience function assumes an ASCII DLL name followed by one UTF-16LE NUL. If the actual allocation has additional capacity bytes, pass those exact bytes to hash_bytes instead of treating this convention as a loader guarantee.

The complete local checks cover nine call constants, length behavior, rotations, and byte order. Actual output follows:

$ python ror13-resolver-model.py
KERNEL32.DLL!LoadLibraryA=0x0726774c
WS2_32.DLL!WSAStartup=0x006b8029
WS2_32.DLL!WSASocketA=0xe0df0fea
WS2_32.DLL!connect=0x6174a599
KERNEL32.DLL!CreateProcessA=0x863fcc79
KERNEL32.DLL!WaitForSingleObject=0x601d8708
KERNEL32.DLL!ExitProcess=0x56a2b5f0
KERNEL32.DLL!GetVersion=0x9dbd95a6
NTDLL.DLL!RtlExitUserThread=0x6f721347
four_module_collisions=0x92af16da
maximum_length_tail_changes_hash=True
signed_jl_boundary_0x7b_0x80=PASS
ror13(0x4b)=0x02580000
ror26(0x4b)+0x45=0x00001305
sockaddr_example=AF_INET,4444,192.0.2.202
PASS: arithmetic only; no shellcode execution

A lookup table should retain every candidate for a hash and use arguments and context to disambiguate them. It is a reverse hash lookup table, not necessarily a rainbow table with a chain-based compression scheme.

Recover handle relationships from the API sequence

The pushed string bytes are 77 73 32 5F 33 32 00 00, or ws2_32. An annotation reading ws3_32 conflicts with those bytes; the bytes decide the interpretation.

This is an address-substituted field example. The first two bytes decode little-endian as AF_INET = 2, the next two decode in network byte order as port 4444, and the address is 192.0.2.202. It demonstrates memory layout, not an observed connection to that example endpoint.

After a successful connection, the sample assigns the same socket to STARTUPINFOA's hStdInput, hStdOutput, and hStdError, sets STARTF_USESTDHANDLES and bInheritHandles = true, and creates cmd. These argument relationships support the remote-command-channel interpretation; connect alone does not. CreateProcessA

The exit tail includes version and hash-selection conditions. The low byte of the displayed ExitProcess constant does not satisfy the comparison that selects the other exit, so distinguish the presence of a RtlExitUserThread branch from this configuration actually taking it.

An equal hash is not an equal module identity

names = ["KERNEL32.DLL", "HERNEL32.DLX", "IERNEL32.DLT", "JERNEL32.DLP"]
assert {module_hash(name) for name in names} == {0x92af16da}
raw = "KERNEL32.DLL\0".encode("utf-16le")
assert hash_bytes(raw, True) != hash_bytes(raw + b"\0", True)
assert hash_bytes(b"\x80", True) == 0x80
assert hash_bytes(b"\x7b", True) == 0x5b

All four module names produce 0x92AF16DA. Rotations and addition can overlap byte contributions: ROR26(0x4B) + 0x45 and ROR26(0x49) + 0xC5 both produce 0x1305. Validate full-name collisions with the byte model rather than inferring them solely from a local bit diagram.

A name collision alone does not redirect a call. The candidate module must also be present, expose the appropriate name, win the search order, and maintain the required calling convention. This concerns a particular shellcode's custom resolver, not a claim that the Windows loader identifies DLLs by this hash.

MaximumLength adds another dependency: spare capacity bytes can change the result without changing the visible name. Modifying loader structures also affects the host, bringing byte-count units, object lifetime, loader locks, concurrent enumeration, and ordinary name comparisons into the compatibility problem.

Separate observer setup from sample execution

Interface illustration: observer setup, a 324-byte sample execution prompt, and a LoadLibraryA / ws2_32 alert.

The recorded order matters. The observation module is present first; the sample then begins resolving APIs and eventually reaches the observation function. This supports a path through one sample and process state, not coverage of every variant or compatibility of a deployable detection product.

The local work here consists of arithmetic and layout checks, not observation-DLL injection or reverse-shell execution. The reusable result is the resolver model: establish input bytes, pointer bases, termination conditions, and calling conventions, then connect the identified APIs to their actual arguments.

Official references

• Rapid7: x86 block_api.asm
• Microsoft: UNICODE_STRING
• Microsoft: PE format
• Microsoft: SOCKADDR_IN
• Microsoft: CreateProcessA