Chapter 5: Memory Protection

Memory corruption vulnerabilities—buffer overflows, use-after-free, type confusion—have been the bread and butter of exploitation for decades. In response, Windows has developed multiple overlapping layers of memory protection, each designed to make exploitation progressively more difficult. Understanding these protections is essential for anyone working in offensive security, as bypassing them is often required for successful exploitation. For defenders, understanding these mechanisms reveals what they protect against and where gaps remain.

This chapter examines each protection layer, how it works, and the techniques attackers use to circumvent it.

---

The Memory Protection Stack

Modern Windows systems implement memory protection as a layered defense, with each layer addressing different aspects of the exploitation process:

┌─────────────────────────────────────────────────────────────────────────┐
│                    MEMORY PROTECTION LAYERS                              │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  Hardware: Intel CET (Shadow Stack + IBT)                        │   │
│  └─────────────────────────────────────────────────────────────────┘   │
│                                │                                         │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  VBS: Hypervisor-enforced Code Integrity (HVCI)                  │   │
│  └─────────────────────────────────────────────────────────────────┘   │
│                                │                                         │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  Kernel: DEP (NX bit), KASLR, SMEP, SMAP                         │   │
│  └─────────────────────────────────────────────────────────────────┘   │
│                                │                                         │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  User Mode: DEP, ASLR, CFG, Stack Canaries, SafeSEH             │   │
│  └─────────────────────────────────────────────────────────────────┘   │
│                                │                                         │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  Application: ACG, CIG, Child Process Policy                     │   │
│  └─────────────────────────────────────────────────────────────────┘   │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

Each layer makes different assumptions about what an attacker can and cannot do. DEP assumes attackers can corrupt memory but shouldn't execute it. ASLR assumes attackers can execute code but don't know where things are. CFG assumes attackers might redirect control flow but shouldn't reach arbitrary destinations. Together, these create defense in depth that requires attackers to solve multiple problems simultaneously.

---

DEP: Data Execution Prevention

Understanding DEP

The classic exploitation technique involved placing shellcode in a buffer (data), corrupting a return address or function pointer, and redirecting execution to the shellcode. DEP (Data Execution Prevention) breaks this model by marking data pages as non-executable. When the CPU attempts to execute code from a non-executable page, it triggers an exception and terminates the process.

MITRE ATT&CK Mitigation: M1050 - Exploit Protection

DEP leverages the NX (No-eXecute) bit in page table entries, a hardware feature available on all modern CPUs. When the NX bit is set, the page cannot be executed regardless of other permissions.

DEP Modes

Windows offers several DEP configurations:

Modern systems typically use OptOut or AlwaysOn, ensuring most processes receive DEP protection by default.

DEP Configuration

# Check DEP status
wmic OS Get DataExecutionPrevention_SupportPolicy

# Values:
# 0 = AlwaysOff
# 1 = AlwaysOn
# 2 = OptIn (default)
# 3 = OptOut

# Configure via bcdedit
bcdedit /set {current} nx AlwaysOn

Bypassing DEP with ROP

Return-Oriented Programming (ROP) revolutionized exploitation when DEP made direct shellcode execution impossible. Instead of injecting new code, ROP chains together existing code snippets—called "gadgets"—that end with a ret instruction. By carefully constructing a sequence of return addresses on the stack, an attacker can perform arbitrary computations using only existing executable code.

DEP prevents direct shellcode execution
ROP chains existing code gadgets

Example ROP Chain:
1. pop rdi; ret         → Set RDI = address
2. pop rsi; ret         → Set RSI = size
3. pop rdx; ret         → Set RDX = PAGE_EXECUTE_READWRITE
4. call VirtualProtect  → Make memory executable
5. ret to shellcode     → Execute payload

The typical ROP strategy is to call VirtualProtect or VirtualAlloc to create an executable memory region, then copy shellcode there and execute it. This requires finding appropriate gadgets in loaded modules—a task automated by tools like ROPgadget and ropper:

# Using ROPgadget
ROPgadget --binary target.exe --ropchain

# Using ropper
ropper --file target.exe --search "pop rdi"

JIT Spraying

Just-In-Time compilers present another DEP bypass opportunity. JIT compilers generate executable code at runtime, and an attacker can influence what code gets generated:

Abuse Just-In-Time compilers
1. Create JavaScript/ActionScript
2. Embed shellcode in constants
3. JIT compiles to executable code
4. Redirect execution to JIT code

Checking DEP Status Per Process

// Query DEP policy for process
DWORD Flags;
BOOL Permanent;
GetProcessDEPPolicy(GetCurrentProcess(), &Flags, &Permanent);

// Flags:
// 0 = DEP disabled
// 1 = DEP enabled
// Permanent = Cannot be changed

---

ASLR: Address Space Layout Randomization

Understanding ASLR

DEP forces attackers to use existing code, but they still need to know where that code is located. ASLR (Address Space Layout Randomization) addresses this by randomizing the locations of key memory regions:

• Executable base address
• DLL base addresses
• Stack
• Heap
• PEB/TEB

Without knowing where things are, attackers cannot reliably construct ROP chains or target specific functions.

ASLR Entropy

The effectiveness of ASLR depends on how much randomization it provides—measured in bits of entropy:

64-bit systems benefit significantly from ASLR because the larger address space allows more randomization. On 32-bit systems, the limited address space constrains entropy, making brute-force attacks more feasible.

ASLR Evolution

┌─────────────────────────────────────────────────────────────────────┐
│                        ASLR EVOLUTION                                │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  Basic ASLR (Vista+)                                                │
│  └─ Image randomization at boot                                     │
│  └─ Same addresses per boot cycle                                   │
│                                                                      │
│  ASLR + High Entropy (Windows 8+)                                   │
│  └─ Requires /HIGHENTROPYVA linker flag                             │
│  └─ More randomization bits                                         │
│                                                                      │
│  Force ASLR (Windows 8+)                                            │
│  └─ Relocates even images without /DYNAMICBASE                      │
│  └─ System-wide setting                                             │
│                                                                      │
│  Bottom-Up ASLR                                                      │
│  └─ Randomizes heap and stack bases                                 │
│  └─ Independent of image ASLR                                       │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

High Entropy ASLR is particularly important—programs must be compiled with the /HIGHENTROPYVA flag to receive full 64-bit randomization. Without this flag, even 64-bit programs receive reduced entropy for compatibility reasons.

Bypassing ASLR

Information Disclosure

The most reliable ASLR bypass is leaking a pointer. If an attacker can read a memory address through any vulnerability (format string, out-of-bounds read, uninitialized memory), they can calculate base addresses:

// Leak pointer to calculate base address
PVOID leaked_ptr = GetLeakedPointer();
PVOID base = (PVOID)((ULONG_PTR)leaked_ptr - KNOWN_OFFSET);

Partial Overwrite

ASLR randomizes upper bits, but lower bits are often predictable due to page alignment:

ASLR only randomizes upper bits
Lower 12-16 bits often predictable

Attack: Overwrite only lower bytes
Result: Jump within same page

This technique works when the attacker can control only part of a pointer—the corruption redirects execution to a different offset within the same or nearby memory region.

Non-ASLR Modules

Legacy or poorly configured DLLs may lack ASLR:

Find DLL without ASLR:
- Legacy DLLs
- Third-party modules
- Java/Flash components (legacy)

Use fixed addresses from non-ASLR module

Heap Spraying

Heap spraying fills memory with shellcode copies, increasing the probability that a corrupted pointer lands on executable code:

// Fill heap with shellcode
var spray = new Array(0x1000);
for (var i = 0; i < spray.length; i++) {
    spray[i] = nop_sled + shellcode;
}
// Probabilistically hit shellcode

Checking ASLR Status

# Check if image has ASLR
dumpbin /headers file.exe | findstr "Dynamic base"

# Using PowerShell
$pe = [System.IO.File]::ReadAllBytes("file.exe")
# Check DLL characteristics for DYNAMIC_BASE (0x40)

---

CFG: Control Flow Guard

Understanding CFG

Even with DEP and ASLR, attackers can redirect execution to existing code. Control Flow Guard (CFG) addresses this by validating that indirect calls (calls through function pointers) target legitimate functions, not arbitrary code.

MITRE ATT&CK Mitigation: M1050 - Exploit Protection

CFG works by maintaining a bitmap of valid call targets. When the compiler encounters an indirect call, it inserts a validation check. At runtime, the target address is checked against the bitmap—if it's not a valid target, the process terminates.

┌─────────────────────────────────────────────────────────────────────┐
│                         CFG MECHANISM                                │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  Compilation:                                                        │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  1. Compiler generates CFG bitmap of valid call targets     │   │
│  │  2. Indirect calls instrumented with guard check            │   │
│  │  3. Bitmap stored in PE image                               │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                      │
│  Runtime:                                                            │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  call [target]                                               │   │
│  │       │                                                      │   │
│  │       ▼                                                      │   │
│  │  __guard_dispatch_icall_fptr(target)                        │   │
│  │       │                                                      │   │
│  │       ▼                                                      │   │
│  │  ntdll!LdrpValidateUserCallTarget                           │   │
│  │       │                                                      │   │
│  │       ├── Valid: Continue call                               │   │
│  │       └── Invalid: Terminate process                         │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

The CFG Bitmap

The bitmap has a granularity of 8 bytes—each bit represents whether the corresponding 8-byte aligned address is a valid call target:

// CFG bitmap structure
// Each bit represents 8 bytes (granularity)
// 1 = Valid call target
// 0 = Invalid call target

// Bitmap lookup:
BOOL IsValidTarget(PVOID target) {
    ULONG_PTR offset = (ULONG_PTR)target >> 3;  // Divide by 8
    ULONG_PTR bit = offset & 0x7;               // Bit within byte
    ULONG_PTR byte_offset = offset >> 3;        // Byte in bitmap

    return (bitmap[byte_offset] >> bit) & 1;
}

Bypassing CFG

Bitmap Corruption

With an arbitrary write primitive, an attacker can mark their target as valid:

// If attacker has arbitrary write:
// Modify CFG bitmap to mark target as valid

PVOID cfg_bitmap = GetCfgBitmap();
ULONG_PTR target_offset = (ULONG_PTR)target >> 3;
ULONG_PTR byte_offset = target_offset >> 3;
ULONG_PTR bit = target_offset & 0x7;

// Set bit in bitmap
cfg_bitmap[byte_offset] |= (1 << bit);

Valid Target Abuse

Many legitimate functions are marked as valid call targets but can be abused:

Find valid call targets that are useful:
- VirtualProtect (change memory protections)
- WinExec (execute commands)
- LoadLibrary (load arbitrary DLLs)

Chain valid targets for exploitation

JIT Code

JIT-compiled code is automatically added to the CFG bitmap:

JIT-compiled code is added to CFG bitmap
1. Trigger JIT compilation
2. Shellcode compiled as JIT code
3. JIT code is marked valid by CFG
4. Call JIT code (CFG allows)

Exception Handlers

Exception handlers may not be fully protected:

Exception handlers may not be CFG-protected
1. Corrupt exception handler chain
2. Trigger exception
3. Execute arbitrary handler

XFG: Extended Flow Guard

XFG extends CFG with type-based checking—the target must not only be a valid function but must have the correct function signature:

XFG adds type-based checking:
- Call targets must match function prototype
- Hash of function signature validated
- Prevents calling wrong function type

Bypass: Find function with matching signature

XFG significantly narrows the attack surface by preventing attackers from calling arbitrary valid functions.

---

CET: Control-flow Enforcement Technology

Understanding CET

Intel Control-flow Enforcement Technology (CET) provides hardware-enforced control flow integrity. While CFG is implemented in software, CET uses CPU features that cannot be bypassed through memory corruption alone.

CET has two components:

• Shadow Stack: Protects return addresses
• IBT (Indirect Branch Tracking): Validates indirect jumps

Shadow Stack

The shadow stack is a hardware-maintained copy of return addresses. When a function is called, the return address is pushed to both the regular stack and the shadow stack. On return, both values are compared—if they don't match, the CPU raises an exception.

┌─────────────────────────────────────────────────────────────────────┐
│                       SHADOW STACK                                   │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  Regular Stack          Shadow Stack                                │
│  ┌─────────────┐        ┌─────────────┐                            │
│  │ Local Vars  │        │             │                            │
│  │ Saved RBP   │        │             │                            │
│  │ Return Addr ├───────▶│ Return Addr │  ← Mirrored                │
│  │ Parameters  │        │             │                            │
│  ├─────────────┤        ├─────────────┤                            │
│  │ Local Vars  │        │             │                            │
│  │ Saved RBP   │        │             │                            │
│  │ Return Addr ├───────▶│ Return Addr │  ← Mirrored                │
│  │ Parameters  │        │             │                            │
│  └─────────────┘        └─────────────┘                            │
│                                                                      │
│  On RET:                                                            │
│  1. Pop return address from regular stack                           │
│  2. Pop return address from shadow stack                            │
│  3. Compare: If mismatch → #CP exception                            │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

This effectively kills ROP—even if an attacker corrupts return addresses on the regular stack, the shadow stack retains the original values.

IBT: Indirect Branch Tracking

IBT ensures indirect jumps and calls land on valid targets marked with the ENDBR instruction:

ENDBR64/ENDBR32 instruction:
- Must be at valid indirect branch targets
- CPU tracks indirect branches
- Landing site must have ENDBR
- Otherwise: #CP exception

Compiler inserts ENDBR at:
- Function entry points
- Jump table targets
- Other indirect targets

CET Bypass Considerations

Shadow Stack Attacks

The shadow stack is extremely difficult to attack:

Shadow stack is hardware-protected
- Cannot be directly modified
- Separate memory region
- Kernel-managed

Attack surface:
- Race conditions during context switch
- Kernel vulnerabilities
- Hypervisor attacks

IBT Attacks

IBT can potentially be bypassed by finding useful code after ENDBR instructions:

All valid targets have ENDBR
Attack: Find ENDBR gadgets

ENDBR32 = 0xF3 0x0F 0x1E 0xFB
ENDBR64 = 0xF3 0x0F 0x1E 0xFA

Look for useful code after ENDBR instruction

However, the limited availability of useful ENDBR gadgets significantly constrains attackers.

Checking CET Status

# Check if CPU supports CET
Get-CimInstance Win32_Processor | Select-Object Name, Caption

# Check if process uses CET
# Requires Windows 11+

---

Stack Protection

Stack Canaries (GS)

Stack canaries (the /GS compiler flag) place a random value between local variables and the return address. Buffer overflows that corrupt the return address will also corrupt the canary, which is detected before the function returns:

// Compiler inserts canary before return address
// /GS compiler flag

void VulnerableFunction(char* input) {
    // Compiler adds:
    // DWORD cookie = __security_cookie ^ RBP;
    // [stack layout: cookie | locals | saved_rbp | return_addr]

    char buffer[64];
    strcpy(buffer, input);  // Overflow possible

    // Compiler adds check before return:
    // if (cookie != (__security_cookie ^ RBP)) {
    //     __report_gsfailure();
    // }
}

Bypassing Stack Canaries

Information Leak

If the canary value can be leaked through another vulnerability, the attacker can include it in their payload:

// Leak canary value through separate vulnerability
// Format string, out-of-bounds read, etc.

printf(user_input);  // If input = "%p %p %p %p"
// May leak canary from stack

Canary Prediction

On Windows, canaries are derived from __security_cookie:

On Windows:
- Canary derived from __security_cookie
- Cookie initialized at process start
- Can be predicted if:
  - Process restart known
  - Fork child inherits cookie

Exception Handler Overwrite

SEH handlers are checked before the canary:

SEH handlers stored before canary check
1. Overflow past canary and return address
2. Overwrite SEH handler
3. Trigger exception
4. Handler executed before canary check

SafeSEH

SafeSEH maintains a list of valid exception handlers and terminates the process if an invalid handler is invoked:

Structured Exception Handler Overwrite Protection
- /SAFESEH linker flag
- Maintains table of valid handlers
- Invalid handler causes termination

Bypass:
- Non-SafeSEH module in process
- Heap-based handlers
- Direct code execution via other means

SEHOP

SEHOP (SEH Overwrite Protection) validates the integrity of the entire exception handler chain:

Validates SEH chain integrity
- Last handler must point to ntdll!FinalExceptionHandler
- Chain must be valid linked list
- OS-level protection (Vista+)

Bypass:
- Requires precise chain reconstruction
- Must know handler addresses

---

Stack Spoofing

Understanding Stack Spoofing

Stack spoofing creates fake call stacks to evade detection. EDR products often inspect thread stacks to identify suspicious activity—a thread that appears to originate from shellcode or unknown memory is suspicious. Stack spoofing makes malicious threads appear to have normal call stacks.

Synthetic Stack Frame Construction

// Build fake stack frame for detection evasion

typedef struct _SPOOF_CONTEXT {
    PVOID ReturnAddress;      // Fake return address
    PVOID Rbp;                // Fake frame pointer
    PVOID Rsp;                // Stack pointer
    PVOID Function;           // Real function to call
    PVOID Arguments[4];       // Function arguments
} SPOOF_CONTEXT;

// Stack layout after spoofing:
// ┌─────────────────────────────────┐
// │ Fake return addr (ntdll func)   │
// │ Fake RBP → next fake frame      │
// │ ... legitimate-looking data ... │
// │ Fake return addr (kernel32)     │
// │ Fake RBP → next fake frame      │
// │ ... more fake frames ...        │
// │ RtlUserThreadStart              │ ← Final frame
// └─────────────────────────────────┘

UNWIND_INFO Parsing

Creating convincing fake frames requires understanding how Windows unwinds stacks. The PE exception directory contains UNWIND_INFO structures that describe each function's stack layout:

// Parse PE exception directory for accurate frame sizes
typedef struct _UNWIND_INFO {
    UCHAR Version : 3;
    UCHAR Flags : 5;
    UCHAR SizeOfProlog;
    UCHAR CountOfCodes;
    UCHAR FrameRegister : 4;
    UCHAR FrameOffset : 4;
    UNWIND_CODE UnwindCode[1];
} UNWIND_INFO;

// Calculate frame size from unwind codes
SIZE_T CalculateFrameSize(PUNWIND_INFO UnwindInfo) {
    SIZE_T size = 0;
    for (int i = 0; i < UnwindInfo->CountOfCodes; i++) {
        // Process each unwind code
        // UWOP_ALLOC_LARGE, UWOP_ALLOC_SMALL, etc.
    }
    return size;
}

Selecting Convincing Return Addresses

Good fake return addresses appear in legitimate thread stacks:

Good return addresses for spoofing:

1. NtWaitForSingleObject + offset
   - Common in legitimate thread stacks

2. WaitForSingleObjectEx + offset
   - Win32 wait wrapper

3. BaseThreadInitThunk + offset
   - Thread entry point

4. RtlUserThreadStart + offset
   - Final frame in all stacks

---

Additional Protections

ACG: Arbitrary Code Guard

ACG prevents dynamic code generation entirely:

Prevents dynamic code generation:
- No PAGE_EXECUTE_READWRITE
- No PAGE_EXECUTE_WRITECOPY
- JIT compilation restrictions

SetProcessMitigationPolicy(ProcessDynamicCodePolicy, ...)

Applications using ACG cannot create executable memory at runtime, effectively blocking JIT-based attacks and many shellcode injection techniques.

CIG: Code Integrity Guard

CIG ensures only properly signed code can execute:

Only signed code can execute:
- Blocks unsigned DLL injection
- Blocks reflective loading

SetProcessMitigationPolicy(ProcessSignaturePolicy, ...)

SMEP and SMAP

These kernel-mode protections prevent the kernel from executing or accessing user-mode memory:

SMEP (Supervisor Mode Execution Prevention):
- Kernel cannot execute user-mode pages
- Prevents ret2usr attacks

SMAP (Supervisor Mode Access Prevention):
- Kernel cannot access user-mode pages
- Prevents data-only kernel attacks

---

Detection and Monitoring

Checking Process Mitigations

# Check process mitigations
Get-ProcessMitigation -Name process.exe

# Check system-wide defaults
Get-ProcessMitigation -System

ROP Detection

EDR products employ various techniques to detect ROP:

EDR detections:
1. Stack pivot detection (RSP far from TEB stack)
2. Return address validation
3. Gadget pattern detection
4. CFG bypass detection

Exploit Prevention Events

---

Summary

Memory protection has evolved from simple DEP to comprehensive hardware-enforced solutions. Each protection addresses specific exploitation techniques:

The trend is clear: hardware-based protections like CET are increasingly difficult to bypass. Exploitation is moving toward logic vulnerabilities, type confusion, and other attacks that don't rely on corrupting control flow. For defenders, enabling all available mitigations—particularly CET and HVCI on supported hardware—dramatically raises the bar for attackers.

For offensive security professionals, understanding these protections guides technique selection and reveals where creative solutions are needed. The era of simple buffer overflows is long past; modern exploitation requires chaining multiple bugs and bypassing multiple mitigations simultaneously.

---

References

• Microsoft Exploit Protection
• Intel CET Specification
• Control Flow Guard
• ROPgadget
• Windows Internals - Memory Management