memory-heap.md

Memory Analysis Tutorial Using x64dbg

This tutorial provides step-by-step instructions for analyzing a simple C program using x64dbg, focusing on memory allocation, initialization, and deallocation. The program dynamically allocates an array, initializes it with squared values, prints the fifth element, and frees the memory. We'll use x64dbg to set breakpoints, track heap allocations, observe array initialization, examine memory contents, and analyze the memory state after deallocation.

Prerequisites

• x64dbg: Download and install from https://x64dbg.com.
• C Compiler: Use a compiler like MinGW to compile the provided C program into a 64-bit executable.
• Program Code:

  #include <stdio.h>
  #include <stdlib.h>
  
  int main() {
      int* numbers = malloc(10 * sizeof(int));
      for (int i = 0; i < 10; i++) {
          numbers[i] = i * i;
      }
      
      printf("Fifth element: %d\n", numbers[4]);
      free(numbers);
      return 0;
  }

• Compile the program (e.g., gcc -o memory_analysis.exe memory_analysis.c) to generate a 64-bit executable.

Step-by-Step Instructions

Step 1: Load the Executable in x64dbg

1. Open x64dbg (ensure you're using the 64-bit version, x64dbg.exe).
2. Go to File > Open and select the compiled memory_analysis.exe.
3. The debugger pauses at the program entry point. The CPU window displays the disassembled code, and the Registers window shows the current state of CPU registers.

Step 2: Set a Breakpoint on malloc

1. Locate the malloc call:
   • In the CPU window, press Ctrl+G to open the "Go to Expression" dialog.
   • Type malloc and press Enter to jump to the malloc function's address in the linked library (e.g., msvcrt.dll).
   • Alternatively, if you know the source code, find the call instruction for malloc in the main function by stepping through the code or searching for the pattern (e.g., call <address> after a mov instruction setting up the size argument).
2. Set a breakpoint at the end of the malloc call:
   • Scroll to the instruction just after the call malloc (typically a ret or the next instruction in main).
   • The return value of malloc (the allocated memory address) is stored in the RAX register.
   • Right-click the instruction after the call malloc and select Toggle Breakpoint (or press F2).
   • Verify the breakpoint in the Breakpoints tab (View > Breakpoints).

Step 3: Set a Breakpoint on free

1. Locate the free call:
   • In the CPU window, press Ctrl+G again and type free to jump to the free function.
   • Alternatively, step through the main function to find the call free instruction.
2. Set a breakpoint:
   • Right-click the call free instruction and select Toggle Breakpoint (or press F2).
   • Confirm the breakpoint in the Breakpoints tab.

Step 4: Run the Program and Track Heap Allocation

1. Run the program:
   • Press F9 to run the program until the first breakpoint (after malloc).
   • The debugger pauses after the malloc call, with the allocated address in RAX.
2. Inspect the RAX register:
   • In the Registers window, note the value in RAX. This is the base address of the allocated heap memory for the numbers array (e.g., 0x000001A2C3D04000).
3. View the allocated memory in the Dump window:
   • In the Dump window (bottom-left in x64dbg), right-click and select Follow in Dump > RAX.
   • The Dump window now shows the memory at the address returned by malloc. It may initially contain uninitialized data (random values).

Step 5: Watch Array Initialization

1. Step through the loop:
   • Press F8 (Step Over) to execute instructions one by one, or set a breakpoint at the loop's store instruction (e.g., mov [rax + offset], value).
   • To focus on numbers[4] = 16:
     • The loop calculates i * i and stores it in numbers[i].
     • For i = 4, the value is 4 * 4 = 16 (0x10 in hexadecimal).
   • Find the instruction that writes to numbers[4] (e.g., mov [rax + 0x10], 0x10, since 4 * sizeof(int) = 16 bytes offset).
   • Set a breakpoint on this instruction by right-clicking and selecting Toggle Breakpoint.
2. Run to the breakpoint:
   • Press F9 to continue execution until the breakpoint.
   • When the debugger pauses, verify in the Dump window that the value 0x10 (16) is written to the memory address RAX + 0x10.

Step 6: Examine Memory Contents After Initialization

1. Inspect the array in the Dump window:
   • After the loop completes (step through or set a breakpoint after the loop), go to the Dump window.
   • Follow the address in RAX again (right-click > Follow in Dump > RAX).
   • The memory should contain the initialized values:
     • numbers[0] = 0 (0x00)
     • numbers[1] = 1 (0x01)
     • numbers[2] = 4 (0x04)
     • numbers[3] = 9 (0x09)
     • numbers[4] = 16 (0x10)
     • numbers[5] = 25 (0x19)
     • numbers[6] = 36 (0x24)
     • numbers[7] = 49 (0x31)
     • numbers[8] = 64 (0x40)
     • numbers[9] = 81 (0x51)
   • Each value is a 4-byte integer, so the values are stored at 4-byte intervals (e.g., RAX, RAX+4, RAX+8, etc.).
2. Verify in the Stack or Memory Map:
   • In the Memory Map tab (View > Memory Map), locate the heap segment containing the allocated memory (look for a .heap region).
   • Double-click the heap region to view its contents, confirming the initialized values.

Step 7: Observe Memory State After free

1. Run to the free breakpoint:
   • Press F9 to continue execution until the call free breakpoint.
2. Step over the free call:
   • Press F8 to execute the free call.
   • The memory at the address in RAX is now deallocated.
3. Inspect the memory in the Dump window:
   • Go back to the Dump window and follow the same address (RAX from the malloc step).
   • The memory may still contain the previous values (e.g., 0x10 for numbers[4]), as free does not zero out the memory. However, this memory is no longer valid for the program to access.
   • In some cases, the heap manager may overwrite the memory with debug patterns (e.g., 0xFEEEFEEE in Windows debug builds).
4. Check the Memory Map:
   • In the Memory Map tab, verify if the heap region is still marked as allocated or if it has been released. The exact behavior depends on the heap manager.

Step 8: Analyze and Conclude

1. Review findings:
   • You tracked the heap allocation by observing the address in RAX after malloc.
   • You confirmed the initialization of numbers[4] = 16 in the memory dump.
   • You examined the full array contents after the loop.
   • You observed that after free, the memory may retain its values but is no longer valid.
2. Optional: Test for use-after-free:
   • If you modify the program to access numbers after free, you can set a breakpoint on that access and observe undefined behavior in x64dbg (e.g., crashes or garbage data).

Tips and Notes

• x64dbg Shortcuts:
  • F2: Toggle breakpoint.
  • F7: Step into (follow function calls).
  • F8: Step over (execute function calls without stepping into them).
  • F9: Run until the next breakpoint.
  • Ctrl+G: Go to address or symbol.
• Heap Behavior: The heap manager may not immediately release memory to the OS after free. The memory contents may remain unchanged until reused.
• Debugging Symbols: If available, debugging symbols (e.g., PDB files) make it easier to locate malloc and free calls. Use gcc -g when compiling to include symbols.
• Safety: Ensure you’re running the program in a safe, isolated environment, as debugging involves low-level memory access.

Conclusion

This exercise demonstrated how to use x64dbg to analyze memory allocation, initialization, and deallocation in a C program. By setting breakpoints on malloc and free, tracking the heap address in RAX, and inspecting the memory dump, you gained insight into the program’s memory management. These skills are essential for reverse engineering and debugging memory-related issues in real-world applications.