CLow-Level

C Security & Hardening: Defeating Buffer Overflows, Format String Attacks & Memory Vulnerabilities

TT
TopicTrick Team
C Security & Hardening: Defeating Buffer Overflows, Format String Attacks & Memory Vulnerabilities

C Security & Hardening: Defeating Buffer Overflows, Format String Attacks & Memory Vulnerabilities


Table of Contents


The Vulnerability Landscape

According to the MITRE CWE Top 25 and NIST vulnerability database, C-based software consistently dominates critical vulnerability counts:

VulnerabilityCWE% of CVEsExploitability
Buffer overflow (stack)CWE-12128%Remote code execution
Heap overflowCWE-12214%RCE via allocator corruption
Integer overflowCWE-19011%Leads to buffer overflow
Format stringCWE-1348%Read/write arbitrary memory
Use-after-freeCWE-4167%Type confusion attacks
NULL dereferenceCWE-47612%Denial of service

Buffer Overflow: Stack Smashing in Detail

The classic exploit: overwrite the return address to redirect execution to attacker-controlled code:

c
// VULNERABLE: classic textbook buffer overflow
void vulnerable_login(void) {
    char username[16];  // 16-byte stack buffer
    
    gets(username);  // NO BOUNDS CHECK — reads until newline
    // Input: "AAAAAAAAAAAAAAAA" * 4 + <return_address> 
    // Overwrites: return address → points to attacker shellcode
    
    printf("Welcome, %s\n", username);
}

// SECURE: bounds-limited input
void secure_login(void) {
    char username[64];
    
    if (!fgets(username, sizeof(username), stdin)) {
        return; // EOF or error
    }
    
    // Remove trailing newline
    username[strcspn(username, "\n")] = '\0';
    
    // Validate length
    if (strlen(username) == 0 || strlen(username) > 32) {
        fprintf(stderr, "Username must be 1-32 characters\n");
        return;
    }
    
    printf("Welcome, %s\n", username);
}

Dangerous → Secure function replacements:

DangerousSecure AlternativeWhy Safer
gets()fgets(buf, size, fp)Size-bounded
strcpy(dst, src)strncpy(dst, src, size-1) + null-terminateSize-bounded
strcat(dst, src)strncat(dst, src, size - strlen(dst) - 1)Size-bounded
sprintf(buf, fmt, ...)snprintf(buf, size, fmt, ...)Size-bounded, no overflow
scanf("%s", buf)scanf("%63s", buf)Width-limited in format string

Heap Overflow: Corrupting Allocator Metadata

Heap overflows are less predictable than stack overflows but often more serious because they corrupt malloc's internal bookkeeping:

c
// VULNERABLE: heap overflow
int *buf = malloc(10 * sizeof(int)); // Allocate 10 integers
for (int i = 0; i <= 10; i++) {     // Bug: writes 11 items (one past end)
    buf[i] = i;                      // buf[10] overwrites glibc heap metadata
}
// After buf[10]: heap's internal "next free block" pointer is corrupted
// Next malloc/free call: undefined behavior, possible RCE via heap spray

// SECURE: strict bounds
int *buf = malloc(10 * sizeof(int));
if (!buf) return -1;
const int COUNT = 10;
for (int i = 0; i < COUNT; i++) {   // Strict < not <=
    buf[i] = i;
}
free(buf);

Detect with AddressSanitizer:

bash
gcc -fsanitize=address -g vulnerable.c -o vulnerable
./vulnerable
# ASan output: heap-buffer-overflow on address 0x... at pc 0x...

Format String Attacks: printf as an Attack Vector

Passing user input directly as a printf format string allows reading and writing arbitrary memory:

c
// CRITICAL VULNERABILITY: user controls format string
void vulnerable_log(const char *user_message) {
    printf(user_message);  // If user inputs "%x %x %x %n":
    // %x %x %x: READ from stack (information disclosure)
    // %n:       WRITE the count of printed chars to the next pointer on stack
    //           → arbitrary write to arbitrary address
}

// ATTACK INPUT: "%08x %08x %08x %n"
// Reads: stack addresses, heap addresses (information disclosure)
// Writes: can overwrite function's GOT entry → execute shellcode

// SECURE: ALWAYS use %s for strings
void secure_log(const char *user_message) {
    printf("%s", user_message);      // user_message treated as data, not format
    fprintf(stderr, "%s\n", user_message); // Same for stderr
    syslog(LOG_INFO, "%s", user_message);  // Same for syslog
}

Rule: Never pass untrusted data as the format argument to any printf-family function. Always use printf("%s", data) or snprintf(buf, size, "%s", data).


Integer Overflow: Silent Arithmetic Errors

Integer overflow causes incorrect results that lead to undersized buffers being allocated:

c
#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>

// VULNERABLE: allocation size overflow
void vulnerable_alloc(size_t width, size_t height) {
    // If width = 65537, height = 65537 on 32-bit:
    // width * height = 65537 * 65537 = 4,295,098,369
    // Truncated to uint32_t: 65,537 (far too small!)
    size_t size = width * height * sizeof(uint32_t); // OVERFLOW!
    uint32_t *pixels = malloc(size);                 // malloc(tiny_value)
    pixels[width * height - 1] = 0;                  // HEAP OVERFLOW
}

// SECURE: check before multiplying
int safe_alloc(size_t width, size_t height, uint32_t **out) {
    // Check: would width * height overflow size_t?
    if (width > 0 && height > SIZE_MAX / width) {
        return -1; // Would overflow
    }
    size_t count = width * height;
    
    if (count > SIZE_MAX / sizeof(uint32_t)) {
        return -1; // Multiplication with sizeof would overflow
    }
    
    *out = malloc(count * sizeof(uint32_t));
    return *out ? 0 : -1;
}

// C23 approach: use __builtin_mul_overflow (GCC/Clang)
bool safe_multiply(size_t a, size_t b, size_t *result) {
    return !__builtin_mul_overflow(a, b, result); // true if no overflow
}

Use-After-Free: Dangling Pointer Attacks

c
#include <stdlib.h>
#include <stdio.h>

// VULNERABLE: use-after-free
void vulnerable_pattern(void) {
    int *p = malloc(sizeof(int));
    *p = 42;
    free(p);
    
    // Bug: p still contains the old address (now freed)
    printf("%d\n", *p);  // Use-after-free — undefined behavior
    *p = 99;              // WRITE to freed memory
    // If malloc re-allocates this block, writing 99 corrupts new object's data
}

// SECURE: null after free, RAII pattern
void secure_pattern(void) {
    int *p = malloc(sizeof(int));
    if (!p) return;
    *p = 42;
    free(p);
    p = NULL; // Immediately null the pointer after free
    
    if (p) {  // This check now protects against use-after-free
        printf("%d\n", *p);
    }
}

// Better: wrapper macro
#define SAFE_FREE(ptr) do { free(ptr); (ptr) = NULL; } while (0)

OS-Level Mitigations

Modern operating systems deploy multiple layers of exploit mitigation:

bash
# Check what mitigations your binary has on Linux:
checksec --file=./your_binary

# Typical output:
# Stack Canary  : ENABLED  → Detects stack overflows before ret
# NX            : ENABLED  → Stack/heap marked non-executable (no shellcode)
# PIE           : ENABLED  → Position-independent; enables ASLR
# ASLR          : ENABLED  → Randomize stack/heap/library addresses
# RELRO         : FULL     → GOT table read-only after init

Stack Canaries: The compiler places a random 8-byte value (the "canary") before the return address. Before a function returns, the canary is checked. If overwritten (by a buffer overflow), the program is immediately terminated with a fault.

ASLR (Address Space Layout Randomization): Randomizes where the stack, heap, and libraries are loaded each run. Without knowing the exact address of the return address or the target shellcode, most exploits fail.

NX/DEP (No-Execute/Data Execution Prevention): Marks the stack and heap as non-executable. Even if an attacker overwrites the return address, they cannot execute code on the stack — they must use Return-Oriented Programming (ROP), which is significantly more complex.


Compiler Hardening Flags

Always compile production C with these security flags:

bash
# Recommended production build flags
gcc -O2 -Wall -Wextra -Wpedantic           \
    -fstack-protector-strong               \  # Stack canaries
    -D_FORTIFY_SOURCE=3                    \  # Runtime bound checking on libc calls
    -fpie -pie                             \  # Enable ASLR
    -Wl,-z,relro,-z,now                    \  # Full RELRO
    -Wl,-z,noexecstack                     \  # NX stack
    -fstack-clash-protection               \  # Stack clash defense
    -fcf-protection=full                   \  # Control flow integrity (Intel CET)
    main.c -o program

# Debug/testing flags (never in production release)
gcc -g -fsanitize=address,undefined,leak   \  # ASan + UBSan + LeakSanitizer
    -fsanitize=thread                      \  # + ThreadSanitizer (not with ASan)
    main.c -o program_debug

# Static analysis
cppcheck --enable=all main.c               # Static analysis
clang --analyze main.c                     # Clang analyzer
scan-build gcc main.c                      # Clang scan-build

-D_FORTIFY_SOURCE=3 (C23 recommended): The compiler replaces unsafe functions like strcpy, memcpy, sprintf with bounds-checking versions that abort() at runtime if an overflow is detected. Zero code change required — just a compile flag.


Dynamic Analysis: ASan and Valgrind

bash
# AddressSanitizer (ASan) — fast, finds most memory errors
gcc -fsanitize=address,undefined -g program.c -o program_asan
./program_asan
# Catches: heap/stack buffer overflow, use-after-free, double-free,
#          use-after-return, invalid free, memory leaks

# Valgrind memcheck — slower but comprehensive
gcc -g program.c -o program_debug
valgrind --tool=memcheck --leak-check=full --track-origins=yes ./program_debug

# libFuzzer — automated security testing (finds inputs that crash your code)
gcc -fsanitize=address,fuzzer program.c -o fuzz_target
./fuzz_target corpus_dir/  # Generates random inputs, tracks crashes

Secure Coding Standards: CERT C and MISRA-C

CERT C Secure Coding Standard (developed by Carnegie Mellon CERT):

  • Rules like MEM30-C: Do not access freed memory.
  • Rules like INT30-C: Ensure unsigned integer operations do not wrap.
  • Used by: US Department of Defense, automotive suppliers, avionics.
  • Tool: cppcheck --addon=cert main.c

MISRA-C 2023 (Motor Industry Software Reliability Association):

  • Originally for automotive safety (ISO 26262 functional safety).
  • Stricter subset of C — forbids goto, dynamic allocation, function pointers.
  • Used by: automotive (BMW, Bosch), aerospace (Boeing), medical devices (FDA regulated).
  • Compliance verified by: PC-lint, LDRA, Polyspace.

Secure Code Checklist

Before shipping any C library or service:

  • All array accesses bounds-checked
  • snprintf used (never sprintf, strcpy, gets)
  • malloc return values always checked for NULL
  • free() followed immediately by ptr = NULL
  • All errno-setting functions have return value checked
  • printf("%s", data) never printf(data)
  • Integer arithmetic checked for overflow before allocation sizing
  • memset_explicit() (C23) used for zeroing credentials/keys
  • Build with: -fstack-protector-strong -D_FORTIFY_SOURCE=3 -pie
  • Tested with: AddressSanitizer on all code paths
  • Tested with: Valgrind for memory leak verification
  • Fuzz tested with: libFuzzer or AFL++ for all input-parsing code

Frequently Asked Questions

Is C fundamentally insecure? C is memory-unsafe by design — the language gives you raw memory access with no bounds checking. This is intentional for performance. However, with disciplined use of safe functions, bounds checking, compiler hardening flags, and static/dynamic analysis, C software can achieve excellent security. The Linux kernel, OpenSSH, and OpenSSL run on billions of systems with acceptable security records.

Why isn't FORTIFY_SOURCE on by default? It has a measurable performance overhead (additional branch per function call) that some applications cannot afford. High-performance C code often runs with -D_FORTIFY_SOURCE=0 and relies on code review and fuzzing instead.

What is the difference between ASan and Valgrind? ASan is ~2-5× slower than native execution and is a compile-time instrumentation. Valgrind is ~10-20× slower and instruments at runtime (no recompilation needed). ASan finds more types of errors (use-after-return, stack buffer overflows) and is faster. Valgrind is more accurate for certain leak analysis and doesn't require recompilation — useful for testing without source code.

Should I prefer Rust over C for new security-sensitive code? For new code where Rust is feasible — yes. Rust's ownership model and borrow checker eliminate entire categories of memory vulnerabilities at compile time. For existing C codebases, FFI boundaries with Rust are a practical path. For systems that must remain in C (OS kernels, embedded with limited tooling), C with aggressive use of analysis tools is still the state of practice.


Key Takeaway

C security is not accidental — it is a Deliberate Engineering Practice. Every buffer using user-supplied size, every pointer cast, every format string, every integer arithmetic involving user data is a potential attack surface. By systematically applying safe functions, compiler hardening flags, and dynamic analysis tools, you transform C from "powerful but dangerous" into "powerful and professionally hardened."

Read next: Project: Heap Memory Allocator — Build malloc & free from Scratch →


Part of the C Mastery Course — 30 modules from C basics to expert systems engineering.