Core Dumps in C/C++: A Comprehensive Guide to Crash Analysis and Debugging

Disclaimer: ChatGPT generated document.

A core dump is a snapshot of a process at or near the moment it crashes. In C and C++, it is one of the most useful artifacts for debugging failures that are hard to reproduce, especially segmentation faults, aborts, illegal instructions, assertion failures, memory corruption, and crashes that occur only in production.

What a core dump contains

Conceptually, imagine freezing a crashed process and saving enough of its state to disk so that a debugger can inspect it later. A core dump can contain the process's virtual memory, stack contents, heap contents, CPU registers, thread states, loaded shared libraries, memory mappings, and other operating-system metadata.

It normally does not contain the executable's source code. To debug effectively, you typically need the core file together with the exact executable and matching shared libraries, ideally built with debugging symbols.

For example:

my_program
    ↓ crashes
Linux kernel
    ↓
core / core.12345
    +
my_program binary
    +
debug symbols
    ↓
gdb
    ↓
stack trace, variables, registers, threads, memory...

Why C and C++ programs generate core dumps

C and C++ provide relatively little runtime protection compared with managed languages. A bad memory access can directly cause a processor exception that the operating system converts into a signal.

Typical examples include:

int* p = nullptr;
*p = 42;

Usually:

SIGSEGV

Or:

std::abort();

Usually:

SIGABRT

Or an assertion:

assert(x != nullptr);

A failed assertion normally calls abort(), which produces SIGABRT.

Other possible causes include division errors, invalid instructions, bus errors, stack overflow, corrupted function pointers, buffer overflows, use-after-free bugs, or deliberately raising a fatal signal.

Signals commonly associated with core dumps

On Unix-like systems, core dumps are closely tied to signals.

Some important ones are:

Signal	Typical cause
`SIGSEGV`	Invalid memory access
`SIGABRT`	`abort()`, failed assertion
`SIGBUS`	Invalid/alignment-related memory access
`SIGILL`	Illegal CPU instruction
`SIGFPE`	Arithmetic exception
`SIGTRAP`	Debugging/breakpoint trap
`SIGQUIT`	User-generated termination, often Ctrl+\

Whether a signal actually produces a core file depends on operating-system configuration, resource limits, signal handlers, container configuration, and security policy.

A minimal crash example

#include <iostream>

void crash()
{
    int* ptr = nullptr;
    *ptr = 123;
}

int main()
{
    std::cout << "About to crash\n";
    crash();
}

Compile it with debugging information:

g++ -g -O0 crash.cpp -o crash

Then run:

./crash

You might see something like:

Segmentation fault (core dumped)

The phrase:

(core dumped)

means the operating system considered the crash core-dump-worthy. It does not always mean that a file named core appeared in the current directory. Modern systems often redirect cores elsewhere.

The `-g` compiler option

This is extremely important.

Compile debugging builds with:

gcc -g program.c -o program

or:

g++ -g program.cpp -o program

-g tells the compiler to include debugging information such as mappings between machine addresses and source files, functions, types, and line numbers.

Without symbols, GDB might show:

#0  0x0000555555555168 in ?? ()

With debugging symbols:

#0  crash() at crash.cpp:6
#1  main() at crash.cpp:12

A core dump itself doesn't magically know your source-level names.

Optimization changes debugging

This:

g++ -g -O0 app.cpp

is easiest to debug.

Production applications often use something like:

g++ -g -O2 app.cpp

or split the debugging information into separate files.

With optimization, the compiler may inline functions, eliminate variables, rearrange instructions, merge code paths, or keep variables only in registers.

Therefore GDB may report:

optimized out

for variables.

That does not mean the core dump is broken. It means the optimized machine code no longer has a simple correspondence to the original source.

Enabling core dumps on Linux

A common reason people don't get a core dump is:

ulimit -c

returning:

That means the maximum core file size is zero.

For the current shell, you can generally enable unlimited core sizes with:

ulimit -c unlimited

Verify:

ulimit -c

Expected:

unlimited

Then run your program from the same shell.

Resource limits are inherited by child processes, which is why the shell setting matters.

Where the core file goes

Historically, Unix might create:

core

or:

core.12345

in the working directory.

Modern Linux systems can instead route dumps through a core handler.

Linux exposes the configuration through:

cat /proc/sys/kernel/core_pattern

Examples might look like:

core

or:

core.%e.%p

where values can encode things like the executable name or PID.

If the value starts with:

the kernel pipes the core dump to another program rather than simply writing a file.

For example, distributions using systemd-coredump may capture crashes centrally.

Debugging a core with GDB

The classic command is:

gdb ./program core

or:

gdb ./program core.12345

Once inside GDB:

bt

means:

backtrace

A typical result:

#0  crash () at crash.cpp:6
#1  main () at crash.cpp:12

That is usually your first clue.

A very useful starting sequence is:

bt
bt full
info threads
thread apply all bt
frame 0
info locals
info args

I'll unpack all of those below.

Stack traces

Every active function call usually corresponds to a stack frame.

Suppose:

void c()
{
    int* x = nullptr;
    *x = 5;
}

void b()
{
    c();
}

void a()
{
    b();
}

int main()
{
    a();
}

A core might give:

#0 c()
#1 b()
#2 a()
#3 main()

The lowest-numbered frame is normally where execution stopped.

But an important debugging lesson is:

The crash location is not necessarily the bug location.

For example, heap corruption might occur 10,000 instructions earlier, and only later cause free() to crash.

Navigating frames

Inside GDB:

frame 0

Select the crash frame.

Or:

frame 3

Then inspect variables:

info locals

Arguments:

info args

Print something:

print ptr

Short form:

p ptr

Dereference:

p *ptr

Pointers:

p/x ptr

The /x asks for hexadecimal representation.

`bt full`

Instead of:

bt

you can use:

bt full

which tries to display local variables for each stack frame.

For production-crash triage, this is often one of the most useful commands.

Multi-threaded programs

Core dumps are particularly valuable for deadlocks and multi-threaded crashes because they may contain the states of many threads, not merely the crashing one.

In GDB:

info threads

Example:

  Id   Target Id         Frame
* 1    Thread ...        crash()
  2    Thread ...        pthread_cond_wait()
  3    Thread ...        epoll_wait()

* indicates the currently selected thread.

Switch:

thread 2

Get every thread's backtrace:

thread apply all bt

For more detail:

thread apply all bt full

This command is extremely useful for production incidents.

Registers

A core dump generally captures processor register values around the crash.

In GDB:

info registers

On x86-64 you'll see things such as:

rax
rbx
rcx
rdx
rsp
rbp
rip

rip is particularly important because it is the instruction pointer: the instruction being executed when the processor faulted.

For ARM64, you'll instead encounter registers such as:

x0 ... x30
sp
pc

Assembly around the crash

You can inspect the current instruction:

x/i $pc

Or disassemble the current function:

disassemble

More conveniently:

disassemble /m

or depending on GDB/version:

disassemble /s

can intermix source and assembly.

This becomes important when symbols are incomplete or when debugging optimized code.

Examining raw memory

GDB's x command means examine memory.

For example:

x/16x $rsp

examines memory near the stack pointer.

A useful pattern is:

x / count format size address

Examples:

x/16xb ptr

16 bytes.

x/16xw ptr

16 words.

x/8gx ptr

8 giant words, typically 64-bit values.

x/s ptr

Interpret memory as a C string.

C++ objects

With debugging symbols, GDB understands many C++ types.

For example:

std::string name = "Alice";
std::vector<int> numbers = {1, 2, 3};

you may inspect:

p name
p numbers

Modern GDB distributions often include pretty-printers for the standard C++ library, producing readable representations of containers.

Without compatible pretty-printers, std::vector, std::map, etc. can look much messier.

Core dumps and exceptions

A normal C++ exception does not generate a core dump:

throw std::runtime_error("bad");

If somebody catches it, execution continues.

But an uncaught exception eventually calls:

std::terminate()

The default terminate handler normally calls:

std::abort()

which typically generates:

SIGABRT

and potentially a core dump.

So a program might die with output such as:

terminate called after throwing an instance of 'std::runtime_error'
Aborted (core dumped)

`std::terminate`

Other situations can invoke std::terminate(), not only uncaught exceptions.

For example:

std::thread t(...);
// t still joinable here

If the std::thread object's destructor runs while the thread remains joinable, the program calls std::terminate().

Core analysis may consequently show frames involving:

std::terminate
std::abort
raise

The useful bug may be several frames above those library/runtime functions.

Assertion failures

For:

assert(value > 0);

a failure commonly results in something like:

Assertion `value > 0' failed.
Aborted (core dumped)

The trace may go through:

abort()
__assert_fail()
your_function()

When debugging, move past the libc frames and inspect your own function.

Null pointers

One classic cause:

Foo* foo = nullptr;
foo->bar();

The actual fault may occur inside Foo::bar(), depending on what the method does.

Inspect:

p this

If you see:

$1 = (Foo * const) 0x0

you've likely found a null this pointer.

Dangling pointers

More dangerous:

Foo* foo = new Foo;
delete foo;
foo->bar();

The pointer isn't necessarily zero.

It could still display a plausible address:

0x5555567c32a0

Yet the memory is no longer owned by foo.

Core dumps can help, but use-after-free bugs are often easier to diagnose with AddressSanitizer.

Heap corruption

Consider:

char* p = new char[8];
p[20] = 'X';

The illegal write may not immediately crash.

Later:

delete[] p;

might crash inside the allocator.

Your backtrace could misleadingly end in:

free()
malloc_consolidate()

or related allocator internals.

This is why interpreting core dumps requires distinguishing:

where the program crashed

from:

where memory became corrupted

For heap corruption, sanitizers are often superior at identifying the original illegal access.

Stack overflow

Infinite recursion:

void recurse()
{
    recurse();
}

eventually exhausts the stack.

The resulting core may show thousands of repeated frames:

#0 recurse()
#1 recurse()
#2 recurse()
#3 recurse()
...

A stack overflow often manifests as SIGSEGV.

Corrupted stacks

Buffer overruns can overwrite return addresses or frame metadata:

void bad()
{
    char buf[8];
    strcpy(buf, "this string is vastly too long");
}

A backtrace may then look nonsensical:

#0  ...
#1  0x4141414141414141
#2  ??

because the unwinding information or return address has been corrupted.

Again, ASan is usually excellent for this kind of problem.

Debug symbols and stripped binaries

Production executables are often stripped to reduce binary size.

For example:

strip my_program

removes much of the symbolic debugging data.

Core debugging then becomes much less readable.

A common production strategy is to keep:

production binary
+
separate debug symbol file

The deployed binary can remain small while engineers retain private symbol packages matching each released build.

The key word is matching.

Your binary must match the core dump

This is one of the most important practical rules.

Do not debug:

core from release 1.4.2

using:

binary from release 1.4.3

Even a tiny rebuild can move code and data addresses.

At minimum, retain the exact executable, debug symbols, relevant shared-library symbols, build ID/version, and ideally source revision associated with each production release.

Shared libraries matter too

Suppose your application loads:

libfoo.so
libssl.so
libstdc++.so
libc.so

The core records mappings and state involving these libraries.

If you analyze it on a machine with substantially different library versions, stack traces may be incorrect or symbols may not resolve.

GDB can be configured with library search paths and sysroots to reproduce the original runtime environment.

ASLR

Modern operating systems use Address Space Layout Randomization.

Therefore an executable or library may be loaded at different virtual addresses each run.

You might see an address such as:

0x7f35c1ab43f2

GDB uses the mappings stored in the core plus the matching ELF binaries to resolve it to something meaningful, such as:

libfoo.so!do_work()+42

You normally do not need to manually compensate for ASLR if the core, executable, and libraries are available.

PIE executables

Modern Linux binaries are often compiled as PIE:

Position Independent Executable

so the executable itself can also be randomized.

Again, good debuggers and matching binary metadata handle this automatically.

ELF and core files

On Linux, core dumps usually use the ELF format, the same broad executable format used for binaries and shared libraries.

You can inspect a core with tools like:

file core

which might report something resembling:

ELF 64-bit LSB core file, x86-64 ...

You can also inspect its ELF structure with:

readelf

For example:

readelf -h core
readelf -n core
readelf -l core

The NOTE sections contain important process metadata.

`file` command

Before debugging, this is often worth running:

file ./my_program
file ./core

It can immediately reveal architecture mismatches such as:

x86-64

versus:

ARM aarch64

or whether the executable is stripped.

`coredumpctl`

On systems using systemd-coredump, core files can be indexed by the operating system instead of appearing in the current directory.

Typical commands include:

coredumpctl list

and:

coredumpctl info

and you can often launch GDB against a selected dump:

coredumpctl gdb

You can also identify crashes by process name, PID, executable, etc.

Exact availability and behavior depend on how the Linux distribution has configured systemd.

Core dump naming

Linux's core_pattern can incorporate placeholders representing process properties.

Administrators often configure names resembling:

core.myapp.18472.1700000000

because plain:

core

is inconvenient when multiple processes crash.

In production, crash-management systems usually attach metadata such as build version, machine/container identity, timestamp, executable name, and process ID.

Core size

Core dumps can be enormous.

If a server process uses:

20 GB

of virtual/resident memory, its core could potentially be very large.

The operating system can omit some mappings or apply filtering, and crash-management systems may compress dumps.

Linux also exposes core dump filtering controls that determine which kinds of mappings are included.

Security concerns

Core dumps can contain extremely sensitive data.

If your process handled:

passwords
API keys
encryption keys
authentication tokens
customer data
database contents
private messages
TLS plaintext
credit-card data

some of it may be sitting directly in the process memory recorded in the dump.

Treat core files like potentially sensitive production data.

They should normally have restricted permissions, limited retention, controlled access, secure transport, and appropriate deletion policies.

This is one reason some security-sensitive services disable core dumps entirely.

Secrets may survive even after use

Suppose:

std::string password = read_password();
authenticate(password);

Even if the function returns, copies of the password may still exist somewhere in heap or stack memory.

A core dump can therefore expose data developers did not realize was still resident.

This is particularly important for cryptographic applications.

`fork()` and core dumps

After fork(), parent and child are separate processes with copy-on-write virtual memory.

If the child crashes, the dump describes the child's address space.

Server architectures that use worker processes may therefore produce separate dumps for individual workers rather than the supervisor.

Signal handlers can affect core generation

Consider:

signal(SIGSEGV, handler);

If your handler intercepts a segmentation fault and exits normally:

void handler(int)
{
    _exit(1);
}

you may lose the normal core dump.

If a program wants logging before termination while still producing a core, signal handling must be designed very carefully.

One common technique is eventually restoring/defaulting and re-raising the signal.

However, crash signal handlers are tricky because most C/C++ and POSIX functionality is not async-signal-safe.

Don't do complex work in `SIGSEGV` handlers

A tempting crash handler is:

void handler(int sig)
{
    std::cerr << "crashed!\n";
    save_to_database();
    malloc(...);
    ...
}

This is unsafe.

If SIGSEGV happened because memory or allocator state is corrupted, invoking complex library code can deadlock or crash again.

Signal handlers should generally perform only operations documented as async-signal-safe.

For postmortem debugging, letting the operating system generate a proper core is often safer than trying to reconstruct crash information inside the process.

`SIGSEGV` isn't a C++ exception

This misunderstanding is common.

This does not generally work:

try {
    int* p = nullptr;
    *p = 5;
}
catch (...) {
    // not reached
}

A segmentation fault is an operating-system/hardware fault, not a standard C++ exception.

catch (...) catches C++ exceptions, not arbitrary POSIX signals.

Core dumps versus sanitizers

These complement each other.

A core dump tells you:

What did the process look like when it died?

AddressSanitizer tells you something closer to:

Where did the illegal memory operation happen?

For debugging development builds, compiling with:

-fsanitize=address -fno-omit-frame-pointer -g

is extraordinarily useful.

Example:

g++ -g -O1 \
    -fsanitize=address \
    -fno-omit-frame-pointer \
    program.cpp -o program

ASan catches problems such as heap buffer overflows, stack buffer overflows, use-after-free, double-free, and many other memory errors.

For production crashes where sanitizers weren't enabled, the core dump may be the only detailed postmortem artifact you have.

UndefinedBehaviorSanitizer

Another useful option is:

-fsanitize=undefined

It can detect certain forms of undefined behavior before they turn into mysterious corruption.

For example:

g++ -g \
    -fsanitize=address,undefined \
    program.cpp -o program

This is often a very effective debugging build.

ThreadSanitizer

For data races:

-fsanitize=thread

ThreadSanitizer often finds problems that would be nearly impossible to diagnose from a single core dump.

Core dumps still help with deadlocks and final thread state, but race conditions frequently require runtime instrumentation to identify the original conflicting accesses.

Valgrind versus core dumps

Valgrind's Memcheck can also detect illegal memory operations:

valgrind ./program

Core dumps are postmortem snapshots, whereas Valgrind monitors execution.

The tools answer somewhat different questions.

A practical debugging toolkit often includes:

GDB + core dumps
ASan/UBSan
TSan
Valgrind
logging/telemetry

rather than choosing exactly one.

`-fno-omit-frame-pointer`

For better stack unwinding, particularly in profiling and crash analysis, production builds often use:

-fno-omit-frame-pointer

Historically, optimized compilers frequently reused the frame-pointer register for general computation.

Modern unwinding can use DWARF metadata without frame pointers, but retaining them can make stack traces more robust and simplify profiling.

A useful production-friendly configuration can therefore include:

-O2 -g -fno-omit-frame-pointer

depending on performance requirements.

DWARF

On Linux/Unix toolchains, debug information is commonly encoded using DWARF.

DWARF contains descriptions of things such as source lines, types, function boundaries, variable locations, call-frame information, and how machine instructions correspond to source code.

GDB uses this metadata to turn:

0x00005555555551ab

into:

Parser::parse()
parser.cpp:287

Core dumps and DWARF are separate things:

core dump = runtime state
DWARF = source/debugging metadata

You need both for excellent debugging.

Build IDs

Production Linux systems often use ELF build IDs.

A build ID uniquely identifies a particular compiled binary.

This makes it possible for crash systems to say essentially:

this core used build ID abcdef...

and find exactly the correct debug symbols from a symbol server/archive.

This is far safer than relying only on filenames such as:

myapp

because thousands of different builds can all have the same executable name.

Core dumps in Docker and containers

Containers complicate things because core generation depends on both container configuration and host-kernel configuration.

Important details can include resource limits, host core_pattern, PID namespaces, filesystem accessibility, security policies, and the process supervisor.

Running:

ulimit -c unlimited

inside the container is not always enough.

The host kernel ultimately handles the fault.

For production Kubernetes environments, crash collection is often handled by external agents or dedicated crash-reporting infrastructure rather than expecting core to appear inside an ephemeral container filesystem.

Core dumps in Kubernetes

The same basic principles apply, but several additional problems appear.

Pods disappear. Containers are ephemeral. Filesystems may be transient. The crashed process may restart immediately. The node, not merely the container, controls important kernel settings.

Therefore production systems frequently collect cores into persistent storage or use crash handlers that annotate dumps with pod, namespace, container image, build ID, and deployment version.

Without this metadata, a core dump from a large cluster can be very difficult to connect to the correct executable and symbols.

macOS

macOS also supports postmortem crash diagnostics, but the ecosystem differs from Linux.

Apple commonly uses crash reports containing stack traces and metadata, with LLDB being the standard debugger.

The analogous symbolic-debugging requirement is still the same:

crash artifact
+
exact binary
+
matching symbols

For Apple binaries, debugging symbols are commonly distributed using .dSYM bundles.

Windows

Windows does not ordinarily call these Unix-style core dumps.

The equivalent concept is typically a crash dump or minidump, commonly represented by .dmp files.

Windows tools include WinDbg and Visual Studio.

A Windows minidump may intentionally contain less memory than a full process dump, which is valuable because full dumps can be enormous.

The conceptual model remains:

saved process state + matching executable/symbols → postmortem debugging

Windows symbols frequently use PDB files.

Minidumps versus full dumps

A full dump can contain most or all useful process memory.

A minidump contains a selected subset, perhaps including threads, stacks, registers, modules, exception information, and some memory ranges.

Minidumps trade completeness for much smaller size.

Large-scale products frequently prefer compact crash reports or minidumps because millions of full core dumps would be expensive and potentially privacy-sensitive.

Breakpad and Crashpad

Applications such as browsers often use dedicated crash-reporting frameworks.

Google's Breakpad and Crashpad, for example, are designed around capturing crash information and processing symbols separately.

Instead of shipping enormous native core dumps from every user machine, crash-reporting infrastructure can collect smaller crash artifacts and symbolicate them centrally.

Symbolication

Symbolication means translating low-level machine addresses:

0x7f827ab41220

into meaningful information:

MyClass::doThing()
foo.cpp:218

It requires accurate symbols corresponding to the exact binary.

Incorrect symbols can produce plausible-looking but completely wrong traces, which is much more dangerous than seeing ??.

GDB command cheat sheet

Here is the compact set I would memorize:

gdb ./program core

bt                     stack trace
bt full                stack trace + locals
info threads           list threads
thread N               switch thread
thread apply all bt    trace every thread

frame N                select frame
up                     caller frame
down                   callee frame

info locals            local variables
info args              function arguments
p expr                 print expression
p/x expr               hexadecimal print

info registers         CPU registers
x/i $pc                instruction at program counter
disassemble            machine code around function

x/16xb addr            examine 16 bytes
x/8gx addr             examine 8 64-bit values
x/s addr               show C string

info sharedlibrary     loaded shared libraries
info proc mappings     memory mappings, where supported

A realistic core-dump debugging workflow

Suppose production reports:

myservice crashed at 03:14

A good investigation starts by preserving the exact core, executable, debug symbols, shared-library/runtime environment, build version or Git commit, and configuration metadata.

Then:

gdb ./myservice core.1234

Run:

bt

then:

thread apply all bt

Identify the crashing thread.

Select the interesting frame:

frame 4

Inspect:

info args
info locals

Then inspect suspicious objects:

p object
p object->field

Check pointers:

p/x object

Look at nearby stack frames:

up
down

If the trace suggests heap corruption rather than an immediate invalid access, attempt to reproduce with ASan.

That combination—core analysis followed by sanitizer reproduction—solves a huge fraction of serious native crashes.

A deceptively important question

Suppose GDB says:

Program terminated with signal SIGSEGV
#0 memcpy()

Do not immediately conclude:

memcpy is broken

memcpy() almost certainly received an invalid pointer or length.

Move upward:

up

and inspect:

info args
info locals

The library function where the CPU faulted is often merely the victim.

The same reasoning applies when the crash occurs in:

free
malloc
std::string
std::vector
pthread_mutex_lock
memcpy
strlen

These functions can crash because your program previously corrupted state or supplied invalid arguments.

Core dumps and deadlocks

You can also deliberately capture process state when the program hasn't crashed.

For example, attaching a debugger to a hung process lets you inspect every thread:

gdb -p PID

then:

thread apply all bt

There are also operating-system/debugger mechanisms for producing dump-like snapshots of live processes.

This is extremely valuable for deadlocks.

Imagine:

Thread 1: waiting for mutex A
Thread 2: waiting for mutex B

and their stacks reveal:

Thread 1 holds B, wants A
Thread 2 holds A, wants B

You have your deadlock cycle.

`gcore`

GDB commonly provides the gcore command or a corresponding utility to create a core dump from a running process without waiting for it to crash.

Conceptually:

gcore PID

produces a core snapshot.

This is useful when a process is hung, stuck, consuming CPU unexpectedly, or behaving incorrectly but still alive.

You can then analyze the snapshot offline.

Crashes from `abort()` are often easier than corruption

Sometimes developers deliberately abort when an invariant becomes impossible:

if (!state_is_valid()) {
    abort();
}

That can actually be very useful.

Instead of allowing bad state to propagate for another 30 seconds and eventually crash inside malloc(), the program creates a core immediately when the invariant first fails.

This is the philosophy behind many assertions.

Why reproducibility matters

Core dumps tell you about one particular execution.

They may prove:

pointer p was null

but not necessarily explain:

why did it become null?

For that you may still need logs, tracing, sanitized reproductions, source inspection, race detectors, heap instrumentation, and telemetry.

A core dump is forensic evidence, not necessarily the entire explanation.

What core dumps cannot tell you

A core is extremely powerful, but it is not a recording of everything that happened before the crash.

Usually it cannot directly tell you:

what this memory contained 5 seconds ago
who last modified this field
which thread freed this pointer
what network packet triggered the bug
the exact previous sequence of instructions

unless that evidence happens to remain in memory or another diagnostic mechanism recorded it.

For historical execution, you need tracing, logging, recording debuggers, sanitizer instrumentation, or similar tools.

Core dump versus debugger attached during crash

If you run:

gdb ./program

and then:

run

GDB catches the fault immediately.

That gives you the live process and can be more flexible than postmortem debugging.

Core dumps shine when the crash happens somewhere you cannot interactively debug:

production server
CI test
customer machine
overnight batch job
rare race condition
remote embedded system

You capture the evidence once and inspect it later.

Compiler recommendations

For a debug build:

-g -O0

For a realistic optimized build that remains fairly debuggable:

-g -O2 -fno-omit-frame-pointer

For memory-bug development/testing:

-g -O1 -fsanitize=address,undefined -fno-omit-frame-pointer

For race hunting:

-g -O1 -fsanitize=thread

Do not generally combine AddressSanitizer and ThreadSanitizer in the same executable.

Production recommendations

If you ship serious C or C++ software, a mature crash-debugging setup usually has a deterministic build identifier, exact symbols archived for every release, core/minidump collection, restricted access to dumps, automatic symbolication, source-revision mapping, retention limits, and integration with crash grouping/telemetry.

The goal is that an address such as:

0x7f2c18ab9204

from a production server can reliably become:

RequestHandler::finish()
request_handler.cpp:417
commit 91ce2af
release 8.13.2

without having to reproduce the crash.

The mental model to remember

A core dump is fundamentally:

memory
+ registers
+ thread states
+ process metadata
at one instant in time

And effective debugging is:

core dump
+ exact executable
+ matching shared libraries
+ matching debug symbols
+ matching source
+ debugger

From there, the central investigation is usually:

Which thread crashed?
        ↓
What instruction faulted?
        ↓
What stack led there?
        ↓
What arguments/locals/pointers were present?
        ↓
Is this the original bug or merely downstream corruption?
        ↓
Can ASan/UBSan/TSan reproduce the root cause?

If you become comfortable with gdb, bt, thread apply all bt, frame, info locals, p, registers, and debug symbols, you'll be able to diagnose a surprisingly large percentage of real-world native crashes.

MangaD/cpp_core_dumps.md

Core Dumps in C/C++: A Comprehensive Guide to Crash Analysis and Debugging

What a core dump contains

Why C and C++ programs generate core dumps

Signals commonly associated with core dumps

A minimal crash example

The -g compiler option

Optimization changes debugging

Enabling core dumps on Linux

Where the core file goes

Debugging a core with GDB

Stack traces

Navigating frames

bt full

Multi-threaded programs

Registers

Assembly around the crash

Examining raw memory

C++ objects

Core dumps and exceptions

std::terminate

Assertion failures

Null pointers

Dangling pointers

Heap corruption

Stack overflow

Corrupted stacks

Debug symbols and stripped binaries

Your binary must match the core dump

Shared libraries matter too

ASLR

PIE executables

ELF and core files

file command

coredumpctl

Core dump naming

Core size

Security concerns

Secrets may survive even after use

fork() and core dumps

Signal handlers can affect core generation

Don't do complex work in SIGSEGV handlers

SIGSEGV isn't a C++ exception

Core dumps versus sanitizers

UndefinedBehaviorSanitizer

ThreadSanitizer

Valgrind versus core dumps

-fno-omit-frame-pointer

DWARF

Build IDs

Core dumps in Docker and containers

Core dumps in Kubernetes

macOS

Windows

Minidumps versus full dumps

Breakpad and Crashpad

Symbolication

GDB command cheat sheet

A realistic core-dump debugging workflow

A deceptively important question

Core dumps and deadlocks

gcore

Crashes from abort() are often easier than corruption

Why reproducibility matters

What core dumps cannot tell you

Core dump versus debugger attached during crash

Compiler recommendations

Production recommendations

The mental model to remember

The `-g` compiler option

`bt full`

`std::terminate`

`file` command

`coredumpctl`

`fork()` and core dumps

Don't do complex work in `SIGSEGV` handlers

`SIGSEGV` isn't a C++ exception

`-fno-omit-frame-pointer`

`gcore`

Crashes from `abort()` are often easier than corruption