implementation sketch for faster integer overflow checking in clang/llvm

overflow.md

I want to speed up integer overflow checking for hardening purposes by keeping a sticky overflow flag and only trapping when necessary. I want to keep it super simple while hopefully giving the optimizers room to do their thing.

In the codegen part of clang:

• each function gets an i1 for storing overflow information, initialized to 0
• each integer overflow check ORs its result into the overflow flag
• before each function call, return instruction, or other side-effecting operation, execude ud2 if overflow is set

Reasonable?

This scheme (saturating has_overflowed bit) will also make
optimizing overflow checks away more difficult. I.e. in

i32 a = ...;
if ((a + 2) sign overflowed) set bit;
...
if ((a + 1) sign overflowed) set bit;

you won't be able to optimize away the second check just by doing
control flow analysis (you'll have to rely on the semantics of set bit), but in

i32 a = ...;
if ((a + 2) sign overflowed) abort();
...
if ((a + 1) sign overflowed) abort();

you can.

Ok, I'm convinced, thanks guys, I'll start reading the passes relevant for check removal.