(This is additional detail for one of two metrics in a recent LLVM RFC. The ideas here were proposed by Stephen Livermore-Tozer.)
In addition to a variable coverage metric (described in our RFC elsewhere), we also plan to measure per-variable line table coverage as well. This is particularly useful for disambiguating cases where the above variable coverage metric shows less than 100% coverage. By also checking line table coverage for the same variable, it becomes clear whether the missing variable coverage is due to source lines missing from the line table or instructions missing from variable location ranges.
For the variable v and build configuration o, we have S(v) is the set of source lines in v's scope at which it is defined, and B(o,v) is the subset of S(v) at which v has a value described by debug info when built with o. Our per-variable coverage metric is then
C(o,v) = |B(o,v)| / |S(v)|
where
S(v) is the set of source lines in v's scope at which it is defined and
B(o,v) is the subset of source lines for which variable v is described in the debug info of build o.
Additionally to capture the line table coverage, we first define
P(o) the set of source lines present in the line table of build o,
and then we define a per-variable "projected" measure of the line table coverage:
L(o,v) = |S(v) ∩ P(o)| / |S(v)|
For any variable v, we have:
L(o,v) = 1 if the line table contains all lines over which v is defined
If we're working without a separate "O0 baseline", this always holds because it means we're approximating |S(v)| by |S(v) ∩ P(o)|. In other words we don't have any "ground truth" for the correct line table.
L(o,v) = C(o,v) if the debug info had perfect coverage of variable v "up to" that possibly incomplete line table.
Otherwise L(o,v) > C(o,v). It means some lines in variable v's defined ranges are in the line table but there's no description of variable v at those lines. In other words, the line table has better coverage of v’s defined range than v’s variable info does.
Putting all that another way, the gap between C(o,v) and L(o,v) tells us, of the shortfall in C, how much can be attributed to the line table.
If L = 1, the line table is blameless.
If L = C, its coverage shortfall explains all of C's shortfall.
C(o,v) and L(o,v) both define values per-variable and share a common denominator, meaning they can be directly compared for each variable (e.g. plotted together, as a way to visualise this "gap").
As a worked example, consider the following code:
1: int phi(unsigned n) {
2: if (n < 2)
3: return 1;
4: unsigned r = 1;
5: for (int i = 2; i < n; i++)
6: if (gcd(i, n) == 1)
7: r++;
8: return r;
9: }`
For the variable r with o = "clang-19 -O3 -g", we have the following:
P = {2, 3, 4, 5, 6, 7, 8} all source lines in r's scope
P(o) = {2, 5, 6, 8} the ones in the optimised line table
S(r) = {5, 6, 7, 8} the ones where r has a defined value
B(o,r) = {5, 6} the subset that happen to be covered in r's debug info
Thus, we compute
|P(o)| = 4
C(o,r) = 2/4 = 0.5
L(o,r) = 3/4 = 0.75
i.e. for variable r, our coverage result pair (C(o,r), L(o,r)) = (0.5, 0.75).
Now let's consider some possible changes to the completeness of the line table P(o). We are holding constant B(o,r), the set of lines where we can in principle describe the variable r (assuming we have those lines in our line table).
If we gain stepping on line 7, r is available there, so our new result is (0.75, 1.0)
(both have gone up! the new line coverage has also enabled new coverage of variable r)
If we gain stepping on line 7, r was unavailable, we get (0.5, 1.0)
(a gain on r's lines, but not on r itself which is unavailable there)
If we gain stepping on line 3, where r is not defined, we get (0.5, 0.75)
(i.e. no change! no line coverage has been gained that is relevant to variable r)
If we lose stepping on line 6, where r was available, we get (0.25, 0.5)
(i.e. both go down)
If we lose stepping on line 8, where r was already unavailable, we get (0.5, 0.5)
(i.e. only the line metric goes down)
If we lose stepping on line 2, where r is not defined, we get (0.5, 0.75)
(again no change)
One disadvantage of making everything per-variable is that in the degenerate case of lines where there are no locals in scope, the L metric doesn't reveal anything about line coverage gains or losses. Such situations are rare, however, and we can easily in addition simply calculate |Po|/|P| for an overall line table coverage figure. It just wouldn't be as useful to quote this number alongside the per-variable coverage numbers.