Skip to content

Instantly share code, notes, and snippets.

@bjacob
Created July 20, 2026 14:35
Show Gist options
  • Select an option

  • Save bjacob/a5cbf589c78a5e6558f2310e95c29b80 to your computer and use it in GitHub Desktop.

Select an option

Save bjacob/a5cbf589c78a5e6558f2310e95c29b80 to your computer and use it in GitHub Desktop.

ConSan Synchronization Identities and the SGEMM Scalability Gap

What is a synchronization identity?

ConSan assigns a stable textual identity to each synchronization event found while analyzing a GPU code object. A representative identity has this shape:

fnv1a64:<code-object-fingerprint>
|kernel=<complete-kernel-symbol>
|event=barrier
|pc=0x0000000000012340
|mnemonic=s_barrier_signal
|occurrence=12

The fields identify:

  • the code object containing the event;
  • the complete kernel or function symbol;
  • the synchronization-event kind;
  • the instruction's text offset;
  • the decoded mnemonic; and
  • the occurrence number used to distinguish otherwise similar events.

ConSan uses these identities in inventories, logs, fault specifications, provenance, and synchronization sequences. Storing identities rather than unstable object addresses also makes evidence reproducible across processes.

Why can an identity be long?

Tensile-generated kernel symbols encode a large number of solution parameters. A symbol can include tile dimensions, data types, workgroup geometry, Stream-K mode, memory-layout choices, and many other tuning flags. The kernel-name field alone can therefore contain hundreds of characters.

Many kernels in one aggregate library have identities with similarly long, nearly identical prefixes. Comparing two identities may consequently inspect substantial portions of their strings before reaching the field that differs.

"Long synchronization identity" refers to this textual representation. It does not mean that the GPU synchronization instruction itself is long or slow.

The old owner-annotation algorithm

A synchronization sequence stores the identities of its member events. During host-side owner annotation, the old implementation looked up each member with a linear search through the complete event inventory, conceptually:

for (sequence : synchronization_sequences) {
  for (identity : sequence.member_event_identities) {
    event = find_if(sync_events, [&](const auto &candidate) {
      return candidate.identity == identity;
    });
    annotate_sequence_from(event);
  }
}

The full SGEMM aggregate contained 115,776 synchronization events and roughly a corresponding number of sequence-member lookups. A linear search for each member can require approximately:

115,776 * 115,776 / 2 = about 6.7 billion identity comparisons

Those comparisons operate on the long strings described above. Sampling the instrumenting process showed it repeatedly performing this lookup and string comparison inside annotate_execution_owners.

This work occurred while ConSan was analyzing and patching the code object on the host. It happened before the affected SGEMM benchmark problem began GPU execution, so the 900-second stall was not evidence that the workload itself or the software GPU was intrinsically that slow.

The fix

The fix builds an identity index once:

std::unordered_map<std::string_view, const ConSanSyncEvent *>
    event_by_identity;

event_by_identity.reserve(sync_events.size());
for (const ConSanSyncEvent &event : sync_events)
  event_by_identity.emplace(event.identity, &event);

Every sequence-member lookup then uses the index:

auto event = event_by_identity.find(identity);

Using emplace preserves the former first-match behavior if duplicate identities ever occur. The string_view keys refer to the existing event identity storage, so the index does not duplicate every long string.

The algorithm now performs work approximately proportional to the number and total byte length of the identities, rather than proportional to the square of the event count. Hashing a long identity still costs time proportional to its length, but it no longer triggers a scan across the entire inventory.

Regression evidence

A focused regression constructs 8,192 synchronization events and sequences, runs owner annotation, and verifies that every sequence receives the expected kernel-local execution owner. With the identity index, that regression completes in about 0.55 seconds. The complete ConSan host suite also passes.

The identity representation remains valuable for stable evidence and fault selection. The scalability defect was the repeated linear search, not the existence of stable textual identities.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment