zlib-ng PR #1977 follow-up: unroll adler32 AVX-512-VNNI x4 for more in-flight VPDPBUSD (verified spill-free + bit-exact vs reference; not yet benchmarked)

zlib-ng-pr1977-adler32-avx512vnni-4x-unroll-README.md

zlib-ng — adler32 AVX-512-VNNI: unroll ×4 for more in-flight VPDPBUSD

Follow-up idea spun off from PR #1977 (AVX2-VNNI adler32).

Why

adler32_avx512_vnni uses only 2 VPDPBUSD accumulators (vs2, vs2_1). VPDPBUSD has ~5-cycle latency, so two in-flight dot products can't cover it and the 128-byte loop is latency-bound at ~5c/128B — the same ceiling as the AVX2-VNNI path.

But AVX-512 has 32 zmm registers, not AVX2's 16. The AVX2 kernel is physically capped at 4 accumulators; the AVX-512 kernel has the budget to unroll wider and simply isn't using it.

Change

Unroll the main loop ×4 (256 B/iter) with four VPDPBUSD accumulators (vs2, vs2_1, vs2_2, vs2_3), and peel the 64-byte remainder down to a 256-byte multiple. Four in-flight dot products cover the latency, roughly halving the latency-bound time to ~5c/256B.

Verification

• Correctness: the restructured integer accounting was modeled scalar-side and is bit-identical to the reference adler32 across 331 sizes, including every block/NMAX boundary (63/64/65, 127/128, 191/192, 255/256/257, 5551/5552/5553, ...) with random initial adler.
• Codegen (clang 22, -mavx512vnni): builds spill-free, 14 of 32 zmm registers used, 4 vpdpbusd in the inner loop. The two-accumulator original uses 12 zmm.

Expectations / caveats

• Up to ~2x on the compute-bound (L1/L2-resident) range where it's VPDPBUSD-latency-bound. Negligible once memory-bandwidth-bound on large buffers.
• Not benchmarked on hardware — I'm on arm64, so this is asm + scalar-model verification only. Needs an avx512vnni part (Ice Lake / Sapphire Rapids / Zen 4+). A/B before merging.
• The same 4-accumulator copy variant should follow if this holds up.

Apply with git apply from the repo root.

zlib-ng-pr1977-adler32-avx512vnni-4x-unroll.diff

diff --git a/arch/x86/adler32_avx512_vnni.c b/arch/x86/adler32_avx512_vnni.c
--- a/arch/x86/adler32_avx512_vnni.c
+++ b/arch/x86/adler32_avx512_vnni.c
@@ -44,34 +44,40 @@
         len -= k;
         __m512i vs1_0 = vs1;
         __m512i vs3 = _mm512_setzero_si512();
-        /* We might get a tad bit more ILP here if we sum to a second register in the loop */
+        /* Four VPDPBUSD accumulators to keep enough dot-product ops in flight to hide the
+         * instruction's ~5-cycle latency. AVX-512 has the register budget (32 zmm) that the
+         * AVX2 path lacks, so it can unroll wider than two. */
         __m512i vs2_1 = _mm512_setzero_si512();
-        __m512i vbuf0, vbuf1;
+        __m512i vs2_2 = _mm512_setzero_si512();
+        __m512i vs2_3 = _mm512_setzero_si512();
+        __m512i vbuf0, vbuf1, vbuf2, vbuf3;
 
-        /* Remainder peeling */
-        if (k % 128) {
-            vbuf1 = _mm512_loadu_si512((__m512i*)src);
+        /* Remainder peeling down to a multiple of 256 bytes */
+        while (k % 256) {
+            vbuf0 = _mm512_loadu_si512((__m512i*)src);
 
             src += 64;
             k -= 64;
 
-            __m512i vs1_sad = _mm512_sad_epu8(vbuf1, zero);
+            __m512i vs1_sad = _mm512_sad_epu8(vbuf0, zero);
             vs1 = _mm512_add_epi32(vs1, vs1_sad);
             vs3 = _mm512_add_epi32(vs3, vs1_0);
-            vs2 = _mm512_dpbusd_epi32(vs2, vbuf1, dot2v);
+            vs2 = _mm512_dpbusd_epi32(vs2, vbuf0, dot2v);
             vs1_0 = vs1;
         }
 
-        /* Manually unrolled this loop by 2 for an decent amount of ILP */
-        while (k >= 128) {
+        /* Manually unrolled this loop by 4 for a decent amount of ILP */
+        while (k >= 256) {
             /*
                vs1 = adler + sum(c[i])
                vs2 = sum2 + 64 vs1 + sum( (64-i+1) c[i] )
             */
             vbuf0 = _mm512_loadu_si512((__m512i*)src);
             vbuf1 = _mm512_loadu_si512((__m512i*)(src + 64));
-            src += 128;
-            k -= 128;
+            vbuf2 = _mm512_loadu_si512((__m512i*)(src + 128));
+            vbuf3 = _mm512_loadu_si512((__m512i*)(src + 192));
+            src += 256;
+            k -= 256;
 
             __m512i vs1_sad = _mm512_sad_epu8(vbuf0, zero);
             vs1 = _mm512_add_epi32(vs1, vs1_sad);
@@ -84,12 +90,24 @@
             vs1_sad = _mm512_sad_epu8(vbuf1, zero);
             vs1 = _mm512_add_epi32(vs1, vs1_sad);
             vs2_1 = _mm512_dpbusd_epi32(vs2_1, vbuf1, dot2v);
+
+            vs3 = _mm512_add_epi32(vs3, vs1);
+            vs1_sad = _mm512_sad_epu8(vbuf2, zero);
+            vs1 = _mm512_add_epi32(vs1, vs1_sad);
+            vs2_2 = _mm512_dpbusd_epi32(vs2_2, vbuf2, dot2v);
+
+            vs3 = _mm512_add_epi32(vs3, vs1);
+            vs1_sad = _mm512_sad_epu8(vbuf3, zero);
+            vs1 = _mm512_add_epi32(vs1, vs1_sad);
+            vs2_3 = _mm512_dpbusd_epi32(vs2_3, vbuf3, dot2v);
             vs1_0 = vs1;
         }
 
         vs3 = _mm512_slli_epi32(vs3, 6);
-        vs2 = _mm512_add_epi32(vs2, vs3);
         vs2 = _mm512_add_epi32(vs2, vs2_1);
+        vs2_2 = _mm512_add_epi32(vs2_2, vs2_3);
+        vs2 = _mm512_add_epi32(vs2, vs2_2);
+        vs2 = _mm512_add_epi32(vs2, vs3);
 
         adler0 = partial_hsum(vs1) % BASE;
         adler1 = _mm512_reduce_add_epu32(vs2) % BASE;