AttnRes: Attention Over the Residual Stream — Experimental Results (2026-03-20)

attnres-results.md

AttnRes: Attention Over the Residual Stream

Overview

AttnRes replaces the standard residual connection in transformers with a depth attention mechanism — instead of simply adding each layer's output to a running sum, the model attends over previous layer outputs to decide what information to carry forward.

Standard transformers use x = x + layer(x) at every layer. AttnRes variants replace this with a learned attention operation across the depth axis: "which previous layers' outputs should I attend to when constructing the input to this layer?"

All experiments use a GPT-2-style decoder-only transformer trained on FineWeb-Edu (10B tokens), with RoPE, SwiGLU, and RMSNorm.

Variants Tested

Core Variants

Variant	Description
standard (baseline)	Normal residual connections. x = x + layer(x).
full_attnres	Each layer attends over all prior layer outputs individually. Most expressive but O(L²) in depth.
control_block	Layers grouped into blocks of 3. Depth attention operates over block summaries (mean of the 3 layer outputs in each completed block) plus the current block's partial state. Reduces depth sequence length from L to L/3.
block_input_dep_query	Like control_block, but with input-dependent queries — a low-rank (rank 16) modulation of the block query based on the current layer's input, giving per-layer query specialization within each block.
block_hybrid_memory	Like control_block, but also maintains a sliding window of the 4 most recent raw layer outputs in addition to block summaries. Combines coarse long-range memory (block summaries) with fine-grained local memory (recent outputs).

Depth Encoding Variants

Variant	Description
block_alibi_depth	ALiBi-style linear bias on depth attention logits based on layer distance. Nearby blocks get higher attention weight, distant blocks are penalized linearly.
block_gqa_depth	Grouped query attention for depth — queries are shared across groups of 4 layers (reducing query parameters by 4×) while keys/values remain per-block.
block_rope_depth	Rotary positional encoding on the depth dimension, giving the model a continuous sense of "how far apart in depth" two blocks are.
block_sparse_depth	Sparse depth attention — each block attends only to the 4 nearest blocks + every 4th older block. Reduces computation for very deep models.
block_value_proj	Learned value projection — projects block summary outputs through a learned matrix before using them as values in depth attention (rather than using raw block outputs).

Deep Variants (48 layers)

Same total parameters as the 24-layer models (~500M), but with 48 narrower layers (d=1080 vs d=1536). Tests whether depth + AttnRes can outperform width.

Variant	Description
deep_baseline	48-layer standard residuals. Pure depth scaling test.
deep_control_block	48-layer with block AttnRes (16 blocks of 3 layers).
deep_hybrid_memory	48-layer with hybrid memory AttnRes.

Results: Big Model (500M params, 10B tokens)

Completed Experiments

Eval loss at step 6000 (held-out data). Lower is better. Baseline = standard residuals.

Variant	Eval Loss	Δ vs Baseline	Train Loss (final)	Seeds	Throughput
block_hybrid_memory	2.682	−0.034	2.623	4 seeds (1337, 42, 123, 456)	197k tok/s
block_gqa_depth	2.691	−0.025	2.628	1 seed (s42 in progress)	248k tok/s
control_block	2.694	−0.022	2.631	2 seeds	249k tok/s
block_alibi_depth	2.704	−0.012	2.642	1 seed	247k tok/s
baseline_standard	2.716	—	2.651	1 seed	459k tok/s
block_input_dep_query	2.717	+0.001	2.657	1 seed	216k tok/s
block_value_proj	3.068	+0.352	3.071	1 seed	242k tok/s
full_attnres	3.215	+0.499	3.180	1 seed	163k tok/s
block_sparse_depth	2.816*	+0.100	2.831*	1 seed	246k tok/s

*Killed early at step 5000 — clearly not converging.

Multi-Seed Results for block_hybrid_memory

Seed	Eval Loss @ 6000	Final Train Loss
42	2.682	2.620
1337	2.691	2.629
123	2.692	2.630
456	2.697	2.635
Mean	2.690	2.628

Deep Variants (48-layer, in progress)

Comparison at step 2500 eval loss (partial results):

Model	Eval @ 2500	Δ vs Shallow Baseline
deep_control_block (48L + attnres)	2.946	−0.028
Shallow baseline_standard (24L)	2.974	—
deep_baseline (48L, standard)	2.998	+0.024

deep_hybrid_memory still too early to compare (step ~1200).

Interpretation

1. AttnRes consistently beats standard residuals

The top 4 variants all outperform the baseline, with block_hybrid_memory showing the largest and most robust improvement (−0.034 eval loss, confirmed across 4 seeds). This is a real architectural improvement, not noise.

2. Block-level granularity is essential

full_attnres (per-layer attention) is dramatically worse (+0.499) — it's too expensive per step and can't train enough in the same token budget. Grouping layers into blocks of 3 and attending over block summaries is the right tradeoff.

3. Hybrid memory is the best single variant

Combining block summaries (coarse long-range) with a sliding window of recent layer outputs (fine-grained local) gives the best results. The intuition: nearby layers benefit from precise information, while distant layers only need compressed summaries.

4. GQA depth is the efficiency champion

block_gqa_depth achieves nearly identical quality to block_hybrid_memory (2.691 vs 2.682 eval) while running at 248k tok/s (vs 197k) and using 89GB memory (vs 120GB). That's 26% faster and 25% less memory for 0.009 eval loss. Shared queries across layer groups lose almost nothing while cutting depth attention cost substantially. Second seed in progress to validate.

5. Depth encoding matters — but not all approaches work

• ALiBi depth (linear bias): modest help (−0.012). Simple inductive bias that nearby layers matter more.
• RoPE depth: crashed at step 80 — sinusoidal positional encoding may not suit the depth dimension well (or needs hyperparameter tuning).
• Sparse depth: no convergence. Restricting which blocks can attend to each other hurts too much.
• Value projection: catastrophically worse. Adding a learned projection on top of block summaries may over-parameterize the depth pathway.

6. Depth + AttnRes > Width > Depth alone

The deep model results (preliminary) tell a compelling story:

• 48 narrow layers with standard residuals underperform 24 wide layers (2.998 vs 2.974 eval @ step 2500)
• But 48 narrow layers with AttnRes outperform both (2.946 eval @ step 2500)

This suggests AttnRes doesn't just help — it's what makes deeper models viable at matched parameter count. Standard residual connections may be the bottleneck preventing effective depth scaling.

7. Input dependence doesn't help

block_input_dep_query (per-layer query modulation) performs no better than baseline despite extra parameters and slower throughput. The depth attention query doesn't need to vary per-input — a learned per-block query is sufficient.

Cost/Benefit Summary

Variant	Quality (eval Δ)	Speed Cost	Memory Cost	Verdict
block_hybrid_memory	★★★★ (−0.034)	−57%	+145%	Best quality, expensive
block_gqa_depth	★★★½ (−0.025)	−46%	+83%	Best quality/cost ratio
control_block	★★★ (−0.022)	−46%	+83%	Solid, simpler
block_alibi_depth	★★ (−0.012)	−46%	+83%	Modest, cheap
block_input_dep_query	☆ (+0.001)	−53%	+99%	Not worth it

Speed cost = reduction in tok/s vs baseline. Memory cost = increase in GPU memory vs baseline (49GB).

What's Next

1. block_gqa_depth seed 42 — in progress, validate multi-seed consistency
2. Deep model completions — deep_baseline, deep_control_block, deep_hybrid_memory all running; full comparison in ~20 hours
3. block_mqa_depth — proposed but not yet implemented. Multi-query depth attention (one shared query, different keys per block). Could push efficiency even further than GQA.
4. Scaling — if depth + AttnRes continues to outperform, test at larger parameter counts

---

Experiments run on 4× NVIDIA B200 (180GB each, 8 GPUs per node). Training framework: PyTorch + torchrun DDP. Dataset: FineWeb-Edu sample-10BT.

Last updated: 2026-03-20