madebyollin

What is the DLSS5 Model Architecture?

Let's attempt to figure out the model architecture from DLSS5's binary! See https://gist.github.com/madebyollin/f87b506b2779c2aeae5e44c81ef0fdee for context.

Warning

The remainder of this gist was authored by Codex 5.6

Architecture

Pipeline summary

madebyollin

What is the DLSS4.5 Model Architecture?

DLSS 4.5 is a transformer that produces nice-looking, temporally-stable videos, frame by frame, in real time, on consumer graphics cards. Amazing! I was curious to know precisely what neural network architecture DLSS4.5 is using. To figure that out, I asked Codex (+Hopper MCP) to analyze the nvngx_dlss_310.7.129 DLL and piece together what's going on.

Warning

The remainder of this gist was authored by Codex 5.5.

Architecture

Pipeline summary

madebyollin

Pure-Play World-Model Companies

• Overworld (Louis Castricato, Shahbuland Matiana, Andrew Lapp...)
• World Labs (Keunhong Park, Jason Bunk, Mohamed El Banani, Julian Quevedo...)
• General Intuition (Eloi Alonso, Vincent Micheli, Adam Jelley, Anthony Hu...)
• OdysseyML (Ahmad Nazeri, Amogh Adishesha, Jenny Seidenschwarz, Richard Shen, Sarah King, Tobiah Rex, Vighnesh Birodkar...)
• Decart (Dean Leitersdorf, Orian Leitersdorf, Moshe Shalev...)
• 💀 Dynamics Lab (Joe Chen, Ruizhou Ding, Yemin Shi): Founder hired by Roblox, site is dead
• 💀 Morpheus (Xun Huang, Guande He, Yongqi Chen...): [Company acquired by Roblox](https://www.reddit.com/r/neuralworlds/comments/1tvtqkk/morpheus_ai_acquired_b

madebyollin

Problem

If you run pytorch ops on CPU you may get spammed with this extremely annoying warning about NNPack:

#!/usr/bin/env python3
import torch as th
import torch.nn as nn
conv = nn.Conv2d(3, 3, 3)
# cpu forward here will spam an nnpack warning

madebyollin

Claude's Mentra Live Hardware Notes

Verified 2026-03-03, firmware asg-client v34.0 (build date 2026-02-04).

ADB

Both USB and WiFi ADB work. Mentra's own blog notes the charging cable "also works as a USB cable so developers are free to access, modify, and replace Mentra Live's software" (https://mentraglass.com/blogs/blog/making-mentra-live).

madebyollin

NWCapture World Recording Instructions

What is a neural world?

Please read https://madebyoll.in/posts/world_emulation_via_dnn/

You can also find more experimental neural worlds at https://neuralworlds.net.

What is the NWCapture App?

madebyollin

jupyterlab_vim adds a truly unfortunate keybinding where, if you are in normal mode and press -, the current cell splits into two. I have never intentionally activated this feature yet have unintentionally activated it many dozens of times. It's awful.

Unfortunately, the logical route to disable the split-cell feature (modifying keybinding config files) doesn't work. I don't know why.

Therefore, I now manually locate the vim_bindings file and patch it:

#!/usr/bin/env python3
from pathlib import Path

madebyollin

Single-pass Superconditioning

Motivation

Guided diffusion sampling typically uses two forward passes per step:

1. One caption-conditional forward pass, to compute E[flow | noisy image, noise level, caption]
2. One unconditional forward pass, to compute E[flow | noisy image, noise level]

These results are then linearly combined to form a single guided/superconditioned flow prediction.

madebyollin

These useful concepts show up in specific areas of NN-training literature but can be applied pretty broadly.

1. Non-leaky augmentations: you can add arbitrary augmentations during training, without substantially biasing in-domain performance, by adding a secondary input that tells the network which augmentations were used. This technique shows up the Karras et al image generation papers (ex. https://arxiv.org/pdf/2206.00364) but it's applicable whenever you want good performance on limited data.
2. Batch-stratified sampling: rather than generating per-sample random numbers with e.g. torch.rand(batch_size), you can use th.randperm(batch_size).add_(th.rand(batch_size)).div_(batch_size) instead, which has the same distribution but lower variance, and therefore trains more stably. This shows up in k-diffusion https://github.com/crowsonkb/k-diffusion/commit/a2b7b5f1ea0d3711a06661ca9e41b4e6089e5707, but it's applicable whenever you're randomizing data across the batch axis.
3. Replay buffers: when y

madebyollin

Reviewing the Claims of DC-AE

TL;DR - I think the paper is a good contribution and basically holds up, but Figure 2 seems suspicious and the released repo doesn't include the pieces (AE training code and pretrained 4096-element AEs) that would be needed to make DC-AE practically competitive with SD/SDXL VAEs.

---

DC-AE is an MIT / Tsinghua / NVIDIA paper about improving generative autoencoders (like the SD VAE) under the high-spatial-compression ratio regime.