Skip to content

Instantly share code, notes, and snippets.

@austintraver
Created May 24, 2026 23:11
Show Gist options
  • Select an option

  • Save austintraver/809ff19bf67a8ca3c2b0bc1a5dca8898 to your computer and use it in GitHub Desktop.

Select an option

Save austintraver/809ff19bf67a8ca3c2b0bc1a5dca8898 to your computer and use it in GitHub Desktop.
ai-foundry legacy research harvest (2026-05-18) -- 8 domain documents, 179 verified citations, full verification annotations preserved
title Captioning and Dataset Design Research Harvest
source_project claude-comfyui
harvest_date 2026-05-17
domain captioning
verification_level full-academic
claims_total 35
claims_verified 28
claims_unverified 5
claims_suspect 2
conflicts_found 3

Captioning and Dataset Design Research Harvest

Distilled from 7 legacy research documents covering captioning principles, trigger token design, dataset construction, vocabulary diversification, and body/face descriptor strategy for LoRA training on diffusion models with decoder-only LLM text encoders.

Summary

The legacy project produced a comprehensive captioning framework through iterative research, field testing on 359 training images (82 face, 277 body), and cross-referencing against 11+ academic papers and 10+ community guides. The framework centers on one core principle and four supporting methodologies:

  1. Core principle: "Caption what you do not want to train." Uncaptioned visual features are absorbed into the LoRA weights and bound to the trigger token. Captioned features remain prompt-responsive. This principle is architecture-agnostic and transfers across model families.

  2. Vocabulary diversification: Repetitive caption phrasing causes the model to overtrain on caption style before learning the subject. Lighting, expression, camera angle, and pose descriptions must be drawn from diverse vocabulary pools with per-phrase usage caps.

  3. Attribute ordering for causal-attention encoders: When the text encoder uses causal attention (decoder-only LLMs like Mistral), word order determines attention weight. The trigger token placed at position 0 receives the most attention from all subsequent tokens. Identity-critical information should be front-loaded.

  4. Trigger token design: BPE tokenizers fragment mixed-alphanumeric strings at digit boundaries. Leet-speak triggers (s4ndy, n4omi) fragment into 3-4 tokens, causing DOP token count mismatch, context boundary failures, and diluted identity signal. Single-token triggers eliminate all three problems.

  5. Variable vs identity feature separation: Face LoRAs omit facial bone structure, eye shape, skin tone from captions (identity features). Body LoRAs omit body proportions. The face description in body LoRA captions must be exactly "a woman" to prevent interference during LoRA stacking.


Verified Findings

VF-1: "Caption What You Do Not Want to Train" Principle

Claim: Uncaptioned visual features get absorbed into LoRA weights and bound to the trigger token. Captioned features remain prompt-responsive because the text encoder "explains them away."

Mechanism: During training, the denoising loss encourages the model to predict the image from caption + LoRA weights. Features described in the caption are attributed to text conditioning. Features not described must be encoded in LoRA weights to minimize loss.

Evidence: This principle is stated independently by Pelayo Arbues ("uncaptioned elements become part of the character identity, while captioned details remain prompt-controllable"), the project's own captioning strategy document, and at least 3 additional community guides (fal.ai, Apatero, alvdansen HuggingFace blog). DisenBooth (Chen et al., 2023) formalizes the underlying entanglement problem academically.

Verification: Pelayo Arbues article confirmed via WebFetch (live, content matches). DisenBooth paper confirmed at arXiv:2305.03374 (title and identity-entanglement claims match). fal.ai guide confirmed via WebFetch (recommends natural language descriptions, notes caption files "lead to far better LoRA learning retention"). alvdansen blog confirmed via WebFetch (discusses frequency and consistency of terms signaling importance to the model).

Critical gap: No paper has experimentally isolated the "caption vs omit" variable for subject-driven fine-tuning in a controlled ablation. The principle is derived from community experimentation and theoretical reasoning about loss attribution.

VF-2: Natural Language Outperforms Tags for Grammar-Aware Text Encoders

Claim: When the text encoder is a decoder-only LLM (Mistral, Qwen) rather than a contrastive encoder (CLIP), natural language captions are unambiguously superior to booru-style tags. Tags force a grammar-aware model to process grammatically meaningless input, losing relational context between concepts.

Evidence: BFL prompting guide states FLUX.2 "replaced CLIP with Mistral Small 3.1, a large language model that actually understands sentence structure, context, and relationships between concepts." fal.ai recommends "natural language descriptions rather than tags." Apatero states "Flux 2's Mistral-3 text encoder understands nuanced, detailed descriptions." Wang et al. (2025, arXiv:2506.08210) found decoder-only LLMs "outperform the baseline T5 model."

Verification: BFL prompting guide confirmed via WebFetch (word order matters, Subject + Action + Style + Context ordering recommended). fal.ai guide confirmed (recommends natural language). Apatero guide confirmed ("Mistral-3 text encoder" language present). Wang et al. confirmed at arXiv:2506.08210 (title and T5 outperformance claims match).

Scope boundary: This finding applies specifically to models using decoder-only LLM text encoders. CLIP-based models (SDXL, Pony) may still benefit from tag-style captions matching their contrastive training data.

VF-3: Attribute Ordering Matters in Causal-Attention Encoders

Claim: In a decoder-only LLM text encoder with causal attention, word order determines attention weight distribution. Earlier tokens receive more attention from all subsequent tokens. Placing the trigger token at position 0 maximizes its attention weight across the full sequence.

Evidence: BFL prompting guide: "word order matters -- FLUX.2 pays more attention to what comes first." This is a direct consequence of causal (left-to-right) attention: token at position 0 is in the attention window of every subsequent token, while the last token is only in its own window.

Verification: BFL prompting guide confirmed via WebFetch. The causal attention mechanism is a fundamental property of decoder-only transformer architectures and does not require model-specific verification.

Generalization: This finding transfers to any diffusion model using a decoder-only LLM text encoder with causal attention. It does not apply to encoder-only (CLIP) or encoder-decoder (T5) architectures where bidirectional attention distributes weight more evenly.

VF-4: Vocabulary Diversification Prevents Style Overfitting

Claim: If training captions use the same sentence structure, length, and vocabulary repeatedly, the model overtrains on the caption style before learning the subject, causing concept bleeding between subjects.

Evidence: SimpleTuner Discussion #634 states: "If you constantly use the same sequence, length and style of sentences, it is very likely that Flux overtrains on that before it learns your subjects." The discussion recommends switching between "vivid and professional descriptions" and using caption complexity proportional to image complexity.

Verification: SimpleTuner Discussion #634 confirmed via WebSearch and GitHub API. The quoted warning appears in a community member's comment within the discussion (not from the SimpleTuner maintainer). The discussion's primary topic is text encoder training for FLUX LoRAs, not captioning strategy; the vocabulary diversification advice appears as incidental guidance within a comment about why trigger words cause concept bleeding. [CITATION-DEPTH: source is a community comment in a tangential discussion (#634 is titled "FLUX LoRA Training: text encoder training?"), not a dedicated captioning resource. The quoted text is accurate but the discussion topic was mischaracterized as being about vocabulary diversification. Clarified.]

Project-internal corroboration: The Sandy v1 training dataset had "gentle shadows" in 90% of captions, "soft warm lighting" in 49%, "loose flowing hair" in 44%. This extreme phrase repetition was identified as a contributing factor to weak prompt responsiveness.

VF-5: BPE Tokenizers Fragment Mixed-Alphanumeric Strings at Digit Boundaries

Claim: The digit 4 in triggers like s4ndy and n4omi is always a separate BPE token. BPE tokenizers never merge digits with adjacent letters because digit-letter bigrams are extremely rare in pre-training corpora. This causes s4ndy to fragment into 4 tokens and n4omi into 3 tokens on both Mistral (Tekken, 131K vocab) and Qwen (byte-level BPE, 151K vocab) tokenizers.

Evidence: The legacy project ran empirical tokenizer experiments using HuggingFace transformers library with add_special_tokens=False. Results were consistent across Mistral-Small-3.1-24B-Instruct, Qwen3-8B, and Qwen3-4B. This is a fundamental property of BPE tokenization: merge operations are learned from training corpus statistics, and digit-letter transitions are statistically rare, so they are never merged.

Verification: BPE tokenization behavior with digits is well-documented in NLP literature. The specific tokenization of s4ndy and n4omi was verified empirically in the legacy project. The general principle (digits fragment mixed-alphanumeric tokens) is a known BPE property that can be verified with any BPE tokenizer.

VF-6: DOP Performs Trigger-to-Class Substitution via String Replace Before Tokenization

Claim: In ai-toolkit, Differential Output Preservation (DOP) replaces the trigger word with the class word using Python str.replace() on the raw caption text BEFORE tokenization. This means: (a) if the trigger is N tokens and the class word is M tokens, the tokenized sequence length changes by N-M tokens, shifting all subsequent positions; (b) a single-token trigger matching the class word's token count eliminates positional shift entirely.

Evidence: ai-toolkit source code SDTrainer.py line ~1628:

dop_prompts = [p.replace(self.trigger_word,
    self.train_config.diff_output_preservation_class)
    for p in conditioned_prompts]

Verification: DeepWiki documentation for ai-toolkit confirmed DOP's three-pass mechanism (prior prediction with class word, training prediction with trigger, preservation prediction with class word). The str.replace() mechanism is confirmed in multiple community guides. Ostris (ai-toolkit creator) stated on X: "DOP and careful captioning are what keep the LoRA 'tied' to a trigger phrase so that non-trigger prompts stay close to the base model."

VF-7: FLUX.2 Text Encoder Uses Multi-Layer Extraction (Layers 10, 20, 30)

Claim: FLUX.2 dev's Mistral text encoder extracts hidden states from layers 10, 20, and 30, concatenates them, producing output shape (batch_size, seq_len, 3 * hidden_dim). Maximum sequence length is 512 tokens.

Evidence: FLUX.2 source code text_encoder.py:

OUTPUT_LAYERS_MISTRAL = [10, 20, 30]
MAX_LENGTH = 512

Verification: DeepWiki page for FLUX.2 text encoders confirmed via WebFetch: multi-layer extraction from layers [10, 20, 30], MAX_LENGTH = 512, output shape (batch_size, sequence_length, 3 * hidden_dim).

VF-8: The aboutme.be Experiment Is SDXL-Only, Not Transferable to FLUX.2

Claim: The Verweirder (2023) experiment that found "only v002 (detailed captions on BOTH training AND reg) prevented concept bleed" was conducted on SDXL with CLIP dual encoders, not FLUX.2/Mistral. The experimental results do not transfer directly due to fundamental architectural differences: CLIP bag-of-tokens vs Mistral causal attention, 77-token vs 512-token capacity, single-layer vs 3-layer extraction.

Evidence: The aboutme.be blog post was confirmed via WebFetch as using "SDXL 1.0 with the baked 0.9 VAE." The experiment's findings about v002 preventing concept bleed were confirmed. The SDXL/CLIP architectural differences from FLUX.2/Mistral are well-established.

Verification: Blog post confirmed live at cited URL. The correction notices in the legacy documents themselves flag this limitation, showing internal quality control was applied.

Transferability note: The directional finding (format matching helps, detailed reg captions prevent bleed) may transfer to FLUX.2 as a reasonable heuristic, but the specific experimental evidence applies only to SDXL/CLIP.

VF-9: "sks" Has Problematic Semantic Loading (Weapons Brand)

Claim: The classic DreamBooth trigger sks carries semantic meaning (associated with a weapons brand/rifle), which can affect generation results. HuggingFace's advanced training guide warns about this.

Evidence: HuggingFace blog sdxl_lora_advanced_script.md: "those tokens usually have other semantic meaning associated with them and can affect your results. The sks example, popular in the community, is actually associated with a weapons brand." GitHub issue #71 on Dreambooth-Stable-Diffusion confirms users observed guns appearing in generations.

Verification: HuggingFace blog confirmed via WebSearch. GitHub issue confirmed to exist at cited URL. The general principle (pre-existing token semantics affect fine-tuning) is well-established in the fine-tuning literature.

VF-10: DreamBooth Established Prior Preservation Loss

Claim: DreamBooth (Ruiz et al., 2022) established "class-specific prior preservation loss" for subject-driven fine-tuning, training on subject images with "[V] [class noun]" captions while simultaneously training on model-generated class images to prevent language drift.

Evidence: DreamBooth paper arXiv:2208.12242.

Verification: Paper confirmed at cited arXiv ID via WebFetch. Authors confirmed as Ruiz et al. The "autogenous class-specific prior preservation loss" is described in the abstract.

VF-11: DisenBooth Formalized the Entanglement Problem

Claim: DisenBooth (Chen et al., 2023) formalized that "identity-relevant information and identity-irrelevant information are entangled in the latent embedding space," leading to generated images "heavily dependent on the irrelevant information."

Evidence: DisenBooth paper arXiv:2305.03374.

Verification: Paper confirmed at cited arXiv ID via WebFetch. Title "DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation" and entanglement claims confirmed.

VF-12: DECOR Validated Orthogonal Projection for Embedding Disentanglement

Claim: DECOR (Jang et al., 2024) proposed "projecting embeddings onto a vector space orthogonal to undesired token vectors" to address overfitting and content leakage during LoRA fine-tuning.

Evidence: DECOR paper arXiv:2412.09169.

Verification: Paper confirmed at cited arXiv ID via WebFetch. Title "DECOR: Decomposition and Projection of Text Embeddings for Text-to-Image Customization" and orthogonal projection claims confirmed.

VF-13: T-LoRA Found Higher Timesteps More Prone to Overfitting

Claim: T-LoRA (Soboleva et al., 2025) found that "higher diffusion timesteps are more prone to overfitting than lower ones," necessitating timestep-sensitive fine-tuning.

Evidence: T-LoRA paper arXiv:2507.05964.

Verification: Paper confirmed at cited arXiv ID via WebFetch. Authors confirmed as Soboleva et al. Timestep overfitting claim confirmed in abstract.

Relevance to captioning: Body proportions are learned at high-noise timesteps (coarse structural features), making body LoRAs more susceptible to overfitting. This connects to the observation that body LoRAs are more forgiving of imperfect captions than face LoRAs (body proportions occupy a simpler geometric manifold than facial identity).

VF-14: RECAP Demonstrated Caption Quality Impact on Training

Claim: RECAP (Segalis et al., 2023) demonstrated that recaptioning training data with higher-quality captions dramatically improved image generation: "FID 14.84 vs. the baseline of 17.87."

Evidence: RECAP paper arXiv:2310.16656.

Verification: Paper confirmed at cited arXiv ID via WebFetch. FID improvement figures confirmed. RECAP is a fine-tuning / continued training study (fine-tuned SD v1.4 for 250k additional steps on recaptioned data), not a pre-training study. [CORRECTED: paper explicitly states fine-tuning, not pre-training. See agent-003-verdicts.md, cite-011.]

VF-15: Concept Sliders Demonstrated Low-Rank Direction Identification

Claim: Concept Sliders (Gandikota et al., 2023) demonstrated that "low-rank parameter directions corresponding to one concept" can be identified "while minimizing interference with other attributes."

Evidence: Concept Sliders paper arXiv:2311.12092.

Verification: Paper confirmed at cited arXiv ID via WebFetch. Title "Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models" and low-rank direction identification confirmed. Note: the claim of "up to 50 simultaneous adapters" was not confirmed in the abstract; the paper describes them as "plug-and-play" and "composed efficiently" without specifying a number. The legacy documents already noted this was corrected to "up to 50."

VF-16: TARA Found LoRA Modules Ignore Rare Tokens in Favor of BOS

Claim: TARA (Peng et al., 2025, AAAI 2026) found token-wise interference among LoRA modules and proposed Token Focus Masking to constrain each module to focus on its associated rare token.

Evidence: TARA paper arXiv:2508.08812.

Verification: Paper confirmed at cited arXiv ID via WebFetch and WebSearch. GitHub repo confirmed at YuqiPeng77/TARA. Token Focus Masking and token-interference claims confirmed. The specific claim about "ignoring rare tokens in favor of BOS" aligns with the paper's description of the interference problem, though the exact "BOS" phrasing may be the legacy document's interpretation rather than the paper's exact wording. [CITATION-DEPTH: AAAI 2026 acceptance was NOT confirmed in the arXiv abstract. The paper was submitted to arXiv on 2025-08-12 with no venue mentioned. The previous harvest's claim of "AAAI 2026 acceptance confirmed" could not be re-verified from the arXiv page alone. Core technical claims are confirmed.]

VF-17: CLIPScore as Caption-Image Alignment Metric

Claim: CLIPScore (Hessel et al., EMNLP 2021) measures caption-image alignment using cosine similarity between CLIP image and text embeddings.

Evidence: CLIPScore paper arXiv:2104.08718.

Verification: Paper confirmed at cited arXiv ID via WebFetch. Title "CLIPScore: A Reference-free Evaluation Metric for Image Captioning" confirmed. The cosine similarity mechanism is the standard CLIP similarity computation.

Important caveat: CLIP's 77-token window means CLIPScore truncates long captions. This is a scoring limitation, not a training caption length limit. The legacy documents correctly note this distinction.

VF-18: Caption Dropout Requires Non-Cached Text Embeddings

Claim: Caption dropout (replacing captions with empty strings during a fraction of training steps) only works when cache_text_embeddings: false. If text embeddings are cached, they are pre-computed at startup and dropout cannot modify them at training time.

Evidence: ai-toolkit source code toolkit/dataloader_mixins.py line 408. The legacy documents note this was a confirmed bug in Sandy v1 training where cache_text_embeddings: true silently disabled caption dropout.

Verification: The mechanism is architecturally straightforward: cached embeddings are computed once, so runtime dropout cannot intervene. The specific ai-toolkit code path was verified in the legacy project's source code analysis.

VF-19: "A Woman" Rule for Body LoRA Face Descriptions

Claim: When training a body LoRA that will be stacked with a separate face LoRA at inference, body LoRA captions must describe the subject's face as only "a woman" (no qualifiers). If body captions include facial feature descriptions, the body LoRA learns to generate those faces, competing with the face LoRA during stacking.

Evidence: This is a logical consequence of the core "caption what you don't want to train" principle applied to multi-LoRA stacking. If the body LoRA learns facial features from captions, those features become text-conditioned (prompt-controllable) in the body LoRA rather than deferred to the face LoRA.

Verification: The principle follows directly from VF-1. No independent experimental validation of this specific multi-LoRA stacking interaction exists, but the theoretical reasoning is sound given the established captioning principle. The legacy project treated this as a design rule derived from first principles.

VF-20: VLM Captions Require Full LLM Review (Not Optional Cleanup)

Claim: VLM-generated captions (e.g., from Qwen3-VL) are raw drafts, not finished training data. In the Naomi captioning run (277 images), 100% of VLM drafts required rewriting by LLM agents. Common failures: doubled trigger tokens, outfit misidentification, hair color leakage, implicit body descriptions, inaccurate framing terms.

Evidence: Project-internal empirical data from the Naomi captioning run. Specific failure rates documented: 100% double trigger, 61% under word minimum, 11% outfit misidentification, 18% hair color leakage, 5% implicit body descriptions, 7% forbidden body terms, 14% inaccurate framing.

Verification: These are project-internal empirical observations. The specific failure rates cannot be verified against external sources, but the general finding that VLMs produce imperfect captions requiring human/LLM review is widely reported in the community. The recommendation of a VLM-draft

  • LLM-review pipeline is a practical workflow finding.

VF-21: Regularization Captions Must Never Contain Trigger Words

Claim: Regularization image captions must never contain the trigger word. Reg images teach the model what the base concept looks like WITHOUT the specific identity. Including the trigger in reg captions defeats regularization.

Evidence: This follows directly from the DreamBooth prior preservation mechanism (VF-10). The aboutme.be experiment (SDXL-only but directionally relevant) confirmed this: v002 used reg images generated from training captions minus the trigger word.

Verification: aboutme.be blog confirmed via WebFetch. The principle is architecturally self-evident: if reg captions contain the trigger, the model cannot learn to distinguish triggered from untriggered generations.

VF-22: Detailed Reg Captions Outperform Generic "A Woman" Captions

Claim: Reg images captioned with detailed descriptions matching the training caption format prevent concept bleed. Generic captions ("a woman") or empty captions allow LoRA identity features to leak into non-triggered generations.

Evidence: aboutme.be experiment found that only v002 (detailed captions on both training and reg) prevented concept bleed. This was confirmed via WebFetch as the experiment's central finding.

Transferability caveat: This finding was experimentally validated on SDXL/CLIP only. Transfer to decoder-only LLM encoders is a reasonable inference (format matching reduces distributional gap between training and reg data regardless of encoder architecture) but is not experimentally proven.

VF-23: Trigger Token Brute-Force Scan Methodology

Claim: Of 17,576 possible three-letter lowercase strings (26^3), 1,983 are single tokens on both Mistral (Tekken, 131K vocab) and Qwen (byte-level BPE, 151K vocab) tokenizers. Candidates were filtered for pronounceability, memorability, low semantic loading, and no collision with domain vocabulary.

Evidence: Empirical scan documented in the trigger token research. The methodology is reproducible: load both tokenizers from HuggingFace Hub, iterate all aaa through zzz strings, filter for single-token on both.

Verification: The methodology is sound and reproducible. The specific count (1,983) and candidate list (zek, kov, zik, nak, zar, nox, rez) are empirical results from the scan. The methodology transfers to any future model with a BPE tokenizer.

VF-24: Mistral Negation Processing Is Unreliable for Image Conditioning

Claim: Captions should never use negation ("no X", "without X", "not X") because decoder-only LLMs like Mistral were not contrastively trained and handle negation unreliably for image conditioning purposes.

Evidence: BFL prompting guide implicitly discourages negation by recommending affirmative descriptions. The legacy project's internal testing (referenced in research/2026-04-22-03-flux2-realism-prompting.md) confirmed Mistral's weak negation performance. CLIP handles negation through contrastive training; Mistral lacks this mechanism.

Verification: The architectural difference (contrastive vs instruction-tuned training) is well-established. Specific negation failure examples from the legacy project are internal empirical data. The BFL guide's avoidance of negation in all examples is circumstantial but consistent evidence.

VF-25: Caption Dropout Rate Interacts with Caption Length

Claim: When captions are very detailed (100+ words), a dropout step creates a large distribution shift (rich conditioning to zero conditioning), producing stronger gradient signals and potential instability. Shorter captions produce gentler dropout transitions.

Evidence: This is a theoretical inference from the mechanism of caption dropout (replacing the full caption with an empty string). The distribution shift magnitude is proportional to the information content of the dropped caption.

Verification: The mechanism is architecturally self-evident. No empirical study has measured the interaction between caption length and dropout gradient magnitude. The legacy project's recommendation of 0.05-0.10 for detailed captions and 0.10-0.15 for shorter captions is a reasonable heuristic but not experimentally calibrated.

VF-26: DOP Signal Strength Depends on Trigger Token Fraction

Claim: The DOP preservation loss operates on the embedding difference between trigger-caption and class-caption. In long captions, the trigger token represents a small fraction of the embedding (~1% in a 100-word caption), potentially weakening the DOP signal. Positional attention partially mitigates this for causal-attention encoders (earlier tokens get more weight).

Evidence: Theoretical inference from DOP mechanism (VF-6) combined with causal attention properties (VF-3). The interaction between token fraction and positional attention is unstudied.

Verification: The mechanism is sound in principle. The specific concern (signal dilution in long captions) is acknowledged in the legacy documents as theoretical with no empirical validation. The positional attention mitigation is a valid counterargument.

VF-27: Wang et al. (2025) Found Layer-Averaged Embeddings Outperform Last-Layer

Claim: Wang et al. (2025, arXiv:2506.08210) found that "the de facto way of using last-layer embeddings as conditioning leads to inferior performance" for decoder-only LLMs as text encoders, and that layer-normalized averaging significantly improves alignment with complex prompts.

Evidence: Paper arXiv:2506.08210.

Verification: Confirmed via WebFetch. The finding supports FLUX.2's multi-layer extraction design (layers 10, 20, 30) as architecturally motivated.

VF-28: BFL Recommends 30-80 Words for Inference Prompts

Claim: BFL's prompting guide recommends short (10-30 words), medium (30-80 words, "usually ideal for most projects"), and long (80+ words) prompt categories.

Evidence: BFL prompting guide.

Verification: Confirmed via WebFetch. Note: this is inference prompt guidance, not training caption guidance. BFL has published no training-specific captioning recommendations.


Unverified Claims

UV-1: CivitAI "Captions vs No-Captions" Article Findings

Claim: CivitAI Article 7203 "reportedly found captioning beneficial for FLUX LoRA quality."

Status: [UNVERIFIED: citation behind login wall]. The article at civitai.com/articles/7203 requires authentication. Content could not be verified via WebFetch. The legacy documents themselves note it was "behind login wall at time of research."

UV-2: CivitAI "Understanding Prompting and Captioning" Article

Claim: CivitAI Article 8487 supports the "caption what you don't want to train" principle.

Status: [UNVERIFIED: citation behind login wall]. The article at civitai.com/articles/8487 requires authentication. Content could not be verified.

UV-3: Specific VLM Failure Rates from Naomi Captioning Run

Claim: Specific failure rates (100% double trigger, 61% under word minimum, 11% outfit misidentification, etc.) from the 277-image Naomi captioning run.

Status: [UNVERIFIED: no upstream source found]. These are project-internal empirical observations that cannot be independently verified. The general finding (VLMs produce imperfect captions) is corroborated by community experience, but the specific percentages are project-specific data.

UV-4: 5% Phrase Repetition Threshold

Claim: No phrase of 4+ words should appear in more than 5% of captions.

Status: [UNVERIFIED: no upstream source found]. The legacy documents themselves flag this: "No threshold exists in any official documentation. Originated from a single community anecdote in SimpleTuner Discussion #634." The body-descriptor-evidence-assessment document explicitly lists this as "DISREGARD."

UV-5: Concept Sliders Composing "Up to 50" Simultaneous Adapters

Claim: Concept Sliders "successfully composed up to 50 simultaneous adapters."

Status: [UNVERIFIED: no upstream source found]. The Concept Sliders paper abstract describes adapters as "plug-and-play" and "composed efficiently" but does not specify a number. The legacy documents already corrected this from "50+" to "up to 50" but the specific number remains unverified in the abstract. The full paper may contain this claim but it was not confirmed.


Suspect Claims

SC-1: Fixed Body Descriptor Phrase Prevents Quality Fork

Claim: Using a verbatim fixed body descriptor phrase ("with very large pendulous breasts, a narrow waist, and wide hips" or later "with massive breasts") in every body LoRA caption prevents the quality fork where loose clothing degrades while tight clothing remains acceptable.

Status: [SUSPECT: no evidence basis for the causal mechanism]. The legacy documents themselves rate this as "MEDIUM -- the hybrid approach is theoretically justified but experimentally unvalidated on FLUX.2." The body descriptor evidence assessment document further classifies it: "HIGH that variation failed. MEDIUM that fixing variation will solve the problem." The Naomi v1 quality fork could have been caused by other factors (rank/LR interaction, content_or_style setting, insufficient reg data).

Note: This deviates from the core "caption what you don't want to train" principle (VF-1). The deviation was a deliberate design choice motivated by a practical failure, not theoretical preference.

SC-2: Trigger Token Fraction Meaningfully Affects DOP Effectiveness

Claim: In very long captions (~180 words), the trigger token represents ~0.6% of the embedding, potentially too diluted for effective DOP constraint.

Status: [SUSPECT: no evidence basis]. The legacy documents themselves acknowledge: "The denoiser receives a time-step-conditioned input and produces a noise prediction. Whether a 3-token positional shift in text conditioning meaningfully affects the noise prediction comparison is unknown." The DOP comparison operates at the noise prediction level, not the text embedding level, so token fraction in the embedding may be irrelevant.


Conflicts

CONFLICT-1: Body Proportions in Captions -- Omit vs Fixed Phrase

Legacy position (earlier docs): Omit body proportions from body LoRA captions entirely, letting the LoRA learn them from pixels alone. This follows the pure "caption what you don't want to train" principle.

Legacy position (later docs): Include a fixed body descriptor phrase in every caption. This is a deliberate deviation from the pure principle, motivated by the Naomi v1 quality fork.

ai-foundry current state (AGENTS.md): The trigger tokens are viv (Sandy) and lux (Naomi), different from both the original (s4ndy/n4omi) and recommended replacements (zek/kov). This suggests the triggers were changed but to different values than the research recommended.

CONFLICT-2: Trigger Token Identity

Legacy research recommends: zek (Sandy) and kov (Naomi) as optimal single-token triggers based on brute-force tokenizer scan.

ai-foundry AGENTS.md states: Triggers are viv (Sandy) and lux (Naomi).

Resolution: Austin approved a different trigger change (to viv/lux) after the research was conducted. The captioning-strategy-synthesis document itself notes: "Austin approved the trigger change from s4ndy/n4omi on 2026-04-28." The final choice of viv/lux over zek/kov represents a decision made after the research, not a contradiction of it. Both viv and lux are likely single tokens on the target encoders (common 3-letter English words), satisfying the core recommendation even if different candidates were chosen.

CONFLICT-3: Naomi Trigger Format (Period vs Comma Separator)

Legacy captioning strategy: Uses n4omi. (period separator) for Naomi and s4ndy, (comma separator) for Sandy. The different separators distinguish the two triggers in stacked-LoRA prompts.

Legacy synthesis document: Shows kov. (period) for Naomi and zek, (comma) for Sandy, preserving the convention.

ai-foundry captioning strategy: Uses lux for Naomi. The separator convention with the current triggers is not specified in AGENTS.md.


Sources Consulted

Academic Papers (all verified at cited arXiv IDs)

Paper arXiv Verification
DreamBooth (Ruiz et al., 2022) 2208.12242 Confirmed
DisenBooth (Chen et al., 2023) 2305.03374 Confirmed
Concept Sliders (Gandikota et al., 2023) 2311.12092 Confirmed (core claim; "50 adapters" not in abstract)
RECAP (Segalis et al., 2023) 2310.16656 Confirmed (fine-tuning study: fine-tuned SD v1.4 for 250k steps)
CLIPScore (Hessel et al., 2021) 2104.08718 Confirmed
DECOR (Jang et al., 2024) 2412.09169 Confirmed
T-LoRA (Soboleva et al., 2025) 2507.05964 Confirmed
Wang et al. (2025) 2506.08210 Confirmed
TARA (Peng et al., 2025) 2508.08812 Confirmed (core claims; AAAI 2026 not confirmed in arXiv abstract)

Community Sources (verified via WebFetch/WebSearch)

Source URL Status
BFL Prompting Guide docs.bfl.ml/guides/prompting_guide_flux2 Live, confirmed
Pelayo Arbues pelayoarbues.com/literature-notes/Articles/... Live, confirmed
aboutme.be experiment blog.aboutme.be/2023/08/10/... Live, confirmed (SDXL-only)
fal.ai FLUX.2 training blog.fal.ai/training-flux-2-loras/ Live, confirmed
Apatero FLUX.2 guide apatero.com/blog/flux-2-lora-training-... Live, confirmed
alvdansen HF blog huggingface.co/blog/alvdansen/... Live, confirmed
SimpleTuner Discussion #634 github.com/bghira/SimpleTuner/discussions/634 Live, confirmed
DeepWiki FLUX.2 text encoders deepwiki.com/black-forest-labs/flux2/... Live, confirmed
HuggingFace "sks" warning huggingface.co/blog/sdxl_lora_advanced_script Live, confirmed
CivitAI Article 7203 civitai.com/articles/7203 Behind login wall
CivitAI Article 8487 civitai.com/articles/8487 Behind login wall

Legacy Source Files (7 files read)

File Domain Key contribution
captioning-strategy.md Captioning rules Core principle, include/exclude lists, VLM review pipeline
2026-04-08-01-image-to-prompt.md Tool survey VLM/captioner ecosystem comparison (42 tools)
2026-04-26-flux2-body-lora-captioning.md Body LoRA captions Field-tested captioning methodology, VLM failure patterns
2026-04-28-captioning-strategy-synthesis.md Caption specs Sandy + Naomi format specs, vocabulary pools, reg captions
2026-04-28-flux2-captioning-methodology.md Research synthesis 10-section methodology covering all 8 captioning dimensions
2026-04-28-trigger-token-design.md Trigger tokens Tokenizer analysis, DOP interaction, brute-force scan
2026-05-03-body-descriptor-evidence-assessment.md Evidence audit Tier classification of all captioning claims

Citation Verification Footer

Verified 2026-05-18 by 36 Opus 4.6 agents. Full verdicts in verification/agent-*-verdicts.md.

Cite ID Source Verdict Key Finding
cite-005 arXiv:2104.08718 CONFIRMED CLIPScore reference-free metric, cosine similarity, EMNLP 2021
cite-009 arXiv:2305.03374 PARTIAL Paper addresses entanglement but "formalizes" overstates rigor
cite-011 arXiv:2310.16656 SUPPORTED* FID figures confirmed; "pre-training" label wrong (fine-tuning study)
cite-016 arXiv:2412.09169 CONFIRMED Title, orthogonal projection, embedding disentanglement match
cite-019 arXiv:2506.08210 CONFIRMED Decoder-only LLMs outperform T5, layer-averaged better, CVPR 2025
cite-022 arXiv:2508.08812 CONFIRMED Token interference, Token Focus Masking, BOS findings all verified

6 citations: 4 CONFIRMED, 1 SUPPORTED*, 1 PARTIAL

Citation Deep Verification Report

Date: 2026-05-18 Run duration: ~21 minutes (05:54 to 06:15 UTC) Citations verified: 179 Agents dispatched: 36 (5 citations each, last agent had 4) Max concurrent: 8 Model: Opus 4.6 Spot-check: 10/10 PASS (no fabricated quotes, no shallow briefs)

Verdict Summary

Verdict Count %
CONFIRMED / VERIFIED / SUPPORTED ~140 ~78%
PARTIAL (overstatement, nuance gap, or misframing) ~18 ~10%
WRONG SOURCE (correct claim, wrong URL) ~12 ~7%
UNREACHABLE (login wall, HTTP 451, SPA) ~5 ~3%
UNSUPPORTED / NOT VERIFIED ~4 ~2%

Key Corrections Found

Factual Errors

  1. Wan2.2-S2V-14B runs at 24fps, not 16fps — model card says it explicitly
  2. T-LoRA / TARA technique conflation — Token Focus Masking belongs to TARA, not T-LoRA (T-LoRA uses timestep-dependent rank masking)
  3. RECAP is a fine-tuning study, not pre-training — paper fine-tuned SD v1.4 for 250k additional steps
  4. rsLoRA "nearly doubles at rank 256" — rank 256 was never tested; improvements are modest (~1.84 vs ~1.86 perplexity)
  5. Z-Image has 30 attention heads, not 32 — model weights show 30 (arXiv paper says 32, weights are authoritative)
  6. kohya-ss issue #1492 inverted — harvest said "loss barely moved" but actual issue reports loss remained HIGH
  7. ai-toolkit issue #751 has TWO bugs — both key naming and missing code path are real independent issues

MPS bf16 Correction

  1. Apple Silicon HAS bf16 since PyTorch ~2.6 — the blanket "no bf16 on MPS" is only correct for Intel/AMD Macs. PyTorch #139386 added bf16 autocast for Apple Silicon (closed as completed).

Source Misattributions (Right Claim, Wrong URL)

  1. Camera body rendering characters — Canon EOS R5, Sony A7R V are NOT in any cited source. BFL guide lists Canon 5D Mark IV, Sony A7IV. The specific rendering characterizations are harvest author interpolation.
  2. Comfy blog cited for Klein architecture — blog is a product announcement, not architecture spec. Block counts come from SimpleTuner FLUX2.md.
  3. DeepWiki page reorganization — several DeepWiki URLs no longer contain the content they originally held
  4. Wan 2.2 bf16 not in ComfyUI docs — neither the examples page nor the tutorial mentions bf16
  5. NVIDIA Blackwell Compatibility Guide — only covers sm_100, not sm_120. The forums thread is the actual source.
  6. BFL Klein training docs — covers LoRA params but not guidance embeddings. That comes from SimpleTuner.

Staleness / Outdated Claims

  1. MegaTTS3 voice cloning now viable — community released WaveVAE encoder at drbaph/MegaTTS3-WaveVAE
  2. VibeVoice TTS code has been restored — removal was temporary; repo is actively developed through May 2026
  3. ComfyUI org transferredcomfyanonymous/ComfyUIComfy-Org/ComfyUI (old URLs redirect)

Editorial / Unsourced Claims

  1. "5-10x longer consistency" — found in ONE source (zimage.net) out of 4 co-cited URLs. Other sources use qualitative language or "70-80% effectiveness."
  2. Five-element realism taxonomy — harvest author synthesis, not extracted from any cited source
  3. Bonnaire "Best Paper" → actually NeurIPS 2025 Oral. The "4 of 21,575" statistic was fabricated.
  4. ConceptPrism citation fabricated for body descriptors — paper is about concept disentanglement, not body descriptors
  5. John Shi article covers FLUX.1 only — contextually misleading in a FLUX.2 section

Per-Document Breakdown

Document Citations Confirmed Partial Wrong Source Other
training-methodology.md 23 18 3 1 1
training-experiments.md 22 18 2 1 1
captioning-and-datasets.md 6 5 1 0 0
inference-techniques.md 17 12 3 2 0
model-architecture.md 38 28 4 4 2
evaluation-methodology.md 13 11 1 0 1
video-generation.md 28 23 2 1 2
hardware-and-operations.md 37 28 4 3 2

Failure Mode Taxonomy

The deep verification revealed four distinct failure categories:

  1. Fabrication (~2%): Claims with no basis in any cited source (Bonnaire "Best Paper" stat, ConceptPrism body descriptor attribution, camera rendering characters)
  2. Wrong Source (~7%): Technically correct knowledge attributed to the wrong URL. Evidence exists elsewhere but the citation chain is broken.
  3. Overstatement (~10%): Source partially supports the claim but the harvest overstates, broadens scope, or adds editorial quantification without marking it as such.
  4. Staleness (~2%): Claims accurate at authoring time but overtaken by events (MegaTTS3, VibeVoice, ComfyUI org transfer).

Spot-Check Results

10 random CONFIRMED verdicts were re-verified by a meta-verification agent:

  • 10/10 quotes exist in the cited sources
  • 10/10 Source Briefs contain genuine body-only details
  • 10/10 Noteworthy Context sections add analytical value
  • 0 agents flagged for systematic concerns

Evidence Inventory

.folio/research/legacy-harvest/verification/
  agent-001-verdicts.md through agent-036-verdicts.md (36 files, 9,149 lines)
  spot-check-report.md
  pre-footer-backup/ (8 harvest doc copies)

Methodology Assessment

The 5-citation-per-agent cap with mandatory Source Briefs was effective:

  • No agent shortcuts detected in spot-check
  • Every agent fetched every source (confirmed by tool-use counts in summaries)
  • Source Briefs consistently demonstrated body-level reading depth
  • The rolling pool eliminated idle time between batch waves
  • 21 minutes wall clock for 179 citations across 36 agent dispatches
title Evaluation Methodology Research Harvest
source_project claude-comfyui
harvest_date 2026-05-17
domain evaluation
verification_level full-academic
claims_total 79
claims_verified 64
claims_unverified 4
claims_suspect 11
conflicts_found 0

Evaluation Methodology Research Harvest

Distilled from 43 legacy source files in /Users/austin/Developer/claude-comfyui/. Focus: transferable evaluation principles, not individual run results.


1. Why Training Loss Is Not a Quality Signal

Verified Core Claim

Training loss (MSE) in diffusion LoRA training is structurally blind to sample quality collapse. This was demonstrated empirically in the Sandy v1 training run (rank 128, LR 2e-4, 82 images): MSE loss at the collapse checkpoint (step 550, loss 0.45) was indistinguishable from the peak quality checkpoint (step 330, loss 0.44).

  • Model scope: model-agnostic (structural argument), validated on FLUX.2 dev LoRA
  • Source: research/2026-04-28-checkpoint-selection-methodology.md, Section 1

Three Compounding Problems

  1. Timestep averaging hides localized failure. Aggregate loss treats all timesteps equally. Degradation at medium timesteps (which control image structure) is diluted by stable loss at extremes.

    • Citation: Dodson et al., "Two Calm Ends and the Wild Middle," arXiv:2602.17846, February 2026. VERIFIED: paper exists, confirms memorization concentrates at medium noise levels while high-noise and low-noise "calm ends" resist it.
    • Confidence: HIGH
  2. The optimal loss floor is non-zero and unknowable in practice. A loss value cannot be interpreted without computing the minimum achievable loss J* for the specific dataset and timestep.

    • Citation: Xu et al., "Diagnosing and Improving Diffusion Models by Estimating the Optimal Loss Value," arXiv:2506.13763, ICLR 2026. VERIFIED: paper exists, confirms optimal loss is non-zero and derives closed-form estimators. Published as ICLR 2026 camera-ready (submitted June 2025, revised April 2026).
    • Confidence: HIGH
  3. MSE-optimal is not perceptually optimal. The Perception-Distortion Tradeoff proves minimizing MSE and maximizing perceptual quality are fundamentally at odds.

    • Citation: Blau & Michaeli, arXiv:1711.06077, CVPR 2018. VERIFIED: paper proves distortion and perceptual quality are at odds for any distortion measure, not just MSE/PSNR. Presented as long oral at CVPR 2018 (arXiv submitted November 2017).
    • Confidence: HIGH

Timestep-Dependent Overfitting

T-LoRA provides direct evidence that LoRA overfitting is non-uniform across timesteps. Higher (noisier) diffusion timesteps are more vulnerable.

  • Citation: Soboleva et al., "T-LoRA," arXiv:2507.05964, AAAI 2026. VERIFIED: paper exists, accepted at AAAI 2026, tested on SDXL (original paper). FLUX.1-dev support added in GitHub repo's February 2026 update, not the original AAAI submission. T-LoRA's technique is timestep-dependent rank masking (not Token Focus Masking, which belongs to TARA, arXiv:2508.08812). [CORRECTED: FLUX.1-dev scope clarified; technique attribution disambiguated from TARA. See agent-005-verdicts.md, cite-021.]
  • Confidence: HIGH for LoRA specifically

Memorization Timescales

Bonnaire et al. identify two distinct training timescales:

  • tau_gen (quality emergence): constant regardless of dataset size
  • tau_mem (memorization onset): scales linearly with dataset size n

For small datasets (Sandy, n=82), tau_mem is proportionally small. For larger datasets (Naomi, n=277), tau_mem is approximately 3-4x larger.

  • Citation: Bonnaire et al., "Why Diffusion Models Don't Memorize," arXiv:2505.17638, NeurIPS 2025. VERIFIED: paper exists, accepted as oral at NeurIPS 2025. [CITATION-DEPTH: source says "accepted as an oral at NeurIPS 2025," harvest claimed "NeurIPS 2025 Best Paper (4 of 21,575 submissions)" -- corrected. Oral acceptance is prestigious but distinct from Best Paper.]
  • Confidence: HIGH for the timescale finding; MEDIUM for direct applicability to LoRA fine-tuning. [CITATION-DEPTH: harvest claimed "studies pre-training from scratch on 32x32 CelebA" but paper abstract says "numerical experiments with standard U-Net architectures on realistic and synthetic datasets" without specifying 32x32 CelebA -- corrected to remove unverified dataset detail.]
  • SUSPECT: The legacy documents extend Bonnaire with a lrrank scaling factor that does not appear in the paper. This extension was caught by the project's own adversarial review. Do not propagate the lrrank factor as source-backed.

Practical Consequence

Loss is useful for exactly two things:

  1. Detecting NaN/infinity (complete divergence)
  2. Detecting sudden 10x+ jumps (catastrophic forgetting)

For everything else -- gradual quality degradation, identity drift, concept bleed -- sample images evaluated by human review and optional automated metrics are the only reliable signal.


2. Multi-Metric Evaluation Framework

2.1 JPEG Compression File Size Trajectory

  • What: Image complexity proxy via JPEG compression at fixed quality (95). JPEG is more efficient on low-detail/uniform images, so collapsing outputs produce smaller files.
  • Model scope: model-agnostic
  • Confidence: MEDIUM (validated empirically by Sandy v1 collapse only; no published validation exists)
  • Thresholds (uncalibrated): Healthy 80-120% of baseline; Warning sustained below 80% for 2+ checkpoints; Critical below 60%.
  • Role: Tripwire/early warning indicator, NOT a quality metric. Weight: 0.10
  • Source: research/2026-04-28-checkpoint-selection-methodology.md, Section 2.1

2.2 ArcFace Identity Similarity (Face-Specific)

  • What: 512-dimensional face embedding cosine similarity between generated faces and a reference centroid computed from 5 diverse training images.
  • Model scope: model-agnostic for the metric; face-specific application
  • Citation: Deng et al., "ArcFace: Additive Angular Margin Loss for Deep Face Recognition," CVPR 2019. Used in DreamBooth, PuLID, Arc2Face evaluation. VERIFIED via InsightFace documentation.
  • Confidence: HIGH that ArcFace is appropriate for face identity in diffusion outputs; MEDIUM on specific thresholds
  • Expected trajectory: Low 0.1-0.3 (early, LoRA not yet learned) -> plateau 0.5-0.75 (identity sweet spot) -> potential increase above 0.75 (memorization) or drop (collapse)
  • Limitations: Trained on real photos, may behave unpredictably on highly stylized outputs. Does not measure face quality, only identity match. No face detected = score 0 (itself a collapse signal).
  • SUSPECT threshold: The target range 0.45-0.70 and memorization warning at 0.75 are uncalibrated estimates from the legacy project.
  • Source: research/2026-04-28-checkpoint-selection-methodology.md, Section 2.2

2.3 DINO Self-Similarity (Identity Stability)

  • What: Self-supervised ViT-S/16 embeddings for holistic subject identity preservation (body shape, pose, clothing style, not just face).
  • Model scope: model-agnostic
  • Citation: Caron et al., "DINO," arXiv:2104.14294, 2021. Primary metric in DreamBooth (arXiv:2208.12242). VERIFIED: DreamBooth uses DINO as primary subject fidelity metric with Pearson correlation 0.32 vs human preference.
  • Confidence: HIGH
  • Role: Primary identity metric for body/proportion characters (e.g., Naomi); supplementary for face characters (e.g., Sandy, where ArcFace is primary).
  • Source: research/2026-04-28-checkpoint-selection-methodology.md, Section 2.3

2.4 CLIP-T (Text-Image Alignment)

  • What: CLIP cosine similarity between prompt text embedding and generated image embedding. Measures whether the model still follows prompts.
  • Model scope: model-agnostic
  • Citation: Standard metric from DreamBooth and the HuggingFace diffusers evaluation framework. VERIFIED.
  • Confidence: HIGH that CLIP-T is a standard metric; MEDIUM on specific threshold values
  • Limitation: CLIP was trained on web-scraped alt-text. It measures semantic alignment coarsely but may not capture nuanced prompt details (exact clothing color, specific pose). For LoRA evaluation specifically, a declining CLIP-T trajectory across checkpoints indicates the model is losing prompt-following ability as it overfits.
  • IMPORTANT AUDIT NOTE: The validation methodology audit (2026-05-03) found CLIPScore tautological when images are generated from their own captions. Threshold of 25 is barely above random noise (24). Use CLIP-T for trajectory comparison across checkpoints, NOT as an absolute quality gate.
  • Source: research/2026-04-28-checkpoint-selection-methodology.md, Section 2.4; research/2026-05-03-validation-methodology-audit.md

2.5 LPIPS Inter-Sample Diversity

  • What: Perceptual distance between image pairs using deep features. Used as pairwise diversity measurement across same-checkpoint, different-prompt samples.
  • Model scope: model-agnostic
  • Citation: Zhang et al., "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric," arXiv:1801.03924, CVPR 2018. VERIFIED via GitHub repository.
  • Confidence: HIGH as a perceptual similarity metric; MEDIUM for its use as a diversity metric in LoRA checkpoint selection (reasonable but not specifically published for this purpose)
  • Role: Canary for memorization. Complete diversity loss is catastrophic; minor reduction is expected as the LoRA specializes.
  • Thresholds (uncalibrated): Warning below 70% of baseline for 2+ checkpoints; Critical below 50%.
  • Source: research/2026-04-28-checkpoint-selection-methodology.md, Section 2.5

2.6 Metrics Found INVALID by Project Audit

The validation methodology audit (2026-05-03) tested and rejected these metrics for LoRA evaluation:

Metric Why Invalid Audit Source
CLIPScore (absolute) Tautological when images generated from their own captions Hessel et al. 2021
PRDC (k=3, N=82) Curse of dimensionality; k-NN unreliable in 768-D with 82 samples Park & Kim 2023 (ICCV)
Vendi Score > 50 Threshold arbitrary; metric sound but needs domain calibration Friedman & Dieng 2022
DINOv2 centroid distance Captures pose/clothing, not body shape; threshold 0.3-0.5 baseless Meta DINOv2 paper
Absolute ArcFace thresholds Calibrated on real photos, not synthetic-to-real comparison FRCSyn Challenge CVPR 2024
t-SNE visualization No actionable signal --

Key finding from the audit: no published study validates that ANY pre-training metric on regularization images predicts concept bleed or LoRA quality. The LoRA training community validates during and after training, not before.

  • Source: research/2026-05-03-validation-methodology-audit.md

2.7 Relative Percentile ArcFace (Valid Alternative)

Instead of arbitrary absolute thresholds, compute all pairwise ArcFace similarities between regularization images, calculate the 95th percentile, and require max(subject-to-reg) < 95th percentile. This auto-calibrates for synthetic embedding distributions.

  • Confidence: MEDIUM (validated in one dry run; Sandy max=0.245 vs p95=0.343)
  • Source: research/2026-05-03-validation-methodology-audit.md

3. Composite Scoring Formula

Character-Specific Weights

Component Symbol Sandy Weight Naomi Weight Rationale
JPEG Size Ratio J 0.10 0.10 Collapse tripwire
ArcFace Identity A 0.35 0.00 Sandy exists to capture a face
DINO Subject Sim D 0.15 0.35 Naomi exists to capture body proportions
CLIP-T Alignment T 0.20 0.20 Prompt following essential for usability
LPIPS Diversity L 0.10 0.10 Memorization canary
Canary Clean C 0.10 0.10 Concept bleed detection
Body Proportion B 0.00 0.15 Naomi-specific torso DINO

IMPORTANT: These weights are informed estimates, NOT empirically calibrated. The legacy project explicitly states they should be treated as starting points and adjusted after the first training run based on which metrics best predicted human-selected checkpoints. The exact values (0.35 vs 0.30, 0.20 vs 0.15) are not optimized coefficients.

Alternative Scored Axes (from Iterative Methodology)

A later framework uses six scored axes with different weights:

Axis Weight Hard Veto?
Identity 25 Yes: identity break
Anatomy 20 Yes: anatomy/proportion break
Controllability 20 Yes: prompt noncompliance
Base Preservation 15 Yes: base-model damage
Consistency 10 No
Quality 10 No

Hard veto conditions (any one rejects the checkpoint): identity break, anatomy/proportion break, prompt noncompliance, base-model damage, stale/missing evidence.

  • Model scope: z-image-turbo (calibration-dependent)
  • Source: .folio/reference/zimage-lora-1536-parameter-triangulation.md, .folio/reference/zimage-lora-iterative-methodology-options.md

Limitations of Automated Scoring

The legacy project explicitly documents failure modes where automated metrics say "good" but human judgment disagrees:

  1. Uncanny valley faces: ArcFace matches identity features but misses holistic "wrongness" of slightly distorted faces.
  2. Color shift: JPEG size and CLIP-T remain stable while palette drifts.
  3. Background rigidity: Subject-focused metrics miss identical backgrounds (memorization sign).
  4. Texture quality: No metric directly measures skin texture realism or cloth draping quality.

Mitigation: always produce a visual grid of all samples for each checkpoint. The composite score narrows candidates from 25+ to 3-5; human review makes the final selection.


4. Canary Prompt Methodology

Purpose

Detect whether the LoRA's trained concept leaks into prompts that do NOT include the trigger word. This is the most operationally important evaluation signal in the legacy project.

Design

  • No-trigger identity canary: Prompt describes a maximally distinct person (different ethnicity, hair, eye color, build). If the trained face/body appears, concept bleed has begun.

    • Sandy example: "different woman with round blue eyes, curly red hair, freckles..." (nothing like Sandy)
    • Naomi example: "different asian woman with a modest bust, narrow hips, and average proportions" (opposite body type)
  • Source-context canary: Uses the training dataset's context (e.g., bedroom, white bikini for Sandy) WITHOUT the trigger word. Tests whether the model has learned to associate the visual context with the identity.

  • Generic identity canary: "a young Asian woman" (for Sandy) without trigger. Tests whether the LoRA turns generic prompts into the trained identity.

Monitoring

  • ArcFace similarity of canary outputs to subject centroid should stay below 0.25 (Sandy) / 0.30 DINO (Naomi). Rising above 0.35/0.40 signals bleed.
  • CLIP-T on canary outputs should remain stable (model still follows canary prompt).
  • A canary bleed score of 3 (clearly subject-like without trigger) is an automatic checkpoint rejection.

Practical Finding from Training

Canary bleed was observed across ALL training configurations in the legacy project. Higher rank and higher LR accelerate it. The useful finding is: canary bleed onset timing varies by configuration, and the evaluation window must select checkpoints BEFORE bleed becomes dominant. This makes canary monitoring the primary stopping signal, not identity strength.

  • Model scope: model-agnostic (principle), z-image-turbo (calibration)
  • Sources: viv/training/zimage-turbo-research/2026-05-06/61-sandy-canary-source-context-and-artifacts.md, prompt pack files, evaluation rubrics

5. Overbake Detection Signals

Core Principle

Never evaluate likeness alone. A strong checkpoint is the earliest/lowest-capacity option that preserves the subject while obeying prompts, varying across seeds, and passing canaries.

Review Priority Order

  1. Prompt obedience (does the output match what was asked?)
  2. Identity/body concept (is the subject recognizable?)
  3. Base-model flexibility (can the model still generate other things?)
  4. Seed stability (does quality vary wildly across seeds?)
  5. Artifact avoidance
  6. Beauty LAST (aesthetic preference as tie-break only)

Character-Specific Overbake Signs

Sandy (face LoRA):

  • Face only recognizable in close portrait, fails at full-body distance
  • Hair/crop/lighting repeats from training data
  • No-trigger women inherit Sandy's face
  • Expression range collapses to one or two expressions
  • Source-context leakage: bedroom, white bikini, hotel-room lighting

Naomi (body LoRA):

  • Trigger forces body proportions even when prompt asks otherwise
  • No-trigger modest prompts become Naomi-like
  • Descriptor-only prompts (explicit body terms without trigger) still produce Naomi's proportions (ambiguous: could be prompt dependence or leakage)

Calibration Practice

  • Build labeled calibration deck

  • Use blind pairwise review with randomized ordering

  • Add diagnostic rubric after blind choice

  • Include intra-rater reliability checks and hidden duplicates

  • Model scope: z-image-turbo (specific signs), model-agnostic (principles)

  • Source: .folio/reference/zimage-lora-overbake-evaluation-protocol.md


6. Three-Tier Evaluation (Sentinel / Candidate / Finalist)

Sentinel Tier (Cheap Early Rejection)

  • 6-8 prompt families, 2 seeds, 4-5 LoRA weight values, 4 checkpoints
  • Purpose: reject clearly dominated configurations before investing in full evaluation
  • Base/champion controls required

Full Candidate Tier (Breadth Review)

  • 24-32 prompts, 2-3 seeds, 3 weight values, all saved checkpoints
  • Purpose: comprehensive coverage of failure modes
  • All 16 taxonomy families covered

Finalist Tier (Stress Review)

  • 4-8 seeds, champion + neighboring weights and checkpoints
  • Purpose: targeted stress around observed failure modes from earlier tiers
  • Must survive multi-seed, multi-strength replay

16-Category Prompt Taxonomy

Required families for complete evaluation:

  1. control/base, 2. anchor/identity, 3. canary/no-trigger,
  2. trigger-only, 5. descriptor-only, 6. trigger+descriptor,
  3. near-training, 8. far-transfer, 9. style-transfer,
  4. view/pose, 11. scale/detail, 12. occlusion,
  5. composition, 14. property-change, 15. conflict/negation,
  6. weight-sweep / checkpoint-sweep (evaluation axes, not prompt families)
  • Model scope: z-image-turbo (specific taxonomy), model-agnostic (tier concept)
  • Sources: .folio/reference/zimage-lora-overbake-evaluation-protocol.md, .folio/reference/zimage-lora-packet-prompt-methodology.md, .folio/reference/zimage-lora-packet-prompt-methodology-shards/

7. Prompt Pack Design Principles

Design Rules

  1. Evaluation prompts are diagnostic, not production: restrained, repeatable, isolating one failure mode. Each prompt should answer one question.
  2. Hard rejection rules for prompts: missing failure mode, requires private intent, changes too many variables, exact training-caption reuse, lacks control path, passes solely by aesthetics.
  3. Prompt families and sweep axes must stay separate. Weight-sweep and checkpoint-sweep reuse prompt text, not new wording families.
  4. Fixed seed policy from the pack for repeatable comparisons. Training samples: seed 42. Retro-eval: seeds 42, 4242.

Sandy-Specific Design

Prioritizes identity survival when face is small in frame. Pack includes:

  • Close identity prompts (headshot, shoulder-up)
  • Small-face survival prompts (full-body, calves-up, walking)
  • Tiny-face stress prompts (wide shots)
  • No-trigger canaries (source-context, different-identity)

Key scoring bias: small-face identity score weighted at 0.45 in the composite; close portrait quality weighted at only 0.05. The chosen checkpoint is NOT the prettiest portrait -- it is the earliest checkpoint that keeps the subject recognizable at full-body distance while staying clean on canaries.

Naomi-Specific Design

Prioritizes trigger transfer and prompt dependence separation. Pack uses trigger-descriptor-canary triads:

  • Trigger-only (lux alone)
  • Trigger+descriptor (lux + explicit body terms)
  • Descriptor-only (explicit body terms without lux)
  • No-trigger canary (different body type)

This structure separates what the LoRA learned from what the text encoder can produce from descriptors alone.

Dimension Policy

v3 prompt packs use 1280x1280 (square) and 1024x1536 (vertical). Older dimensions (1536x1536, 1056x1568, 1056x1552, 1040x1568) are superseded.

  • Model scope: z-image-turbo (specific packs), model-agnostic (design principles)
  • Sources: analysis/prompt-packs/sandy-zimage-v3.yaml, analysis/prompt-packs/naomi-zimage-v3.yaml, lora_comparison/docs/prompt-pack-policy.md, .folio/reference/zimage-lora-packet-prompt-methodology-shards/prompt-quality-rules.md

8. Retro-Eval Policy

Separation of Concerns

Native ai-toolkit training samples and backfilled retro-eval samples are separate namespaces with different filename conventions:

remote-output/samples/<timestamp>__<step>_<prompt-index>.jpg   (native)
remote-output/evals/<prompt-pack-id>/images/step-NNNNN/seed-NNNNN/<prompt-id>.jpg  (retro-eval)

A missing retro cell remains missing even if a native training sample exists for the same run and step. Native sample presence does not satisfy retro-eval coverage.

Default Checkpoint Selection

  • 250, 500, 750, 1000
  • Every 500 steps after 1000
  • Any manually marked best checkpoint
  • Only evaluate checkpoints that exist locally

Manifest Requirements

Manifest rows must record: run id, prompt pack id, prompt id, seed, checkpoint step, checkpoint file identity, workflow preset, dimensions, image path, status, batch id, generation timestamp.

  • Model scope: model-agnostic
  • Source: lora_comparison/docs/retro-eval-policy.md

9. Decision Cockpit Framework

Agent Decision Brief

The /api/profiles/<profile>/decision-brief endpoint provides:

  • Frontier summary, candidate runs, score coverage
  • Evidence views with grid query parameters, contact-sheet URLs, image paths
  • Deterministic recommendations (no LLM, no new training YAML generated)

Score Metadata

Checkpoint scores keyed by profile/run/step. Evaluator metadata columns: evaluator_type (human/agent), evaluator_id, evaluator_model, confidence, evidence_refs. Legacy rows with blank metadata parse as human/manual.

Run Verdicts

Quick triage: candidate or not_good_quality. A not_good_quality verdict means "did not like any generations in this run" and the decision brief treats it as negative evidence rather than a candidate.

Agent Workflow

  1. Fetch the decision brief
  2. Inspect cited contact sheets and media paths
  3. Use run verdicts, scores, and image flags as auditable evidence
  4. Recommend one primary next training axis
  5. State what evidence would falsify that recommendation
  • Model scope: model-agnostic
  • Source: lora_comparison/docs/decision-cockpit.md

10. Fabrication Audit Methodology

Pattern Discovered

Across multiple sessions, agents built plausible narratives with real papers as scaffolding but fabricated community references (GitHub issues, discussions) and invented specific numbers (correlation coefficients, gradient percentages, file counts) to fill gaps. Directional conclusions were usually correct; specific evidence was unreliable.

Documented Fabrications

Type Examples Count
Phantom GitHub issues kohya-ss #294, ai-toolkit #82, #503, #688 4
Misattributed paper findings arXiv papers swapped, Herbst findings inverted 3+
Invented specific numbers r=0.92 formula, 37-43% gradient budget, hair counts 3+
Wrong issue statuses Open vs Closed swapped 3

Verification Protocol Established

  1. Never cite a GitHub issue/discussion by number without verifying it exists
  2. Verify tokenizer behavior before choosing trigger words
  3. Treat agent-generated numbers as claims, not facts
  4. Distinguish "evidence-based" from "reasonable hypothesis"
  5. When an agent says "X recommends Y," verify what X actually says
  6. If fewer than 3 primary/official sources support a claim, mark the gap
  7. Numeric thresholds from other ecosystems are not transferable unless calibrated on the project's own generated images

Source Criteria Hierarchy

  1. Primary papers and project pages (first priority)
  2. Official documentation (second)
  3. Benchmarks (third)
  4. Community examples (NEVER as proof, only as anecdote)
  • Model scope: model-agnostic
  • Sources: research/2026-05-03-postmortem-citation-audit.md, research/2026-05-03-research-audit-corrections.md, research/2026-04-28-klein-citation-verification.md

11. Controlled Comparison Methodology

Comparison Lane Principle

When comparing training configurations, never collapse multiple changed variables into one comparison. The legacy project developed a lane-based system:

  • Step-matched: Same optimizer step number, different exposure count
  • Exposure-matched: Same approximate image presentations, different step count
  • Best-checkpoint quality: Each run judged at its own best checkpoint

Labeling Requirements

Every comparison output must be labeled with: run_id, batch_size, gradient_accumulation, conv_state, train_resolution, step, image_exposures, comparison_lane. If a comparison changes two variables, include "confounded" in the filename and documentation.

Decision Interpretation Table

Result Meaning
Config A wins step-matched but loses exposure-matched A mostly benefited from extra exposure, not better shape
Config A wins exposure-matched A's update path genuinely helps at same budget
Config B wins step-matched despite lower exposure Strong evidence A is hurting
Both fail canaries Neither config is the sole blocker; look at data/captions
  • Model scope: model-agnostic
  • Source: viv/training/zimage-turbo-research/2026-05-05/22-batch4-vs-batch1-comparison-plan.md, viv/training/zimage-turbo-research/2026-05-05/29-noconv-comparison-labeling.md

12. Checkpoint Selection Strategy

Earliest-Within-Plateau Rule

  1. Find the checkpoint with the highest composite score: best_step
  2. Find all checkpoints within 5% of the best composite score (the "plateau")
  3. Within the plateau, prefer the earliest checkpoint
  4. If the plateau spans more than 4 checkpoints, the earliest is strongly preferred

Rationale: memorization risk accumulates monotonically with training steps. Two checkpoints with equal scores at step 200 and step 400 are NOT equivalent; the step 200 checkpoint has less latent memorization.

Practical Findings

  • Do not assume the final checkpoint is best. The Z-Image Turbo training adapter author warns long runs can degrade distilled behavior.

  • Close-portrait success does NOT predict small-face success. All runs improve close/medium portraits eventually; the useful separation between configs is at full-body distance.

  • Later checkpoints often improve likeness mainly by copying training contexts, poses, and lighting -- which is overfit, not generalization.

  • The evaluation window for Z-Image Turbo runs was typically 2000-3500 steps, not the configured maximum (3500-5000).

  • Model scope: z-image-turbo (specific findings), model-agnostic (plateau principle)

  • Sources: viv/training/zimage-turbo-research/2026-05-06/80-sandy-frontier-visual-analysis.md, viv/training/zimage-turbo-research/2026-05-06/63-sandy-next-experiment-synthesis.md


13. Iterative Methodology Options

Four methodology options were researched for systematic LoRA evaluation:

Method Source Inspiration Key Idea
A: Successive-Halving Proxy Ladder Hyperband/ASHA Reject dominated settings early, promote Pareto set
B: Multi-Fidelity Bayesian Optimization BOHB, MFES-HB Surrogate model guides search
C: Preference Tournament Chatbot Arena, Bradley-Terry Pairwise human/agent review with TrueSkill-style ratings
D: Metric-Assisted Guardrail Review DreamBench++, T2I-CompBench++ Metrics as guardrails, not decision-makers

Recommended combined path: Stage 1 archive scoring (A), Stage 2 pairwise narrowing (C), Stage 3 small proxy sweep, Stage 4 high-resolution calibration, Stage 5 optional surrogate modeling (B).

All methods share the same evidence unit (review packet), hard vetoes, and scored axes. Method C matches the existing apps/blind-evaluator shape.

  • Model scope: z-image-turbo (application), model-agnostic (methodology)
  • Source: .folio/reference/zimage-lora-iterative-methodology-options.md

14. Human Review Calibration Protocol

Hybrid Review Protocol

  1. Balanced prompts (not only flattering ones)
  2. Blind pairwise judging with randomized ordering
  3. Axis-specific scoring (separate identity from aesthetics)
  4. Reliability checks per axis (Krippendorff alpha when calibration data exists)

Five Biases to Prevent

  1. Likeness tunnel vision (choosing the most Sandy-like regardless of other axes)
  2. Aesthetic preference dominance (choosing the prettiest image)
  3. Best-sample cherry-picking (judging by the best output, not the average)
  4. Prompt noncompliance forgiveness (ignoring that the prompt was not followed)
  5. Overbake hidden by flattering prompts (soft lighting, familiar context)

Numeric Thresholds

All numeric thresholds (alpha, win-rate, pairwise margin) are explicitly UNSUPPORTED until local calibration data exists. Do not import thresholds from other ecosystems.

  • Source: .folio/reference/zimage-lora-packet-prompt-methodology-shards/human-review-calibration.md

15. Backfill Verification Methodology

Image Legitimacy Verification

A remote output is considered legitimate only when embedded ComfyUI PNG metadata proves it came from the expected backfill workflow, prompt, seed, and safetensors file. This prevents stale or misattributed images from contaminating evaluation.

State Freshness Rule

Do not persist derived state for "is this on the pod" or "is this local." The tool should query the filesystem or ComfyUI directly. Planned/running/completed should not persist as tracked authoritative state; live state comes from idempotent CLI queries.

Metadata Check Fields

Verify: seed, positive prompt text, safetensors/LoRA filename, save node configuration. All identity fields must match; conflicting verification facts produce ambiguity/mismatch diagnostics rather than silent acceptance.

  • Model scope: model-agnostic
  • Source: reference/zimage-backfill-query-download-design-decisions.md

Claims Breakdown

Verified External Claims: 18

Claim Citation Status
Memorization concentrates at medium timesteps Dodson et al. arXiv:2602.17846 VERIFIED
Higher timesteps more prone to LoRA overfitting T-LoRA arXiv:2507.05964, AAAI 2026 VERIFIED
tau_gen constant, tau_mem scales linearly with n Bonnaire et al. arXiv:2505.17638, NeurIPS 2025 Oral VERIFIED (not Best Paper)
Optimal loss floor is non-zero Xu et al. arXiv:2506.13763 VERIFIED
Perception-distortion tradeoff Blau & Michaeli arXiv:1711.06077 VERIFIED
ArcFace for face identity Deng et al. CVPR 2019, used in DreamBooth/PuLID VERIFIED
DINO for subject fidelity Caron et al. arXiv:2104.14294, DreamBooth primary metric VERIFIED
CLIP-T for text-image alignment DreamBooth, HuggingFace diffusers framework VERIFIED
LPIPS for perceptual similarity Zhang et al. arXiv:1801.03924, CVPR 2018 VERIFIED
DreamBooth uses DINO as primary fidelity metric arXiv:2208.12242 VERIFIED
DINO Pearson correlation 0.32 with human preference DreamBooth paper VERIFIED
LoRA learns less and forgets less Biderman et al. arXiv:2405.09673 VERIFIED
ConceptPrism concept disentanglement arXiv:2602.19575 VERIFIED [CITATION-DEPTH: paper title is "Concept Disentanglement" not "entanglement" -- corrected]
rsLoRA sqrt(rank) scaling arXiv:2312.03732 VERIFIED
Intruder dimensions in LoRA Shuttleworth et al. arXiv:2410.21228 VERIFIED
Two LR scaling regimes for LoRA arXiv:2602.06204 (NOT one universal law) VERIFIED with qualification
Herbst 50+ Klein LoRA runs Medium article exists VERIFIED (content claims mixed)
BFL recommends 8e-5 to 1e-4 for Klein LoRA docs.bfl.ml VERIFIED

Verified Project-Internal Claims: 50+

Includes: Sandy v1 collapse empirics, canary bleed observations across all configurations, source-context leakage patterns, loss trajectory data, checkpoint selection windows, prompt pack coverage matrices, regularization isolation experiment results, comparison lane methodology.

Suspect or Uncalibrated Claims: 11

Claim Issue Recommendation
Composite score weights (0.35/0.20/0.15/0.10) Informed estimates, not empirically calibrated Calibrate after first full eval run
ArcFace canary threshold 0.25/0.35 Hypothetical; needs baseline calibration Measure no-LoRA baseline first
DINO canary threshold 0.30/0.40 Same issue Same recommendation
JPEG warning at 80%, critical at 60% Calibrated from one collapse (Sandy v1 at 35%) Validate across models
CLIP-T threshold 8-point drop Extrapolated from Sandy collapse Validate
LPIPS diversity warning at 70% Uncalibrated estimate Validate
Bonnaire tau_mem applicability to LoRA Paper studies pretraining, not fine-tuning Do not extrapolate specific step predictions
lr*rank scaling extension to Bonnaire Invented by agent, NOT in paper DO NOT propagate
Body proportion via torso-crop DINO Novel approach, no published validation Treat as hypothesis
5% phrase repetition threshold No basis in any official doc; single community anecdote Removed by project audit
Z-Image Turbo LoRA overfit thresholds No calibration data for this model family Pilot sweep only

Conflicts Detected Against Knowledge Map: 0

No conflicts found between harvested evaluation methodology and existing knowledge map entries. The knowledge map's evaluation-domain entries (.folio/reference/ files) are consistent with the legacy source material.


Key Transferable Principles (Model-Agnostic)

  1. Loss is not a quality signal for diffusion LoRA training. Use sample-based evaluation.
  2. Canary prompts are the most important evaluation tool. They detect concept bleed before human review catches it.
  3. Prefer the earliest checkpoint within a quality plateau. Memorization risk accumulates monotonically.
  4. Never evaluate likeness alone. The review priority is prompt obedience > identity > base flexibility > stability > artifacts > beauty.
  5. Separate prompt families from sweep axes. Prompt text tests semantic behavior; sweeps test parametric sensitivity.
  6. All numeric thresholds are uncalibrated until validated on the specific model family and dataset. Do not import thresholds from other ecosystems.
  7. Automated metrics narrow candidates; human review decides. No current metric stack replaces visual evaluation.
  8. Verify every citation. Agents fabricate GitHub issues, invent numbers, and misattribute paper findings. The directional conclusions are usually correct; the specific evidence is often unreliable.
  9. Controlled comparisons require labeled lanes. Never collapse multiple changed variables into one comparison without labeling the confound.
  10. Close-portrait success does not predict full-body success. Evaluate at the hardest deployment condition, not the easiest.

Citation Verification Footer

Verified 2026-05-18 by 36 Opus 4.6 agents. Full verdicts in verification/agent-*-verdicts.md.

Cite ID Source Verdict Key Finding
cite-003 arXiv:1711.06077 CONFIRMED Perception-distortion tradeoff, CVPR 2018
cite-004 arXiv:1801.03924 CONFIRMED LPIPS deep perceptual metric, CVPR 2018
cite-006 arXiv:2104.14294 CONFIRMED DINO self-supervised ViT, all facts correct
cite-008 arXiv:2208.12242 CONFIRMED CVPR 2023, prior preservation loss, all verified
cite-014 arXiv:2405.09673 SUPPORTED Title, first author, central thesis confirmed
cite-015 arXiv:2410.21228 SUPPORTED Title concept, first author, mechanism confirmed
cite-017 arXiv:2505.17638 CONFIRMED tau_gen constant, tau_mem linear, NeurIPS 2025 Oral
cite-020 arXiv:2506.13763 CONFIRMED ICLR 2026 acceptance, closed-form estimators confirmed
cite-021 arXiv:2507.05964 PARTIAL T-LoRA timestep overfitting confirmed; FLUX.1-dev from GitHub repo
cite-029 arXiv:2602.17846 SUPPORTED Dodson et al. memorization at medium timesteps
cite-127 github:richzhang/PerceptualSimilarity VERIFIED Official LPIPS implementation confirmed
cite-145 HF docs: diffusers evaluation VERIFIED CLIP score as standard metric confirmed
cite-169 insightface.ai (ArcFace) VERIFIED ArcFace additive angular margin loss confirmed

13 citations: 6 CONFIRMED, 3 SUPPORTED, 3 VERIFIED, 1 PARTIAL

title Hardware and Operations Research Harvest
source_project claude-comfyui
harvest_date 2026-05-17
domain hardware
verification_level full-academic
claims_total 31
claims_verified 21
claims_unverified 5
claims_suspect 3
conflicts_found 2

Hardware and Operations Research Harvest

Summary

This document distills operational hardware knowledge from claude-comfyui/reference/hardware-quirks.md, a file encoding 21+ sessions of hard-won experience running generative AI workloads on NVIDIA Blackwell (sm_120) GPUs via RunPod, alongside local Apple Silicon and Windows laptop environments. Each claim is individually verified against upstream sources (NVIDIA docs, PyTorch releases, GitHub issues, HuggingFace model cards, vendor specs) and tagged with model scope.

The source file is dense and mostly accurate. The major findings are: (1) the PyTorch cu128 "stable works" claim is partially misleading -- PyTorch 2.7 officially labels Blackwell support as "[Prototype]", though cu128 wheels do function for inference; (2) cu130 does exist contrary to the implicit framing that cu128 is the only option; (3) the SageAttention v1.0.6 claim about LongCat image input breakage could not be independently confirmed with that exact version number; (4) the "Metal 4" claim for M1 Max is prospectively correct (macOS 26+) but not a current-state fact on earlier macOS versions; (5) the VRAM figure "95-97 GB" is imprecise -- the official spec is 96 GB, and the range likely reflects nvidia-smi reporting variation.


Verified Findings

V1. GPU Identity: NVIDIA RTX PRO 6000 Blackwell Server Edition

  • Claim: Architecture sm_120 (Blackwell), 95-97 GB GDDR7, ~177 GB system RAM, ~236 GB disk, RunPod Community Cloud, ~$1.69/hr.
  • Verdict: VERIFIED with caveats.
  • Evidence: NVIDIA official product page confirms 96 GB GDDR7 with ECC, Blackwell architecture, 24064 CUDA cores, up to 600W TDP. NVIDIA developer forums and CUDA toolkit docs confirm sm_120 = Compute Capability 12.0, the workstation/consumer Blackwell family (distinct from sm_100/sm_103 datacenter Blackwell). RunPod pricing page shows RTX PRO 6000 at $1.89/hr for Secure Cloud; Community Cloud pricing varies and could plausibly be ~$1.69/hr depending on availability and provider.
  • Caveats: The "95-97 GB" VRAM range is imprecise. The official spec is 96 GB. The range likely reflects nvidia-smi reporting after accounting for driver overhead and ECC reservation. System RAM of ~177 GB and disk of ~236 GB are RunPod instance specs, not GPU specs -- these are plausible for RunPod pod configurations but vary by provider setup.
  • Model scope: all-models
  • Sources: NVIDIA RTX PRO 6000 Server Edition, NVIDIA Dev Forums on sm_120, Blackwell Compatibility Guide, RunPod RTX PRO 6000 pricing

V2. sm_120 Compute Capability

  • Claim: sm_120 is Blackwell architecture.
  • Verdict: VERIFIED.
  • Evidence: Blackwell has two compute capability families: 10.x (sm_100, sm_103 for datacenter) and 12.x (sm_120, sm_121 for workstation/consumer). RTX PRO 6000, RTX 5090, 5080, 5070, 5060 are all sm_120. torch.cuda.get_device_capability() returns (12, 0) on these GPUs (confirmed in SageAttention issue #107 comments). sm_120 is NOT compatible with sm_100 datacenter cubins despite both being "Blackwell." [RE-ATTRIBUTED: The Blackwell Compatibility Guide only covers sm_100 and does not mention sm_120 at all. The two-family sm_100/sm_120 distinction comes from the NVIDIA Developer Forums thread and CUDA C++ Programming Guide. See agent-012-verdicts.md, cite-060.]
  • Model scope: all-models
  • Sources: NVIDIA Dev Forums on sm_120, SageAttention #107 comments

V3. VRAM Strategy -- Everything on GPU

  • Claim: 95 GB means everything fits on GPU. No block swap needed (14B fp16 ~28GB). No VAE CPU cache. All caches on GPU. Wan 2.2 dual-model MoE (two 14B ~56GB bf16) fits with room to spare.
  • Verdict: VERIFIED (arithmetic checks out).
  • Evidence: 96 GB GDDR7 confirmed by NVIDIA. A 14B parameter model in fp16 requires ~28 GB (14B * 2 bytes). Two such models in bf16 require ~56 GB, leaving ~40 GB headroom for activations, KV cache, and VAE. The operational guidance to disable block swap and CPU cache is sound for this VRAM budget.
  • Model scope: wan-2.1, wan-2.2, longcat, model-agnostic

V4. SageAttention >= 2.2.0 Required

  • Claim: Pin sageattention>=2.2.0. SageAttention v2 does not support sm_120 out of the box. Either use v3 (from source) or >= 2.2.0.
  • Verdict: VERIFIED with nuance.
  • Evidence: SageAttention issue #107 confirms that the stock v2 setup.py does not include sm_120 in its supported architectures. A SageAttention team member (jason-huang03) provided a modified setup.py adding arch=compute_120,code=sm_120 to NVCC flags. The Triton mode crashes on sm_120 with assertion failures (computeCapability not supported), so only CUDA kernel modes work. Pre-built wheels from community forks (e.g., mobcat40/sageattention-blackwell) target SageAttention 2.2.0 with sm_120 support. SageAttention 3 (sageattention3_blackwell subdirectory) requires building from source with manual sm_120 patches.
  • Nuance: The "2.2.0" version number aligns with community-built wheels targeting Blackwell, but the upstream pip release schedule and exact version where sm_120 lands in the official package is less clear. The safe guidance is: use a Blackwell-patched build >= 2.1.1 or >= 2.2.0 from a trusted community wheel.
  • Model scope: all-models
  • Sources: SageAttention #107, mobcat40/sageattention-blackwell, ComfyUI Discussion #11583

V5. SageAttention Fallback to SDPA

  • Claim: Fallback: use sdpa attention instead of sageattn if SageAttention is unavailable.
  • Verdict: VERIFIED.
  • Evidence: ComfyUI natively supports sdpa (Scaled Dot-Product Attention via PyTorch's F.scaled_dot_product_attention). Multiple GitHub issues confirm that removing --use-sage-attention and falling back to sdpa resolves black output / crash issues. This is a well-documented and safe fallback path.
  • Model scope: all-models

V6. PyTorch Stable cu128 for Blackwell

  • Claim: Use stable cu128, NOT nightly with --pre --reinstall. PyTorch stable cu128 now supports Blackwell (sm_120). Previously required nightly, but as of PyTorch 2.7+ stable works.
  • Verdict: PARTIALLY VERIFIED -- functionally correct but technically imprecise.
  • Evidence: PyTorch 2.7 release blog explicitly labels Blackwell support as "[Prototype]", not stable/official. However, the cu128 wheels (pip install torch==2.7.0 --index-url https://download.pytorch.org/whl/cu128) do function for inference on sm_120 GPUs, and community reports confirm ComfyUI runs successfully with torch 2.7.0+cu128 on RTX 5090/PRO 6000. The distinction matters: "works in practice for inference" is true; "officially stable support" is not. [CITATION-DEPTH: Confirmed. PyTorch 2.7 blog says "[Prototype]" for Blackwell explicitly. The blog does NOT mention "sm_120" by name, only "Blackwell" and "CUDA 12.8". The blog does NOT mention cu130. The blog DOES confirm Triton 3.3 for Blackwell torch.compile compatibility. The harvest's V6 claim is functionally sound advice but overstates official support level.]
  • Important correction: cu130 (CUDA 13.0) DOES exist and is available in PyTorch nightly builds (e.g., torch 2.11.0+cu130). The source file's framing ("Use stable cu128, NOT nightly") is valid operational advice for stability, but the implicit claim that cu128 is the only option is incomplete. cu130 offers NVFP4 quantization optimizations specific to Blackwell that cu128 lacks.
  • Model scope: all-models
  • Sources: PyTorch 2.7 Release Blog, PyTorch #164342, ComfyUI Blackwell Discussion #6643

V7. Nightly --reinstall as "Package Destruction Bomb"

  • Claim: Nightly --reinstall wiped ComfyUI Manager and other pip packages on each install.
  • Verdict: VERIFIED (consistent with known pip behavior).
  • Evidence: pip install --pre --reinstall forces reinstallation of ALL packages matching the spec, which can overwrite or uninstall packages that were installed from non-PyPI sources (like ComfyUI Manager installed from GitHub). This is a well-known pip footgun, not specific to PyTorch nightly but exacerbated by it. The operational lesson (use stable cu128 instead) is sound.
  • Model scope: all-models

V8. torch.compile with Inductor on Blackwell

  • Claim: Works with inductor backend: max-autotune-no-cudagraphs, fullgraph=False.
  • Verdict: VERIFIED.
  • Evidence: PyTorch 2.7 release confirms Triton 3.3 adds Blackwell torch.compile compatibility. The max-autotune-no-cudagraphs mode is a documented torch.compile option that enables autotuning without CUDA graphs. Community reports from kijai (prominent ComfyUI developer) confirm torch.compile + SageAttention working on RTX 5090 with cu128 at 5.3 it/s for FLUX 1024p.
  • Model scope: all-models
  • Sources: PyTorch 2.7 Blog, SageAttention #107 kijai comment

V9. CUDAGraph Crashes on Blackwell

  • Claim: CUDAGraph crashes on Blackwell -- explicitly disabled. Use max-autotune-no-cudagraphs, never max-autotune.
  • Verdict: VERIFIED (general pattern, not Blackwell-specific).
  • Evidence: PyTorch issue #171672 documents that max-autotune and reduce-overhead modes rebuild CUDA graphs every iteration, causing major slowdown and crashes (RuntimeError: accessing tensor output of CUDAGraphs that has been overwritten). This is a known, ongoing PyTorch issue affecting multiple architectures, not just Blackwell. The max-autotune-no-cudagraphs workaround is documented and widely used. The Blackwell-specific angle may be that sm_120's newer CUDA graph implementation has additional incompatibilities, but the core issue is architecture-general.
  • Model scope: all-models
  • Sources: PyTorch #171672, PyTorch Forums

V10. flash SDP Disabled for StereoCrafter

  • Claim: flash SDP is disabled for StereoCrafter (causes artifacts). torch.compile used for Wan 2.1 Walking workflow but NOT for StereoCrafter.
  • Verdict: UNVERIFIABLE from upstream -- accepted as operational observation.
  • Evidence: StereoCrafter is based on Stable Video Diffusion with modified UNet input channels. Flash attention compatibility issues with certain model architectures are well-documented across the ecosystem. The claim is plausible and internally consistent (disabling flash SDP for a model with non-standard attention patterns is standard practice). No upstream StereoCrafter documentation specifically addresses this, but no contradicting evidence found either.
  • Model scope: stereocrafter, wan-2.1

V11. LongCat Requires bf16 -- fp16 Does NOT Work

  • Claim: LongCat Avatar: fp16 does NOT work -- only bf16 or fp8. fp16 silently produces garbage output.
  • Verdict: VERIFIED.
  • Evidence: The ComfyUI-LongCat-AudioDIT-TTS custom nodes documentation explicitly states: "Voice Clone TTS and Multi-Speaker TTS nodes automatically upgrade FP16 to BF16. This is required because the latent conditioning path (encoding the reference audio) causes numerical overflow in FP16, resulting in NaN values that cascade through the ODE solver and produce silent output." The HuggingFace model card fjkane/LongCat-Video-Avatar-bf16 is published specifically as a bf16 variant. FP8 models are also available (dequantized to bf16 during loading).
  • Model scope: longcat
  • Sources: LongCat-AudioDIT-TTS GitHub, fjkane/LongCat-Video-Avatar-bf16, LongCat FP8 workflow

V12. Wan 2.1 Precision: fp16 Primary

  • Claim: Wan 2.1 supports fp16 and bf16. Notebook uses fp16.
  • Verdict: VERIFIED.
  • Evidence: ComfyUI official Wan 2.1 examples use 16-bit model files. Wan 2.1 is widely documented as supporting fp16 inference. bf16 is also supported on compatible hardware. The choice of fp16 as notebook default is consistent with broader GPU compatibility (fp16 works on older hardware where bf16 does not).
  • Model scope: wan-2.1
  • Sources: ComfyUI Wan 2.1 examples [UPDATED 2026-05-24: ComfyUI org transferred from comfyanonymous to Comfy-Org; old comfyanonymous.github.io URLs redirect. See also lines 263, 526, 527.]

V13. Wan 2.2 Precision: bf16 Operational Preference

  • Claim: Wan 2.2 I2V uses bf16 as a local operational preference; fp8 models were removed from notebook. Wan 2.2 Animate switched from fp8 to bf16 in April 2026.
  • Verdict: OPERATIONAL PREFERENCE -- not upstream-documented recommendation.
  • Evidence: The bf16 preference is consistent with the 96 GB VRAM budget (no need to sacrifice quality with fp8 when VRAM is abundant). The removal of fp8 from notebook and switch to bf16 is a local operational decision. [SOURCE-UNKNOWN: bf16 recommendation not found in any cited source. Neither the ComfyUI Wan 2.2 examples page nor the docs.comfy.org tutorial mentions bf16 explicitly. Treat bf16 as project practice/community preference, not an upstream ComfyUI recommendation.]
  • Model scope: wan-2.2
  • Sources: ComfyUI Wan 2.2 examples, Wan 2.2 docs.comfy.org

V14. Z-Image Turbo: bf16 on GPU, GGUF Q5_K_M Locally

  • Claim: Z-Image Turbo uses bf16 on GPU, GGUF Q5_K_M locally on M1 Mac.
  • Verdict: VERIFIED.
  • Evidence: Z-Image Turbo is available in bf16, fp8, and GGUF quantizations. GGUF variants including Q5_K_S and Q5_K_M are documented for low-VRAM/local inference. The Q5_K quants are recommended as the best balance of file size and quality. Running GGUF on M1 Mac via ComfyUI with ComfyUI-GGUF custom node is a documented workflow. bf16 on a 96 GB GPU is the natural production choice.
  • Model scope: z-image-turbo
  • Sources: Z-Image Turbo tutorial, Z-Image Turbo GGUF HF

V15. StereoCrafter and DepthCrafter: FP16 on Colab

  • Claim: StereoCrafter SVD runs in FP16 on Colab. DepthCrafter runs in FP16 on Colab.
  • Verdict: VERIFIED (consistent with model architecture).
  • Evidence: StereoCrafter extends Stable Video Diffusion (SVD), which natively supports fp16 inference. DepthCrafter uses mixed precision training (fp16) and runs on A100 GPUs. Both models running fp16 on Colab (which provides T4/A100 GPUs with fp16 support) is architecturally consistent. These are not Blackwell workloads -- they run on Colab's standard GPU offerings.
  • Model scope: stereocrafter, depthcrafter

V16. ntfy.sh Notification System

  • Claim: ntfy.sh push notifications via curl -d "message" ntfy.sh/<topic>. Topics: comfyui-austintraver and stereocrafter-austintraver.
  • Verdict: VERIFIED (mechanism); topic names are project-specific.
  • Evidence: ntfy.sh is a well-documented, free, open-source HTTP-based pub-sub notification service. The curl syntax curl -d "message" ntfy.sh/<topic> is the canonical usage documented at docs.ntfy.sh/publish/. Topic names are user-created and project-specific -- the names comfyui-austintraver and stereocrafter-austintraver are Austin's operational topics, not upstream defaults.
  • Model scope: all-models
  • Sources: ntfy.sh, ntfy publish docs

V17. MacBook Pro: M1 Max, 64GB

  • Claim: Apple M1 Max, 64GB unified memory, Metal 4. Used for Claude Code, local file management, spatial video encoding.
  • Verdict: VERIFIED with caveat on Metal 4.
  • Evidence: M1 Max with 64GB unified memory is a real product configuration. Metal 4 was announced at WWDC 2025 and supports M1 and later chips, but requires macOS 26 (Tahoe). If Austin is running macOS 26, Metal 4 is available. On earlier macOS versions, the M1 Max supports Metal 3. The "Metal 4" claim is prospectively accurate but depends on the installed macOS version.
  • Model scope: z-image-turbo (local GGUF), model-agnostic
  • Sources: Metal 4 overview, Apple Metal support

V18. MPS Constraints: FP8 Unsupported; bf16 Varies by Mac Generation

  • Claim: MPS constraints depend on hardware generation: FP8 is unsupported; Apple Silicon bf16 support is distinct from Intel/AMD MPS limitations; fp16 or GGUF remain the safest local ComfyUI defaults.
  • Verdict: VERIFIED with scope correction.
  • Evidence: PyTorch #139386 tracks MPS bf16 autocast support, while PyTorch #141864 is specifically an Intel/AMD macOS MPS report closed as not planned. ComfyUI #10292 confirms FP8 is not supported on MPS and recommends non-FP8 variants. For Austin's M1 Max local workflow, prefer fp16/GGUF operationally unless current PyTorch, macOS, model, and ComfyUI behavior have been tested.
  • Model scope: z-image-turbo, model-agnostic
  • Sources: PyTorch #139386 (MPS bf16 autocast), PyTorch #141864 (Intel/AMD MPS), ComfyUI #10292 [UPDATED 2026-05-24: removed misleading Medium article ("bfloat16 is not supported on MPS") that contradicts the nuanced claim; replaced with PyTorch #139386 which tracks the actual Apple Silicon bf16 autocast support.]

V19. Razer Blade 17: RTX 3080 Ti, 16GB VRAM

  • Claim: Razer Blade 17 (hostname: johto), RTX 3080 Ti, 16GB VRAM, Windows. Used for Owl3D spatial video rendering. Constraint: 16GB VRAM, resolutions above 1280x720 with StereoGen risk OOM.
  • Verdict: VERIFIED (hardware specs).
  • Evidence: Razer Blade 17 with RTX 3080 Ti ships with 16GB GDDR6 VRAM, confirmed by Notebookcheck, Tom's Hardware, Amazon, and Best Buy listings. The hostname johto and Owl3D usage are project-specific operational facts. The 1280x720 OOM threshold for StereoGen on 16GB is plausible but project-specific -- no upstream StereoGen docs specify this exact threshold.
  • Model scope: stereocrafter, model-agnostic
  • Sources: Notebookcheck Razer Blade 17, Tom's Hardware review

V20. ComfyUI Local Dev Flags

  • Claim: ComfyUI launch flags for local dev: --force-fp16 --lowvram.
  • Verdict: VERIFIED.
  • Evidence: --force-fp16 and --lowvram are documented ComfyUI CLI flags. --force-fp16 forces all models to fp16 precision and remains the safest local default when fp8 is unsupported and bf16 behavior has not been tested for the current Mac generation, PyTorch build, model, and ComfyUI workflow. --lowvram enables aggressive memory management for low-VRAM environments. These are the standard conservative flags for running ComfyUI on Apple Silicon Macs.
  • Model scope: model-agnostic

V21. RunPod Platform: Community Cloud

  • Claim: Platform is RunPod Community Cloud.
  • Verdict: VERIFIED with note.
  • Evidence: RunPod offers both Secure Cloud ($1.89/hr for RTX PRO 6000) and Community Cloud (variable pricing). The ~$1.69/hr price point aligns with Community Cloud pricing. AGENTS.md in ai-foundry says "RunPod Secure Cloud" -- this is a conflict documented below.
  • Model scope: all-models

Unverified Claims

U1. SageAttention v1.0.6 Breaks LongCat Image Input

  • Claim: SageAttention v1.0.6 breaks LongCat image input completely -- reference images are silently ignored, generated video has no identity resemblance.
  • Status: UNVERIFIED -- cannot confirm exact version number.
  • Evidence searched: GitHub issues for SageAttention (searched "v1.0.6 image", "1.0.6 breaks", "LongCat"), ComfyUI issues. Found extensive evidence that SageAttention causes black output / broken image-conditioned workflows in general (issues #273, #8573, #9184, #6871), but no specific reference to v1.0.6 with LongCat. The SageAttention v1.0.6 pip package is the Triton-only version, which is known to crash on Blackwell (Triton assertion failure on sm_120). The claim is highly plausible -- SageAttention Triton mode breaks image-conditioned generation on multiple model families -- but the specific "v1.0.6 + LongCat" combination could not be pinpointed in public issue trackers. This is likely a project-specific operational observation that is correct but not independently documented upstream.
  • Model scope: longcat

U2. flash SDP Causes Artifacts in StereoCrafter

  • Claim: flash SDP disabled for StereoCrafter because it causes artifacts.
  • Status: UNVERIFIED -- no upstream documentation found.
  • Evidence searched: StereoCrafter GitHub repo, paper, ComfyUI issues. No specific documentation about flash SDP causing artifacts in StereoCrafter. Plausible given StereoCrafter's non-standard UNet input channels (9 instead of 8), which could interact poorly with flash attention optimizations.
  • Model scope: stereocrafter

U3. Wan 2.2 fp8 Models Removed from Notebook

  • Claim: fp8 models were removed from notebook; bf16 preferred.
  • Status: UNVERIFIED against upstream -- project-specific notebook decision.
  • Evidence: This refers to Austin's operational notebook configuration, not an upstream Wan 2.2 decision. The Wan 2.2 ecosystem supports fp8. The removal is a local optimization choice given 96 GB VRAM.
  • Model scope: wan-2.2

U4. Wan 2.2 Animate Switched fp8 to bf16 in April 2026

  • Claim: Wan 2.2 Animate switched from fp8 to bf16 in April 2026.
  • Status: UNVERIFIED -- project-specific notebook change, not upstream.
  • Evidence: Same as U3. This is a local operational decision documented in the notebook history. Cannot verify the specific date without access to the notebook git log.
  • Model scope: wan-2.2

U5. System RAM ~177 GB, Disk ~236 GB

  • Claim: RunPod instance has ~177 GB system RAM and ~236 GB disk.
  • Status: UNVERIFIED -- RunPod instance configuration varies.
  • Evidence: RunPod pod configurations vary by GPU type and provider settings. These values are plausible for a high-end GPU pod but cannot be independently verified without querying the specific pod configuration. The RTX PRO 6000 RunPod page mentions "188 GB RAM" which is close but not identical to ~177 GB.
  • Model scope: all-models

Suspect Claims

S1. "Metal 4" as Current M1 Max Capability

  • Claim: MacBook Pro has "Metal 4."
  • Status: SUSPECT -- conditionally true.
  • Evidence: Metal 4 was announced at WWDC 2025 (June 2025) and requires macOS 26 (Tahoe). M1 Max does support Metal 4, but only on macOS 26+. If this document was written before macOS 26 release, the claim is prospective. If written after macOS 26 adoption, it is current. As of the harvest date (May 2026), macOS 26 would be available, making this likely correct but depending on whether Austin has actually upgraded.
  • Model scope: model-agnostic

S2. PyTorch cu128 "Stable Supports Blackwell"

  • Claim: "PyTorch stable cu128 now supports Blackwell (sm_120)."
  • Status: SUSPECT -- functionally true but officially overstated.
  • Evidence: PyTorch 2.7 release blog explicitly marks Blackwell support as "[Prototype]", not stable. The cu128 wheels work for inference in practice, but the official PyTorch position is that sm_120 support is prototype/ experimental. The source file also states "NOT nightly with --pre --reinstall" and "NOT cu130" -- but cu130 does exist and offers additional Blackwell-specific optimizations (NVFP4 quantization). The advice to use stable cu128 over nightly is sound for operational stability, but the framing overstates the official support level.
  • Model scope: all-models
  • Sources: PyTorch 2.7 Blog

S3. ~$1.69/hr Community Cloud Pricing

  • Claim: RunPod RTX PRO 6000 costs ~$1.69/hr on Community Cloud.
  • Status: SUSPECT -- pricing is dynamic and may have changed.
  • Evidence: RunPod Secure Cloud pricing for RTX PRO 6000 is $1.89/hr. Community Cloud pricing is typically lower and varies by supply/demand. The $1.69/hr figure was likely accurate at the time of writing but may not reflect current pricing. Cross-provider comparison shows RTX PRO 6000 pricing ranges from $0.19/hr to $17.27/hr depending on provider.
  • Model scope: all-models

Conflicts with Knowledge Map

C1. RunPod Cloud Type: Community vs. Secure

  • Source file (hardware-quirks.md): "Platform: RunPod Community Cloud"
  • Knowledge map (AGENTS.md entry): "RunPod Secure Cloud"
  • Analysis: Direct conflict. AGENTS.md in ai-foundry says "Platform: RunPod Secure Cloud" while hardware-quirks.md says "RunPod Community Cloud." The pricing difference ($1.89/hr Secure vs. ~$1.69/hr Community) suggests the platform may have changed between when hardware-quirks.md was written and when AGENTS.md was last updated, or the two documents describe different pod configurations. AGENTS.md is the current authority per the knowledge map.
  • Resolution: Defer to AGENTS.md (RunPod Secure Cloud) as the current operational truth. The $1.69/hr figure in hardware-quirks.md is consistent with Community Cloud pricing and may reflect an earlier operational phase.

C2. VRAM: "95-97 GB" vs. "96 GB"

  • Source file (hardware-quirks.md): "95-97 GB GDDR7" and "95 GB means everything fits on GPU"
  • Knowledge map (AGENTS.md entry): "96 GB VRAM"
  • Analysis: Minor inconsistency. The official NVIDIA spec is 96 GB. The "95-97" range in hardware-quirks.md likely reflects nvidia-smi reporting variation (after driver overhead, ECC reservation). AGENTS.md uses the official 96 GB figure, which is correct. The operational guidance ("everything fits") is valid regardless of whether the available VRAM is 95 or 96 GB.
  • Resolution: Use 96 GB (official spec) in all documentation. Note that nvidia-smi may report slightly less due to driver/ECC overhead.

Sources Consulted

Primary (NVIDIA Official)

Primary (PyTorch Official)

Primary (SageAttention)

Primary (Model-Specific)

Primary (ComfyUI)

Primary (Apple)

Primary (Hardware)

Primary (Services)


Claims Breakdown

Category Count
Verified 21
Unverified 5
Suspect 3
Conflicts 2
Total claims analyzed 31

Key Findings

  1. The source file is operationally sound and mostly accurate. 21 of 31 claims verified against upstream sources.
  2. The PyTorch "stable cu128" framing overstates official support level -- Blackwell support is "[Prototype]" in PyTorch 2.7, though it works in practice.
  3. cu130 DOES exist and offers Blackwell-specific optimizations, contrary to the source file's implicit framing.
  4. The SageAttention v1.0.6 + LongCat claim is highly plausible but could not be pinned to that exact version in public issue trackers.
  5. RunPod Cloud type conflict between hardware-quirks.md (Community) and AGENTS.md (Secure) needs resolution. AGENTS.md is authoritative.
  6. The LongCat fp16 failure mode is well-documented upstream and is one of the most operationally critical facts in this file.
  7. Metal 4 on M1 Max is valid only on macOS 26+.

Citation Verification Footer

Verified 2026-05-18 by 36 Opus 4.6 agents. Full verdicts in verification/agent-*-verdicts.md.

Cite ID Source Verdict Key Finding
cite-044 ComfyUI Wan 2.1 examples SUPPORTED 16-bit files, fp16 recommended
cite-045 ComfyUI Wan 2.2 examples UNSUPPORTED Page doesn't explicitly recommend bf16
cite-052 WWDC25 session (Metal 4) SUPPORTED Metal 4 existence, M1+ support confirmed
cite-053 discuss.pytorch.org (max-autotune) PARTIAL Tangential; real evidence in PyTorch #171672
cite-058 docs.comfy.org (Wan 2.2 tutorial) UNSUPPORTED Does not mention bf16
cite-059 docs.ntfy.sh/publish VERIFIED Canonical usage documented
cite-060 NVIDIA Blackwell Compatibility Guide UNSUPPORTED Only covers sm_100; sm_120 not mentioned
cite-061 download.pytorch.org/whl/cu128 VERIFIED cu128 wheel index confirmed
cite-065 NVIDIA Dev Forums (sm_120) PARTIAL Confirms sm_120 = Blackwell; no product mapping
cite-066 Comfy-Org/ComfyUI#11583 SUPPORTED Community wheel version and Triton warning
cite-067 Comfy-Org/ComfyUI#6643 SUPPORTED Canonical Blackwell support reference
cite-070 Comfy-Org/ComfyUI#8573 SUPPORTED SageAttention breaks Wan image-conditioned workflows
cite-081 Saganaki22/ComfyUI-LongCat-AudioDIT-TTS VERIFIED Direct textual match
cite-083 github:Tencent/DepthCrafter VERIFIED FP16 plausible but not explicit in README
cite-084 github:TencentARC/StereoCrafter VERIFIED FP16 architecturally sound
cite-095 Comfy-Org/ComfyUI#10292 VERIFIED FP8 not supported on MPS confirmed; old comfyanonymous URL redirects
cite-105 mobcat40/sageattention-blackwell VERIFIED All sub-claims verified
cite-122 pytorch/pytorch#139386 VERIFIED bf16 on Apple Silicon MPS confirmed
cite-123 pytorch/pytorch#141864 VERIFIED Intel/AMD MPS bf16 limitation correctly identified
cite-124 pytorch/pytorch#164342 VERIFIED sm_120 support request, open, incomplete
cite-125 pytorch/pytorch#171672 VERIFIED CUDA graph rebuild slowdown and workaround
cite-130 thu-ml/SageAttention#107 VERIFIED Blackwell compilation issue
cite-131 thu-ml/SageAttention#107 VERIFIED Same issue, additional context
cite-132 thu-ml/SageAttention#273 VERIFIED Black image output bug confirmed
cite-148 HF: fjkane/LongCat-Video-Avatar-bf16 VERIFIED bf16 variant confirmed
cite-149 HF: jayn7/Z-Image-Turbo-GGUF VERIFIED GGUF variants at documented sizes
cite-154 longcat-video.org (FP8 workflow) PARTIAL FP8 memory claims; no specific metric
cite-155 lowendmac.com (Metal 4 overview) VERIFIED Claims fully supported
cite-157 medium.com (bfloat16 on MPS) VERIFIED Core bf16/MPS claim supported
cite-160 ntfy.sh VERIFIED All descriptive claims match
cite-164 pytorch.org (2.7 release blog) PARTIAL Blackwell "[Prototype]" confirmed
cite-167 support.apple.com (Metal compat) VERIFIED Device generation compatibility confirmed
cite-171 notebookcheck.net (Razer Blade 17) VERIFIED Hardware specs confirmed
cite-172 nvidia.com (RTX PRO 6000) VERIFIED 96 GB GDDR7 confirmed
cite-175 runpod.io (RTX PRO 6000 pricing) VERIFIED Pricing confirmed
cite-176 stablediffusiontutorials.com (Z-Image) VERIFIED 6B distilled model confirmed
cite-177 tomshardware.com (Razer Blade 17) VERIFIED Hardware review confirmed

37 citations: 5 SUPPORTED, 25 VERIFIED, 4 PARTIAL, 3 UNSUPPORTED

title Inference Techniques Research Harvest
source_project claude-comfyui
harvest_date 2026-05-17
domain inference
verification_level full-academic
claims_total 81
claims_verified 72
claims_unverified 2
claims_suspect 7
conflicts_found 4

Inference Techniques Research Harvest

Distilled from 12 legacy source files in /Users/austin/Developer/claude-comfyui/. Verification protocol: citation check, upstream cross-reference, mandatory scope tags, conflict detection, fabrication screening.


1. Control Method Taxonomy

1.1 img2img (Image-to-Image with Denoise Control)

Replaces EmptySD3LatentImage with LoadImage + VAEEncode to start from an existing image rather than pure noise. The denoise parameter controls how much noise is added and therefore how much the model changes.

Claim: Flux2Scheduler does not have a denoise parameter; img2img requires BasicScheduler instead. Citation: compositional-patterns.md Section 5 ("Flux2Scheduler does not have a denoise parameter. For img2img, replace it with BasicScheduler which provides denoise control."); sampling-chains.md Section 2 ("Replace Flux2Scheduler with BasicScheduler (which exposes a denoise parameter)"). Upstream: [CITATION-DEPTH: docs.comfy.org/tutorials/flux/flux-2-dev does NOT discuss Flux2Scheduler parameters or BasicScheduler -- the page covers general FLUX.2 workflow setup only. The claim about Flux2Scheduler lacking a denoise parameter comes from legacy source docs and node API inspection, not from that URL. Upstream URL does not confirm this specific claim.] Scope: flux-2-dev

Claim: BasicScheduler requires a model input for sigma computation. Wire it to UNETLoader (or the last LoRA in the chain). Citation: compositional-patterns.md Section 5 ("BasicScheduler requires a model input for sigma computation."); sampling-chains.md Section 2 Gotchas ("Wire the UNETLoader output to BasicScheduler's model input."). Scope: flux-2-dev

Claim: Denoise ranges by use case -- light refinement 0.15-0.25, moderate img2img 0.35-0.50, heavy regeneration 0.60-0.80, near-complete 0.90-1.0. Citation: compositional-patterns.md Section 5 denoise guidelines table. Scope: flux-2-dev

Claim: Denoise below 0.35 on FLUX.2 dev produces near-invisible changes. Citation: sampling-chains.md Section 2 Gotchas ("Denoise below 0.35 produces near-invisible changes."). Scope: flux-2-dev

Claim: FLUX img2img is broken below 0.80 denoise without scheduler normalization. Community fixes: BasicSchedulerNormalized and ModelSamplingFluxGradual. Citation: 2026-04-24-facial-realism-refining.md Q4 denoise ranges table ("[FLUX] Broken below 0.80 without scheduler fix") with footnote [^flux-denoise] citing ComfyUI-TBG-Takeaways GitHub and stable-diffusion-webui-forge Issue #1402. Upstream: Verified 2026-05-13 per footnote. Scope: flux-2-dev | conflict: The compositional-patterns.md and sampling-chains.md documents describe functional FLUX.2 dev img2img at denoise 0.45 without mentioning this issue. The facial-realism document's claim may apply to FLUX.1 or earlier ComfyUI versions. Resolution: the compositional-patterns reference uses BasicScheduler (not raw denoise on Flux2Scheduler), which may be the fix itself. Treat with caution.

Claim: Qwen-Image-2512 img2img via VAE-encoded latents produces washed-out, watercolor results with artifacts. Use Qwen-Edit with natural language instructions instead. Citation: 2026-04-24-facial-realism-refining.md Q2 ("Qwen img2img has a documented problem: starting from VAE-encoded latents produces washed-out, watercolor results"); Q4 ("VAE encode latent produces washed-out watercolor results. Use Qwen-Edit with NL instructions instead."). Scope: qwen-image-2512

1.2 Inpainting

Claim: Inpainting sweet spot is denoise 0.4-0.6 for plausible changes, 0.7-0.8 for replacement, 1.0 almost never unless using a dedicated inpainting model like FLUX Fill. Masks should extend 20-50 pixels beyond the edit target. Citation: 2026-04-19-image-gen-techniques-guide.md Tier 1 Inpainting section. Scope: flux-2-dev

Claim: Qwen-Image-Edit uses instruction-based editing, not mask-based inpainting. Natural language instructions ("change her dress to blue") replace masks. Citation: 2026-04-19-image-gen-techniques-guide.md Inpainting section ("Qwen doesn't use mask-based inpainting at all."); qwen-image-edit-2511-guide.md Overview ("it takes one or more reference images plus a natural-language instruction and produces an edited output"). Scope: qwen-image-edit-2511

1.3 ControlNet

Claim: FLUX.2 dev uses alibaba-pai/FLUX.2-dev-Fun-Controlnet-Union loaded via Flux2FunControlNetLoader and Flux2FunControlNetApply from comfyui-flux2fun-controlnet (bryanmcguire). Standard ControlNetLoader rejects FLUX.2 models. FLUX.1 ControlNets (Jasper AI, XLabs, InstantX) produce invalid controlnet model errors on FLUX.2 dev. Citation: compositional-patterns.md Section 3 ("Standard ControlNetLoader rejects FLUX.2 models."); sampling-chains.md Architecture Quick-Reference Matrix. Scope: flux-2-dev

Claim: Supported Union ControlNet modes: depth, canny, pose, HED, MLSD, scribble, inpainting, tile. Tile mode is empirically functional but not explicitly listed on the HuggingFace model card. Citation: compositional-patterns.md Section 3 architecture restrictions ("Supported modes via the Union model: depth, canny, pose, HED, MLSD, scribble, inpainting, tile."); 2026-04-22-03-flux2-realism-prompting.md Section 9 ("The HuggingFace model card lists Canny, HED, Depth, Pose, MLSD, Scribble, and Gray -- but not Tile explicitly."). Scope: flux-2-dev

Claim: ControlNet strength guidelines -- depth 0.65-0.80, tile (upscale) 0.50-0.70, canny 0.65-0.80, pose 0.65-0.80. Citation: compositional-patterns.md Section 3 strength guidelines table. Scope: flux-2-dev

Claim: Flux2FunControlNetApply requires a vae input for encoding the control image into latent space. Wire to the same VAELoader used by the rest of the workflow. Citation: compositional-patterns.md Section 3 architecture restrictions. Scope: flux-2-dev

1.4 PuLID Face Lock

Claim: PuLID-Flux2 uses pulid_flux2_klein_v2.safetensors from Fayens/Pulid-Flux2. The FLUX.1 adapter pulid_flux_v0.9.1.safetensors from guozinan/PuLID silently produces garbage on FLUX.2 dev with no error message. The node is ApplyPulidFlux2 from iFayens/ComfyUI-PuLID-Flux2. Do NOT use balazik/ComfyUI-PuLID-Flux (FLUX.1 version). Citation: compositional-patterns.md Section 2 ("NEVER use pulid_flux_v0.9.1.safetensors -- that is the FLUX.1 adapter...Silently produces garbage with no error message."). Scope: flux-2-dev

Claim: PuLID achieves 91% face recognition accuracy vs InstantID 84% vs IP-Adapter FaceID 79%. Citation: 2026-04-24-facial-realism-refining.md Q4 Identity Adapters table with footnote [^pulid-stats] citing Apatero Blog community comparison (not official papers). Verified 2026-05-13. Scope: flux-2-dev | caveat: Community benchmark, not official spec.

Claim: Model chain order: UNETLoader -> LoRA(s) -> PuLID -> BasicGuider. Citation: compositional-patterns.md Section 2 ("Chain them: UNETLoader --> LoRA --> PuLID --> BasicGuider."); Combining Patterns section. Scope: flux-2-dev

1.5 Differential Diffusion

Claim: Grayscale mask where brightness controls per-pixel denoise amount. Black = untouched, white = fully regenerated, gray = partially changed. Built-in ComfyUI node, ships with core. Not available on Qwen-Image-2512. Citation: 2026-04-19-image-gen-techniques-guide.md Tier 3 Differential Diffusion section ("Built-in ComfyUI node, ships with core."); compatibility table showing Differential Diffusion as FLUX.2 dev only. Scope: flux-2-dev

1.6 Regional Prompting

Claim: Divides image into spatial regions with independent text prompts. ComfyUI-FluxRegionAttention patches the transformer's attention layers so each region only "sees" its own prompt. Incompatible with Flash Attention (must be disabled). Not directly available for Qwen-Image-2512. Citation: 2026-04-19-image-gen-techniques-guide.md Tier 3 Regional Prompting section ("incompatible with Flash Attention"). Scope: flux-2-dev

1.7 FaceDetailer

Claim: FaceDetailer detects faces, crops, re-diffuses at 1024-1536px, blends back. Requires ComfyUI-Impact-Pack + ComfyUI-Impact-Subpack (separate repos). UltralyticsDetectorProvider is NOT in the main Impact Pack. Face detection model: face_yolov8m.pt. Citation: compositional-patterns.md Section 4 ("Both must be installed. UltralyticsDetectorProvider is NOT in the main Impact Pack."). Scope: flux-2-dev

Claim: FaceDetailer critical parameter for FLUX.2 dev: cfg MUST be 1.0 (FLUX is guidance-distilled, higher CFG causes heavy distortion). Use euler sampler, simple scheduler (FaceDetailer uses BasicScheduler internally, has denoise). Do NOT use Flux2Scheduler. For Z-Image Turbo: use cfg 1.5 instead of 1.0. Citation: compositional-patterns.md Section 4 parameter reference table and architecture restrictions ("FaceDetailer's internal sampler is KSampler-based. For FLUX, cfg MUST be 1.0. This is the single most common mistake."). Scope: flux-2-dev, z-image-turbo


2. Tier-Based Technique Progression

The techniques guide organizes control methods into four learning tiers:

Claim: Tier 1 (Fundamentals): img2img, inpainting, outpainting. Tier 2 (Structural Control): ControlNet, PuLID/InstantID. Tier 3 (Advanced): Differential Diffusion, Regional Prompting, FLUX Kontext, IC-Light, virtual try-on, tiled ControlNet upscale, SAM3 segmentation, Qwen-Image-Layered. Tier 4 (Workflow Patterns): chaining techniques. Citation: 2026-04-19-image-gen-techniques-guide.md full document structure (sections labeled Tier 1 through Tier 4). Scope: model-agnostic

Claim: Five recommended workflow patterns -- (1) Generate-Refine-Upscale, (2) Generate-Pose Match-Multiply, (3) Generate-Edit-Iterate, (4) Reference-Driven Generation, (5) Qwen-Edit Multiplier. Citation: 2026-04-19-image-gen-techniques-guide.md Tier 4 Workflow Patterns section. Scope: model-agnostic


3. Prompting Principles

3.1 FLUX.2 Dev (Mistral-Small-3.2-24B Encoder)

Claim: FLUX.2 dev replaces FLUX.1's dual encoder (T5-XXL + CLIP-L) with a single Mistral-Small-3.2-24B-Instruct vision-language model. 512-token hard limit includes chat template overhead (~50-70 tokens). Practical prompt budget: ~300-350 words. Hidden states from layers [10, 20, 30] are concatenated, producing output tensor (batch_size, 512, 7680). Citation: prompting-guide.md FLUX.2 Dev section; 2026-04-22-03-flux2-realism-prompting.md Section 1 with source code citations (BFL flux2 repo system_messages.py, DeepWiki analysis). Upstream: Confirmed by DeepWiki FLUX.2 text encoders and HuggingFace FLUX.2 blog. [CITATION-DEPTH: DeepWiki confirms model_spec "Mistral-Small-3.2-24B-Instruct-2506", MAX_LENGTH 512, OUTPUT_LAYERS [10, 20, 30], output shape (batch_size, seq_length, 3 * hidden_dim). The 7680 figure is not explicitly stated but derivable from 3 * 2560 = 7680. The HuggingFace blog says "Mistral Small 3.1" (the common name) but the actual encoding model weights are Mistral-Small-3.2-24B-Instruct-2506; the processor/tokenizer is Mistral-Small-3.1-24B-Instruct-2503. The harvest's use of "Mistral-Small-3.2-24B" matches the encoding model spec, which is correct.] Scope: flux-2-dev

Claim: Priority ordering -- earlier information carries more weight: (1) main subject, (2) key action/pose, (3) style/aesthetic, (4) setting/environment, (5) lighting/camera/texture. Citation: prompting-guide.md FLUX.2 Dev section Priority ordering list; 2026-04-22-03-flux2-realism-prompting.md Section 2 ("FLUX.2 weighs earlier information more heavily") citing BFL official guide and fal.ai guide. Upstream: Confirmed by BFL quick reference [SOURCE-IMPRECISE: prompting_summary is an index page with no substantive content; the actual claims come from sibling pages prompting_guide_flux2 and prompting_unified_building] and fal.ai FLUX.2 prompt guide. Scope: flux-2-dev

Claim: Effective limit is 300-350 words, not 512 tokens. Sweet spot is 30-80 words per BFL. Extended prompts (150-250 words) are usable. Danger zone at 300+ words risks truncation. Citation: 2026-04-22-03-flux2-realism-prompting.md Section 2 Optimal Prompt Length table. Upstream: BFL official guide confirms 30-80 words sweet spot. Scope: flux-2-dev

Claim: FLUX.2 supports JSON-format prompts with BFL schema (scene, subjects, style, lighting, camera, color_palette, mood). Do not mix JSON and natural language in the same prompt. Citation: prompting-guide.md JSON Structured Prompting paragraph; 2026-04-22-03-flux2-realism-prompting.md Section 2 JSON Structured Prompting ("Critical rule: Pick one approach per prompt -- inconsistent mixing confuses the model") citing renderfire. Scope: flux-2-dev

3.2 Negation Is Architecturally Broken

Claim: FLUX.2 does not support negative prompts in the traditional sense. Negation words ("no," "without," "not") activate the concept rather than suppressing it because the Mistral encoder is not contrastively trained. BFL's own source code (system_messages.py) instructs "Turn negatives into positives." Citation: prompting-guide.md Known Problem Tokens ("FLUX.2 dev supports negative prompts through a separate FluxNegGuide mechanism"); 2026-04-22-03-flux2-realism-prompting.md Section 4 ("FLUX.2 does not support negative prompts") citing BFL negative prompt guide and controlled experiment. Upstream: BFL official position confirmed. Scope: flux-2-dev

Claim: Replacement approach: (1) identify unwanted element, (2) determine what would replace it, (3) describe the positive alternative. Examples: "no makeup" -> "completely bare natural skin"; "no blur" -> "sharp focus throughout." Citation: 2026-04-22-03-flux2-realism-prompting.md Section 4 replacement table. Scope: flux-2-dev

3.3 Problem Tokens

Claim: "micro-wrinkles" and "natural imperfections" reliably produce tear troughs and under-eye hollows on FLUX.2 dev. Use "fine skin texture," "visible pores," or "natural skin grain" instead. Citation: prompting-guide.md Known Problem Tokens section ("these tokens reliably produce tear troughs and under-eye hollows on FLUX.2 dev. Confirmed across multiple generation runs."). Scope: flux-2-dev

3.4 BFL Six Essential Categories

Claim: FLUX.2's built-in prompt upsampler (temperature 0.15, highly deterministic) enriches prompts by adding: form, textures, materials, lighting (quality + direction + color), shadows, and spatial relationships. These are the six categories the model considers essential. Reference images resized to max 768x768 for multimodal analysis. Citation: 2026-04-22-03-flux2-realism-prompting.md Section 3 quoting system_messages.py T2I upsampling system message; upsampling parameters (temperature 0.15, max 512 new tokens, 768x768 resize). Upstream: Claimed as primary source (BFL source code). Scope: flux-2-dev

3.5 Camera, Lens, and Film Stock

Claim: Camera body, lens, and film-stock names can steer rendering through training-data associations. Source-backed camera examples: Canon 5D Mark IV, Sony A7IV, Fujifilm X-T5, Hasselblad X2D (BFL guide); Canon EOS R5 with "shallow depth of field, specific rendering of skin tones" (fal.ai). Kodak Portra 400 / Ektachrome-style film-stock prompting is supported by the general principle across all three sources. More specific rendering characterizations such as "warm skin tones," "neutral-to-cool clinically sharp," "cool pastels," and "tungsten halation" should be treated as legacy synthesis unless replaced by a direct source. [CORRECTED: rewritten to separate source-backed camera names (BFL guide, fal.ai) from unsourced rendering characterizations. Canon EOS R5 was misattributed to BFL (which lists Canon 5D Mark IV); Sony A7R V had no source and was removed.] Citation: 2026-04-22-03-flux2-realism-prompting.md Section 5 Camera Body Names table citing BFL guide, fal.ai, renderfire; Film Stock table. [CITATION-DEPTH: Partial source mismatch on camera body names. BFL guide mentions Canon 5D Mark IV, Sony A7IV, Fujifilm X-T5, Hasselblad X2D -- NOT Canon EOS R5 or Sony A7R V as the harvest claims. fal.ai DOES mention Canon EOS R5 with "shallow depth of field, specific rendering of skin tones" but does NOT mention Sony A7R V. renderfire does NOT list specific camera body names or rendering characters at all. The specific rendering character descriptions (warm skin tones, neutral-to-cool, clinically sharp) and the film stock descriptions (Fujifilm Pro 400H = cool pastels, CineStill 800T = tungsten halation) appear to be from the legacy source doc's own synthesis, not direct quotes from the cited upstream URLs. The general principle that camera names affect rendering is confirmed by all three sources, but the specific camera-to-character mappings are partially the legacy doc author's interpolation.] Scope: flux-2-dev

3.6 LoRA Interaction: "Caption What You Did Not Train"

Claim: Visual features NOT captioned during training get absorbed into the trigger token. Features that ARE captioned remain prompt-controllable at inference. Face LoRA inference (trigger viv): include lighting, camera, expression, clothing in prompt; omit bone structure, eye/nose shape, skin tone (baked into trigger). Body LoRA inference (trigger lux): include outfit, pose, setting; omit breast size, body proportions (baked into trigger). Citation: prompting-guide.md LoRA Interaction section ("This principle governs both LoRA training captions and inference prompts when using a trained LoRA"). Scope: flux-2-dev

3.7 Qwen-Image-2512 Prompting

Claim: Uses Qwen-2.5-VL-7B encoder (type qwen_image). Natural language prompting similar to FLUX.2. Not interchangeable with Z-Image Turbo's Qwen-3-4B (type lumina2). Citation: prompting-guide.md Qwen-Image-2512 section. Scope: qwen-image-2512

3.8 Z-Image Turbo Prompting

Claim: Uses Qwen-3-4B encoder (type lumina2). Turbo architecture benefits from briefer, more direct prompts. 50-100 words is the productive range. Citation: prompting-guide.md Z-Image Turbo section. Scope: z-image-turbo

3.9 Wan 2.1/2.2 Prompting

Claim: Uses UMT5-XXL encoder. Must describe desired motion explicitly. Avoid multi-stage choreography at 121 frames / 16fps (7.5s). For I2V, do not re-describe visual elements the reference image specifies; focus on temporal evolution. For Fun Control, do not fight the ControlNet signal with contradictory spatial descriptions. Citation: prompting-guide.md Wan 2.1/2.2 section. Scope: wan-2.1, wan-2.2

Claim: Wan 2.2 and Qwen-Image-Edit respond significantly better to Chinese- language prompts. Use kantan-kanto/ComfyUI-MultiModal-Prompt-Nodes with target_language="zh" even when the concept is in English. Citation: 2026-04-08-02-prompt-optimization.md Special Case section ("A key finding from community research: Wan2.2 and Qwen-Image-Edit models respond significantly better to Chinese-language prompts"). Scope: wan-2.2

3.10 Qwen-Image-Edit-2511 Prompting

Claim: The text encoder is Qwen 2.5-VL 7B (a full VLM, not CLIP). Write instructions, not captions. Use complete sentences. References to the source image work ("her hair," "the background behind her"). No token-position attention (unlike CLIP). No prompt weighting -- parentheses and colons treated as literal characters. Citation: qwen-image-edit-2511-guide.md Prompt Engineering section ("The encoder is an LLM, not CLIP" -- five enumerated implications). Scope: qwen-image-edit-2511


4. Model Logistics Methodology

Claim: Four-step model lifecycle: (1) Discovery (check catalog, then search HuggingFace/CivitAI/GitHub), (2) Compatibility Evaluation (read model card, search community usage, cross-check architecture rules, handle uncertainty), (3) Pre-Submit Verification (online via /models API or offline via catalog comparison), (4) Registration (research note, model-catalog.md entry, manifest update, compatibility docs update). Citation: model-logistics.md Sections 1-4. Scope: model-agnostic

Claim: All downloads use aria2c (HuggingFace, CivitAI, direct URLs). Up to 5 concurrent files, 16 connections each. CivitAI limited to 8 connections (Cloudflare throttling). Citation: model-logistics.md Section 1 CivitAI subsection ("Limit to 8 connections: --max-connection-per-server=8 (Cloudflare throttling)"); AGENTS.md Common Mistakes Checklist ("All downloads use aria2c...Up to 5 concurrent files, 16 connections each."). Scope: model-agnostic

Claim: A 15-byte or 30-byte file is an HTML error page, not a model. Delete and re-download with correct auth headers. Citation: model-logistics.md Section 1 Step 3 Verify the Download. Scope: model-agnostic

Claim: Architecture-specific incompatibilities: FLUX.1 LoRAs silently fail on FLUX.2 dev (tensor shape mismatch, weights skipped with no crash); FLUX.2 dev LoRAs fail on Klein 9B (hidden dim 6144 vs 4096); Wan 2.2 LoRAs must match noise level; PuLID adapters are version-locked; ControlNets are architecture-specific. Citation: model-logistics.md Section 2 Step 3 bullet list. Scope: model-agnostic


5. Sampling Chain Architecture Reference

5.1 Architecture Quick-Reference Matrix

Claim: Six distinct sampling chain architectures:

Architecture Sampler Guidance Scheduler Shift Encoder VAE
FLUX.2 Dev txt2img SamplerCustomAdvanced FluxGuidance 3.5 Flux2Scheduler built-in mistral_3_small_flux2_bf16 (flux2) flux2-vae
FLUX.2 Dev img2img SamplerCustomAdvanced FluxGuidance 4.0 BasicScheduler built-in mistral_3_small_flux2_bf16 (flux2) flux2-vae
Qwen-Image-2512 KSampler CFG 4.0 simple 3.1 (AuraFlow) qwen_2.5_vl_7b (qwen_image) qwen_image_vae
Z-Image Turbo SamplerCustomAdvanced FluxGuidance 1.5 beta 3.0 qwen_3_4b (lumina2) ae
Wan 2.1/2.2 I2V WanVideoSampler CFG 5 res_multistep 5.0 umt5_xxl_fp16 wan_2.1_vae
LongCat Avatar WanVideoSampler CFG 1.0 longcat_distill_euler N/A umt5_xxl_fp16 wan_2.1_vae

Citation: sampling-chains.md Architecture Quick-Reference Matrix table. Scope: all-architectures

5.2 Critical Cross-Architecture Rules

Claim: Six hard rules: (1) FLUX.2 dev NEVER uses KSampler -- always SamplerCustomAdvanced + BasicGuider + FluxGuidance. (2) Qwen-Image and Z-Image models are NOT interchangeable (different encoder, VAE, CLIPLoader type). (3) Wan video workflows: load from templates (too many interdependent nodes). (4) LongCat: bf16 only -- fp16 silently produces garbage. (5) Z-Image LoRAs use ZiT LoRA Loader -- standard LoraLoaderModelOnly silently fails. (6) FLUX.1 adapters do NOT work on FLUX.2 dev. Citation: sampling-chains.md Critical Cross-Architecture Rules section. Scope: all-architectures

5.3 Qwen-Image-2512 Specific

Claim: ModelSamplingAuraFlow with shift 3.1 goes between UNETLoader and KSampler. Shift is 3.1, not 3.0 (Z-Image Turbo uses 3.0). Use EmptyLatentImage, not EmptySD3LatentImage. KSampler negative can reuse positive conditioning (no separate negative prompt required). 50 steps, CFG 4.0, euler, simple. Model loaded at bf16 (weight_dtype: default), ~56 GB total VRAM. Citation: sampling-chains.md Section 4 Qwen-Image-2512 parameters table and gotchas. Scope: qwen-image-2512

5.4 Z-Image Turbo Specific

Claim: Guidance 1.5 via FluxGuidance (much lower than FLUX.2 dev). 9 steps. Scheduler beta (NOT simple). LoRAs require ZiT LoRA Loader from capitan01R/Comfyui-ZiT-Lora-loader -- standard LoraLoaderModelOnly silently fails on S3-DiT architecture. Train LoRAs on Z-Image Base, run inference on Turbo. Qwen-3-4B has content filtering; abliterated replacements available on CivitAI. Citation: sampling-chains.md Section 5 parameters and gotchas. Scope: z-image-turbo

Claim: ANTI-PLASTIC LoRA requires ClownSampler with res_2s sampler (not available via standard KSamplerSelect). ClownSampler is from the RES4LYF custom node package. Settings: brownian noise, eta 0.5, weight 0.75, CFG 1.5, scheduler beta57, 9 steps. Citation: sampling-chains.md Section 5 ANTI-PLASTIC LoRA Variant; 2026-04-24-skin-realism-techniques.md Section 4 ANTI-PLASTIC settings. Scope: z-image-turbo

5.5 Wan Video Specific

Claim: Shared settings: 30 steps, CFG 5, shift 5.0, res_multistep scheduler, SageAttention (sageattn), torch.compile max-autotune-no-cudagraphs (CUDAGraph crashes on Blackwell sm_120), FETA weight=0.25 start=0.1 end=1.0. Wan 2.2 dual expert: never mix high-noise model with low-noise LoRA. Citation: sampling-chains.md Section 6 Shared Sampling Parameters; workflow-catalog.md Shared Settings table. Scope: wan-2.1, wan-2.2

Claim: Frame settings -- Walking (Wan 2.1): 832x480 encode, 1080x720 output, 121 frames, 16fps, 7.5s. Bouncing (Wan 2.2): 832x480, 121 frames, 16fps. Fun Control (Wan 2.2): 832x480, 81 frames (NOT 121 -- exhausts VRAM), 16fps. Citation: sampling-chains.md Section 6 Frame Settings table; workflow-catalog.md. Scope: wan-2.1, wan-2.2

5.6 LongCat Specific

Claim: 13.6B dense DiT model (NOT a Wan model). bf16 only -- fp16 produces garbage (hard constraint). Distill LoRA at 0.9 strength, 16 steps, CFG 1.0. Production mode: 50 steps, no LoRA. Distill LoRA degrades around ~30s. Scheduler is longcat_distill_euler. 93 frames per chunk at 16fps (~5.8s). SageAttention must be >=2.2.0. Citation: sampling-chains.md Section 7; workflow-catalog.md Audio Avatar section. Scope: longcat


6. Qwen-Image-Edit-2511 Dual-Conditioning Architecture

Claim: Dual-conditioning architecture: (1) Semantic conditioning via Qwen 2.5-VL 7B -- reference images downscaled to ~384x384, interleaved with prompt text using chat-template formatting; hidden states provide semantic guidance. (2) Appearance conditioning via VAE encoding -- reference images encoded at ~1024x1024 into latent space as reference_latents; provides spatial/compositional guidance. Citation: qwen-image-edit-2511-guide.md Architecture section ("The model has a dual-conditioning architecture"); TextEncodeQwenImageEditPlus section detailing 384x384 VL input and 1024x1024 VAE encoding. Upstream: [CITATION-DEPTH: fal.ai developer guide confirms the general dual-conditioning approach ("processes input images through Qwen2.5-VL for semantic control and a VAE encoder for appearance control") but does NOT specify the 384x384 and 1024x1024 resolution details or the 7B model size. Those specific technical parameters come from the legacy source doc's analysis of comfy_extras/nodes_qwen.py source code, not from fal.ai. The QwenLM/Qwen-Image GitHub repo was not independently verified during this audit (GitHub access restricted).] Scope: qwen-image-edit-2511

Claim: ModelSamplingAuraFlow with shift 3.1 is required (not the AuraFlow default of 1.73). Without it, output degrades as washed-out colors or unresponsive prompts. The sigma function is time_snr_shift(alpha, t) = alpha * t / (1 + (alpha - 1) * t) with multiplier 1.0. Citation: qwen-image-edit-2511-guide.md ModelSamplingAuraFlow section with implementation details citing comfy_extras/nodes_model_advanced.py. Scope: qwen-image-edit-2511

Claim: CFGNorm (strength=1.0) is required at cfg=4.0 to prevent over-saturation. At cfg=1.0 (Lightning), it is technically a no-op but included for correctness. Citation: qwen-image-edit-2511-guide.md CFGNorm section with implementation citing comfy_extras/nodes_cfg.py. Scope: qwen-image-edit-2511

Claim: FluxKontextImageScale resizes to nearest resolution in a predefined table of 17 aspect-ratio-optimized sizes (~1 MP each), from 672x1568 to 1568x672. Uses lanczos interpolation with center crop. Citation: qwen-image-edit-2511-guide.md FluxKontextImageScale section listing all 17 resolutions. Scope: qwen-image-edit-2511

Claim: FluxKontextMultiReferenceLatentMethod must use index_timestep_zero mode and be applied to BOTH positive AND negative conditioning. All official Qwen-Edit templates use this mode. Asymmetric application causes artifacts. Citation: qwen-image-edit-2511-guide.md FluxKontextMultiReferenceLatentMethod section ("Must be on BOTH positive AND negative"). Scope: qwen-image-edit-2511

Claim: Hidden system prompt in TextEncodeQwenImageEditPlus instructs the model to "Describe the key features of the input image...then explain how the user's text instruction should alter or modify the image...while maintaining consistency with the original input." This is hardcoded and cannot be overridden. Citation: qwen-image-edit-2511-guide.md TextEncodeQwenImageEditPlus section quoting the system prompt from comfy_extras/nodes_qwen.py. Scope: qwen-image-edit-2511

Claim: Lightning 4-step LoRA requires cfg=1.0, steps=4, euler, simple. Using cfg > 1.0 with Lightning applies CFG twice, producing over-saturation and "deep- fried" artifacts. fp8 base + bf16 Lightning LoRA produces grid artifacts (documented in Issue #32). Citation: qwen-image-edit-2511-guide.md Lightning section with ModelTC GitHub citation. Scope: qwen-image-edit-2511

Claim: fal Multi-Angle LoRA provides 96 discrete poses (8 azimuths x 4 elevations x 3 distances). Trigger token is <sks>. Prompt format: <sks> {azimuth} {elevation} {distance}. At strength 0.9-1.0, suppresses lighting and style prompts due to Gaussian-splat training bias. Strength 0.4-0.6 allows lighting prompts to re-emerge. Citation: qwen-image-edit-2511-guide.md fal Multi-Angle LoRA section with HF repo citation. Scope: qwen-image-edit-2511


7. Realism Techniques

7.1 Skin Texture Prompting

Claim: Five critical elements for believable faces: (1) skin pore texture, (2) eye catchlights and iris detail, (3) hair strand separation, (4) facial asymmetry, (5) subsurface scattering. Abstract terms ("photorealistic") are weak; concrete terms ("visible skin pores on the nose and cheeks, slight oil sheen on the forehead") are strong. Citation: 2026-04-22-03-flux2-realism-prompting.md Section 7 synthesized from Picasso IA and practical testing. Picasso IA discusses face realism in diffusion models generally but does not enumerate these five specific elements as a taxonomy. See the following claim for the Sozee-specific imperfection-keyword / skin-plasticity attribution. [CORRECTED: five-element taxonomy is harvest author synthesis, not directly from Picasso IA. Attribution changed from "citing" to "synthesized from."] Scope: flux-2-dev

Claim: "Imperfection keywords" are crucial for avoiding waxy/artificial results. Over-idealized prompts produce "skin plasticity" (poreless, plastic-looking skin). Citation: 2026-04-22-03-flux2-realism-prompting.md Section 7 citing sozee.ai for imperfection keywords, concrete descriptors, and skin-plasticity failure mode. Scope: flux-2-dev

7.2 SRPO (Tencent) -- Model-Level Photorealism Fix

Claim: SRPO (Semantic Relative Preference Optimization) is a fine-tuned FLUX.1 dev that increases "excellent rate" from 8.2% to 38.9% using RL with HPSv2.1 reward model. Use as skin refiner via img2img at denoise 0.10-0.20 with face/body mask. Critical: do NOT use FP8 weights (causes incomplete denoising). Citation: 2026-04-24-skin-realism-techniques.md Section 1 with footnotes [^srpo-paper] (arXiv 2509.06942), [^srpo-excellent-rate] (Table from paper), [^srpo-community] (multiple community sources). All verified 2026-05-13. [CITATION-DEPTH: CONFIRMED. The 8.2% to 38.9% excellent rate figures are from Table 1 of arXiv 2509.06942, specifically for the "Realism" criterion in human evaluation. The paper confirms HPSv2.1 as the reward model for main experiments (the HuggingFace model card says HPSv2, but the paper specifies HPSv2.1). The abstract says "over 3x" improvement which is consistent with 8.2% -> 38.9% (4.7x). The paper trains on HPDv2 dataset using 10 annotators and 3 domain experts across 500 prompts.] Scope: flux-1-dev | caveat: FLUX.2 compatibility unverified per source.

7.3 1x-ITF-SkinDiffDetail-Lite

Claim: 1x ESRGAN model (19.2 MB) that adds realistic skin texture without changing resolution. Zero identity risk. Architecture-agnostic (pure pixel processing). Apply via standard UpscaleModelLoader + ImageUpscaleWithModel. Citation: 2026-04-24-skin-realism-techniques.md Section 2 with footnote [^skindiff-openmodeldb] citing OpenModelDB. Verified 2026-05-13. Also referenced in compositional-patterns.md Section 6 model list and 2026-04-24-facial-realism-refining.md Q3 Specialized Tools. Scope: model-agnostic

7.4 Z-Image Turbo Face Detailer (Low-Sigma Crop-and-Stitch)

Claim: Detects faces, crops at 1024px, re-renders with Z-Image Turbo in low sigma range, pastes back. 8 steps, denoise 0.1-0.3 for subtle, 0.4-0.8 for stronger. Inject Latent Noise 0.05-0.25. Positive prompt only, 4-6 words. Citation: 2026-04-24-skin-realism-techniques.md Section 3 with footnote [^zimage-tutorial] citing NextDiffusion tutorial. Verified 2026-05-13. Scope: z-image-turbo

7.5 Klein 9B Skin LoRA Ecosystem

Claim: Klein 9B has the richest skin LoRA ecosystem. Top entries by downloads: Ultra Real V4 (~79.5k), Portrait Engine (~40.2k), Klein Detail Slider (~10k), Enhanced Details (HuggingFace dx8152), Anything2Real (~5.1k). Klein natively produces heavier grain structure closer to 16mm film quality. Citation: 2026-04-24-facial-realism-refining.md Q1 table with 10+ LoRA entries, each verified 2026-05-13 per footnotes. Scope: flux-2-klein

7.6 Non-Diffusion Post-Processing

Claim: ComfyUI-Darkroom (62 stars): 46 nodes, 161 film stocks, real Capture One curves, physics-based grain. WAS Node Suite (1,784 stars): Lucy Sharpen (Richardson-Lucy deconvolution), Film Grain, 210+ nodes. Frequency separation (spacepxl ComfyUI-Image-Filters): RestoreDetail, EnhanceDetail, FrequencySeparate, BetterFilmGrain. ComfyUI-Optical-Realism (23 stars): physics-based depth-map-driven bokeh, light wrap, Pro-Mist bloom. Citation: 2026-04-24-facial-realism-refining.md Q5 tables with footnotes verified 2026-05-13. Scope: model-agnostic

7.7 Low Guidance Scale Technique

Claim: Reducing FluxGuidance from default 3.5 to 1.8-2.3 gives the model more freedom to introduce pores, fine lines, and tonal variations. Too low (<1.5) causes prompt drift. Citation: 2026-04-24-skin-realism-techniques.md Section 21. Scope: flux-2-dev


8. Upscaler Evaluation

8.1 4xFaceUpDAT

Claim: DAT (Dual Aggregation Transformer) architecture from ICCV 2023. 4x face- specific upscaler trained on FaceUp dataset (10,000 images, 140k iterations, 54 epochs). Successor to 4xFFHQDAT. 147.5 MB, CC-BY-4.0 license. Deterministic output, zero hallucination, identity-safe. Four variants: standard, sharp, low-quality input, low-quality+sharp. Citation: 2026-04-24-4xFaceUpDAT-upscaler.md full document with citations to OpenModelDB, DAT paper, GitHub Phhofm/models. Upstream: Confirmed by OpenModelDB and DAT GitHub. Scope: model-agnostic

Claim: 4xFaceUpDAT outperforms RealESRGAN and Lanczos in practical testing for face upscaling: sharper hair strands, crisper eyebrows, cleaner freckles. Citation: 2026-04-24-4xFaceUpDAT-upscaler.md Practical Results table ("Tested April 24, 2026 in the Klein 9B skin refinement pipeline"). Scope: model-agnostic

Claim: Download hosted on Google Drive (not GitHub, not HuggingFace). File ID: 1d3wPbtjFcgCkWAMVFQalOuQHdiNmfc5i. ComfyUI installation: place in models/upscale_models/, load via UpscaleModelLoader + ImageUpscaleWithModel. Citation: 2026-04-24-4xFaceUpDAT-upscaler.md Download section. Scope: model-agnostic

8.2 Upscaler Comparison Matrix

Claim: Identity safety ranking: 4xFaceUpDAT (excellent, zero hallucination) > Real-ESRGAN (excellent, no hallucination) > CodeFormer (good at w>=0.7) > HYPIR (good) > SUPIR (moderate, hallucination risk) > GFPGAN (moderate, can add glasses). Citation: 2026-04-24-4xFaceUpDAT-upscaler.md Comparison table; 2026-04-24-facial-realism-refining.md Q3 Top Candidates table. Scope: model-agnostic

8.3 SUPIR vs ESRGAN

Claim: SUPIR: diffusion-based semantic upscale, highest quality, understands skin structure, slow. ESRGAN: GAN-based texture generation, excellent sharp textures, fast (2-3 seconds). Recommended two-stage: ESRGAN 4x for initial resolution bump, then SUPIR for final skin quality pass. Citation: 2026-04-22-03-flux2-realism-prompting.md Section 8 comparison table. Scope: model-agnostic

8.4 Tiled ControlNet Upscale

Claim: Higher quality than simple upscaling: (1) upscale 2-4x with traditional upscaler, (2) split into overlapping tiles, (3) each tile through img2img at low denoise 0.2-0.4 with ControlNet Tile, (4) blend tiles. ControlNet strength below 0.7, denoise 0.2-0.4. Higher values cause inter-tile inconsistencies. Citation: 2026-04-19-image-gen-techniques-guide.md Tier 3 Tiled ControlNet Upscale section; compositional-patterns.md Section 6 ControlNet tiled upscale subsection. Scope: flux-2-dev


9. Identity Preservation Metrics

Claim: ArcFace (InsightFace buffalo_l) cosine similarity >= 0.80 for same-image refinement. Standard verification threshold is 0.30-0.45 at FMR 1e-4 to 1e-5 (dataset-dependent). Before/after refinement should score 0.85-0.95. Citation: 2026-04-24-facial-realism-refining.md Q6 with footnote [^arcface-threshold] correcting the original document's 0.68 threshold. InsightFace official guide cited, verified 2026-05-13. Scope: model-agnostic

Claim: NTIRE evaluation protocol uses AdaFace (not ArcFace) as identity gate with dataset-specific thresholds (0.3/0.5/0.6). Quality scoring formula: CLIPIQA + MANIQA + (MUSIQ/100) + (Q-ALIGN/5) + max(0, (10-NIQE)/10) + max(0, (100-FID)/100). Citation: 2026-04-24-facial-realism-refining.md Q6 NTIRE section with footnote [^ntire-eval] citing NTIRE 2025 Challenge paper. Verified 2026-05-13. Scope: model-agnostic

Claim: ComfyUI in-pipeline identity metric: ComfyUI_FaceAnalysis (cubiq, 542 stars) computes face embedding distances. Supports InsightFace + AuraFace. Citation: 2026-04-24-facial-realism-refining.md Q6 Automation section with footnote [^faceanalysis]. Verified 2026-05-13. Scope: model-agnostic


10. Prompt Optimization Ecosystem

Claim: Six prompt optimization approaches: (1) dedicated small models (SuperPrompt-v1, Fooocus, Flux-Prompt-Enhance), (2) local LLM via Ollama/LM Studio, (3) local LLM via GGUF/llama.cpp, (4) cloud API LLM, (5) style templates (no AI), (6) wildcards/stochastic variation. Citation: 2026-04-08-02-prompt-optimization.md Section 1 verbal summary. Scope: model-agnostic

Claim: SuperPrompt-v1 is actively harmful for FLUX and Wan. Its output is CLIP keyword soup that the Mistral/T5 encoders cannot properly utilize. Flux-Prompt- Enhance (gokaygokay) is the only dedicated Flux prompt enhancer model. Citation: 2026-04-08-02-prompt-optimization.md Section 3 Approach 1 ("Actively harmful for Flux -- Flux's T5 encoder responds to natural language, and SuperPrompt's output looks like CLIP keyword soup when fed to T5."). Scope: flux-2-dev, wan-2.1, wan-2.2


11. Compositing Pipeline Reference

Claim: Full production pipeline node graph (Sandy-Naomi FLUX.2 Dev Pipeline) combines all compositional patterns: model chain (UNETLoader -> face LoRA 0.7 -> body LoRA 0.6 -> PuLID -> BasicGuider), conditioning chain (CLIPTextEncode -> FluxGuidance 4.0 -> ControlNet depth 0.70 -> BasicGuider), preprocessor chain (LoadImage -> DepthAnythingV2 -> ControlNet), post-decode chain (VAEDecode -> FaceDetailer cfg=1.0 euler simple -> SaveImage). Citation: compositional-patterns.md Combining Patterns section, Full production pipeline node graph. Scope: flux-2-dev

Claim: LoRA strength guidelines: face identity (viv) 0.6-0.8, body style (lux) 0.5-0.7, skin realism 0.4-0.6 (above 0.7 introduces artifacts), stacked (2+) 0.5-0.7 each (never 1.0 on stacked LoRAs). LoRAs on FLUX.2 dev more than double inference time (ComfyUI issue #11058). Citation: compositional-patterns.md Section 1 strength guidelines table and architecture restrictions. Scope: flux-2-dev


Conflict Register

Conflict 1: FLUX.2 Dev img2img Denoise Threshold

compositional-patterns.md and sampling-chains.md describe functional FLUX.2 dev img2img at denoise 0.45 using BasicScheduler. 2026-04-24-facial-realism-refining.md Q4 claims FLUX img2img is "broken below 0.80 without scheduler fix" (BasicSchedulerNormalized or ModelSamplingFluxGradual). The resolution is likely that the facial-realism document's claim applies to raw Flux2Scheduler (which has no denoise param at all), while the working examples use BasicScheduler which inherently handles the denoise schedule correctly. Alternatively, the facial-realism claim may apply to FLUX.1, not FLUX.2. Evidence is insufficient to fully resolve.

Conflict 2: ControlNet Tile Mode Support

compositional-patterns.md lists tile as a supported mode for the alibaba-pai Union model. 2026-04-22-03-flux2-realism-prompting.md Section 9 notes "the HuggingFace model card lists Canny, HED, Depth, Pose, MLSD, Scribble, and Gray -- but not Tile explicitly." Tile support may be implemented at the ComfyUI node level rather than in the model itself.

Conflict 3: FLUX.2 Dev txt2img Guidance Value

sampling-chains.md Section 1 specifies guidance 3.5 for txt2img and 4.0 for img2img. compositional-patterns.md full pipeline uses 4.0 with ControlNet. 2026-04-22-03-flux2-realism-prompting.md says 3.5-4.0 range. Not a hard conflict -- the difference is use-case-dependent (plain txt2img vs img2img vs ControlNet pipelines).

Conflict 4: Skin Realism LoRA FLUX.2 Compatibility

Multiple sources catalog skin realism LoRAs (SRPO, CORE PHYSICS, Photorealistic Skin No Plastic, FluxRealSkin, etc.) but all are trained on FLUX.1. FLUX.2 dev has a different architecture (Mistral-24B encoder vs T5-XXL + CLIP-L). Every source marks these as "needs testing on FLUX.2" or "FLUX.2 compatibility unverified."


Fabrication Screening

Items Flagged as Unverified by Sources

  1. Qwen "index time step zero" technique from AIStudyNow: 2026-04-24-skin-realism-techniques.md footnote [^qwen-timestep-unverified] states: "Could not find a primary source for the 'Qwen index time step zero' technique attributed to AIStudyNow."

  2. Qwen-Image-2512 "9/10 best skin texture" rating: 2026-04-24-facial-realism-refining.md footnote [^qwen-skin-unverified] states: "The specific '9/10' rating and 'best skin texture' quote could not be found."

  3. "8/10 NTIRE 2025 finalists used DiffBIR": 2026-04-24-facial-realism-refining.md footnote [^diffbir-ntire] correction: "The NTIRE 2025 paper confirms multiple top teams used DiffBIR, but the paper does not state a specific count of 8/10."

Corrections Applied by Source Authors

  1. ArcFace threshold corrected from 0.68 to 0.30-0.45 (footnote [^arcface-threshold]).
  2. GPEN architecture corrected from "StyleGAN2 prior" to "GAN trained similarly to StyleGAN" (footnote [^gpen-arch]).
  3. HYPIR ComfyUI node attribution corrected from "11dogzi" to "EricRollei" (footnote [^hypir-comfyui]).
  4. 4xFaceUpDAT training dataset corrected from FFHQ to FaceUp (footnote [^faceupdat]).
  5. SRPO described as "Semantic Relative Preference Optimization" not generic "RL on human preference" (footnote [^srpo]).
  6. CivitAI Anything2Real model ID corrected from version ID to model ID (footnote [^anything2real-id]).
  7. Z-Image denoise range corrected from "0.2-0.3 best" to "0.1-0.3 subtle, 0.4-0.8 stronger" (footnote [^zimage-denoise-correction]).

Report Summary

Claims Breakdown

Category Verified Upstream-Confirmed Cautioned Unverified
Control method taxonomy 18 12 2 0
Prompting principles 16 10 1 0
Sampling chains 14 8 0 0
Qwen-Edit architecture 9 6 0 0
Realism techniques 11 7 2 2
Upscaler evaluation 6 4 0 0
Model logistics 4 2 0 0
Identity metrics 3 3 0 0
Total 81 52 5 2

Key Findings

  1. The FLUX.2 dev sampling chain is fundamentally different from all other architectures: SamplerCustomAdvanced + BasicGuider + FluxGuidance, never KSampler. This is the single most common mistake across sources.

  2. Encoder/VAE/ControlNet components are NOT interchangeable between architectures even within the same model family (Qwen-Image vs Z-Image, FLUX.1 vs FLUX.2). Silent failures with no error messages are the norm for wrong pairings.

  3. FLUX.2 dev's Mistral-24B encoder fundamentally changes prompting from CLIP-era keyword soup to natural language prose. The 512-token budget with ~300-350 usable words, priority ordering, and affirmative-only framing are well-sourced from BFL official documentation.

  4. The Qwen-Image-Edit-2511 dual-conditioning architecture (semantic via VL at 384px

    • appearance via VAE at 1024px) is thoroughly documented with ComfyUI source code citations. Both paths must be wired for identity preservation.
  5. For face upscaling, 4xFaceUpDAT (DAT transformer, zero hallucination) is the recommended identity-safe option over GFPGAN/CodeFormer/SUPIR which all carry hallucination risk.

  6. The skin realism LoRA ecosystem is almost entirely FLUX.1-trained. FLUX.2 compatibility is unverified across the board. This is a material gap.

Suspects (Requiring Caution)

  • FLUX.1 skin LoRA compatibility claims for FLUX.2 dev (all marked unverified)
  • FLUX img2img "broken below 0.80" claim (may apply to FLUX.1 or old scheduler only)
  • Community-sourced PuLID accuracy percentages (benchmark-specific, not official)
  • ControlNet Tile mode on the Union model (empirically working but not on model card)

Citation Verification Footer

Verified 2026-05-18 by 36 Opus 4.6 agents. Full verdicts in verification/agent-*-verdicts.md.

Cite ID Source Verdict Key Finding
cite-010 arXiv:2308.03364 PARTIAL DAT architecture and ICCV 2023 confirmed; face upscaler from community
cite-038 blog.picassoia.com (FLUX realism) CONFIRMED Five realism elements verified; prompting advice confirmed
cite-048 deepwiki.com (FLUX.2 text encoder) SUPPORTED Core specs confirmed; 7680 derived not explicit
cite-051 dev.to (negation prompting trick) SUPPORTED Controlled experiment confirms negation activates concepts
cite-055 docs.bfl.ml/prompting_guide_flux2 PARTIAL Camera names affect rendering; specific models not from BFL
cite-056 docs.bfl.ml/prompting_guide_t2i_negative VERIFIED FLUX.2 no traditional negative prompts confirmed
cite-057 docs.bfl.ml/prompting_summary IMPRECISE Priority ordering supported; URL is index page
cite-062 fal.ai (FLUX.2 prompt guide) PARTIAL 5-tier hierarchy is harvest elaboration of source's 4-tier
cite-063 fal.ai (how to use FLUX) PARTIAL Sony A7R V not in source; film stocks absent
cite-076 github:Phhofm/models VERIFIED 4xFaceUpDAT model source confirmed
cite-134 github:zhengchen1999/DAT VERIFIED Official DAT implementation, ICCV 2023
cite-142 HF blog: FLUX.2 VERIFIED Block architecture claim exact match
cite-161 openmodeldb.info (SkinDiffDetail) VERIFIED ESRGAN architecture confirmed
cite-162 openmodeldb.info (4xFaceUpDAT) VERIFIED DAT architecture, 4x scale, 147.5 MB confirmed
cite-165 renderfire.com (FLUX.2 prompting) UNSUPPORTED General guide; specific claims not from this source
cite-166 sozee.ai (Replicate FLUX) PARTIAL Compares traditional vs Sozee workflows
cite-170 nextdiffusion.ai (Z-Image Turbo tutorial) VERIFIED Comprehensive Z-Image face detail guide

17 citations: 1 CONFIRMED, 2 SUPPORTED, 7 VERIFIED, 5 PARTIAL, 1 UNSUPPORTED, 1 IMPRECISE

Legacy Research Harvest: Inventory Summary

Generated: 2026-05-17 Total artifacts inventoried: 238 Source project: claude-comfyui

Triage Breakdown

Recommendation Count %
harvest 103 43%
skip-operational 87 37%
skip-obsolete 26 11%
needs-review 22 9%

Domain Breakdown

Domain Count
training 115
evaluation 61
inference 23
hardware 14
architecture 11
video 7
captioning 6
tokenization 1

Models Mentioned

Model Appearances
z-image-turbo 151
flux2-dev 62
klein-9b 21
longcat 11
wan-2.2 11
qwen-image 10
wan-2.1 9

Needs-Review Items (22)

These items have ambiguous scope or mixed content. Austin should promote each to harvest or skip.

  • analysis/frontier-job-drafts/2026-05-06-1620-auto-refill-sandy-exp20/README.md: Contains the lower-LR Sandy containment matrix table (EXP17-20 rank/LR/reg grid). The matrix itself is useful methodology but embedded in operational job queueing.
  • analysis/frontier-job-drafts/2026-05-06-fullpack-v3-replacements/README.md: Documents the policy change from subset to full prompt-pack sampling, 1024 resolution, and 500-step cadence. Policy decisions may have reusable training methodology.
  • analysis/frontier-job-drafts/2026-05-06-highres-frontier-start/README.md: Documents the 512-to-1024 resolution transition rationale and starting branch selection for both profiles. Methodology for high-res frontier design.
  • analysis/config-distance/2026-05-06/historical-config-distance-manifest.json: Config distance analysis comparing current and historical runs. Contains structured run metadata with config diffs. May have reusable config comparison methodology.
  • .folio/backlog/2026-05-04-agent-provisioning-behavioral-testing.md: Documents seven distinct agent failure modes during pod provisioning with root cause analysis. Has embedded research context about behavioral eval design, but is primarily infrastructure task-tracking.
  • .folio/backlog/project-reorganization.md: Detailed project restructuring plan with module consolidation table and file-level mapping. The reorganization partially happened (ai-foundry exists). Embedded architecture decisions may be relevant.
  • .folio/backlog/subagent-model-effort-config.md: Documents Claude Code Agent tool limitations for model/effort control, with design for subagent definition files. References GitHub issue anthropics/claude-code#31027. Has embedded research context about the Agent tool API.
  • .folio/backlog/video-dubbing-workflows.md: Three video dubbing workflow variants (InfiniteTalk, I2V+LatentSync, FantasyTalking). LongCat model is listed in AGENTS.md as an active workflow. References archived handoffs with implementation specs.
  • .folio/backlog/2026-05-17/location-cleanup/.claude__worktrees__admiring-sammet-91d0de__.folio__backlog/2026-04-25-regularization-dataset-generation.md: Regularization dataset generation methodology for Sandy face LoRA. While FLUX.2 specific, the general regularization methodology (caption strategy, diversity requirements, generation workflow) may be transferable.
  • .folio/backlog/2026-05-17/location-cleanup/.claude__worktrees__admiring-sammet-91d0de__.folio__backlog/2026-04-26-regularization-dataset-execution.md: Detailed execution plan for regularization dataset generation with ComfyUI workflow design. Pod setup and workflow methodology may be transferable.
  • .folio/backlog/2026-05-17/location-cleanup/.claude__worktrees__admiring-sammet-91d0de__.folio__backlog/deferred-techniques.md: List of deferred experimental techniques with unblock conditions. Short but may reference techniques not yet explored.
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-22-03-flux2-realism-prompting.md: FLUX.2-specific Mistral encoder deep-dive with architecture details. Some prompting principles transfer but the encoder-specific details are FLUX-only.
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-24-facial-realism-refining.md: Klein 9B skin LoRA ecosystem and img2img workflow. Klein-specific details but some skin realism technique knowledge transfers.
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-24-klein9b-face-swap-techniques.md: Klein 9B sequence concatenation face swap architecture with verified tensor shapes. Klein-specific but architectural understanding may transfer.
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-25-klein-9b-face-lora-training-report.md: Klein 9B architecture analysis with detailed parameter comparison tables. Architecture details are Klein-specific but training methodology transfers. Modified 2026-05-03.
  • viv/training/zimage-turbo-research/2026-05-05/05-live-training-monitor-and-batch-headroom.md: Massive live monitoring log with real GPU/VRAM/throughput data, checkpoint timing, and overhead calculations. Contains transferable monitoring methodology but is mostly operational observation data.
  • viv/training/zimage-turbo-research/2026-05-05/30-live-run-conv-signal.md: Contains live conv-on evidence (LoRA module counts, log lines) mixed with operational observation. Conv-on vs conv-off methodology may have transferable value.
  • viv/training/zimage-turbo-research/2026-05-05/34-first-real-run-folder-metadata.md: Metadata schema for durable training run folders (identity fields, aliases, config summary, artifact manifest). May have transferable value for run folder organization.
  • reference/qwen-image-edit-2511-guide.md: Deep Qwen-Image-Edit architecture analysis with dual-conditioning explanation, node chain internals, LoRA ecosystem catalog, prompt engineering for LLM-based encoders. Contains transferable principles (LLM vs CLIP encoding, dual-conditioning, denoise behavior maps) but heavily tied to one model family.
  • reference/zimage-backfill-query-download-design-decisions.md: Design decisions for backfill CLI tool: source-of-truth policy, recovery semantics, error philosophy (fail-fast), batch state as provenance-only. Contains transferable software design principles (idempotent queries, metadata verification, derived vs persisted state) mixed with Z-Image operational specifics.
  • reference/zimage-lora-frontier-heartbeat-procedure.md: Heartbeat agent procedure for keeping training pods productive. Contains transferable frontier exploration methodology (one-axis experiments, visual evidence requirements, queue depth targets, correction handling protocol) but heavily tied to Z-Image pod operations.
  • lora_comparison/docs/README.md: Mixed operational and methodological content. The launch commands, canonical locations, and retro-eval boundary sections are operational. However, the profile-driven evaluation architecture, score durability policy, evaluator metadata auditability, decision brief workflow, and retro-eval manifest indexing describe transferable evaluation framework design. Needs review to separate operational from harvestable content.

Overlap Candidates (89 harvest items overlap with existing ai-foundry refs)

  • analysis/agent-reports/2026-05-06-sandy-parameter-tweaks/01-config-inventory.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • analysis/agent-reports/2026-05-06-sandy-parameter-tweaks/02-result-findings.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/backfill-missing-samples/2026-05-07T01-06-45Z/INDEX.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/completed-run-evals/2026-05-06-1658-zimage-frontier-pass/README.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/completed-run-evals/2026-05-06-1658-zimage-frontier-pass/naomi-v6-promptpack-v2-r32-lr1e4-steps3000.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/completed-run-evals/2026-05-06-1658-zimage-frontier-pass/naomi-v6-promptpack-v2-r32-lr7e5-steps3500.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/completed-run-evals/2026-05-06-1658-zimage-frontier-pass/naomi-v6b-promptpack-v2-r16-lr7e5-steps3500.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/completed-run-evals/2026-05-06-1658-zimage-frontier-pass/naomi-v6b-promptpack-v2-r24-lr7e5-steps3500.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/completed-run-evals/2026-05-06-1658-zimage-frontier-pass/naomi-v6b-promptpack-v2-r32-lr5e5-steps4000.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/completed-run-evals/2026-05-06-1658-zimage-frontier-pass/sandy_EXP18_noconv_r64_lr4e5_res512_reglight.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/completed-run-evals/2026-05-06-1839-zimage-frontier-pass/README.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/completed-run-evals/2026-05-06-1839-zimage-frontier-pass/naomi_v7_fullpack_v3_r16_lr5e5_res1024_steps4000.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/completed-run-evals/2026-05-06-1839-zimage-frontier-pass/sandy_EXP23_fullpack_v3_r64_lr4e5_res1024_reglight.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/prompt-packs/sandy-zimage-v2.yaml overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/prompt-packs/sandy-zimage-v3.yaml overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/prompt-packs/naomi-zimage-v2.yaml overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/prompt-packs/naomi-zimage-v3.yaml overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • analysis/training-run-local-artifact-audits/2026-05-06-lora-runs/INDEX.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • .folio/backlog/z-image-lora-training-pilot.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-08-02-prompt-optimization.md overlaps with: ComfyUI PNG Metadata Reference
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-19-identity-porting-report.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-19-image-gen-techniques-guide.md overlaps with: ComfyUI PNG Metadata Reference
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-24-4xFaceUpDAT-upscaler.md overlaps with: ComfyUI PNG Metadata Reference
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-24-skin-realism-techniques.md overlaps with: ComfyUI PNG Metadata Reference
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-25-klein-9b-body-lora-training-report.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-25-lora-training-guide.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-25-zimage-body-lora-training-report.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-25-zimage-face-lora-training-report.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-26-naomi-hyperparameter-audit.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-26-sandy-flux2-lora-post-mortem.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-27-dop-multi-lora-composability.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-27-multi-gpu-lora-training-investigation.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-27-naomi-lora-v2-optimization.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-27-reg-dataset-composition-research.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-27-sandy-lora-quality-diagnosis.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-28-checkpoint-selection-methodology.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-28-dataset-composition-adversarial-review.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-28-klein-citation-verification.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-28-zimage-adversarial-review.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-04-28-zimage-citation-verification.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-05-03-lr-scheduler-research.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-05-03-postmortem-citation-audit.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-05-03-research-audit-corrections.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-05-03-validation-methodology-audit.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • .folio/research/2026-05-17/location-cleanup/research/2026-05-03-white-balance-decision.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/01-ai-toolkit-yaml-guardrails.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/02-step-lr-checkpoint-sweep.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/03-small-face-evaluation-grid.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • viv/training/zimage-turbo-research/2026-05-05/04-optimizer-batch-rank-risks.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/06-prelaunch-validator-spec.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/08-small-face-evaluation-prompt-rubric.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • viv/training/zimage-turbo-research/2026-05-05/09-batch4-stop-or-stress-test.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/10-training-speed-levers.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/11-batch-size-training-dynamics.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/12-next-run-decision-tree.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/16-dedicated-fact-check-report.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/17-batch-size-plain-english.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/20-next-run-risk-register.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/22-batch4-vs-batch1-comparison-plan.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • viv/training/zimage-turbo-research/2026-05-05/26-batch1-step-ceiling-decision.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/29-noconv-comparison-labeling.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • viv/training/zimage-turbo-research/2026-05-05/31-conv-preset-fact-check.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/33-after-noconv-branch-rules.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • viv/training/zimage-turbo-research/2026-05-05/41-six-sample-triage-set.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • viv/training/zimage-turbo-research/2026-05-05/43-synced-sample-progression-read.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • viv/training/zimage-turbo-research/2026-05-05/44-naomi-zimage-sample-prompts.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • viv/training/zimage-turbo-research/2026-05-06/60-sandy-visual-identity-retention.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • viv/training/zimage-turbo-research/2026-05-06/61-sandy-canary-source-context-and-artifacts.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • viv/training/zimage-turbo-research/2026-05-06/62-sandy-log-config-metrics.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • viv/training/zimage-turbo-research/2026-05-06/63-sandy-next-experiment-synthesis.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • viv/training/zimage-turbo-research/2026-05-06/80-sandy-frontier-visual-analysis.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • viv/training/zimage-turbo-research/2026-05-06/81-sandy-frontier-config-analysis.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • reference/cheap-test-runs.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • reference/model-evaluation-template.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • reference/model-logistics.md overlaps with: ComfyUI PNG Metadata Reference
  • reference/regularization-datasets.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • reference/workflow-catalog.md overlaps with: ComfyUI PNG Metadata Reference
  • .claude/skills/lora-training/references/ai-toolkit-setup.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .claude/skills/lora-training/references/lux_flux2.yaml overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .claude/skills/lora-training/references/operational-footguns.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .claude/skills/lora-training/references/regularization-guide.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .claude/skills/lora-training/references/training-config-guide.md overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .claude/skills/lora-training/references/viv_flux2.yaml overlaps with: Safetensors Metadata Reference, Normalized Training Import Layout, Z-Image LoRA 1536 Parameter Triangulation (+2 more)
  • .claude/skills/comfyui-workflow-builder/references/compositional-patterns.md overlaps with: ComfyUI PNG Metadata Reference
  • .claude/skills/comfyui-workflow-builder/references/prompting-guide.md overlaps with: ComfyUI PNG Metadata Reference
  • .claude/skills/comfyui-workflow-builder/references/sampling-chains.md overlaps with: ComfyUI PNG Metadata Reference
  • lora_comparison/docs/decision-cockpit.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • lora_comparison/docs/prompt-pack-policy.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
  • lora_comparison/docs/retro-eval-policy.md overlaps with: Agentic LoRA Run Evaluation Use Cases (T2), LoRA Packet Calibration Notes, LoRA Packet Pilot Policy (+17 more)
title Model Architecture Research Harvest
source_project claude-comfyui
harvest_date 2026-05-17
domain architecture
verification_level full-academic
claims_total 48
claims_verified 35
claims_unverified 7
claims_suspect 6
conflicts_found 3

Model Architecture Research Harvest

Summary

This document distills verified knowledge about LoRA cross-compatibility rules, tensor shape mismatch patterns, ComfyUI silent failure modes, Z-Image training conventions, and multi-GPU training status from four legacy source files. It also captures transferable project architecture decisions and agent tooling limitations.

The primary source (model-architecture-matrix.md) is a high-quality reference with strong factual grounding. Most architectural claims verify against upstream sources. Key corrections: the FLUX.2 dev text encoder is Mistral-Small-3.2-24B (not 3.1), the VRAM breakdown figures are approximate estimates rather than verified measurements, and the claim that ai-toolkit issue #654 was a "bug fix" oversimplifies what happened (the issue was closed with "resolved in the latest version" but the thread shows training collapse was partly a user-configuration problem and partly a toolkit issue).

The two backlog files (project-reorganization.md, subagent-model-effort-config.md) contain project-management context and agent-tooling observations that are transferable as operational knowledge but not architecture facts.


Verified Findings

1. LoRA Cross-Incompatibility Between FLUX.2 Dev, Klein 9B, and Z-Image Turbo

[model-scope: flux-2-dev, klein-9b, z-image-turbo]

Every pairwise LoRA combination between these three models is incompatible. This is verified through multiple independent lines of evidence:

  • FLUX.2 dev uses hidden_size of approximately 6144 with 8 double blocks and 48 single blocks. Klein 9B uses hidden_size 4096 with 8 double blocks and 24 single blocks. The tensor shapes at every layer differ (e.g., linear2.weight would be [6144, 18432] vs [4096, 16384]).
    • Source: SimpleTuner FLUX2.md (confirms dev: 8+48 blocks, Klein 9B: 8+24 blocks), HuggingFace Klein 9B model card [CITATION-DEPTH: Comfy blog was cited as architecture source but contains no architectural details like hidden_size or block counts -- replaced with SimpleTuner which explicitly tabulates these.]
  • Z-Image Turbo uses an entirely different architecture family: S3-DiT (Scalable Single-Stream DiT) with hidden_size 3840 and fused QKV attention. It has no structural equivalent to the double-block / single-block Rectified Flow Transformer design.

Verdict: VERIFIED. The incompatibility claim is well-supported by architectural differences and community evidence. The specific tensor shapes cited in the source document are plausible given the verified hidden sizes but were not individually confirmed from model config files.

2. ComfyUI Silently Skips Mismatched LoRA Weights

[model-scope: model-agnostic]

When ComfyUI encounters a LoRA with tensor shapes that do not match the loaded model, it logs ERROR lora warnings for each mismatched key but does not crash or raise a user-facing error. Execution continues, and the output is the unmodified base model. This makes failures hard to detect because the image looks "fine" but has zero LoRA influence.

  • Source: ComfyUI issue #11863 documents this behavior and requests fail-fast instead of silent skip. Issue #5868 shows the exact error log pattern: ERROR lora diffusion_model.img_in.weight shape '[3072, 64]' is invalid for input of size 393216.

Verdict: VERIFIED. The silent-skip behavior is a confirmed, documented ComfyUI design choice. The recommendation to always check logs for ERROR lora after loading is sound operational advice.

3. Z-Image Turbo Architecture: S3-DiT with 6B Parameters

[model-scope: z-image-turbo]

Z-Image Turbo uses the Scalable Single-Stream Diffusion Transformer (S3-DiT) architecture with the following verified specifications:

  • 6.15B total parameters

  • 30 transformer layers (single-stream only; no double blocks)

  • Hidden size: 3840 (intermediate: 10240)

  • 30 attention heads (head_dim=128, 30 * 128 = 3840)

  • Fused QKV attention (qkv.weight shape [11520, 3840])

  • Text encoder: Qwen3-4B (hidden_size 2560, 36 layers)

  • 8 NFE (Number of Function Evaluations) for distilled Turbo variant

  • Source: Z-Image arXiv (2511.22699), Z-Image GGUF whitepaper, ZiT LoRA Loader README

Verdict: VERIFIED with correction. The arXiv paper states 32 attention heads, but the actual model weights inspected by the ZiT LoRA Loader show n_heads=30 n_kv_heads=30 head_dim=128 with dim=3840, confirmed by the fused QKV weight shape [11520, 3840] (11520 = 3 * 30 * 128). The model has 30 heads, not 32. [CITATION-DEPTH: arXiv paper says 32 heads, actual model weights show 30 heads -- corrected to match model reality.] The "RMSNorm throughout" claim was not confirmed; the arXiv paper mentions QK-Norm and Sandwich-Norm, and the ZiT loader shows attention_norm and ffn_norm layer weights but does not label the norm type.

4. Train LoRAs on Z-Image Base, Infer on Turbo

[model-scope: z-image-turbo]

Community testing established that LoRAs trained on Z-Image Base produce better results when used for inference on Turbo than LoRAs trained directly on Turbo. Base-trained LoRAs maintain consistency longer than Turbo-trained ones; one zimage.net guide reports a 5-10x longer consistency multiplier, while the other co-cited sources support the directional claim without that exact multiplier. Turbo-trained LoRAs tend to produce overly bright, unnatural colors with lower similarity scores.

Verdict: VERIFIED. Multiple independent community sources confirm this pattern. The source document's claim of "~100% similarity" for base-trained LoRAs on Turbo is overstated (community reports 60-80% effectiveness compared to base inference), but the directional claim is solid.

5. ZiT LoRA Loader Custom Node Requirement

[model-scope: z-image-turbo]

Z-Image LoRAs require the ZiT LoRA Loader custom node (github.com/capitan01R/Comfyui-ZiT-Lora-loader) because LoRAs trained against Z-Image are commonly exported with separate Q, K, V projections (standard diffusers format), but Z-Image's native architecture stores attention as a single fused QKV matrix. The standard ComfyUI LoRA loader cannot bridge this gap, so LoRA weights are loaded but never applied.

Verdict: VERIFIED. The node exists, the problem it solves is real, and the mechanism (Q/K/V fusion) is documented in the repo. This is the same silent- failure pattern as cross-architecture mismatches: weights are loaded but not applied, producing unmodified base model output.

6. Klein 9B Training Collapse in ai-toolkit (Issue #654)

[model-scope: klein-9b]

Klein 9B LoRA training in ai-toolkit produced collapse/garbage output, reported January 2026. The issue was closed March 29, 2026 by jaretburkett (ostris maintainer) with the comment "This should be resolved in the latest version."

  • Source: GitHub issue #654, opened 2026-01-19, closed 2026-03-29.
  • The thread shows multiple users confirming collapse by step 250-500, with comments from users who got better results with ModelScope, Fal.ai, and TensorArt (which do not use ai-toolkit for Klein training).

Verdict: VERIFIED with corrections. The issue existed and was closed. However, the source document's framing ("broken until March 29, 2026") oversimplifies. The thread shows this was a combination of toolkit issues and user configuration problems (some users were training on the distilled model instead of base). The close comment does not specify what changed. The claim of "30+ Klein 9B LoRAs on HuggingFace" as post-fix validation was not independently verified.

7. FLUX.2 Dev Text Encoder: Mistral-Small 24B

[model-scope: flux-2-dev]

FLUX.2 dev uses a Mistral-Small 24B text encoder. The ai-toolkit tokenizer regex bug (issue #613) is real: the tekken.json tokenizer ships a regex pattern that Python's re module rejects.

Verdict: VERIFIED with version correction. See Suspect Claim #1 below for the version discrepancy.

8. Klein 9B Has No Guidance Embedding

[model-scope: klein-9b]

Klein 9B was trained without guidance embedding. The FluxGuidance node has no effect on Klein 9B inference. This is confirmed by SimpleTuner's FLUX2.md quickstart and multiple community/training sources.

  • Source: SimpleTuner FLUX2 quickstart states: "Klein models use the Qwen3 8B text encoder [...] with no guidance embeddings" and "Klein models do not have guidance embeddings. The following guidance options only apply to dev." [RE-ATTRIBUTED: BFL Klein training docs cover LoRA params but not guidance embeddings. SimpleTuner FLUX2.md is the verified source for this claim. See agent-018-verdicts.md, cite-088.]

Verdict: VERIFIED. Multiple upstream sources confirm this. The operational advice to not waste time tuning guidance values on Klein 9B is correct.

9. Klein 9B and FLUX.2 Dev Share the Same VAE

[model-scope: klein-9b, flux-2-dev]

Both models use ae.safetensors. Klein 9B training documentation from musubi- tuner explicitly states to download ae.safetensors from the FLUX.2-dev repository. VAE latents are interchangeable between the two models, but LoRA weights are not.

Verdict: VERIFIED. The shared VAE is well-documented. The distinction between "latents interchangeable" and "LoRA weights incompatible" is important and correctly stated.

10. Klein 9B Uses Qwen3-8B Text Encoder (Not Mistral)

[model-scope: klein-9b]

Klein 9B uses Qwen3-8B as its text encoder, not the Mistral tokenizer. The Mistral tokenizer regex bug (ai-toolkit #613) does not apply to Klein 9B.

Verdict: VERIFIED. This is correct and operationally important: no tokenizer patch needed for Klein 9B training.

11. musubi-tuner Supports Z-Image LoRA Training

[model-scope: z-image-turbo]

musubi-tuner is the recommended tool for Z-Image LoRA training. ai-toolkit has no S3-DiT support. musubi-tuner confirmed Z-Image Base LoRA and fine-tuning support in its January 2026 update.

Verdict: VERIFIED. The tooling landscape claim is accurate. musubi-tuner is the primary Z-Image training tool, with SECourses and Realtime LoRA Trainer as community wrapper alternatives.

12. ai-toolkit Multi-GPU: split_model_over_gpus Provides No Speedup

[model-scope: flux-2-dev]

The existing split_model_over_gpus in ai-toolkit is naive pipeline parallelism that splits the model across GPUs but does not parallelize training steps. It provides no practical training speed benefit.

  • Source: The split functionality is referenced in DeepWiki ai-toolkit advanced features [SOURCE-MOVED: DeepWiki reorganized; this page originally covered ai-toolkit split_model_over_gpus but now covers Wan 2.2 content after wiki regeneration] and ai-toolkit issue #562. The mechanism replaces forward methods with device-aware versions, distributing parameters by estimated memory, which is model-parallel (not data-parallel).

Verdict: VERIFIED. The claim that this provides no practical speed benefit is consistent with how naive pipeline parallelism works for LoRA training where the bottleneck is sequential forward/backward passes, not memory.

13. ai-toolkit FSDP PR #774 is Unmerged

[model-scope: flux-2-dev]

PR #774 implements real multi-GPU training via FSDP v2. It was opened 2026-04-02 by luke9705 and remains open with zero maintainer comments as of verification date.

  • Source: GitHub PR #774. State: open. Comments: 0. The PR description confirms it was "largely tested on Flux.2 DEV" and implements accelerate launch-based distributed training with LoRA params replicated (not sharded) and text encoder offloaded.

Verdict: VERIFIED. The source document stated "Zero maintainer response as of April 2026" which remains accurate. The PR has 2 thumbs-up reactions but no review or merge activity.

14. SimpleTuner Has Native FSDP2 for FLUX.2 Dev

[model-scope: flux-2-dev]

SimpleTuner by bghira has first-class FSDP v2 support, documented in documentation/FSDP2.md. Multi-GPU distributed training is essentially required for FLUX.2-dev in SimpleTuner due to the combined size of the Mistral-3 text encoder and transformer.

Verdict: VERIFIED with qualification. SimpleTuner is a real alternative framework with native FSDP2 support. However, FSDP2 in SimpleTuner is for full-model fine-tuning only, not LoRA. [CITATION-DEPTH: SimpleTuner FSDP2.md states "FSDP2 can only be enabled when model_type is full. PEFT/LoRA style runs continue to use standard single-device paths." The harvest implied FSDP2 was available for all training modes including LoRA -- corrected.]

15. Z-Image Text Encoder Content Filtering and Abliterated Replacements

[model-scope: z-image-turbo]

The Qwen3-4B text encoder has built-in content filtering that blocks certain prompts. Abliterated (filter-removed) replacement encoders exist on CivitAI.

  • Source: CivitAI model 2193783 confirms the abliterated version exists. The description states it was "created with abliteration" to remove refusals.

Verdict: VERIFIED. CivitAI model ID 2193783 confirmed. Model ID 2194550 was not directly verified but is plausible as a variant version.

16. Klein 9B Face Swap is Reference-Conditioned Regeneration

[model-scope: klein-9b]

Klein 9B's face swap mechanism is sequence concatenation: reference image is VAE-encoded, patchified, concatenated to the generation tokens, and processed through full attention in all 8 double blocks and 24 single blocks. Only generation tokens are kept in the output. This is fundamentally different from IP-Adapter, PuLID, or ControlNet approaches.

  • Source: Verified against ComfyUI source code referenced in the original document. The mechanism of torch.cat([img, kontext], dim=1) is consistent with BFL's architecture documentation. The Comfy blog is retained only as product/context evidence for Klein editing capabilities, not as an architecture source.

Verdict: VERIFIED. The architectural mechanism is correct. The distinction from IP-Adapter and PuLID is accurately described.

17. FLUX.2 Dev: 8 Double Blocks, 48 Single Blocks

[model-scope: flux-2-dev]

Confirmed from multiple sources including the HuggingFace diffusers blog and BFL's own documentation.

Verdict: VERIFIED. SimpleTuner FLUX2.md explicitly confirms "56 (8+48)" total blocks for dev. This differs from FLUX.1 which used 19 double / 38 single blocks.


Unverified Claims

These claims appear plausible but could not be confirmed against upstream sources during this verification pass.

U1. FLUX.2 Dev Hidden Size is Exactly 6144

[model-scope: flux-2-dev]

The source document states hidden_size = 6144. This is plausible given 32B total parameters, 48 attention heads, and the known block counts, but the exact number was not found in any public config.json or official specification document. BFL's gated model repository may contain this in the transformer config, but it was not accessible for verification.

U2. Specific Tensor Shape Examples

[model-scope: flux-2-dev, klein-9b]

The source document cites specific shapes like linear2.weight: [6144, 18432] vs [4096, 16384] and Z-Image's [11520, 3840] for fused QKV. These are plausible given the verified hidden sizes and standard MLP expansion ratios (3x for SwiGLU) but were not individually confirmed from model files.

U3. Klein 9B Context Dim is 12288, Attention Heads is 32

[model-scope: klein-9b]

The source document states context_dim = 12288 (labeled "80% of dev") and 32 attention heads (labeled "67% of dev"). Search results reference 4096 hidden dimensions and txt_in projection from 12288 to 4096, which is consistent. The exact head count of 32 was not individually confirmed from a config file.

U4. Klein 9B Training Time ~1 Hour on RTX PRO 6000

[model-scope: klein-9b]

Plausible given the 3x smaller parameter count compared to FLUX.2 dev, but this is project-specific to Austin's hardware and dataset configuration. Not independently verifiable without matching conditions.

U5. FLUX.2 Dev Training Cost ~$8-10 Per Run

[model-scope: flux-2-dev]

This is project-specific to Austin's RunPod usage (RTX PRO 6000 Blackwell pricing, ~3+ hours per run). Plausible but specific to runtime and pricing at time of writing. Not a universal fact.

U6. 30+ Klein 9B LoRAs on HuggingFace Post-Fix

[model-scope: klein-9b]

The claim that "Post-fix validation: 30+ Klein 9B LoRAs exist on HuggingFace" was not independently verified by counting. HuggingFace search does show multiple Klein 9B LoRA models, but the exact count was not confirmed.

U7. FLUX.2 Dev VRAM Breakdown: 64.4 GB DiT + 35.6 GB Text Encoder

[model-scope: flux-2-dev]

The source document claims 64.4 GB DiT + 35.6 GB text encoder at bf16. The official docs state even an H100 (80 GB) cannot hold text encoder + transformer

  • VAE simultaneously in bf16, and the remote text encoder saves ~20 GB VRAM. The specific breakdown values (64.4 + 35.6 = 100 GB) are internally consistent but the individual component numbers were not confirmed from any official source. The ~100 GB total is plausible but not precisely documented.

Suspect Claims

These claims contain errors, exaggerations, or misleading framing that should be corrected before use.

S1. Text Encoder Version: "Mistral-Small-3.1-24B" (AMBIGUOUS)

[model-scope: flux-2-dev]

The source document states the text encoder is "Mistral-Small-3.1-24B." Multiple upstream sources including DeepWiki's analysis of the black-forest-labs/flux2 repository identify the text encoder as Mistral-Small-3.2-24B (specifically Mistral-Small-3.2-24B-Instruct-2506). The ai-toolkit issue #613 references Mistral-Small-3.1-24B-Instruct-2503 in its bug report from December 2025, which was the version at time of FLUX.2's initial release. BFL appears to have updated to 3.2 in a subsequent release.

Correction: The version is contested across sources. SimpleTuner FLUX2.md (bghira, actively maintained) says "Mistral-Small-3.1-24B." DeepWiki analysis of the BFL repo references "Mistral-Small-3.2-24B-Instruct-2506." ai-toolkit issue #613 references "Mistral-Small-3.1-24B-Instruct-2503." The initial FLUX.2 release likely used 3.1; BFL may have updated to 3.2 in a subsequent release. Both versions may be correct for different model checkpoints. [CITATION-DEPTH: original correction claimed "3.1 is stale" but SimpleTuner's current FLUX2.md still says 3.1 -- ambiguity preserved rather than forcing a single answer.]

S2. "~100% similarity" for Base-Trained LoRAs on Turbo

[model-scope: z-image-turbo]

The source document states: "Base-trained LoRAs on Turbo inference achieve ~100% similarity to the training images." Community sources report 60-80% effectiveness for base-trained LoRAs on Turbo, with character/face LoRAs at 50-70%. The directional claim is correct (base-trained > turbo-trained), but "~100% similarity" is an exaggeration.

Correction: Base-trained LoRAs work better on Turbo than Turbo-trained LoRAs, but expect reduced effectiveness (60-80% for style, 50-70% for face identity) compared to base inference.

S3. Issue #654 "Bug Fix" Framing Oversimplifies

[model-scope: klein-9b]

The source document states: "Klein 9B LoRA training was broken in ai-toolkit until March 29, 2026 (issue #654)." The actual issue thread reveals a more nuanced picture:

  • Some users were training on the distilled model instead of base (user error)
  • Other training frameworks (ModelScope, Fal.ai, TensorArt) produced better results with the same datasets during the same period
  • The close comment says "resolved in the latest version" without specifying what changed
  • The issue was a combination of toolkit bugs and user configuration problems

Correction: Klein 9B training had significant issues in ai-toolkit, likely involving both toolkit bugs and user confusion about distilled vs base models. The fix was not a single identifiable bug fix but rather improvements in "the latest version."

S4. Z-Image VAE Labeled "qwen_image_vae (different)" in Architecture Table

[model-scope: z-image-turbo]

The source document's architecture table says Z-Image uses "qwen_image_vae (different)" as its VAE. Z-Image does use a different VAE from FLUX models (it is not ae.safetensors), but the name "qwen_image_vae" was not found in any upstream Z-Image documentation. The actual VAE file naming may differ.

Correction: Z-Image uses a different VAE than FLUX.2 dev / Klein 9B, but the exact canonical name "qwen_image_vae" should be verified against the actual model repository before use.

S5. Source Document Claims ai-toolkit Issue #654 Fix Date of "March 29, 2026"

[model-scope: klein-9b]

The close date is confirmed as March 29, 2026. However, the source document frames this as the date a specific fix was merged. The closing comment says "This should be resolved in the latest version" which could mean the fix landed days or weeks earlier. The date is the issue-close date, not necessarily the fix-merge date.

Minor concern: The distinction matters if someone is checking whether their ai-toolkit installation includes the fix. The close date is the issue triage date, not a commit date.

S6. Klein 9B VAE Latent Channels Claimed as 128

[model-scope: klein-9b]

The Klein face swap research document states: "VAE encodes the reference to [1, 128, H, W] latents (128 channels, not 16 like standard Flux)." This claim about 128 channels for the VAE encoding needs careful distinction: this likely refers to pre-patchification spatial latent dimensions, not the VAE's native channel count. The standard FLUX VAE (ae.safetensors, shared between FLUX.2 dev and Klein) uses 16 channels. The "128" may be a misunderstanding of the post-VAE representation or an artifact of internal tensor reshaping for Klein's patchification step.

Correction: The FLUX VAE natively outputs 16 channels. The 128-channel tensor referenced is likely a post-processing artifact of the patchification step. This claim should not be used as-is without verification against the actual model code.


Conflicts with Knowledge Map

C1. Text Encoder Version

The knowledge map entry for AGENTS.md does not specify the text encoder version. However, the legacy source document's claim of "Mistral-Small-3.1-24B" conflicts with upstream documentation showing Mistral-Small-3.2-24B. If this version string is used anywhere in ai-foundry code or configuration, it should be updated.

C2. VRAM Figures

The knowledge map entry for AGENTS.md states "96 GB VRAM" for the RTX PRO 6000 runtime. The source document claims FLUX.2 dev at bf16 requires ~100 GB (DiT + TE), which exceeds the available 96 GB. This is consistent with the AGENTS.md note: "Use fp8mixed only when model physically exceeds VRAM (e.g., FLUX.2 dev full bf16 = 100 GB > 96 GB)." No conflict, but the specific breakdown figures (64.4 GB DiT + 35.6 GB TE) remain unverified.

C3. Z-Image Turbo Parameter Count

The source document states "6B" for Z-Image Turbo. The arXiv paper says 6.15B. The knowledge map does not specify a parameter count for Z-Image Turbo. Minor discrepancy (rounding), not a functional conflict.


Transferable Project Architecture Decisions

From project-reorganization.md:

  • Monorepo with single pyproject.toml. The legacy project chose a single pyproject.toml with 4 entry points over uv workspaces. The rationale was "one big ball" is simpler than workspace coordination. This decision carried forward into ai-foundry's loralab package structure.
  • Circular dependency breaking via model extraction. The legacy lora_training and lora_comparison had 43 import edges on shared types. The solution was extracting 22 frozen dataclasses into a shared models/ package with zero dependencies. This pattern is relevant if loralab develops similar coupling.
  • Step-based checkpoint layout. The normalized layout uses step-000500/model.safetensors + step-000500/samples/. This is now implemented in ai-foundry's training import layout.

From subagent-model-effort-config.md:

  • Agent tool model parameter was removed between versions 2.1.66 and 2.1.69. The parameter previously accepted sonnet, opus, haiku but was dropped from the Agent tool schema. This regression is documented in claude-code issue #31027 (now closed). The issue detailed the schema diff showing model present in 2.1.66 but absent in 2.1.69. [CITATION-DEPTH: harvest claimed the tool "accepts" the model parameter; the issue actually reports the parameter was removed -- corrected.]
  • Subagent inherits parent model if not specified. A session running Sonnet silently spawns Sonnet subagents unless the caller explicitly overrides.
  • No built-in way to verify subagent model/effort. The only option is reading raw JSONL transcripts at ~/.claude/projects/{hash}/{session}/subagents/agent-{id}.jsonl.

Sources Consulted

Upstream Documentation

GitHub Issues and PRs

Training Tools

Custom Nodes

Community Testing

Agent Tooling

Citation Verification Footer

Verified 2026-05-18 by 36 Opus 4.6 agents. Full verdicts in verification/agent-*-verdicts.md.

Cite ID Source Verdict Key Finding
cite-026 arXiv:2511.22699 SUPPORTED* Z-Image S3-DiT specs confirmed; paper says 32 heads, weights show 30
cite-034 bfl.ai/blog/flux-2 SUPPORTED Latent flow matching, Mistral-3 VLM, model variants confirmed
cite-036 blog.comfy.org (Klein 4B/9B) PARTIAL Product announcement; no architectural details attributed to it
cite-041 civitai: Z-Image abliterated Qwen3 SUPPORTED Abliterated encoder exists
cite-042 civitai: Z-Image abliterated Qwen3 SUPPORTED Abliteration with refusal removal confirmed
cite-043 civitai: ZiT LoRA Loader SUPPORTED QKV fusion solver for Z-Image LoRAs
cite-046 comfyui.dev (checkpoints guide) UNSUPPORTED Generic VAE advice, not shared-VAE claim
cite-047 deepwiki.com (FLUX.2 text encoder) UNSUPPORTED Architecture overview, defers to section 3.2
cite-049 deepwiki.com (ai-toolkit) SUPPORTED Valid general reference
cite-050 deepwiki.com (ai-toolkit advanced) UNSUPPORTED Covers Wan 2.2 now; content moved
cite-054 docs.bfl.ai (Klein training) SUPPORTED No guidance embeddings mention; SimpleTuner is source
cite-064 flux-2.dev INACCESSIBLE HTTP 451; claim verified via co-cited HF blog
cite-068 Comfy-Org/ComfyUI#11863 SUPPORTED Fail-fast request and silent-skip behavior documented
cite-069 Comfy-Org/ComfyUI#5868 SUPPORTED Verbatim error string match
cite-086 anthropics/claude-code#31027 VERIFIED Schema regression evidence confirmed
cite-087 bghira/SimpleTuner source VERIFIED First-class support and LoRA exclusion confirmed
cite-088 bghira/SimpleTuner FLUX2.md VERIFIED Block counts, guidance, multi-GPU confirmed
cite-089 black-forest-labs/flux2 VERIFIED Canonical upstream source
cite-090 black-forest-labs/flux2 source VERIFIED File exists with upstream documentation
cite-092 capitan01R/Flux2Klein-Enhancer VERIFIED Active repo with Klein architecture evidence
cite-093 capitan01R/Comfyui-ZiT-Lora-loader VERIFIED Claims exactly supported
cite-094 Comfy-Org/ComfyUI VERIFIED Claim accurately stated; old comfyanonymous URL redirects
cite-100 kohya-ss/musubi-tuner VERIFIED Z-Image support documented, Jan 2026 timing matches
cite-101 kohya-ss/musubi-tuner source VERIFIED All sub-claims confirmed
cite-102 kohya-ss/musubi-tuner source VERIFIED Z-Image support; ai-toolkit negative needs separate check
cite-114 ostris/ai-toolkit#562 PARTIAL Issue exists but no technical content for parallelism claims
cite-116 ostris/ai-toolkit#613 VERIFIED Issue accurately represented
cite-117 ostris/ai-toolkit#654 VERIFIED Unusually thorough; self-auditing of suspect claims
cite-121 ostris/ai-toolkit#774 VERIFIED All factual sub-claims confirmed
cite-137 HF: Tongyi-MAI/Z-Image#D18 VERIFIED Base-trained LoRAs outperform Turbo-trained confirmed
cite-139 HF: FLUX.2-dev PARTIAL Top-level identity confirmed; some details from other sources
cite-140 HF: FLUX.2-klein-9B PARTIAL Model identity confirmed; some details not in model card
cite-142 HF blog: FLUX.2 VERIFIED Block architecture claim exact match
cite-151 HF: Mistral-Small discussion #84 VERIFIED Tokenizer regex bug confirmed
cite-153 lilting.ch (Z-Image LoRA) PARTIAL Base-trained superior confirmed; no 5-10x figure
cite-168 apatero.com (Z-Image compatibility) PARTIAL Dedicated guide; supports directional claim
cite-178 z-image.me (GGUF whitepaper) PARTIAL Marketing-oriented; not detailed architecture doc
cite-179 zimage.net (training guide) VERIFIED Base-trained advocacy and 5-10x consistency multiplier confirmed

38 citations: 8 SUPPORTED, 1 SUPPORTED*, 18 VERIFIED, 7 PARTIAL, 3 UNSUPPORTED, 1 INACCESSIBLE

title Training Experiments Research Harvest
source_project claude-comfyui
harvest_date 2026-05-17
domain training
verification_level full-academic
claims_total 46
claims_verified 38
claims_unverified 6
claims_suspect 2
conflicts_found 3

Training Experiments Research Harvest

Distilled from 21 source files in the legacy claude-comfyui project covering Z-Image Turbo LoRA training experiments conducted 2026-05-05 through 2026-05-06. Focus: experiment design methodology, batch size dynamics, optimizer/rank/resolution interaction effects, monitoring methodology, fact-check methodology, and convergence-on vs convergence-off findings. The methodology transfers beyond Z-Image Turbo.

Summary

The legacy research documents a systematic LoRA training experiment campaign on the Z-Image Turbo model using the Ostris training adapter (v2) in ai-toolkit. The work spans 25+ distinct experiment configs (EXP01 through EXP25 for Sandy, plus Naomi variants) across rank (16, 32, 48, 64, 128), learning rate (1e-4, 5e-5, 4e-5, 3e-5), training resolution (512, 512/768 mixed, 768, 1024), batch size (1, 4), conv LoRA (on/off), and regularization (none, light). The most transferable findings are the experiment design methodology, the batch-size exposure calculus, the prelaunch validator spec, the risk register pattern, and the dedicated fact-check methodology.


Verified Findings

1. ai-toolkit YAML Architecture Key Normalization

Claim: The ai-toolkit UI preset writes model.arch: zimage:turbo into exported YAML, but runtime ModelConfig strips the colon suffix before model dispatch, so zimage:turbo becomes zimage at runtime. The Z-Image model class declares arch = "zimage". If no class matches, get_model_class() falls back to legacy StableDiffusion.

Verification: Confirmed against pinned ai-toolkit commit 963a9f42b2.

  • ZImageModel.arch = "zimage" at z_image.py#L40-L42 (source)
  • ModelConfig strips colon tag at config_modules.py#L712-716: if ':' in self.arch: self.arch = self.arch.split(':')[0]
  • get_model_class() exact-matches then falls back at get_model.py#L44-50 (source)
  • UI preset is zimage:turbo at options.ts#L594-616 (source)

Note: Early research notes (01, 06) overstated this as a hard runtime failure. The dedicated fact-check (note 16) corrected this: zimage:turbo is a normalization/style issue for hand-written YAML, not a runtime dispatch failure.

Sources: 01-ai-toolkit-yaml-guardrails.md:9-11, 16-dedicated-fact-check-report.md:9-27, ai-toolkit source at pinned commit.

2. assistant_lora_path Is the Only Runtime Adapter Key

Claim: The runtime config field for loading the Z-Image training adapter is model.assistant_lora_path, not model.training_adapter. Using training_adapter silently skips adapter loading.

Verification: Confirmed.

  • ModelConfig reads assistant_lora_path at config_modules.py#L585-613 (source)
  • Z-Image loads adapter only when assistant_lora_path is not None at z_image.py#L173-175 (source)
  • UI field label is "Training Adapter Path" mapped to assistant_lora_path at SimpleJob.tsx#L293-303

Sources: 01-ai-toolkit-yaml-guardrails.md:24-25, 06-prelaunch-validator-spec.md:ZT-YAML-007/008.

3. Adapter Card Warning: Long Runs Degrade Distilled Behavior

Claim: The Ostris adapter card explicitly warns that direct training on a step-distilled model breaks distillation quickly, and the adapter only slows this breakdown for shorter runs (styles, concepts, characters). Long runs still risk artifact production when the adapter is removed.

Verification: Confirmed via ostris/zimage_turbo_training_adapter. The card states: "When you train directly on a step distilled model, the distillation breaks down very quickly" and "doing a long training run will likely lead to the distillation breaking down to a point where artifacts will be produced when the adapter is removed."

Sources: 02-step-lr-checkpoint-sweep.md:16, 04-optimizer-batch-rank-risks.md:31, adapter model card.

4. Z-Image Turbo Model: 8 NFEs, Guidance 0, Not Fine-Tunable

Claim: Z-Image Turbo uses 8 DiT forwards (NFEs), guidance_scale=0.0 for Turbo inference, and the model card marks Turbo as not fine-tunable (fine-tuning is N/A) while Z-Image Base is marked as "Easy" to fine-tune.

Verification: Confirmed via Tongyi-MAI/Z-Image-Turbo. The card specifies num_inference_steps=9 (resulting in 8 DiT forwards), guidance_scale=0.0, and explicitly says "Guidance should be 0 for the Turbo models." Fine-Tunability is listed as "N/A" for Turbo but "Easy" for Base.

Note: The legacy research used guidance_scale: 1 and sample_steps: 8 for training-time samples, following ai-toolkit UI defaults, not the model card's inference guidance. This is a deliberate ai-toolkit choice for training sample generation, not an error.

Sources: 02-step-lr-checkpoint-sweep.md:15-17, Z-Image-Turbo model card.

5. Batch Size Exposure Calculus

Claim: At fixed optimizer steps, increasing batch size proportionally increases total image exposures. The formulas are:

effective_batch_size = batch_size * gradient_accumulation
image_exposures = steps * effective_batch_size
dataset_passes = image_exposures / dataset_size

For Sandy's 82-image dataset: batch 4, steps 5000 = 20,000 exposures (243.9 passes). batch 1, steps 5000 = 5,000 exposures (61.0 passes). batch 1, gradient_accumulation 4, steps 5000 = 20,000 exposures (not batch-1 exposure).

Verification: Confirmed against ai-toolkit source and standard ML principles.

  • TrainConfig reads batch_size at config_modules.py#L371, reads gradient_accumulation at #L438-441
  • Outer loop counts steps and appends gradient_accumulation batches per step at BaseSDTrainProcess.py#L2128-2224
  • Bucketed datasets form batches internally at data_loader.py#L640-718
  • Arithmetic is standard: 82 images / 82 = 1 pass, confirmed by multiple sources

Key insight: batch_size: 1 + gradient_accumulation: 4 preserves effective batch 4 and 20,000 exposures. It reduces peak VRAM and avoids batch-shape bugs, but does NOT reduce exposure. Agents must not describe this as "batch-1 exposure."

Sources: 11-batch-size-training-dynamics.md:13-27, 17-batch-size-plain-english.md:14-23, 09-batch4-stop-or-stress-test.md:49-64.

6. lr Key Required, Not learning_rate

Claim: ai-toolkit TrainConfig reads self.lr = kwargs.get('lr', 1e-6). Using learning_rate instead silently falls back to 1e-6.

Verification: Confirmed.

  • TrainConfig at config_modules.py#L359 reads lr
  • GitHub issue #751: reporter identified TWO independent root causes:
    1. Config key confusion: TrainConfig (config_modules.py:357) accepts lr, not learning_rate. Using learning_rate silently defaults to 1e-6.
    2. Zimage code path bug: Even with the correct lr key, the zimage adapter does not pass default_lr to prepare_optimizer_params(), unlike other adapters (control_lora, subpixel, mean_flow, i2v). Both bugs independently cause the optimizer to use 1e-6 regardless of config. [CORRECTED: original harvest attributed only key-name confusion; re-verification found zimage-specific bug. Deep verification found both are real independent issues. See agent-024-verdicts.md, cite-119.]

Sources: 04-optimizer-batch-rank-risks.md:47-49, 16-dedicated-fact-check-report.md:47-49.

7. Conv LoRA: UI Hides, Runtime Accepts

Claim: The Z-Image Turbo UI preset hides network.conv via disableSections: ['network.conv'], but the runtime accepts both omitted conv (defaults to None, skips Conv2d LoRA creation) and explicit conv values. Conv on vs conv off is a real experiment axis, not a validity axis.

Verification: Confirmed.

  • UI disables conv section at options.ts#L615
  • NetworkConfig defaults conv to None at config_modules.py#L182
  • LoRA creation skips Conv2d when conv_lora_dim is None at lora_special.py#L306-308 and #L421-430
  • Live run log confirmed conv applied: "apply LoRA to Conv2d with kernel size (3,3). dim (rank): 16, alpha: 16" and "create LoRA for U-Net: 240 modules"

Sources: 31-conv-preset-fact-check.md:9-13, 30-live-run-conv-signal.md:9-16.

8. Cached Text Embeddings Disable Caption Dropout

Claim: When cache_text_embeddings: true, ai-toolkit skips caption_dropout_rate and token_dropout_rate. Enabling text embedding caching in a config with active caption dropout (e.g., 0.05) silently changes training semantics.

Verification: Confirmed.

  • Source applies dropout only when dataset_config.cache_text_embeddings is false
  • Active manual config had caption_dropout_rate: 0.05 with cache_text_embeddings: false, making dropout active
  • Prepared first-pass config had caption_dropout_rate: 0 with caching on, making the semantic split real

Sources: 04-optimizer-batch-rank-risks.md:73-76, 10-training-speed-levers.md:61-70, 16-dedicated-fact-check-report.md:75-84.

9. Batch > 1 with Z-Image Text Embeddings: Known Fragile Path

Claim: GitHub issue #554 documents Z-Image Turbo failures with batch_size > 1: "Batch size of latents must be the same or half the batch size of text embeddings." PR #649 attempted a fix but was closed unmerged.

Verification: Confirmed.

  • Issue #554 documents the error
  • PR #649 was closed with mergedAt: null, not an ancestor of the pinned commit
  • However, empirical evidence showed batch 4 running successfully at 512px with cache_text_embeddings: false, reaching step 500+ without the error

Sources: 04-optimizer-batch-rank-risks.md:36-44, 09-batch4-stop-or-stress-test.md:90-96.

10. Z-Image Flowmatch Scheduler with Shift 3.0

Claim: Z-Image uses CustomFlowMatchEulerDiscreteScheduler with shift: 3.0 for both training and sampling.

Verification: Confirmed at z_image.py#L12-14 and #L33-62.

Sources: 01-ai-toolkit-yaml-guardrails.md:29.

11. OOM Handling: Three Consecutive Strikes Abort

Claim: ai-toolkit catches CUDA OOM, skips the batch, zeroes gradients, and flushes CUDA memory. After more than three consecutive OOMs, it raises RuntimeError("OOM during training step 3 times in a row, aborting training").

Verification: Confirmed in the training loop at the pinned source commit.

Sources: 09-batch4-stop-or-stress-test.md:66-69.

12. Quantization Disables Network Merge-In

Claim: BaseSDTrainProcess disables network merge-in when model.quantize or model.layer_offloading is true. The Z-Image loader also rewrites qtype to float8 when an assistant LoRA is present with qfloat8.

Verification: Confirmed. Source comment notes merging into quantized weights is unresolved. A PR #649 contributor warned quantization made results "very bad" in batch-size testing.

Sources: 04-optimizer-batch-rank-risks.md:86-93, 10-training-speed-levers.md:42.

13. Community Guide Recommendations for Z-Image Turbo LoRA

Claim: A Hugging Face community guide recommends approximately 3000 steps for 5-15 images, batch 1-2, LR 1e-4 to 5e-5, rank 8-16, 1024 resolution, and periodic samples every 200-300 steps.

Verification: Confirmed via content-and-code/training-a-lora-for-z-image-turbo. The guide recommends ~3000 steps, batch 1-2, LR 1e-4 to 5e-5, rank 8-16, 1024 resolution, and sample every 250 steps.

Note: This is a community guide, not the model author's recommendation. Reliability is medium. The guide covers 5-15 image datasets, not 82-image identity datasets.

Sources: 02-step-lr-checkpoint-sweep.md:22-23.

14. Gradient Accumulation vs Direct Batch: Mechanically Close, Not Identical

Claim: batch_size: 4, gradient_accumulation: 1 and batch_size: 1, gradient_accumulation: 4 both produce effective batch 4 per optimizer update, but they differ in: peak activation memory, microbatch execution path, loss logging, numerical precision effects, and cached embedding shape assumptions.

Verification: Confirmed against ai-toolkit source and the Hugging Face Accelerate gradient accumulation guide (docs). The guide confirms gradient accumulation "accumulates gradients over several batches, and only stepping the optimizer after a certain number of batches." ai-toolkit's outer loop appends gradient_accumulation batches before the training hook.

Sources: 09-batch4-stop-or-stress-test.md:72-88, 11-batch-size-training-dynamics.md:96-115.

15. Large-Batch Generalization Gap

Claim: Keskar et al. found that large-batch training tends to converge toward sharp minima, leading to poorer generalization. Goyal et al. showed large minibatches can work with linear LR scaling and warmup. Smith et al. showed batch size growth can substitute for learning rate decay.

Verification: All three papers confirmed.

  • Keskar et al., "On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima" (arxiv:1609.04836): confirmed title and thesis about sharp minima
  • Goyal et al., "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour" (arxiv:1706.02677): confirmed linear scaling rule and warmup
  • Smith et al., "Don't Decay the Learning Rate, Increase the Batch Size" (openreview:B1Yy1BxCZ): confirmed batch-size/LR-decay equivalence thesis

Note: These are general deep learning results. The legacy notes correctly caution against directly applying ImageNet-scale linear LR scaling to 82-image identity LoRAs on a distilled diffusion model.

Sources: 11-batch-size-training-dynamics.md:73-77,131-136.

16. LoRA Paper: Parameter-Efficient Adaptation

Claim: Hu et al. LoRA paper demonstrates low-rank adaptation that freezes base model weights and injects trainable rank decomposition matrices, reducing trainable parameters by 10,000x.

Verification: Confirmed via arxiv:2106.09685. Title: "LoRA: Low-Rank Adaptation of Large Language Models." Confirms freezing pretrained weights and injecting trainable rank decomposition matrices.

Sources: 11-batch-size-training-dynamics.md:140-141.


Verified Methodology Findings

M1. Prelaunch Validator Pattern

The research produced a comprehensive prelaunch YAML validator specification with 35 named check IDs (ZT-YAML-001 through ZT-YAML-035) and 6 runtime log evidence checks (ZT-LOG-001 through ZT-LOG-006), organized by severity (error, warn, info, runtime-error, runtime-warn, runtime-info).

Key transferable design elements:

  • Separate prelaunch YAML validation from runtime log evidence. YAML errors must not be retroactively cleared by runtime evidence.
  • Distinguish UI-facing presets from runtime architecture values (UiModelPreset vs RuntimeModelArch).
  • Require save_every == sample_every so each checkpoint has matching samples.
  • Check for silent no-op keys (e.g., training_adapter instead of assistant_lora_path).
  • Emit findings as stable IDs with path, actual value, expected predicate, severity, and source reference.

Sources: 06-prelaunch-validator-spec.md:1-136.

M2. Risk Register Pattern

A structured risk register with columns: Risk, Likelihood, Impact, Detection signal, Mitigation, Source notes. Applied to the proposed batch-1 next run with 28 distinct risk rows covering launch YAML, adapter, dataset, trigger, underfit, conv artifacts, resolution, LR, step ceiling, sample grid, cadence overhead, caching, dropout, latent cache, gradient checkpointing, quantization, optimizer key, save/sample mismatch, runtime instability, sync gaps, guidance lanes, and checkpoint selection bias.

Key transferable design elements:

  • Highest-priority prelaunch gates listed separately from the full register.
  • Every risk row cites the source research notes that motivated it.
  • Detection signals are concrete log patterns or metric thresholds, not vague descriptions.
  • The register distinguishes "this would change the experiment" from "this would break the run."

Sources: 20-next-run-risk-register.md:1-37.

M3. Decision Tree for Checkpoint Evaluation

A structured decision tree for post-checkpoint evaluation with branching on: broken runtime, weak/generic identity, early identity with overfit/bleed, strong close-ups but weak small-face, and clean small-face winner. Each branch specifies: immediate action (stop/continue), and next run configuration.

Key transferable elements:

  • Do not select checkpoints by final step or prettiest close-up.
  • Rank by small-face identity first, canary cleanliness second, source-context leakage third, naturalness fourth, close portraits last.
  • Stop-early criteria: canary bleed, source copying, plasticity, or close-up improvement without small-face improvement.

Sources: 12-next-run-decision-tree.md:37-69.

M4. Dedicated Fact-Check Methodology

The research produced a dedicated artifact-by-artifact fact-check report (note 16) that systematically verified each earlier note against pinned source code, runtime logs, and live pod state. This caught the highest-impact correction: the zimage:turbo runtime dispatch claim was overstated in 5 earlier notes.

Key transferable elements:

  • Pin a specific upstream commit for reproducible verification.
  • Use GitHub CLI (gh) for source reading, not training-data assumptions.
  • Classify each claim as: Confirmed, Contradicted, Stale/live-state-dependent, Unsupported/inferential, or Judgment/opinion.
  • Track confusion flags separately: identify the top 2-3 systematic confusions across the note set.
  • Include a "Future Live-Verification Checklist" for the next agent session.

Sources: 16-dedicated-fact-check-report.md:1-624.

M5. Experiment Isolation Methodology

The experiment campaign followed a sequential isolation discipline:

  1. First, prove the adapter path works (smoke runs).
  2. Then isolate batch size (batch 4 vs batch 1).
  3. Then isolate conv LoRA (conv-on vs conv-off).
  4. Then isolate resolution (512 vs mixed 512/768 vs 768 vs 1024).
  5. Then isolate regularization (no-reg vs light-reg, controlling for confounds like subject repeats).
  6. Then isolate learning rate (1e-4 vs 5e-5 vs 4e-5 vs 3e-5).
  7. Then test rank (16, 32, 48, 64, 128).

The key rule: one branch, one reason. Do not change LR, rank, buckets, and caption/cache together.

Sources: 33-after-noconv-branch-rules.md:41-46, 01-config-inventory.md:80-87.

M6. Speed Levers Classification

Clean separation of training speed levers into two categories:

Do not invalidate experiment: sample cadence, sample resolution, sample count, skip baseline sample, save cadence, max_step_saves_to_keep, cache_latents_to_disk, gradient_checkpointing.

Change training semantics: batch size, training resolution, rank, conv LoRA, quantization, quantize_te, low_vram, cache_text_embeddings (when caption dropout > 0).

Sources: 10-training-speed-levers.md:46-134.

M7. Monitoring Methodology

Live monitoring with structured observation entries containing: health status, process identification, GPU metrics (VRAM used/total, utilization, power, temp), progress (step/total, elapsed, it/s, loss), sample/checkpoint inventory, error signature grep, and assessment/recommendation.

The monitoring detected a mid-session job transition from sandy default settings_copy to sandy 2026-05-05 06:27:14 and correctly labeled stale heartbeat entries as stale-target artifacts rather than evidence of run failure.

Sources: 05-live-training-monitor-and-batch-headroom.md:1-200.


Verified Experiment Results

E1. Run Inventory Summary

The campaign produced 25+ distinct Sandy experiment configs:

Range Variables tested Key finding
EXP01-06 Conv off, rank 16/32/64/128, LR 1e-4/5e-5, resolution 512 vs 512/768 Identity can bind without conv; rank 128 is over-forceful and bleed-prone
EXP07-14 Frontier: rank 32/64, LR 5e-5/3e-5, resolution 512/512+768/768, regularization 5e-5 is the practical LR frontier; pure 768 not clearly worth runtime cost; EXP10 reg confounded by 4x repeats; EXP14 is clean reg isolation
EXP16-20 Lower LR (4e-5), rank 48/64, light regularization Completes the containment matrix at the lower-LR branch
EXP21-25 High-res (1024), full prompt-pack v3 sampling Resolution frontier with 500-step cadence and complete prompt packs

Sources: 01-config-inventory.md:34-78.

E2. Conv-On Produces Trainable Modules, Not Dead Config

Claim: When network.conv: 16 and network.conv_alpha: 16 are present, the runtime creates Conv2d LoRA modules. Log evidence showed "create LoRA for U-Net: 240 modules" with conv vs "create LoRA for U-Net: [fewer] modules" without.

Verification: Live run log confirmed: "apply LoRA to Conv2d with kernel size (3,3). dim (rank): 16, alpha: 16" and "create LoRA for U-Net: 240 modules."

Sources: 30-live-run-conv-signal.md:60-82.

E3. VRAM Headroom at Batch 4, 512px, BW6000

Claim: Batch 4 at 512px training on the RTX PRO 6000 Blackwell (96 GB) used approximately 24.7 GiB out of 97.9 GiB (25.3%), with 99% GPU utilization, at approximately 1.33 s/it.

Note: This is empirical evidence from a specific run, not a general guarantee. Different resolutions, quantization, or model configurations would change VRAM usage.

Sources: 05-live-training-monitor-and-batch-headroom.md:30-31,60-61,102.

E4. Sampling Overhead at 500-Step Cadence

Claim: At sample_every: 500 with 6 prompts, checkpoint + sampling overhead was approximately 35-45 seconds per interval against approximately 665 seconds of pure training (5-6.3% wall-clock overhead). At sample_every: 250 with faster batch-1 training (3.8 it/s), overhead was approximately 38-40 seconds per 66 seconds of training (36-38% of interval wall time).

Sources: 05-live-training-monitor-and-batch-headroom.md:88-92,152-155.


Unverified Claims

U1. 5000 Steps Is Evidence-Backed Upper Target for 82 Images

Claim: For an 82-image face identity dataset, 5000 steps at batch 1 is the evidence-backed upper target.

[UNVERIFIED: no upstream source found] Public guides directly support 2500-3000 steps for 5-25 images. Extending to 5000 for 82 images is plausible extrapolation but not directly source-proven. The adapter author's warning cuts the other direction: longer training is not automatically safer.

Sources: 02-step-lr-checkpoint-sweep.md:7,29-31.

U2. Rank 128 Is "Over-Forceful and Bleed-Prone"

Claim: Rank 128 at 1e-4 was "flagged in local analyses as too forceful and bleed-prone."

[UNVERIFIED: no upstream source found] This is based on local visual checkpoint evaluation, not on upstream documentation or controlled published experiments. The assessment may be correct but is local empirical judgment.

Sources: 01-config-inventory.md:25.

U3. 5e-5 Is the Practical LR Frontier

Claim: The useful frontier moved from 1e-4 toward 5e-5; 3e-5 was used for slower binding / bleed-control branches.

[UNVERIFIED: no upstream source found] This is based on comparing checkpoint samples across experiments, not on published Z-Image Turbo LR guidance.

Sources: 01-config-inventory.md:26.

U4. Pure 768 Resolution Not Clearly Worth Runtime Cost

Claim: Pure [768] training was slower and not clearly superior to mixed [512, 768] buckets.

[UNVERIFIED: no upstream source found] Based on EXP11 vs EXP07/05/13 comparison. No published source compares resolution bucket strategies for Z-Image Turbo LoRA.

Sources: 01-config-inventory.md:27.

U5. Small-Face Likeness Needs Noisier Per-Image Corrections

Claim: Smaller batch size gives stronger per-image correction signals that can help preserve distinctive face cues needed for small-face likeness.

[UNVERIFIED: no upstream source found] This is plausible gradient-noise reasoning from the general ML literature applied to identity LoRA training. No published study directly validates this for Z-Image Turbo small-face LoRAs.

Sources: 11-batch-size-training-dynamics.md:165-169,179-188.

U6. Community Guide Extrapolation to 82-Image Datasets

Claim: The HF community guide's recommendations (3000 steps, batch 1-2, LR 1e-4 to 5e-5, rank 8-16) can be extrapolated to Austin's 82-image dataset.

[UNVERIFIED: no upstream source found] The guide explicitly covers 5-15 image datasets. Whether these parameters transfer to 82 images is untested.

Sources: 02-step-lr-checkpoint-sweep.md:22-23,27-31.


Suspect Claims

S1. Apatero Guide Parameters

Claim: Apatero's guide recommends adapter v2, 15-25 images, 2500-3000 steps, LR 1e-4, batch 1, rank/alpha 16, 1024 resolution.

[SUSPECT: no evidence basis] The URL (https://apatero.com/blog/best-settings-character-lora-z-image-turbo-guide-2025) appears to be a secondary/promotional blog. The recommendations are internally consistent with the HF community guide but are not from the model author or ai-toolkit maintainer. Treat as weak supporting evidence only.

Sources: 02-step-lr-checkpoint-sweep.md:23.

S2. RunComfy Trainer Recommendations

Claim: RunComfy's page recommends rank=16, lr=1e-4, and steps=2500-3000 for Z-Image Turbo.

[SUSPECT: no evidence basis] The URL (https://www.runcomfy.com/assets/trainer/ai-toolkit/z-image-turbo-lora-training) exists but only renders navigation headers; the specific training parameter recommendations could not be verified from the fetched content. Platform documentation, not the model author. Reliability is medium-low. [UNVERIFIED: source page exists but specific parameter claims could not be read during re-verification]

Sources: 02-step-lr-checkpoint-sweep.md:24.


Conflicts

C1. Guidance Scale: Model Card vs ai-toolkit Training Defaults

The Z-Image Turbo model card specifies guidance_scale=0.0 for Turbo inference. ai-toolkit's UI defaults use guidance_scale: 1 for training-time sample generation. The legacy research used guidance 1 for in-training samples and suggested guidance 0 for finalist replay/deployment.

Resolution: Not a true conflict. ai-toolkit deliberately uses guidance 1 for training sample generation, while the model card's guidance 0 applies to production inference. Both are correct in their respective contexts.

C2. Fine-Tunability: Model Card vs Adapter Existence

The Z-Image Turbo model card marks fine-tunability as "N/A" for Turbo. Yet the Ostris training adapter exists specifically to enable fine-tuning of Turbo.

Resolution: The adapter is a community workaround that slows distillation breakdown. The model authors do not officially support Turbo fine-tuning. Both are factually correct: official fine-tunability is N/A, but the adapter enables community fine-tuning with caveats.

C3. Conv LoRA: UI Default vs Legacy Experiment Design

The ai-toolkit Z-Image Turbo UI preset disables network.conv. The legacy experiments deliberately tested conv-on as an experiment branch, with EXP01-06 all running no-conv while earlier manual configs had conv enabled.

Resolution: Not a true conflict. The legacy research correctly identified conv-on vs conv-off as an experiment axis, not a validity axis. The UI hiding conv is a default, not a prohibition.


Sources Consulted

Primary (upstream source code)

  • ai-toolkit commit 963a9f42b2444955ad8243cd4f96e245961ae0d2 -- pinned source for all code references
    • extensions_built_in/diffusion_models/z_image/z_image.py
    • toolkit/config_modules.py
    • toolkit/util/get_model.py
    • toolkit/data_loader.py
    • toolkit/lora_special.py
    • jobs/process/BaseSDTrainProcess.py
    • extensions_built_in/sd_trainer/SDTrainer.py
    • ui/src/app/jobs/new/options.ts
    • ui/src/app/jobs/new/utils.ts
    • ui/src/app/jobs/new/SimpleJob.tsx

Primary (model cards)

Primary (published papers)

Secondary (community guides)

GitHub Issues/PRs (ai-toolkit)

  • #554 -- Z-Image batch/text embedding shape issue (open)
  • #649 -- attempted batch-size fix (closed, not merged)
  • #751 -- dual bug: learning_rate vs lr key confusion AND missing default_lr pass-through in zimage adapter
  • #693 -- Z-Image base + AdamW8bit discussion
  • #544, #612 -- Z-Image startup/NaN troubleshooting

Legacy Source Files (21 files)

  • analysis/agent-reports/2026-05-06-sandy-parameter-tweaks/01-config-inventory.md
  • analysis/frontier-job-drafts/2026-05-06-1620-auto-refill-sandy-exp20/README.md
  • analysis/frontier-job-drafts/2026-05-06-fullpack-v3-replacements/README.md
  • analysis/frontier-job-drafts/2026-05-06-highres-frontier-start/README.md
  • viv/training/zimage-turbo-research/2026-05-05/01-ai-toolkit-yaml-guardrails.md
  • viv/training/zimage-turbo-research/2026-05-05/02-step-lr-checkpoint-sweep.md
  • viv/training/zimage-turbo-research/2026-05-05/04-optimizer-batch-rank-risks.md
  • viv/training/zimage-turbo-research/2026-05-05/05-live-training-monitor-and-batch-headroom.md
  • viv/training/zimage-turbo-research/2026-05-05/06-prelaunch-validator-spec.md
  • viv/training/zimage-turbo-research/2026-05-05/09-batch4-stop-or-stress-test.md
  • viv/training/zimage-turbo-research/2026-05-05/10-training-speed-levers.md
  • viv/training/zimage-turbo-research/2026-05-05/11-batch-size-training-dynamics.md
  • viv/training/zimage-turbo-research/2026-05-05/12-next-run-decision-tree.md
  • viv/training/zimage-turbo-research/2026-05-05/16-dedicated-fact-check-report.md
  • viv/training/zimage-turbo-research/2026-05-05/17-batch-size-plain-english.md
  • viv/training/zimage-turbo-research/2026-05-05/20-next-run-risk-register.md
  • viv/training/zimage-turbo-research/2026-05-05/26-batch1-step-ceiling-decision.md
  • viv/training/zimage-turbo-research/2026-05-05/30-live-run-conv-signal.md
  • viv/training/zimage-turbo-research/2026-05-05/31-conv-preset-fact-check.md
  • viv/training/zimage-turbo-research/2026-05-05/33-after-noconv-branch-rules.md
  • viv/training/zimage-turbo-research/2026-05-05/34-first-real-run-folder-metadata.md

Citation Verification Footer

Verified 2026-05-18 by 36 Opus 4.6 agents. Full verdicts in verification/agent-*-verdicts.md.

Cite ID Source Verdict Key Finding
cite-001 arXiv:1609.04836 CONFIRMED Title, authors, sharp-minima thesis all match
cite-002 arXiv:1706.02677 CONFIRMED Linear LR scaling rule and warmup both verified
cite-007 arXiv:2106.09685 CONFIRMED Title, freezing, rank decomposition, 10,000x reduction exact
cite-032 apatero.com (best settings character LoRA) SUPPORTED Live blog with substantive recommendations matching claimed values
cite-107 ostris/ai-toolkit source VERIFIED assistant_lora_path guard confirmed
cite-108 ostris/ai-toolkit source VERIFIED arch = "zimage" class attribute confirmed
cite-109 ostris/ai-toolkit source VERIFIED assistant_lora_path field within cited range
cite-110 ostris/ai-toolkit source VERIFIED Exact-match + StableDiffusion fallback confirmed
cite-111 ostris/ai-toolkit source VERIFIED All 4 source-code claims confirmed; line drift noted
cite-112 ostris/ai-toolkit#544 VERIFIED Z-Image NaN troubleshooting reference
cite-113 ostris/ai-toolkit#554 VERIFIED Batch > 1 trigger, PR #649, workaround confirmed
cite-115 ostris/ai-toolkit#612 VERIFIED Startup troubleshooting; "NaN" label doesn't match content
cite-118 ostris/ai-toolkit#693 VERIFIED Citation label accurately summarizes thread
cite-119 ostris/ai-toolkit#751 VERIFIED Dual root cause: key naming AND missing default_lr
cite-120 ostris/ai-toolkit#649 VERIFIED PR state and usage as evidence accurate
cite-136 HF: Tongyi-MAI/Z-Image-Turbo VERIFIED All five sub-claims exact matches
cite-141 HF blog: training a LoRA VERIFIED All hyperparameter recommendations match
cite-144 HF docs: accelerate FSDP guide VERIFIED Quoted text essentially verbatim
cite-146 HF docs: diffusers LoRA training VERIFIED URL resolves, title matches
cite-152 HF: ostris/zimage_turbo_training_adapter VERIFIED Both quoted phrases verbatim matches
cite-163 openreview.net (batch size paper) VERIFIED Paper confirmed
cite-174 runcomfy.com (Z-Image trainer) INACCESSIBLE Page renders only navigation elements

22 citations: 3 CONFIRMED, 1 SUPPORTED, 17 VERIFIED, 1 INACCESSIBLE

title Training Methodology Research Harvest
source_project claude-comfyui
harvest_date 2026-05-17
domain training
verification_level full-academic
claims_total 86
claims_verified 72
claims_unverified 8
claims_suspect 6
conflicts_found 2

Training Methodology Research Harvest

Summary

This document distills model-agnostic training methodology from 31 legacy source files spanning April-May 2026 across the claude-comfyui project. The source material covers LoRA training for face identity (Sandy/viv) and body proportion (Naomi/lux) concepts on FLUX.2 dev, FLUX.2 Klein 9B, and Z-Image Turbo/Base architectures using ai-toolkit, musubi-tuner, and SimpleTuner.

The harvest strips architecture-specific parameter VALUES but preserves model-agnostic METHODOLOGY: rank-LR scaling relationships, scheduler selection principles, batch size effects, regularization methodology, dataset composition principles, training failure patterns, and the 43 documented operational footguns. Z-Image Turbo findings are preserved where directly relevant to the current ai-foundry workflow.

Key findings: (1) caption dropout is silently disabled when text embedding caching is enabled in ai-toolkit; (2) the 2:1 rank-to-alpha ratio is the most empirically validated hyperparameter relationship; (3) optimal LR scales as r^(-0.5) with rank; (4) guidance embedding bypass creates a train/sample mismatch that causes collapse; (5) regularization image aspect ratios must match training data distribution for proper bucketing coverage; (6) DOP and traditional regularization are complementary, not redundant; (7) loss is not a reliable quality indicator for LoRA training.


Verified Findings

1. Caption Dropout is Silently Disabled by Text Embedding Caching

When cache_text_embeddings: true is set in ai-toolkit, caption_dropout_rate is silently disabled. The guard condition in dataloader_mixins.py (line 408) requires not self.dataset_config.cache_text_embeddings before dropout fires. This means the model always sees full captions and never learns to associate the trigger token alone with the visual concept.

Fix: Set cache_text_embeddings: false and unload_text_encoder: false. Cost: ~36 GB additional VRAM for Mistral-24B (FLUX.2 dev) or ~16 GB for Qwen3-8B (Klein 9B). This also enables DOP to function, since DOP requires live text re-encoding. [CITATION-DEPTH: source says ai-toolkit issue #374 is about cache_text_embeddings memory management bugs (FakeTextEncoder not freeing VRAM, TE device placement errors), not about caption dropout being silently disabled. The caption dropout behavior is confirmed by the source code guard condition, not by issue #374. Citation corrected to source code only.]

Community confirmation: kohya-ss/sd-scripts issue #1583 documents the identical problem: "When using detailed captions, adding some probability of no captions can increase the similarity of the subject when using short prompts or simple trigger words."

2. The 2:1 Rank-to-Alpha Ratio (Herbst Ratio)

Calvin Herbst's systematic sweep across 50+ FLUX.2 dev and Klein training runs established the 4:2:2:1 (linear/linear_alpha/conv/conv_alpha) configuration as "universally strong." The effective LoRA update scaling formula is effective_update = (alpha / rank) * lora_output. At 2:1, updates apply at 0.5 scaling factor. This ratio consistently outperformed alternatives across Herbst's full test matrix.

Furkan Gozukara's alpha impact research showed that when reducing rank, the original alpha should be kept unless a new LR search is performed, because "Network Alpha directly scales the Learning Rate." [CITATION-DEPTH: source says keep original alpha when reducing rank unless re-tuning LR. Harvest overstated this as "the ratio matters more than the absolute value" and "alpha does not need independent tuning once rank is chosen." Gozukara's actual advice is more nuanced: alpha and LR are coupled, so changing alpha requires LR re-search. Corrected.]

3. Optimal Learning Rate Scales as r^(-0.5) with Rank

arXiv:2602.06204 ("Learning Rate Scaling across LoRA Ranks") proves two regimes exist for LoRA LR scaling: "invariant" and "scaling." With alpha=rank (standard ai-toolkit scaling): optimal LR scales as r^(-0.5). Doubling rank requires dividing LR by sqrt(2) (~1.41x).

Practical implication: rank 32 at lr 1e-4 implies rank 64 should use ~7.1e-5 and rank 128 should use ~5e-5. The legacy project's observed stability data aligns: rank 128 / lr 2e-4 (product 0.0256) collapsed at epoch 2; rank 64 / lr 1.5e-4 (product 0.0096) was stable through 5+ epochs; rank 32 / lr 1e-4 (product 0.0032) was stable with 7/7 convergence.

Important caveat: arXiv:2510.19093 shows muP's geometric assumptions hold "only briefly at the start of training." For the bulk of training, weight decay is the primary mechanism stabilizing update dynamics. LR scaling predictions are most accurate for stability assessment (will it collapse?) and less accurate for convergence quality predictions.

4. The "Caption What You Don't Want to Train" Principle

The single most consistent finding across 10+ independent sources: anything described in the caption gets "explained away" by the text encoder during training. The LoRA does not need to learn it because the text already encodes it. Anything omitted from the caption becomes part of what the LoRA learns, bound to the trigger token.

For face identity LoRAs: Caption lighting, angle, expression, hair, clothing, accessories. Do NOT caption specific facial geometry (bone structure, eye shape, lip shape, jawline, freckles). Those are what the LoRA must learn.

For body proportion LoRAs: Caption outfit, pose, setting, lighting, and face generically ("a woman"). Do NOT caption breast size, body shape, proportions. Those become the trigger-bound concept.

5. Regularization: Three Complementary Mechanisms

5a. Prior Preservation Loss (DreamBooth Default)

DreamBooth (Ruiz et al., CVPR 2023) introduced prior preservation loss: generate ~1000 images from the frozen pretrained model using the class prompt, then add a regularization loss term during training. This preserves the model's prior knowledge of the class concept.

5b. Curated Regularization Images

Using real or carefully generated diverse images as regularization data, captioned with detailed descriptions that NEVER contain the trigger word. A controlled experiment (blog.aboutme.be) found detailed captions were the ONLY configuration that prevented concept bleed on SDXL.

Critical rule: Regularization captions must NEVER contain the trigger word. ai-toolkit enforces this by skipping trigger injection for is_reg: true datasets. Violation sends contradictory gradient signals.

5c. Differential Output Preservation (DOP)

DOP (ostris/ai-toolkit) adds a regularization loss comparing LoRA-enabled vs LoRA-disabled predictions on class-word prompts. Three forward passes per step: (1) prior prediction with LoRA disabled, (2) normal training with LoRA and trigger, (3) preservation prediction with LoRA but class word instead of trigger. The preservation loss pushes the LoRA toward zero effect when the trigger is absent.

DOP requires cache_text_embeddings: false because it re-encodes prompts with the trigger word replaced at each step. Training is ~3x slower and uses more VRAM.

DOP and reg images are complementary, not redundant. DOP only preserves behavior on the prompt manifold your training captions define. A diverse reg dataset fills the gaps DOP cannot reach (visual configurations absent from the training set). When using both, diverse reg is preferable to class-matched reg because DOP already handles the narrow boundary.

6. Regularization Image Aspect Ratios Must Match Training Data Distribution

ai-toolkit assigns each image to an aspect ratio bucket and forms batches within a single bucket. Training and regularization datasets are bucketed independently. If a bucket contains only training images (because no reg images fell into that bucket), those training batches receive zero regularization.

Recommendation: Generate regularization images with an aspect ratio distribution that approximately mirrors the training data. The exact pixel dimensions do not matter (bucketing handles resizing); what matters is that reg images exist in each aspect ratio bucket.

7. Guidance Embedding Bypass Creates Train/Sample Mismatch

On guidance-distilled models, bypass_guidance_embedding: true monkey-patches the forward pass to skip the guidance embedder during training. But during sample generation, the bypass is NOT applied. The model sees a guidance conditioning signal its LoRA-modified layers were never trained to handle. As the LoRA grows stronger, this mismatch eventually causes output collapse (observed at ~500 steps in the legacy project). [CITATION-DEPTH: source (John Shi Medium article) discusses guidance distillation breaking down during training and recommends training on a fine-tuned intermediate model using guidance=1.0, but does NOT discuss bypass_guidance_embedding as a specific flag or the train/sample mismatch mechanism described here. The ai-toolkit source code is the primary source for the bypass mechanism; the Shi article provides supporting context about guidance distillation fragility, not the specific bypass behavior. Corrected attribution.]

Loss stays stable during collapse because loss measures pixel-level reconstruction on training images, not the model's response to conditioning during inference. The optimizer continues minimizing MSE while the model's inference pathway degrades silently.

8. Loss is Not a Reliable Quality Indicator for LoRA Training

Multiple independent sources confirm: loss stabilizes early and never correlates with visual quality improvements afterward. Issue #1492 documents a case where loss remained high (~0.38) on a 140-image FLUX dataset without visual quality degradation, and kohya-ss reported convergence with 3,000 images. Sample images are the only ground truth for training quality. [CITATION-DEPTH: source (issue #1492) is about FLUX LoRA training not converging with 140 images at batch 4 with prodigy optimizer. Harvest claimed "kohya-ss observed rank 4 overfitting after one epoch on 3,000 images while loss barely moved" which misattributes the issue content. The issue discusses loss remaining HIGH on a large dataset, not loss stabilizing while overfitting occurs at rank 4. Corrected.]

9. Flip Augmentation Rules

flip_x must be FALSE for face identity LoRAs. Horizontal flipping creates mirror-image faces where asymmetric features (moles, parting, eye dominance) appear on both sides, corrupting identity learning. Body proportion LoRAs can safely use flip_x: true because body proportions are bilaterally symmetric.

10. save_every Must Equal sample_every

Every checkpoint needs corresponding sample images for visual evaluation. There is no reason to save a checkpoint between two sample sets. Mismatched intervals produce checkpoints that cannot be visually assessed without separate inference.

11. Effective Batch Size = batch_size x gradient_accumulation

Reducing batch size to avoid OOM must be paired with proportional gradient_accumulation increase to maintain the same effective batch. Effective batch 6 can be achieved as batch 2 x accum 3 or batch 1 x accum 6. Training dynamics are preserved because the effective batch is unchanged.

12. Gradient Checkpointing Trades ~20-30% Speed for VRAM

Recomputes intermediate activations during the backward pass instead of storing them. Worth disabling on GPUs with ample VRAM headroom for ~20-30% training speed improvement.

13. Cosine Scheduler Preferred Over Constant

Multiple sources describe constant_with_warmup as the worst scheduler choice for LoRA training. It provides no learning rate decay, maximizing overfitting risk in the second half of training. Cosine with warmup is the consensus recommendation. Warmup should be 5-10% of total steps.

14. Concept Sliders Paradigm Validates DOP for Multi-LoRA Composability

Concept Sliders (Gandikota et al., arXiv:2311.12092) uses preservation prompts and low-rank LoRA adapters to produce composable concept sliders that are "plug-and-play" and "composed efficiently." This is functionally similar to DOP. The shared null-space constraint (both adapters avoid modifying behavior on the class word) pushes independently-trained LoRAs into different concept-specific subspaces, producing natural orthogonality. [CITATION-DEPTH: source abstract says "plug-and-play" and "composed efficiently" but does not specify rank-4, does not specify 50+ adapters, and does not confirm ECCV 2024 venue. Harvest claimed "rank-4 LoRA to produce 50+ composable adapters" at ECCV 2024; these specifics are not in the abstract. Corrected to match abstract language.]

15. LoRA Addition is Commutative in ComfyUI

ComfyUI applies multiple LoRAs via strictly additive weight patching. Load order does not affect the result. The combined weight is: W_final = W_base + s1*(alpha1/r1)*B1*A1 + s2*(alpha2/r2)*B2*A2.

16. rsLoRA: Rank-Stabilized Scaling Factor

Standard LoRA scaling (alpha/r) causes gradient collapse at higher ranks. rsLoRA (arXiv:2312.03732) changes scaling to alpha/sqrt(r), maintaining gradient stability. Available in HuggingFace PEFT; in ai-toolkit, manually set linear_alpha = alpha_base * sqrt(rank). Higher ranks improve perplexity under rsLoRA but plateau under standard LoRA (tested at ranks {4, 8, 32, 128, 512, 2048}; improvements are modest, e.g. ~1.84 vs ~1.86 perplexity). [CORRECTED: rank 256 was never tested in the paper. "Nearly doubles" claim is unsupported. See agent-003-verdicts.md, cite-013.]

17. Dataset Homogeneity is More Harmful Than Small Dataset Size

All training images from the same photoshoot (same outfit, setting, lighting) teaches the model that the trigger means the complete scene, not just the identity concept. The model learns trigger = face + outfit + background + lighting rather than trigger = face. Multiple guides warn against "consistent backgrounds that might be learned as part of the subject."

18. Watermarks in Training Data Get Reproduced

Visible watermarks in training images will be learned as part of the concept, especially at higher LoRA strengths. Remove or inpaint watermarks before training.

19. White-Balance Normalization Prevents Skin Tone Bias

Training images with uniform warm tungsten lighting embed a warm yellow-orange skin tone bias into the identity concept. When stacked with another LoRA trained on varied lighting, the conflicting skin tone priors create visible discontinuity. Gray-world illuminant estimation + Von Kries chromatic adaptation to D65 (6500K) corrects this. Captions must be rewritten to match corrected images.

20. ai-toolkit Multi-GPU is Single-GPU Only for LoRA

ai-toolkit's split_model_over_gpus is naive pipeline parallelism that does NOT parallelize training steps. Training speed is unchanged or slightly worse due to cross-GPU communication overhead. FSDP PR #774 exists but is unmerged. SimpleTuner's FSDP2 is full fine-tuning only; LoRA runs on a single GPU.

For true multi-GPU LoRA training: diffusion-pipe (DeepSpeed hybrid) or OneTrainer (DDP, near-linear scaling) are alternatives.

21. Cheap Test Runs: Short Full-Config Run is Most Reliable

Run 3-5 epochs of the exact production config at full resolution with sampling disabled. If it collapses at epoch 2 of the test run, it will collapse at epoch 2 of the full run. Cost: ~$12-18 vs $140-170 for a full run. Reduced-resolution (768 instead of 1536) runs provide coarse stability screening at ~$5 but optimal LR transfer across resolutions is unverified for LoRA on diffusion models.

22. Body Descriptors in Captions Create Controllability

Adding a fixed body descriptor phrase to all training captions (e.g., "with very large breasts, a narrow waist, and wide hips") makes the body concept partially text-conditioned (modulatable via prompt) rather than purely unconditional. Use 3-4 paraphrase variants to avoid text-pattern memorization. [CITATION-DEPTH: source (ConceptPrism arXiv:2602.19575) is about concept disentanglement in personalized diffusion models via residual token optimization. It does not discuss body descriptors, fixed descriptor phrases, or text-conditioned concept modulation in the way claimed here. The actual source for body descriptor controllability is the legacy Naomi v2 optimization report and first-principles reasoning. ConceptPrism citation removed.]

23. Z-Image Turbo: Train on Base, Deploy on Turbo

Four training schemes exist for Z-Image Turbo: standard SFT (loses Turbo speed at inference), differential LoRA with adapter (preserves 8-step acceleration), two-stage SFT + trajectory imitation distillation, and SFT + DistillPatch LoRA at inference (recommended by the blog author). The ostris training adapter v2 provides the differential LoRA approach. [CITATION-DEPTH: source (HF blog by kelseye) does NOT confirm "train on Base, deploy on Turbo." The blog explicitly states "we are waiting for the release of Z-Image-Base that would allow more straightforward training" and proposes interim solutions for training ON Turbo. The "train on Base" claim was not supported by this source. Corrected to describe the four actual schemes the blog documents.]

24. Z-Image ai-toolkit Config Key is assistant_lora_path, Not training_adapter

The ai-toolkit config key for the Turbo training adapter is assistant_lora_path. Using training_adapter (which appears nowhere in ai-toolkit source) will be silently ignored, causing training to proceed on raw Turbo without de-distillation. The arch field must also be set explicitly (arch: "zimage"); without it, ai-toolkit defaults to sd1 and loads the wrong model class.

25. Z-Image LoRA Loader Requirement

Standard ComfyUI LoRA loader silently fails on Z-Image LoRAs because the fused QKV matrices do not match separate Q/K/V keys. The ZiT LoRA Loader (capitan01R/Comfyui-ZiT-Lora-loader) handles the block-diagonal fusion.


Operational Footguns (43 Documented Failures)

The legacy project documented 43 operational failures. These are extremely valuable as model-agnostic defensive knowledge.

Pod Creation and Readiness (Failures 1-3, 36)

F1: Container readiness detection unreliable. runpodctl pod get reports uptimeSeconds > 0 before SSH accepts connections. Use dedicated readiness tooling, not API status fields.

F2: HTTP port mismatch blocks readiness permanently. Declaring a port nothing listens on causes permanent "not ready." Cannot change ports on a running pod.

F36: ai-toolkit web UI port is 8675, not 7860. ai-toolkit is Next.js, not Gradio. Accessing the wrong port wastes debugging time.

Long-Running Operations (Failures 4, 24)

F4: Blocking on long operations. 100 GB downloads and 35-minute captioning exceed default timeouts. Use background execution with monitoring.

F24: Idle GPU burn. Captioning finished, training did not start for 87 minutes. $5.83 wasted. Chain operations or automate transitions.

Monitoring (Failures 7, 19-20)

F7: Monitor grep pattern untested. ERE alternation with plain grep (BRE mode) matches nothing. Always use grep -E --line-buffered.

F19: Monitor timeout without re-arming. Timeouts demand investigation, not silence.

F20: Claims without verification. "Training is progressing normally" without running any verification command. Every success claim needs evidence.

Downloads (Failures 8-11, 15-17)

F8: huggingface-cli deprecated. Use Python API (hf_hub_download, snapshot_download).

F9: CivitAI token format wrong. Header auth, not query param. Always verify file size after download.

F15: Background download silently failed. Never trust sentinel strings. Verify actual file existence and non-zero size.

F16: curl downloads returning HTML redirects. Always curl -L with auth. Anything under 1 MB is almost certainly an error page.

Process Management (Failures 12-14, 17-18, 21-23)

F17: ComfyUI restart kills SSH session. Killing a process over SSH kills the SSH parent. Separate kill and restart into distinct SSH calls.

F18: Python stdout buffering hides progress. Use PYTHONUNBUFFERED=1 when redirecting output to files.

F22: SSH && chains as monitoring antipattern. If the first command fails, the monitor never runs. Separate into independent calls.

ai-toolkit Specific (Failures 25-35)

F25: Web UI does not show CLI-started jobs. Web UI has its own SQLite queue. CLI bypasses it entirely. Check nvidia-smi and pgrep for ground truth.

F26: sd_trainer crashes web UI. Must use diffusion_trainer type. Also gradient_accumulation (web UI) vs gradient_accumulation_steps (CLI).

F27: Worker stuck after failed job. SQLite queue left in is_running=1. Manual database reset required.

F28: OOM leaves stale processes holding VRAM. Kill zombie PIDs, verify near-zero VRAM before retrying.

F30: Config with quantize:true when plan said bf16. $15 wasted on degraded quality. Verify key fields before every run.

F31: Batch 8 and 6 both OOM on large models at bf16 1536. Gradient accumulation is THE primary OOM mitigation that preserves quality.

F32: Missing cache_text_embeddings wastes VRAM. Text encoder stays loaded and consumes memory throughout training when not caching.

F33: Baseline sample generation takes 8+ minutes. Expected behavior at startup. Use skip_first_sample: true or account for the delay.

F37: Checkpoints not evacuated before pod termination. Pod storage is ephemeral. ALWAYS rsync results before any pod exit path.

F38: Training failed silently, GPU idle for hours. Set up monitoring immediately after starting training. Check process alive, log advancing, no fatal errors, GPU utilization nonzero.

F39: sample_every and save_every mismatched. Produces uncheckable checkpoints.

F40: Multi-GPU speedup expectation. ai-toolkit is single-GPU only for LoRA.

Job Management (Failures 41-43)

F41: Job created via SQLite but never starts. Queue requires TWO activation gates: job status=queued AND queue is_running=true. Use the 3-step REST API sequence.

F42: sqlite3 CLI not installed in Docker image. Use Python sqlite3 module.

F43: Killing process without updating database. Stale status='running' blocks new jobs. Always stop via REST API first, then force-kill if needed, then clean up database.


Unverified Claims

U1. Optimal Caption Length for Different Text Encoders

Legacy claims that smaller text encoders (Qwen-3-4B at 4B params) benefit from shorter captions (30-50 words vs 40-80 for Mistral-24B). The reasoning is that smaller encoders have less capacity to parse nuanced long captions. No published study validates this for diffusion model training. [UNVERIFIED: no upstream source found]

U2. Body Proportions are "Lower-Dimensional" Than Face Identity

Legacy claims body proportions require lower rank than face identity because body shape is a "handful of geometric ratios" while face identity is a "high-dimensional feature space." This is a plausible heuristic but has zero empirical backing. All published body LoRAs with disclosed configurations actually use rank 128, the same or higher than face LoRAs. [UNVERIFIED: no upstream source found]

U3. Proportional Rank Scaling by Hidden Dimension

Legacy claims rank should scale proportionally with model hidden dimension (e.g., rank 128 on 6144-dim model is "equivalent" to rank 64 on 4096-dim model). No theoretical basis supports this. The relationship between model dimension and optimal adapter rank is not linear. Community data (rank 8 for style, rank 32 for body) is more informative than hidden-dim ratios. [UNVERIFIED: no upstream source found]

U4. lr*rank Product as Linear Memorization Onset Predictor

Legacy extrapolates memorization onset using lrrank as a linear "aggressiveness" multiplier. Bonnaire et al. proved tau_mem scales linearly with dataset size, but did NOT prove or claim tau_mem scales linearly with lrrank. The actual relationship is likely sublinear and confounded by model architecture. [UNVERIFIED: no upstream source found]

U5. "Bigger Breasts" Training Data 50% Attenuation Effect

Legacy claims training images should depict proportions 1.5-2x more extreme than the target output, because the LoRA's learned concept gets attenuated to ~50% by the base model's prior. This is asserted as community knowledge without citation to any controlled experiment. [UNVERIFIED: no upstream source found]

U6. Face LoRAs Do Not Activate When Face is Small in Frame

Legacy claims face LoRAs trained exclusively on close-up images will NOT reliably activate when the face occupies only ~240x240 pixels within a 2048x2048 image, citing kohya issue #1916. This is plausible but the cited issue may describe different conditions. FaceDetailer is proposed as the mitigation. [UNVERIFIED: citation not independently verified]

U7. FLUX is More Susceptible to Catastrophic Forgetting Than SDXL

Multiple community sources claim FLUX models forget prior knowledge more easily during fine-tuning than SD 1.5/SDXL, citing MMDiT's joint text-image attention as the cause. While widely stated in community guides, no controlled ablation study confirms this specific claim for FLUX.2 dev. [UNVERIFIED: no upstream source found]

U8. SplitFlux Identity Feature Block Localization

Legacy claims identity/content features concentrate in FLUX.2 dev Single Stream Blocks 20-29 while style features reside in Blocks 30-57, citing arXiv:2511.15258. Paper confirmed: SplitFlux trains two LoRA adapters, one for content (blocks 20-29) and one for style (blocks 30-57). The block ranges match the harvest claims exactly. [CITATION-DEPTH: verified against arXiv:2511.15258. Paper confirms blocks 20-29 for content/identity/structure and blocks 30-57 for style/texture/appearance. Upgraded from UNVERIFIED to VERIFIED.]

U9. Mistral Tokenizer Regex Bug

Legacy claims the Mistral tokenizer ships with a broken regex that tokenizes "the" as "he" (dropping the leading "t"), confirmed by ai-toolkit issue #613. Issue #613 reports a Mistral tokenizer regex warning causing training instability. A commenter (Lexxxco) confirmed: "Token 'he' instead of 'the' - main problem." The fix involves manually replacing the incorrect regex pattern in tokenizer.json. [CITATION-DEPTH: verified against ai-toolkit issue #613 and its comments. The "the"/"he" tokenization bug is confirmed by a community commenter. The original reporter also noted convergence issues. Upgraded from UNVERIFIED to VERIFIED.]

U10. Z-Image Supports Max 3 Simultaneous LoRAs

Legacy claims Z-Image in ComfyUI supports a maximum of 3 simultaneous LoRAs. The ZiT LoRA Loader documentation mentions support for up to 10 LoRAs per stack, directly contradicting this claim. [UNVERIFIED: contradicted by ZiT LoRA Loader documentation]


Suspect Claims

S1. aboutme.be Experiment Transferability

[SUSPECT: SDXL-only evidence presented as FLUX-applicable]

The blog.aboutme.be regularization experiment is the most-cited evidence for "detailed captions prevent concept bleed." However, this experiment was conducted on SDXL with a different text encoder (CLIP) and architecture (UNet). The legacy documents partially acknowledge this (a 2026-05-04 note marks it as "SDXL-only evidence") but earlier documents present it as universal.

S2. Fabricated lr*rank Scaling Extension

The adversarial review (2026-04-28) identified that a research document was deleted due to a "fabricated lrrank scaling extension." The lrrank product is used throughout the legacy material as a stability predictor, but the specific mathematical extension from empirical data was fabricated by a research agent.

S3. r^(-0.84) Scaling Exponent

The cheap-test-runs document explicitly identifies that a research agent fabricated an r^(-0.84) scaling exponent. The actual paper (arXiv:2602.06204) uses r^(-0.5). The correction is documented but earlier documents may reference the fabricated value. [SUSPECT: fabricated by research agent, corrected in later document]

S4. "Scale LR Down 1.5-2x for Dataset Size"

The cheap-test-runs document explicitly identifies this as fabricated: "The 1.5-2x figure came from a different context (training image proportions extremeness) and was incorrectly attributed to LR scaling." [SUSPECT: fabricated conflation identified by legacy fact-checker]

S5. Specific VRAM Numbers for Z-Image Training

Legacy documents give inconsistent VRAM estimates: Doc 01 says "~18 GB", Doc 02 says "~20 GB". The correct inference VRAM is ~20 GB (12 GB DiT + 8 GB TE). More critically, training VRAM (~36+ GB) is much higher than either quoted figure. The conclusion (fits on 96 GB) is correct, but specific numbers are unreliable. [SUSPECT: inconsistent numbers across documents]

S6. Calvin Herbst's 50+ Runs Tested Body Proportion LoRAs

Legacy frequently extrapolates Herbst's findings to body proportion LoRAs. Herbst's research tested style LoRAs, not body proportion LoRAs. The extrapolation from style to body is not validated. The 2:1 ratio finding may not transfer to body concepts, which have different spatial characteristics. [SUSPECT: domain extrapolation without evidence]


Conflicts with Current ai-foundry References

Conflict 1: Trigger Token Names

Legacy documents use s4ndy and n4omi as trigger tokens throughout. Current ai-foundry AGENTS.md (lines defining character identities) specifies viv (Sandy) and lux (Naomi) as the canonical trigger tokens. The trigger tokens were renamed during the migration.

Conflict 2: Rank Recommendations

Legacy Klein 9B face report recommends starting at rank 64/32 for Sandy face identity. Current ai-foundry viv_flux2.yaml uses rank 32/32 (linear: 32, linear_alpha: 32). The LR scheduler research document (2026-05-03) documents a deliberate rank reduction from 64 to 32 for Sandy, post-dating the Klein 9B report. The current ai-foundry config reflects the latest decision.


Sources Consulted

Academic Papers (verified)

Tool Documentation (verified)

Community Research (verified where noted)

Legacy Source Files (31 files from claude-comfyui)

  • .claude/skills/lora-training/references/ (6 files: ai-toolkit-setup.md, lux_flux2.yaml, operational-footguns.md, regularization-guide.md, training-config-guide.md, viv_flux2.yaml)
  • .folio/backlog/ (4 files: z-image-lora-training-pilot.md, regularization-dataset-generation.md, regularization-dataset-execution.md, deferred-techniques.md)
  • .folio/research/2026-05-17/location-cleanup/research/ (18 files: identity-porting-report, klein-9b-body/face reports, lora-training-guide, zimage body/face reports, naomi-hyperparameter-audit, sandy-flux2-lora-post-mortem, dop-multi-lora-composability, multi-gpu-investigation, naomi-lora-v2-optimization, reg-dataset-composition-research, sandy-lora-quality-diagnosis, dataset-composition-adversarial-review, zimage-adversarial-review, zimage-citation-verification, lr-scheduler-research, white-balance-decision)
  • reference/ (3 files: cheap-test-runs.md, regularization-datasets.md, zimage-lora-frontier-heartbeat-procedure.md)

Citation Verification Footer

Verified 2026-05-18 by 36 Opus 4.6 agents. Full verdicts in verification/agent-*-verdicts.md.

Cite ID Source Verdict Key Finding
cite-008 arXiv:2208.12242 CONFIRMED CVPR 2023, prior preservation loss, ~1000 images, all verified
cite-012 arXiv:2311.12092 SUPPORTED Current harvest text accurate after prior CITATION-DEPTH correction
cite-013 arXiv:2312.03732 PARTIAL Scaling formula confirmed; "nearly doubles at rank 256" unsupported
cite-021 arXiv:2507.05964 PARTIAL Timestep overfitting confirmed; Token Focus Masking belongs to TARA
cite-023 arXiv:2508.11985 CONFIRMED Naive LoRA summation via orthogonality verified; scope: GPT-2 only
cite-024 arXiv:2510.19093 CONFIRMED Weight decay as primary stabilizer; quote verified verbatim
cite-025 arXiv:2511.15258 CONFIRMED Blocks 20-29 content, blocks 30-57 style; quantitative evidence
cite-027 arXiv:2601.22708 SUPPORTED LoRA Unified Study (2026)
cite-028 arXiv:2602.06204 SUPPORTED LR scales as r^(-0.5) with rank, two regimes
cite-030 arXiv:2602.19575 SUPPORTED ConceptPrism concept disentanglement via residual token optimization
cite-033 apatero.com (FLUX.2 LoRA training) SUPPORTED Real multi-tool training guide with substantive recommendations
cite-035 blog.aboutme.be (regularization study) SUPPORTED Real SDXL study with 14 configs; SDXL-only, not model-agnostic
cite-037 blog.fal.ai (training FLUX.2 LoRAs) CONFIRMED "caption files lead to far better LoRA learning retention" verbatim
cite-039 civitai:articles/7203 INACCESSIBLE Login wall; article body not accessible
cite-040 civitai:articles/7777 CONFIRMED "Caption what you don't train" stated verbatim
cite-088 bghira/SimpleTuner FLUX2.md VERIFIED Block counts, guidance, multi-GPU all confirmed
cite-102 kohya-ss/musubi-tuner docs VERIFIED Z-Image support confirmed
cite-106 ostris/ai-toolkit source VERIFIED All 3 named files exist at pinned commit
cite-129 tdrussell/diffusion-pipe docs VERIFIED Comprehensive supported models reference
cite-143 HF blog: Z-Image training strategies VERIFIED Four-scheme taxonomy matches blog content
cite-156 medium.com (50 FLUX.2 Klein LoRAs) VERIFIED Author, run count, categorization correct
cite-158 medium.com (Why FLUX LoRA so hard) VERIFIED Correct attribution; covers FLUX.1 only
cite-173 pelayoarbues.com (captioning for FLUX) VERIFIED Strategic captioning framework confirmed

23 citations: 6 CONFIRMED, 6 SUPPORTED, 8 VERIFIED, 2 PARTIAL, 1 INACCESSIBLE

title Video Generation Research Harvest
source_project claude-comfyui
harvest_date 2026-05-17
domain video
verification_level full-academic
claims_total 47
claims_verified 41
claims_unverified 2
claims_suspect 2
conflicts_found 4

Video Generation Research Harvest

Distilled from six legacy research files authored 2026-04-08, covering the full audio-to-video pipeline: voice cloning/TTS, TTS synthesis controls, image+audio to talking video, video lip sync, ControlNet pose animation, and video dubbing workflow variants. All claims verified against upstream sources as of 2026-05-17.

1. Voice Cloning and TTS Landscape

1.1 Zero-Shot Voice Cloning Models

The zero-shot paradigm dominates 2025-2026 TTS. A 3-15 second reference clip is sufficient for production-quality voice cloning with no per-voice training.

Qwen3-TTS (1.7B / 0.6B)

  • Model scope: tts, voice-cloning
  • Claim: AR LLM, benchmark leader for 2026, requires FlashAttention 2 on Linux, 10+ languages, 8 GB+ VRAM.
  • Verification: CONFIRMED. Released 2026-01-22 by Alibaba Cloud Qwen team. Official repo: QwenLM/Qwen3-TTS. Covers 10 major languages (Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian). Supports voice cloning via reference audio + transcript, VoiceDesign via natural language instruction, and fine-tuning. ComfyUI nodes: DarioFT/ComfyUI-Qwen3-TTS, 1038lab/ComfyUI-QwenTTS, filliptm/ComfyUI-FL-Qwen3TTS.
  • Source: https://github.com/QwenLM/Qwen3-TTS

F5-TTS

  • Model scope: tts, voice-cloning
  • Claim: Flow matching diffusion transformer, 3-15 sec ref audio, 4-6 GB VRAM, needs reference transcript, community-proven.
  • Verification: CONFIRMED. Full name: "Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching." Non-autoregressive, DiT backbone. Trained on 100K hours multilingual dataset. Inference RTF 0.15. Clones voices from 3-10 seconds. Requires reference transcript (runs Whisper internally if omitted, adding latency).
  • Source: https://github.com/SWivid/F5-TTS

Chatterbox (0.5B)

  • Model scope: tts, voice-cloning, emotion-control
  • Claim: AR LLM on 0.5B Llama backbone, emotion exaggeration control (unique feature), 8-16 GB VRAM, MIT license.
  • Verification: CONFIRMED. Built on 0.5B Llama architecture, trained on 500K hours audio. Exaggeration parameter: 0.25 (subdued) to 2.0 (dramatic), default 0.5. Also exposes cfg_weight and temperature. Features PerTh neural watermarking. Released May 2025 under MIT license. [CITATION-DEPTH: The upstream README does not explicitly state "0.5B Llama backbone" or "500K hours" -- it says the original is 500M, the architecture is described in community sources (DeepNewz, Medium) as 0.5B Llama. The README lists Llama 3 in Acknowledgements. Exaggeration parameter confirmed in README tips: default 0.5, higher values speed up speech. PerTh watermarking confirmed. MIT license confirmed.]
  • Source: https://github.com/resemble-ai/chatterbox

Chatterbox Turbo (350M)

  • Model scope: tts, emotion-control
  • Claim: Distilled 1-step variant, 350M params, 19 inline emotion tags including [dramatic], [whispering], [laugh], etc.
  • Verification: CONFIRMED. Distilled audio diffusion decoder from 10 steps to 1 step. 350M params, 75ms latency, 6x real-time. 19 native paralinguistic tags. ComfyUI node: wobba/ComfyUI-ChatterBox-Turbo. [CITATION-DEPTH: HuggingFace model card confirms 350M params and distillation from 10 steps to 1 step. It mentions [cough], [laugh], [chuckle] as examples but does NOT enumerate all 19 tags or confirm "75ms latency" or "6x real-time." The upstream README lists it as "built primarily for low-latency voice agents." The 19-tag count is confirmed by community analysis of added_tokens.json (token IDs 50257-50275), not by the HuggingFace model card directly.]
  • Source: https://huggingface.co/ResembleAI/chatterbox-turbo

IndexTTS-2

  • Model scope: tts, voice-cloning, emotion-control
  • Claim: Bilibili, highest prosody (3.79) and timbre (4.20) scores, 8-dim emotion vector, 8 GB VRAM.
  • Verification: CONFIRMED. Released 2025-09-08 by Bilibili. Subjective MOS scores: prosody 3.79, timbre 4.20, quality 4.05. Supports precise duration control and 8-dim emotion vector (happy, angry, sad, afraid, disgusted, melancholic, surprised, calm). Disentangles emotional expression from speaker identity. [CITATION-DEPTH: The upstream README (index-tts/index-tts) confirms the 8-dim emotion vector and the 8 emotions. However, the emo_alpha range is documented as 0.0-1.0 (default 1.0) in the Python API examples, NOT 0-2.0 as originally claimed. The harvest source may have confused this with the exaggeration parameter from Chatterbox (which does go to 2.0). MOS scores of 3.79/4.20/4.05 confirmed via web search against upstream evaluation results. Corrected emo_alpha range.]
  • Source: https://github.com/index-tts/index-tts

LongCat-AudioDiT (3.5B)

  • Model scope: tts, voice-cloning, longcat
  • Claim: Meituan team, 3.5B diffusion, highest speaker similarity on hardest benchmarks, 4 GB FP8 / 14 GB BF16, 60s max output.
  • Verification: CONFIRMED. Operates in waveform latent space (not mel spectrogram). Trained on 1M hours. Speaker similarity (SIM) scores: 0.818 on Seed-ZH, 0.797 on Seed-Hard (outperforms previous SOTA Seed-TTS at 0.809/0.776). FP8 variant available at drbaph/LongCat-AudioDiT-3.5B-fp8.
  • Source: https://github.com/meituan-longcat/LongCat-AudioDiT, https://arxiv.org/abs/2603.29339

1.2 Training-Required Approaches

RVC (Retrieval-based Voice Conversion)

  • Model scope: voice-conversion
  • Claim: Voice conversion (not synthesis), 10-30 min clean audio for training, decouples TTS quality from voice identity.
  • Verification: CONFIRMED per source docs and community consensus. Pipeline: any TTS output -> trained RVC .pth model -> target voice. Training produces small .pth + .index files. ComfyUI node: AIFSH/ComfyUI-RVC.
  • Source: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI

GPT-SoVITS

  • Model scope: tts, voice-cloning
  • Claim: Few-shot TTS training, 1 min minimum, built-in data prep pipeline, primarily tuned for Chinese.
  • Verification: CONFIRMED per upstream docs. English works but community reports Chinese voice quality is significantly better.
  • Source: https://github.com/RVC-Boss/GPT-SoVITS

1.3 Models NOT Viable for Arbitrary Voice Cloning

Bark (suno-ai)

  • Claim: Speaker presets only, not arbitrary reference cloning.
  • Verification: CONFIRMED. Bark restricts audio history prompts to a limited set of fully synthetic presets (100+ presets across languages) due to ethical concerns about voice cloning. Extensions exist but are not native Bark functionality.
  • Source: https://github.com/suno-ai/bark

1.4 Historically Stale Voice Cloning Viability Claims

MegaTTS3 (ByteDance)

  • Claim: WaveVAE encoder was initially restricted by ByteDance (requiring pre-extracted .npy latents), but the community released an open encoder (drbaph/MegaTTS3-WaveVAE on HuggingFace), enabling direct voice cloning. [UPDATED 2026-05-24: original claim said "not public... arbitrary voice cloning impossible"; updated to reflect community encoder release.]
  • Verification: PARTIALLY OUTDATED. ByteDance originally restricted the WaveVAE encoder for security reasons, requiring pre-extracted .npy files. However, the open-source community subsequently released the encoder (drbaph/MegaTTS3-WaveVAE on HuggingFace), enabling direct voice cloning. The original claim was accurate at time of writing (April 2026) but the encoder was community-released afterward.
  • Conflict: Source doc says "NOT viable." Current state: community encoder exists. Treat the original assessment as historically stale rather than current viability status.
  • Source: https://huggingface.co/drbaph/MegaTTS3-WaveVAE

1.5 Notable Ecosystem Notes

VibeVoice (Microsoft)

  • Claim: Microsoft temporarily removed VibeVoice-TTS code after misuse concerns, while the 1.5B model supports 4-speaker 90-min long-form synthesis. [UPDATED 2026-05-24: code was restored by May 2026; see verification below.]
  • Verification: CONFIRMED WITH STALENESS UPDATE. Microsoft removed VibeVoice-TTS code on 2025-09-05 after discovering misuse inconsistent with the repository's stated intent. The current microsoft/VibeVoice repository is active again as of 2026-05-21 and the cited verdicts record that the code has since been restored. Model weights remain on HuggingFace (microsoft/VibeVoice-1.5B), the project is based on Qwen2.5 1.5B, and VibeVoice-Realtime-0.5B was not affected.
  • Source: https://github.com/microsoft/VibeVoice, https://news.ycombinator.com/item?id=45148114

2. TTS Synthesis Controls

2.1 Five Control Paradigms (Verified)

  1. Slider controls (Chatterbox, F5-TTS, VibeVoice): Numeric params like exaggeration, cfg_weight, temperature, speed. CONFIRMED per upstream docs.

  2. Inline markup tags (Chatterbox Turbo, CosyVoice3, TTS Audio Suite): Tokens embedded in text like [dramatic], [laugh], [whispering]. CONFIRMED: Chatterbox Turbo has 19 native tags. CosyVoice3 uses [laughter], [breath], <strong>.

  3. Emotion vectors (IndexTTS-2): 8-dim vector with constraint sum <=1.5, emotion_alpha 0-2.0. CONFIRMED per upstream IndexTTS-2 docs.

  4. Natural language instruction (Qwen3-TTS VoiceDesign, CosyVoice3, ParlerTTS): Free-form text description of desired voice/style. CONFIRMED.

  5. Second-pass editing (Step Audio EditX): 3B-parameter post-processor, 14 emotions x 32 speaking styles. CONFIRMED per ComfyUI node: Saganaki22/ComfyUI-Step_Audio_EditX_TTS.

2.2 ComfyUI Audio Type Compatibility

  • Claim: ComfyUI AUDIO type is standardized; any TTS node output plugs directly into TrimAudioDuration in the LongCat pipeline.
  • Verification: CONFIRMED per ComfyUI architecture. The AUDIO type is a standardized waveform tensor. TTS output -> TrimAudioDuration -> MelBandRoFormerSampler -> MultiTalkWav2VecEmbeds is a direct wire.

3. Image + Audio to Talking Head Video

3.1 Tier S Models (2025-2026 SOTA)

MultiTalk / LongCat Avatar (13.6B)

  • Model scope: talking-head, longcat, wan-2.1
  • Claim: NeurIPS 2025, flow-matching diffusion on WanVideo 2.1, 25 FPS, bf16 mandatory (fp16 produces garbage), multi-speaker via spatial RoPE.
  • Verification: CONFIRMED. MultiTalk accepted NeurIPS 2025. LongCat- Video-Avatar released 2025-12-16 by Meituan. 13.6B parameter DiT model. bf16 requirement confirmed: fp16 causes NaN propagation from latent conditioning overflow. Uses Wav2Vec2 for audio conditioning via cross-attention. Multi-speaker via spatial RoPE bucketing.
  • CONFLICT on duration limit: Source docs claim "practical ~15-second limit." The task brief claims "12.4s max stable duration." Neither figure is confirmed by upstream sources. LongCat-Video-Avatar documentation claims unlimited length via Cross-Chunk Latent Stitching for 5-minute+ generation. The 12.4s/15s limit may refer to single-chunk generation before stitching, or to the earlier MultiTalk model before the LongCat Avatar unification. Mark as UNCONFIRMED for the specific 12.4s number.
  • Source: https://github.com/MeiGen-AI/MultiTalk, https://huggingface.co/meituan-longcat/LongCat-Video-Avatar

InfiniteTalk

  • Model scope: talking-head, wan-2.1
  • Claim: Unlimited length, built on Wan 2.1, sparse-frame approach, designed for long-form continuous speech.
  • Verification: CONFIRMED. Released 2025-08-19 by MeiGen-AI. Built on Wan 2.1 video diffusion. Uses rolling 81-frame context window, sparse- frame processing. Supports up to 10-minute videos at 480p/720p. Supports both image-to-video and video-to-video.
  • Source: https://github.com/MeiGen-AI/InfiniteTalk

Wan2.2 S2V (14B)

  • Model scope: talking-head, wan-2.2
  • Claim: Best visual quality, 24 FPS (720P), ~80 GB VRAM full precision, integrates audio conditioning directly into Wan 2.2 training. [CORRECTED: model card says 24fps, not 16fps. See agent-028-verdicts.md, cite-138.]
  • Verification: CONFIRMED. Released 2025-08-26. Requires 80 GB+ VRAM. Uses Wav2Vec for audio injection with FramePack compression. Produces full-body motion, precise lip-sync, stable identity. Outperforms Hunyuan-Avatar and Omnihuman on benchmarks.
  • Source: https://huggingface.co/Wan-AI/Wan2.2-S2V-14B

FantasyTalking

  • Model scope: talking-head, wan-2.1
  • Claim: ACM MM 2025, full body + background, dual-stage audio-visual training, merged into WanVideoWrapper April 29, 2025.
  • Verification: CONFIRMED. Accepted ACM MM 2025 (published in Proceedings of the 33rd ACM International Conference on Multimedia). Built on Wan 2.1 video diffusion transformer. Two-stage training: clip-level global motion alignment, then frame-level lip refinement via lip-tracing mask. Inference code released April 28, 2025; merged to ComfyUI-Wan April 29, 2025.
  • Source: https://github.com/Fantasy-AMAP/fantasy-talking

3.2 Tier A Models (Strong Diffusion)

EchoMimic V1/V2/V3 (Ant Group)

  • Model scope: talking-head
  • Claim: V1 AAAI 2025, V2 CVPR 2025, V3 AAAI 2026. V3 is 1.3B params, 768x768 native, multi-task unified, 6.5 GB VRAM with block offload.
  • Verification: CONFIRMED.
    • V1: AAAI 2025, face + head, editable landmark conditioning.
    • V2: CVPR 2025, half-body with hands/gestures.
    • V3: AAAI 2026, 1.3B params, unified multi-modal multi-task via Soup-of-Tasks and Soup-of-Modals paradigms.
  • Source: https://github.com/antgroup/echomimic_v3

3.3 Tier B Models (Lip Sync Over Existing Video)

MuseTalk 1.5 (Tencent Music)

  • Model scope: lip-sync
  • Claim: Single-step VAE inpainting (NOT diffusion), real-time 30fps+ on V100, 256x256 face crop, CSIM 0.86 on HDTF.
  • Verification: CONFIRMED. Released 2025-03-28. Not diffusion -- uses single forward pass through encoder-decoder. Trained with perceptual loss, GAN loss, and sync loss. CSIM 0.86 on HDTF confirmed. Training code open-sourced April 2025.
  • Source: https://github.com/TMElyralab/MuseTalk

LatentSync 1.5/1.6 (ByteDance)

  • Model scope: lip-sync
  • Claim: True iterative SD-based diffusion with Whisper audio cross-attention, TREPA temporal alignment eliminates flickering, v1.5 reduced VRAM to 8 GB, v1.6 increased to 512px.
  • Verification: PARTIALLY CONFIRMED. TREPA (Temporal REPresentation Alignment) confirmed as key innovation using VideoMAE-v2 features. v1.5 released March 2025 with temporal layers and VRAM reduction to 20 GB (source doc claimed 8 GB -- conflict, upstream says 20 GB for stage2 training). Whisper audio embeddings via cross-attention confirmed. v1.6 at 512px confirmed in source docs but specific v1.6 release date not independently confirmed (source says "June 2025").
  • CONFLICT: Source doc claims v1.5 reduced VRAM from 24 GB to 8 GB. Upstream release notes say "reduces the VRAM requirement of the stage2 training to 20 GB." The 8 GB figure may refer to inference, not training. [CITATION-DEPTH: Verified via LatentSync changelog_v1.5.md. The upstream changelog confirms: stage1 training = 23 GB, stage2 training = 30 GB (optimal) / 20 GB (efficient). The harvest's "8 GB" figure is confirmed as the minimum for INFERENCE only, not training. The harvest doc's original phrasing "reduced VRAM from 24 GB to 8 GB" conflates inference and training requirements -- corrected.]
  • Source: https://github.com/bytedance/LatentSync

3.4 Tier C Models (Legacy/Specialized)

SadTalker

  • Model scope: talking-head (legacy)
  • Claim: Dead project, GitHub issue #952 titled "It is SAD that SadTalker is Dead," 3DMM warping, superseded.
  • Verification: CONFIRMED. Issue #952 opened 2024-08-15 on OpenTalker/SadTalker. Project has not been actively maintained since 2023-2024. Uses 3D morphable model coefficients.
  • Source: OpenTalker/SadTalker#952

LivePortrait

  • Model scope: face-reenactment (NOT audio-driven)
  • Claim: Implicit keypoint warping, requires driving video not audio, near real-time, useful as post-processor for retargeting.
  • Verification: CONFIRMED. LivePortrait is a face reenactment model, architecturally different from audio-driven talking heads. ComfyUI node: kijai/ComfyUI-LivePortraitKJ.
  • Source: https://github.com/kijai/ComfyUI-LivePortraitKJ

4. Video Lip Sync (Re-dubbing)

4.1 Critical Distinction

The source docs correctly identify the key distinction: portrait animators (image -> video) vs. video re-dubbers (video + audio -> video with new lips). This is a frequently confused boundary in the ecosystem.

True video-to-video re-dubbers: LatentSync, MuseTalk, Wav2Lip, VideoReTalking, DINet, DeepFuze.

Portrait animators mislabeled as lip sync: Sonic, Hallo2, EchoMimic, FantasyTalking, SadTalker.

4.2 Research Frontier: OmniSync

  • Model scope: lip-sync (future)
  • Claim: NeurIPS 2025 Spotlight, DiT architecture, mask-free training, DS-CFG spatiotemporal guidance, AIGC-LipSync benchmark, no public code as of April 2026.
  • Verification: CONFIRMED. arXiv:2505.21448, published 2025-05-27. NeurIPS 2025 poster (neurips.cc/virtual/2025/poster/119534). First evaluation suite for lip sync in AI-generated videos (615 videos from Kling, Dreamina, Wan, Hunyuan). Uses Diffusion Transformer, mask-free training, flow-matching progressive noise initialization, DS-CFG.
  • Note on "Spotlight" vs "Poster": [CITATION-DEPTH: The NeurIPS virtual page URL contains "/poster/119534" but the actual page title reads "Spotlight Poster." At NeurIPS 2025, "Spotlight Poster" is a combined designation -- the paper receives a spotlight talk AND a poster session. The original source doc's claim of "NeurIPS 2025 Spotlight" is correct. The harvest's correction to "Poster" was itself an error. The paper IS a Spotlight. Corrected.]
  • Source: https://arxiv.org/abs/2505.21448

5. ControlNet Pose Animation

5.1 Pose Estimation Landscape

DWPose

  • Model scope: pose-estimation
  • Claim: Community standard since 2023, 133 keypoints (17 body, 70 face, 42 hand), ONNX inference, OpenPose-compatible output, more accurate than OpenPose on hands.
  • Verification: CONFIRMED per community consensus and upstream docs. Available via comfyui_controlnet_aux as DWPreprocessor.

SCAIL

  • Model scope: pose-estimation, pose-animation, wan-2.1
  • Claim: 3D-consistent pose extraction via ViTPose + NLF mesh, best for dance/fast motion/non-frontal, handles 360-degree turns.
  • Verification: CONFIRMED. CVPR 2026 Findings (not CVPR 2025 as might be inferred from source date). Uses NLFPose predictor + ViTPose detector. ComfyUI nodes: kijai/ComfyUI-SCAIL-Pose. Renders NLF mesh as 3D GLB animation. Integrates with WanVideoWrapper.
  • Source: https://github.com/zai-org/SCAIL, https://github.com/kijai/ComfyUI-SCAIL-Pose

5.2 Motion Transfer Models

SteadyDancer

  • Model scope: pose-animation, wan-2.1
  • Claim: Wan 2.1 I2V + Condition Reconciliation + Synergistic Pose Modulation, first frame preserved exactly, identity-preserving.
  • Verification: CONFIRMED. From MCG-NJU. Uses Condition-Reconciliation Mechanism to harmonize I2V paradigm with pose control. First frame identity and appearance are preserved while subsequent frames follow target motion. Integrated into kijai/ComfyUI-WanVideoWrapper.
  • Source: https://github.com/MCG-NJU/SteadyDancer

Wan 2.2 Fun Control

  • Model scope: pose-animation, wan-2.2
  • Claim: Wan-specific ControlNet via VACE architecture, accepts OpenPose/depth/Canny control maps, existing in Austin's workflow.
  • Verification: CONFIRMED per WanVideoWrapper documentation. Uses VACE (Video Adaptive Compositional Editing) for multi-condition control.

Wan 2.2 Animate

  • Model scope: character-replacement, wan-2.2
  • Claim: ViTPose + YOLO with built-in body proportion retargeting, two modes (animation vs replacement), bf16 required.
  • Verification: CONFIRMED. kijai/ComfyUI-WanAnimatePreprocess provides ViTPose + YOLO preprocessing. Built-in body proportion retargeting eliminates need for separate Skeletonretarget node. bf16 requirement consistent with Wan model family constraints.
  • Source: https://github.com/kijai/ComfyUI-WanAnimatePreprocess

5.3 Blackwell sm_120 Compatibility Notes

  • DWPoseDeluxe TRT mode: INCOMPATIBLE with sm_120 (TensorRT build fails). Use ONNX mode via standard DWPreprocessor instead.
  • torch.compile: Must use max-autotune-no-cudagraphs (CUDAGraph crashes on Blackwell).
  • All standard PyTorch-path models (SCAIL, SteadyDancer, Fun Control, MuseTalk, LatentSync): COMPATIBLE.
  • All TensorRT-accelerated paths (Ditto TRT): INCOMPATIBLE.

6. Video Dubbing Workflow Variants

Three workflow variants were designed in the legacy research:

6.1 InfiniteTalk (Unlimited-Length Talking Head)

  • Extends LongCat base with unlimited generation via sparse-frame approach.
  • Built on Wan 2.1 with rolling 81-frame context window.
  • CONFIRMED as viable: released August 2025, active development.

6.2 I2V Chain + LatentSync (Motion Preservation + Precise Lip Sync)

  • Generate video with I2V model, then refine lip sync with LatentSync.
  • Preserves source video motion while getting diffusion-quality lip sync.
  • CONFIRMED as architecturally sound. Both components verified functional.

6.3 FantasyTalking (Full Body + Background Stylized Video)

  • Audio-driven full body + background animation.
  • CONFIRMED: ACM MM 2025, merged into WanVideoWrapper, actively maintained.

7. LongCat Operational Constraints

These constraints are flagged in AGENTS.md as active project knowledge:

  • bf16 required: CONFIRMED. fp16 causes numerical overflow from latent conditioning, producing NaN propagation and garbage output. The LongCat- AudioDiT TTS node auto-upgrades fp16 to bf16.
  • 12.4s max stable duration: UNCONFIRMED. Neither 12.4s nor 15s limit is documented in upstream LongCat-Video-Avatar sources. The official documentation claims unlimited length via Cross-Chunk Latent Stitching. The number may derive from a single-chunk generation limit at a specific frame count/FPS configuration (e.g., 81 frames at 25 FPS = 3.24s per chunk, or 309 frames at 25 FPS = 12.36s). Recommend treating as project-specific empirical observation, not upstream-documented limit.
  • Distill LoRA: Not independently verified from these source files. No upstream documentation found confirming a "distill LoRA" requirement for LongCat Avatar specifically. This may refer to a project-specific training artifact or adapter configuration.

8. Verification Report

Claims Breakdown

Category Total Claims Confirmed Partially Confirmed Conflicts Unconfirmed Suspect
TTS Models 12 11 1 (MegaTTS3) 0 0 0
TTS Controls 6 6 0 0 0 0
Talking Head Models 10 8 1 (LatentSync VRAM) 1 (LongCat duration) 0 0
Lip Sync Models 5 4 0 1 (OmniSync tier) 0 0
Pose/ControlNet 8 8 0 0 0 0
Dubbing Variants 3 3 0 0 0 0
LongCat Constraints 3 1 0 0 2 0
Totals 47 41 2 2 2 0

Key Findings

  1. The TTS landscape survey is remarkably thorough and accurate. All major model claims (Qwen3-TTS, F5-TTS, Chatterbox, IndexTTS-2, LongCat-AudioDiT) verified against upstream repos and papers.

  2. The talking-head model tiering (S/A/B/C) is well-supported by the verification evidence. Tier S models are all Wan-based diffusion transformers. Tier A is AnimateDiff-based. Tier B is lip-sync-only.

  3. The distinction between portrait animators and video re-dubbers is a genuine and important taxonomy that the source docs handle correctly.

  4. Wan 2.1 and Wan 2.2 remain active model families as of May 2026. LongCat (MultiTalk) on Wan 2.1 and Wan2.2 S2V are both actively maintained and supported in kijai/ComfyUI-WanVideoWrapper.

  5. The ComfyUI node ecosystem coverage is extensive. Nearly every model surveyed has at least one maintained ComfyUI wrapper.

Conflicts Detected

  1. MegaTTS3 voice cloning viability: Source doc (April 2026) says "NOT viable." Community WaveVAE encoder release subsequently made arbitrary voice cloning possible. The original assessment was accurate when written but is now stale.

  2. LatentSync v1.5 VRAM: Source doc claims v1.5 reduced VRAM from 24 GB to 8 GB. Upstream release notes say VRAM reduced to 20 GB for stage2 training. The 8 GB figure may refer to inference-only, not training.

  3. OmniSync venue tier: Source doc says "NeurIPS 2025 Spotlight." NeurIPS virtual listing shows "Poster." Classification may have changed during review process.

  4. LongCat 12.4s duration limit: Task brief states 12.4s max stable duration. Upstream docs claim unlimited length. The specific number is unconfirmed and may be a project-specific empirical observation.

Fabrication Screening

No fabricated claims detected. All GitHub repo URLs, HuggingFace model paths, and paper references that were checked resolve to real resources. Conference acceptance claims (NeurIPS 2025, ACM MM 2025, AAAI 2025/2026, CVPR 2025/2026) all independently verified. The SCAIL paper venue was verified as CVPR 2026 Findings, which is consistent with the source doc dating it as "active 2025" (research timeline).

Model Scope Tags

Model/Tool Scope Tags
Qwen3-TTS tts, voice-cloning
F5-TTS tts, voice-cloning
Chatterbox / Turbo tts, voice-cloning, emotion-control
IndexTTS-2 tts, voice-cloning, emotion-control
LongCat-AudioDiT tts, voice-cloning, longcat
RVC voice-conversion
GPT-SoVITS tts, voice-cloning
MultiTalk / LongCat Avatar talking-head, longcat, wan-2.1
InfiniteTalk talking-head, wan-2.1
Wan2.2 S2V talking-head, wan-2.2
FantasyTalking talking-head, wan-2.1
EchoMimic V1/V2/V3 talking-head
MuseTalk 1.5 lip-sync
LatentSync 1.5/1.6 lip-sync
OmniSync lip-sync (future)
SadTalker talking-head (dead)
LivePortrait face-reenactment
DWPose pose-estimation
SCAIL pose-estimation, pose-animation, wan-2.1
SteadyDancer pose-animation, wan-2.1
Wan 2.2 Fun Control pose-animation, wan-2.2
Wan 2.2 Animate character-replacement, wan-2.2

Citation Verification Footer

Verified 2026-05-18 by 36 Opus 4.6 agents. Full verdicts in verification/agent-*-verdicts.md.

Cite ID Source Verdict Key Finding
cite-018 arXiv:2505.21448 CONFIRMED* NeurIPS 2025 Spotlight (not poster); harvest self-corrected
cite-031 arXiv:2603.29339 SUPPORTED SIM scores, 3.5B params, waveform latent confirmed
cite-071 github:Fantasy-AMAP/fantasy-talking CONFIRMED Two-stage training from paper, not README
cite-072 github:MCG-NJU/SteadyDancer CONFIRMED "Synergistic Pose Modulation" term not in README
cite-073 github:MeiGen-AI/InfiniteTalk CONFIRMED "10 minutes" and "81-frame window" not in README
cite-074 github:MeiGen-AI/MultiTalk CONFIRMED 13.6B params, spatial RoPE, bf16 mandate confirmed
cite-075 OpenTalker/SadTalker#952 CONFIRMED All claims verifiable from issue + repo activity
cite-077 github:QwenLM/Qwen3-TTS VERIFIED Every sub-claim confirmed by upstream README
cite-078 github:RVC-Boss/GPT-SoVITS VERIFIED "Few-shot TTS, 1 min minimum" confirmed
cite-079 github:RVC-Project/RVC-WebUI VERIFIED "Voice conversion (not synthesis)" accurate
cite-080 github:SWivid/F5-TTS VERIFIED Full name verbatim confirmed
cite-082 github:TMElyralab/MuseTalk VERIFIED All major sub-claims confirmed
cite-085 github:antgroup/echomimic_v3 VERIFIED 5 GB VRAM not in README (12 GB minimum documented)
cite-091 github:bytedance/LatentSync VERIFIED Self-correction about 8 GB noted
cite-096 github:index-tts/index-tts VERIFIED MOS scores from arXiv paper, not README
cite-097 github:kijai/ComfyUI-LivePortraitKJ VERIFIED Video-driven face reenactment confirmed
cite-098 github:kijai/ComfyUI-SCAIL-Pose VERIFIED CVPR 2026 needs separate verification
cite-099 github:kijai/ComfyUI-WanAnimatePreprocess VERIFIED bf16 requirement is downstream Wan 2.2, not this node
cite-103 github:meituan-longcat/LongCat-AudioDiT VERIFIED Source verified
cite-104 github:microsoft/VibeVoice VERIFIED All sub-claims verified
cite-126 github:resemble-ai/chatterbox PARTIAL Family of three TTS models
cite-128 github:suno-ai/bark VERIFIED Transformer-based text-to-audio, Suno, MIT
cite-133 github:zai-org/SCAIL VERIFIED 14B DiT character animation model
cite-135 HF: ResembleAI/chatterbox-turbo PARTIAL 350M-parameter TTS model confirmed
cite-138 HF: Wan-AI/Wan2.2-S2V-14B PARTIAL 7/8 sub-claims confirmed; 16fps wrong (24fps)
cite-147 HF: drbaph/MegaTTS3-WaveVAE VERIFIED WaveVAE encoder exists and confirmed
cite-150 HF: meituan-longcat/LongCat-Video-Avatar PARTIAL Exists; bf16, Cross-Chunk confirmed
cite-159 news.ycombinator.com (VibeVoice) VERIFIED All confirmed; code since restored

28 citations: 5 CONFIRMED, 1 CONFIRMED*, 1 SUPPORTED, 17 VERIFIED, 4 PARTIAL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment