Skip to content

Instantly share code, notes, and snippets.

@pingud98
Created June 23, 2026 22:13
Show Gist options
  • Select an option

  • Save pingud98/205874250b915f86cf18d16a8286168f to your computer and use it in GitHub Desktop.

Select an option

Save pingud98/205874250b915f86cf18d16a8286168f to your computer and use it in GitHub Desktop.
Dutch Masters Family Portrait — FLUX.1-dev + per-person LoRAs on RTX 4090

Family Portrait Studio — Dutch Masters

Generates a high-resolution family portrait in Dutch Masters style (Rembrandt, Vermeer) using FLUX.1-dev + per-person LoRAs.

Hardware requirements

  • RTX 4090 (24GB VRAM) ✓
  • 32GB RAM ✓
  • ~80GB disk space

Directory layout

portrait-studio/
├── ComfyUI/                     # Generation UI (http://localhost:8188)
├── SimpleTuner/                 # LoRA training framework
├── training-data/
│   ├── Alice/                   # Photos for Alice (add yours here)
│   ├── Bob/                     # Photos for Bob
│   └── ...
├── loras/
│   ├── Alice/                   # Training outputs for Alice
│   └── Bob/
├── models/                      # (unused — models live in ComfyUI/models/)
└── output/                      # Generated images

Step-by-step guide

1. Fix GPU (one time)

Secure Boot is blocking the NVIDIA driver. Disable Secure Boot in BIOS/UEFI:

  • Reboot → press Del/F2/F12 at the manufacturer logo
  • Find Security → Secure Boot → set to Disabled
  • Save and reboot
  • Verify: nvidia-smi should show your RTX 4090

2. Download models

# First: accept the FLUX.1-dev license at https://huggingface.co/black-forest-labs/FLUX.1-dev
# Then: create a token at https://huggingface.co/settings/tokens (Read access is enough)
HF_TOKEN=hf_yourtoken ./download_models.sh

3. Add each family member

./add_person.sh Alice
./add_person.sh Bob
./add_person.sh Carol
# etc.

Each command creates training-data/Alice/ and the training config.

4. Add photos

Copy 15–30 photos of each person into their training directory:

training-data/Alice/
  close_face_smiling.jpg
  half_body_outdoor.jpg
  different_angle.jpg
  ...

Photo tips:

  • 15–30 images minimum (more = better identity)
  • Mix of head shots, half-body, and 3/4 shots
  • Varied lighting (indoor, outdoor, flash, natural)
  • Different expressions (neutral, smiling, serious)
  • Eyes clearly visible
  • Minimum 512×512 resolution; 1024×1024+ preferred

5. Train LoRAs (one per person)

./train_lora.sh Alice    # ~1-2 hours on RTX 4090
./train_lora.sh Bob
./train_lora.sh Carol

Monitor training progress at http://localhost:6006 (TensorBoard).

After training, each LoRA is automatically copied to: ComfyUI/models/loras/<name>_flux.safetensors

6. Generate the portrait

./start_comfyui.sh

Open http://localhost:8188 in your browser, then:

  1. Click Load → select dutch_masters_portrait_workflow.json
  2. In the LoraLoader nodes, set each person's LoRA file
  3. In the positive prompt, replace portrait_PERSON1 etc. with the trigger words from add_person.sh output (e.g. portrait_alice)
  4. Click Queue Prompt

The workflow generates at 1024×1280 then upscales 4× to 4096×5120 pixels (~13×17 inches at 300 DPI). For a 16×20 print, run a second 1.5× upscale pass in ComfyUI or use Topaz Gigapixel.

Trigger words

Each person's trigger word is: portrait_<firstname_lowercase>

  • Alice → portrait_alice
  • Bob → portrait_bob

Generation tips

Dutch Masters style prompts:

  • "oil painting in the style of Rembrandt van Rijn, dramatic chiaroscuro lighting"
  • "Dutch Golden Age portrait, Vermeer-style soft diffused light, sfumato"
  • "17th century Netherlands, rich dark background, ornate period clothing"

Composition:

  • Generate multiple seeds (change the seed in KSampler) — composition varies a lot
  • For very large families (5+ people), consider generating 2-3 at a time and compositing

LoRA weights:

  • Start at 0.85 per person
  • If someone's identity is weak, increase their weight to 0.9–1.0
  • If someone overrides the style, lower their weight to 0.7–0.75
  • Style stays at FLUX default (no explicit style LoRA needed — the prompt handles it)

Steps to get print-ready:

  1. Generate with the ComfyUI workflow → 4096×5120 base
  2. Optional: Second 1.5× upscale pass for 6144×7680 (20×25" at 300 DPI)
  3. Touch up faces with Photoshop or GIMP if needed
  4. Export as TIFF for printing

Conda environments

  • flux-portrait — ComfyUI (Python 3.11, PyTorch 2.6+CUDA12.4)
  • simpletuner — LoRA training (Python 3.12, PyTorch 2.6+CUDA12.4)
#!/usr/bin/env bash
# Sets up a training directory and SimpleTuner config for one family member.
# Usage: ./add_person.sh <firstname>
#
# After running this:
# 1. Copy 15-30 photos of the person into training-data/<firstname>/
# 2. Run: ./train_lora.sh <firstname>
#
# Photo guidelines:
# - 15-30 images (more is better, ~25 is the sweet spot)
# - Mix of close-up face shots, half-body, and 3/4 body shots
# - Varied lighting: indoor, outdoor, flash, natural window light
# - Different expressions: neutral, smiling, serious
# - Eyes clearly visible, no heavy occlusion
# - Minimum 512x512 resolution; 1024x1024+ preferred
set -e
if [ -z "$1" ]; then
echo "Usage: ./add_person.sh <firstname>"
exit 1
fi
NAME="$1"
NAME_LOWER="${NAME,,}"
TRIGGER="portrait_${NAME_LOWER}"
TRAIN_DIR="/home/jimmy/portrait-studio/training-data/$NAME"
ST_CONFIG_DIR="/home/jimmy/portrait-studio/SimpleTuner/config/$NAME"
LORA_OUT="/home/jimmy/portrait-studio/loras/$NAME"
VAE_CACHE="/home/jimmy/portrait-studio/training-data/${NAME}_vae_cache"
EMBED_CACHE="/home/jimmy/portrait-studio/training-data/${NAME}_embed_cache"
mkdir -p "$TRAIN_DIR" "$ST_CONFIG_DIR" "$LORA_OUT" "$VAE_CACHE" "$EMBED_CACHE"
# Write SimpleTuner main training config (config.json is read from $ST_CONFIG_DIR)
cat > "$ST_CONFIG_DIR/config.json" <<EOF
{
"--resume_from_checkpoint": "latest",
"--data_backend_config": "config/${NAME}/databackend.json",
"--aspect_bucket_rounding": 2,
"--seed": 42,
"--minimum_image_size": 512,
"--output_dir": "${LORA_OUT}",
"--lora_type": "standard",
"--max_train_steps": 2000,
"--num_train_epochs": 0,
"--checkpoint_step_interval": 250,
"--checkpoints_total_limit": 4,
"--tracker_project_name": "family-portrait",
"--tracker_run_name": "${NAME}-lora",
"--report_to": "tensorboard",
"--model_type": "lora",
"--lora_rank": 32,
"--lora_alpha": 16,
"--pretrained_model_name_or_path": "/home/jimmy/portrait-studio/ComfyUI/models/diffusion_models/flux1-dev.safetensors",
"--model_family": "flux",
"--flux_lora_target": "all",
"--train_batch_size": 1,
"--gradient_checkpointing": "true",
"--gradient_accumulation_steps": 4,
"--caption_dropout_probability": 0.1,
"--resolution_type": "pixel_area",
"--resolution": 1048576,
"--validation_seed": 42,
"--validation_step_interval": 250,
"--validation_resolution": "1024x1024",
"--validation_guidance": 3.5,
"--validation_guidance_rescale": "0.0",
"--validation_num_inference_steps": "20",
"--validation_prompt": "A photorealistic portrait of ${TRIGGER}, Dutch Masters oil painting style, dramatic Rembrandt chiaroscuro lighting, dark background, 17th century Netherlands",
"--mixed_precision": "bf16",
"--optimizer": "adamw_bf16",
"--learning_rate": "4e-4",
"--lr_scheduler": "cosine",
"--lr_warmup_steps": 100,
"--vae_path": "/home/jimmy/portrait-studio/ComfyUI/models/vae/ae.safetensors",
"--text_encoder_path": "/home/jimmy/portrait-studio/ComfyUI/models/text_encoders/clip_l.safetensors",
"--text_encoder_2_path": "/home/jimmy/portrait-studio/ComfyUI/models/text_encoders/t5xxl_fp8_e4m3fn.safetensors",
"--validation_torch_compile": "false",
"--logging_dir": "${LORA_OUT}/logs"
}
EOF
# Write data backend config (paths must be absolute)
cat > "$ST_CONFIG_DIR/databackend.json" <<EOF
[
{
"id": "${NAME_LOWER}-photos",
"type": "local",
"instance_data_dir": "${TRAIN_DIR}",
"crop": false,
"minimum_image_size": 512,
"maximum_image_size": 1536,
"target_downsample_size": 1024,
"resolution": 1048576,
"resolution_type": "pixel_area",
"prepend_instance_prompt": true,
"instance_prompt": "${TRIGGER}",
"only_instance_prompt": false,
"caption_strategy": "filename",
"cache_dir_vae": "${VAE_CACHE}",
"vae_cache_clear_each_epoch": false,
"probability": 1.0,
"repeats": 10,
"text_embeds": "${NAME_LOWER}-embed-cache"
},
{
"id": "${NAME_LOWER}-embed-cache",
"dataset_type": "text_embeds",
"default": true,
"type": "local",
"cache_dir": "${EMBED_CACHE}"
}
]
EOF
echo "Person configured: $NAME"
echo "Trigger word: $TRIGGER"
echo ""
echo "NEXT STEPS:"
echo " 1. Copy 15-30 photos of ${NAME} into:"
echo " $TRAIN_DIR/"
echo ""
echo " File names become captions, so be descriptive:"
echo " close_face_smiling.jpg"
echo " half_body_outdoor_natural_light.jpg"
echo " serious_expression_indoor.jpg"
echo " (The trigger word '${TRIGGER}' is prepended automatically)"
echo ""
echo " 2. After adding photos, train the LoRA:"
echo " ./train_lora.sh ${NAME}"
#!/usr/bin/env bash
# Downloads FLUX.1-dev and all required models for portrait generation.
# Prerequisites:
# 1. Accept the FLUX.1-dev license at: https://huggingface.co/black-forest-labs/FLUX.1-dev
# 2. Create a token at: https://huggingface.co/settings/tokens
# 3. Run: HF_TOKEN=hf_yourtoken ./download_models.sh
set -e
if [ -z "$HF_TOKEN" ]; then
echo "ERROR: HF_TOKEN not set."
echo " Create a token at https://huggingface.co/settings/tokens"
echo " Then run: HF_TOKEN=hf_yourtoken ./download_models.sh"
exit 1
fi
COMFY_MODELS="/home/jimmy/portrait-studio/ComfyUI/models"
echo "Downloading models to $COMFY_MODELS ..."
HF_ARGS="--token $HF_TOKEN"
# FLUX.1-dev transformer (main model, ~24GB)
echo ""
echo "==> FLUX.1-dev transformer (~24GB) ..."
conda run -n flux-portrait huggingface-cli download \
black-forest-labs/FLUX.1-dev \
flux1-dev.safetensors \
--local-dir "$COMFY_MODELS/diffusion_models" \
$HF_ARGS
# FLUX VAE (~335MB)
echo ""
echo "==> FLUX VAE ..."
conda run -n flux-portrait huggingface-cli download \
black-forest-labs/FLUX.1-dev \
ae.safetensors \
--local-dir "$COMFY_MODELS/vae" \
$HF_ARGS
# CLIP-L text encoder (~246MB)
echo ""
echo "==> CLIP-L text encoder ..."
conda run -n flux-portrait huggingface-cli download \
comfyanonymous/flux_text_encoders \
clip_l.safetensors \
--local-dir "$COMFY_MODELS/text_encoders"
# T5-XXL text encoder fp8 (~5GB, fp8 saves VRAM vs fp16)
echo ""
echo "==> T5-XXL text encoder (fp8, ~5GB) ..."
conda run -n flux-portrait huggingface-cli download \
comfyanonymous/flux_text_encoders \
t5xxl_fp8_e4m3fn.safetensors \
--local-dir "$COMFY_MODELS/text_encoders"
# Real-ESRGAN 4x upscaler for final print resolution
echo ""
echo "==> Real-ESRGAN 4x upscaler ..."
mkdir -p "$COMFY_MODELS/upscale_models"
wget -q --show-progress \
"https://huggingface.co/ai-forever/Real-ESRGAN/resolve/main/RealESRGAN_x4.pth" \
-O "$COMFY_MODELS/upscale_models/RealESRGAN_x4.pth"
# Face restoration model (for touch-ups after upscaling)
echo ""
echo "==> GFPGAN face restoration ..."
mkdir -p "$COMFY_MODELS/face_restore"
wget -q --show-progress \
"https://huggingface.co/ai-forever/Real-ESRGAN/resolve/main/GFPGANv1.4.pth" \
-O "$COMFY_MODELS/face_restore/GFPGANv1.4.pth" || \
echo "(Optional — skipping GFPGANv1.4 if download fails)"
echo ""
echo "All models downloaded."
echo ""
echo "Next step: train LoRAs for each family member with:"
echo " ./add_person.sh <firstname>"
{
"_note": "Dutch Masters family portrait workflow for ComfyUI. Load this via Load Workflow in the UI. Before using: 1) Edit the LoraLoader nodes to point to each person's LoRA. 2) Edit the CLIPTextEncode prompt with your family members' trigger words. 3) Run.",
"last_node_id": 30,
"last_link_id": 40,
"nodes": [
{
"id": 1,
"type": "UNETLoader",
"pos": [0, 0],
"size": [300, 60],
"outputs": [{"name": "MODEL", "type": "MODEL", "links": [1]}],
"properties": {},
"widgets_values": ["flux1-dev.safetensors", "fp8_e4m3fn"]
},
{
"id": 2,
"type": "DualCLIPLoader",
"pos": [0, 100],
"size": [300, 80],
"outputs": [{"name": "CLIP", "type": "CLIP", "links": [2]}],
"properties": {},
"widgets_values": ["clip_l.safetensors", "t5xxl_fp8_e4m3fn.safetensors", "flux"]
},
{
"id": 3,
"type": "VAELoader",
"pos": [0, 200],
"size": [300, 60],
"outputs": [{"name": "VAE", "type": "VAE", "links": [3]}],
"properties": {},
"widgets_values": ["ae.safetensors"]
},
{
"id": 10,
"type": "LoraLoader",
"title": "Person 1 LoRA",
"pos": [350, 0],
"size": [300, 100],
"inputs": [
{"name": "model", "type": "MODEL", "link": 1},
{"name": "clip", "type": "CLIP", "link": 2}
],
"outputs": [
{"name": "MODEL", "type": "MODEL", "links": [10]},
{"name": "CLIP", "type": "CLIP", "links": [11]}
],
"widgets_values": ["person1_flux.safetensors", 0.85, 0.85],
"_comment": "Change person1_flux.safetensors to your first person's LoRA filename"
},
{
"id": 11,
"type": "LoraLoader",
"title": "Person 2 LoRA",
"pos": [350, 120],
"size": [300, 100],
"inputs": [
{"name": "model", "type": "MODEL", "link": 10},
{"name": "clip", "type": "CLIP", "link": 11}
],
"outputs": [
{"name": "MODEL", "type": "MODEL", "links": [12]},
{"name": "CLIP", "type": "CLIP", "links": [13]}
],
"widgets_values": ["person2_flux.safetensors", 0.85, 0.85],
"_comment": "Change person2_flux.safetensors to your second person's LoRA filename"
},
{
"id": 12,
"type": "LoraLoader",
"title": "Person 3 LoRA (duplicate node for more people)",
"pos": [350, 240],
"size": [300, 100],
"inputs": [
{"name": "model", "type": "MODEL", "link": 12},
{"name": "clip", "type": "CLIP", "link": 13}
],
"outputs": [
{"name": "MODEL", "type": "MODEL", "links": [14]},
{"name": "CLIP", "type": "CLIP", "links": [15]}
],
"widgets_values": ["person3_flux.safetensors", 0.85, 0.85],
"_comment": "Add more LoraLoader nodes in a chain for additional people"
},
{
"id": 20,
"type": "CLIPTextEncode",
"title": "Positive Prompt",
"pos": [700, 0],
"size": [500, 200],
"inputs": [{"name": "clip", "type": "CLIP", "link": 15}],
"outputs": [{"name": "CONDITIONING", "type": "CONDITIONING", "links": [20]}],
"widgets_values": ["A Dutch Golden Age family portrait, portrait_PERSON1 and portrait_PERSON2 and portrait_PERSON3 seated together, oil painting in the style of Rembrandt van Rijn, dramatic chiaroscuro lighting from the left, warm candlelight, rich dark background, ornate period clothing, dignified aristocratic composition, masterwork detail, visible brushwork, sfumato, glazing technique, golden hour tonality, museum quality painting, 17th century Netherlands, photorealistic faces"],
"_comment": "Replace portrait_PERSON1 etc. with your trigger words from add_person.sh"
},
{
"id": 21,
"type": "CLIPTextEncode",
"title": "Negative Prompt",
"pos": [700, 220],
"size": [500, 150],
"inputs": [{"name": "clip", "type": "CLIP", "link": 15}],
"outputs": [{"name": "CONDITIONING", "type": "CONDITIONING", "links": [21]}],
"widgets_values": ["cartoon, anime, illustration, deformed, ugly, blurry, low quality, watermark, signature, modern clothing, photograph, photo, digital art, neon, bright colors, flat lighting"]
},
{
"id": 22,
"type": "EmptyLatentImage",
"pos": [700, 400],
"size": [300, 100],
"outputs": [{"name": "LATENT", "type": "LATENT", "links": [22]}],
"widgets_values": [1024, 1280, 1],
"_comment": "1024x1280 is portrait aspect ratio, good for a family portrait"
},
{
"id": 23,
"type": "KSampler",
"pos": [1050, 0],
"size": [350, 250],
"inputs": [
{"name": "model", "type": "MODEL", "link": 14},
{"name": "positive", "type": "CONDITIONING", "link": 20},
{"name": "negative", "type": "CONDITIONING", "link": 21},
{"name": "latent_image", "type": "LATENT", "link": 22}
],
"outputs": [{"name": "LATENT", "type": "LATENT", "links": [23]}],
"widgets_values": [42, "euler", "simple", 30, 3.5, "disable", 1.0],
"_comment": "seed=42, steps=30, cfg=3.5 work well for FLUX. Try different seeds for composition variety."
},
{
"id": 24,
"type": "VAEDecode",
"pos": [1450, 0],
"size": [200, 80],
"inputs": [
{"name": "samples", "type": "LATENT", "link": 23},
{"name": "vae", "type": "VAE", "link": 3}
],
"outputs": [{"name": "IMAGE", "type": "IMAGE", "links": [24, 25]}]
},
{
"id": 25,
"type": "UpscaleModelLoader",
"pos": [1050, 300],
"size": [300, 60],
"outputs": [{"name": "UPSCALE_MODEL", "type": "UPSCALE_MODEL", "links": [26]}],
"widgets_values": ["RealESRGAN_x4.pth"]
},
{
"id": 26,
"type": "ImageUpscaleWithModel",
"pos": [1450, 120],
"size": [300, 80],
"inputs": [
{"name": "upscale_model", "type": "UPSCALE_MODEL", "link": 26},
{"name": "image", "type": "IMAGE", "link": 24}
],
"outputs": [{"name": "IMAGE", "type": "IMAGE", "links": [27]}],
"_comment": "4x upscale: 1024x1280 -> 4096x5120 pixels (~13x17 inches at 300 DPI)"
},
{
"id": 27,
"type": "SaveImage",
"title": "Save Upscaled Portrait",
"pos": [1800, 120],
"size": [300, 80],
"inputs": [{"name": "images", "type": "IMAGE", "link": 27}],
"widgets_values": ["dutch_masters_portrait", "png", true]
},
{
"id": 28,
"type": "SaveImage",
"title": "Save Base (before upscale)",
"pos": [1800, 0],
"size": [300, 80],
"inputs": [{"name": "images", "type": "IMAGE", "link": 25}],
"widgets_values": ["dutch_masters_portrait_base", "png", false]
}
],
"links": [
[1, 1, 0, 10, 0, "MODEL"],
[2, 2, 0, 10, 1, "CLIP"],
[3, 3, 0, 24, 1, "VAE"],
[10, 10, 0, 11, 0, "MODEL"],
[11, 10, 1, 11, 1, "CLIP"],
[12, 11, 0, 12, 0, "MODEL"],
[13, 11, 1, 12, 1, "CLIP"],
[14, 12, 0, 23, 0, "MODEL"],
[15, 12, 1, 20, 0, "CLIP"],
[20, 20, 0, 23, 1, "CONDITIONING"],
[21, 21, 0, 23, 2, "CONDITIONING"],
[22, 22, 0, 23, 3, "LATENT"],
[23, 23, 0, 24, 0, "LATENT"],
[24, 24, 0, 26, 1, "IMAGE"],
[25, 24, 0, 28, 0, "IMAGE"],
[26, 25, 0, 26, 0, "UPSCALE_MODEL"],
[27, 26, 0, 27, 0, "IMAGE"]
],
"groups": [],
"config": {},
"extra": {"ds": {"scale": 0.7, "offset": [0, 0]}}
}
#!/usr/bin/env bash
# Launches ComfyUI on http://localhost:8188
# The --highvram flag keeps models in VRAM permanently for faster generation.
# Open your browser to http://localhost:8188 after launch.
cd /home/jimmy/portrait-studio/ComfyUI
conda run -n flux-portrait python main.py \
--highvram \
--listen 0.0.0.0 \
--port 8188 \
"$@"
#!/usr/bin/env bash
# Trains a FLUX LoRA for one family member.
# Usage: ./train_lora.sh <firstname>
#
# Expected runtime on RTX 4090: ~1-2 hours per person
set -e
if [ -z "$1" ]; then
echo "Usage: ./train_lora.sh <firstname>"
exit 1
fi
NAME="$1"
ST_DIR="/home/jimmy/portrait-studio/SimpleTuner"
CONFIG_DIR="$ST_DIR/config/$NAME"
TRAIN_DIR="/home/jimmy/portrait-studio/training-data/$NAME"
LORA_OUT="/home/jimmy/portrait-studio/loras/$NAME"
if [ ! -f "$CONFIG_DIR/config.json" ]; then
echo "ERROR: No config found for $NAME. Run ./add_person.sh $NAME first."
exit 1
fi
PHOTO_COUNT=$(find "$TRAIN_DIR" -name "*.jpg" -o -name "*.png" -o -name "*.jpeg" -o -name "*.webp" 2>/dev/null | wc -l)
if [ "$PHOTO_COUNT" -lt 5 ]; then
echo "ERROR: Only $PHOTO_COUNT photos found in $TRAIN_DIR"
echo "Please add at least 15 photos before training."
exit 1
fi
echo "Found $PHOTO_COUNT photos for ${NAME}."
# Launch TensorBoard in background
conda run -n simpletuner tensorboard --logdir "$LORA_OUT" --port 6006 &>/tmp/tb_${NAME}.log &
echo "TensorBoard started at http://localhost:6006"
echo ""
echo "Starting FLUX LoRA training for: $NAME"
echo ""
# SimpleTuner reads config from config/<ENV>/config.json when CONFIG_BACKEND=json
# The ENV variable tells it which subdirectory to use
cd "$ST_DIR"
DISABLE_UPDATES=1 ENV="$NAME" CONFIG_BACKEND="json" \
conda run -n simpletuner bash simpletuner/train.sh \
2>&1 | tee "$LORA_OUT/training.log"
# Copy final LoRA to ComfyUI
FINAL_LORA=$(find "$LORA_OUT" -name "*.safetensors" | grep -v "optimizer" | sort | tail -1)
if [ -n "$FINAL_LORA" ]; then
DEST="/home/jimmy/portrait-studio/ComfyUI/models/loras/${NAME}_flux.safetensors"
cp "$FINAL_LORA" "$DEST"
echo ""
echo "Training complete! LoRA copied to: $DEST"
echo "Trigger word for prompts: portrait_${NAME,,}"
else
echo "Training finished. Check $LORA_OUT for output .safetensors files."
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment