Skip to content

Instantly share code, notes, and snippets.

@devrim

devrim/partfield-image-to-mesh-research.md

Created April 21, 2026 07:51
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save devrim/7d16280bc3c3ea773d7a02387a0e99ac to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save devrim/7d16280bc3c3ea773d7a02387a0e99ac to your computer and use it in GitHub Desktop.
Download ZIP
Image-to-Mesh for PartField — ComfyUI ecosystem landscape + recommendation (April 2026)
Raw
partfield-image-to-mesh-research.md

Image-to-Mesh for PartField — Landscape & Recommendation

Deep-research note on whether to add a ComfyUI-style image-to-mesh backend to the PartField repo, and if so, which one(s). Written April 2026.

TL;DR

We already have it. The PartField repo has Microsoft TRELLIS-image-large fully wired up via trellis_manager.py and the POST /trellis/image_to_3d FastAPI endpoint (api.py:430), with GLB + Gaussian-PLY output, TTL-based GPU offload, and an async job queue. This is the same model most "serious" ComfyUI image-to-3D workflows use today.

What's worth actually adding alongside it — in order of ROI for this repo:

  1. Upgrade to TRELLIS.2-4B (Dec 2025, MIT, same family, native PBR textures, better topology, up to 1536³ resolution) — minimal refactor, large quality jump.
  2. Add Hunyuan3D-2.1 as a second backend — genuinely different strengths (PBR paint model is best-in-class among open weights, accepts image+text+multi-view). License has teeth (EU/UK/KR excluded, 1M MAU cap) — check before production.
  3. Part-aware pair: PartCrafter + HoloPart — genuinely novel capability that slots into PartField's segmentation thesis (image → part-separated 3D mesh, then complete occluded part geometry). Nothing else in our stack does this.

Skip: TripoSR, InstantMesh, SF3D, ComfyUI-3D-Pack, Unique3D, CRM, Wonder3D, OpenLRM — either outclassed, license-blocked, or bundling-overhead without payoff.

1. Current state in this repo

Piece Location Notes
Model microsoft/TRELLIS-image-large (v1, Dec 2024) MIT
Pipeline holder trellis_manager.py:33 Singleton, TrellisManager class
HTTP entry point api.py:430 (POST /trellis/image_to_3d) Async job pattern
Output formats glb, gaussian_ply api.py:399-414
Output dir exp_results/trellis/{job_id}/
GPU lifecycle TTL-idle offload (default 300 s), preload/offload endpoints trellis_manager.py:142-164
Vendored source trellis_src/ — ~96 MB, 350 files Precedent for full-tree vendoring
Env PyTorch 2.4.0 + CUDA 12.4 environment.yml

Practical takeaway: the <backend>_manager.py + POST /<backend>/image_to_3d pattern is already proven. Adding a second backend is a copy-paste-and-adapt job, not an architectural change.

2. ComfyUI ecosystem map (what the question was really about)

Comfy.org's "official" image-to-3D story is two-layered:

Everything else is community wrappers. No official Microsoft, Stability, or VAST ComfyUI node exists for their own models.

The canonical community wrappers most ComfyUI users pick in April 2026:

Backend Canonical ComfyUI node Status
TRELLIS v1 if-ai/ComfyUI-IF_Trellis, smthemex/ComfyUI_TRELLIS Mature
TRELLIS.2 visualbruno/ComfyUI-Trellis2, PozzettiAndrea/ComfyUI-TRELLIS2 New (Dec 2025)
Hunyuan3D 2.x kijai/ComfyUI-Hunyuan3DWrapper (de-facto standard, ~1 k ★) Very active
TripoSG/TripoSR VAST-AI-Research/ComfyUI-Tripo (official from VAST) Active
InstantMesh jtydhr88/ComfyUI-InstantMesh Legacy
Kitchen sink MrForExample/ComfyUI-3D-Pack Maintenance mode (stuck on torch 2.4/CUDA 12.4, same as us)

The community has moved away from the mega-pack toward per-model wrappers because the 3D-Pack's dep matrix keeps rotting. For us (FastAPI, not ComfyUI), per-model wrappers aren't directly reusable anyway — we'd integrate the underlying HuggingFace pipelines the same way trellis_manager.py does today.

3. Open-weight model landscape (self-hostable only)

Tier S — the three that matter in 2026

Model Org Released License VRAM Speed Output
TRELLIS.2-4B Microsoft Dec 2025 MIT ~24 GB ~3 s @ 512³, ~60 s @ 1536³ (H100) PBR GLB (diffuse/normal/roughness/metallic), 3DGS, RF
Hunyuan3D-2.1 Tencent Jun 2025 Tencent Hunyuan Community ~10 GB shape / ~21 GB texture / ~29 GB combined ~30–60 s PBR textured mesh, configurable 40 K–1.5 M polys
TripoSG VAST-AI Mar 2025 MIT ~8 GB Fast Clean mesh geometry

Tier A — part-aware (highly relevant for PartField)

Model License Why it matters here
PartCrafter (PKU/ByteDance/CMU, NeurIPS 2025) Research (verify before commercial) Single image → part-segmented multi-mesh in one pass. TRELLIS outputs a monolithic asset; PartCrafter doesn't. Directly complementary to PartField's segmentation thesis.
HoloPart (VAST-AI, Apr 2025) MIT Amodal part completion — given a segmented 3D shape, fills the occluded geometry inside each part. Perfect downstream of PartField's clustering.
Hunyuan3D-Part (P3-SAM + X-Part) Tencent community Combines native 3D part segmentation with part generation — overlaps with PartField on one axis.
PartPacker (NVlabs, Jun 2025) Non-commercial Alternative to PartCrafter, but the license blocks production use.

Tier B — legacy / outclassed

Model Why skip
TripoSR (Mar 2024) Speed niche only; quality far below TRELLIS/Hunyuan
InstantMesh (Apr 2024) Nice FlexiCubes topology, but geometry quality below TRELLIS
Stable Fast 3D / SPAR3D License gate: Stability Community License ($1 M revenue cap)
Unique3D, CRM, Era3D, Wonder3D, Zero123++ Research artifacts, mostly subsumed by above
OpenLRM / GS-LRM / Long-LRM Historical lineage; subsumed by TRELLIS/TripoSG
CraftsMan3D, Direct3D, MV-Adapter, Hi3DGen Specialized / component-level; not end-to-end replacements

Commercial (NOT self-hostable — noted, skipped)

Meshy, Tripo AI (service, distinct from TripoSG weights), Rodin Gen-1/Gen-2, CSM.ai, Hyper3D, Luma Genie, Hunyuan3D 2.5 and 3.0 (weights unreleased), Hitem3D. Tripo 3.0 (Feb 2026) is the current subjective quality leader but isn't open.

4. License details worth writing down before choosing

Model License Commercial use Gotchas
TRELLIS v1 / TRELLIS.2 MIT ✅ None
TripoSG / TripoSR / HoloPart MIT / Apache-2.0 ✅ None
InstantMesh Apache-2.0 ✅ None
Hunyuan3D 2.x Tencent Hunyuan 3D Community License ⚠️ Excludes EU/UK/South Korea; separate license above 1 M MAU
Stable Fast 3D Stability AI Community License ⚠️ Free up to $1 M annual revenue, enterprise license above
PartPacker Non-commercial ❌ for production Blocker
PartCrafter Research ⚠️ License needs a direct read of the repo before shipping

5. Concrete recommendation for PartField

Phase 1 — low-risk quality bump (1 day). Upgrade the existing TRELLIS integration from TRELLIS-image-large to microsoft/TRELLIS.2-4B. Same pipeline API shape, MIT license, PBR textures, better topology. Change is roughly:

  • TRELLIS_MODEL default in trellis_manager.py:30 → microsoft/TRELLIS.2-4B
  • Verify trellis_src/ is compatible, or update the vendored tree to the TRELLIS.2 source
  • Expose the new resolution knob (512/1024/1536) as a form field in POST /trellis/image_to_3d

Phase 2 — add a second, complementary backend (a few days). Add Hunyuan3D-2.1 as hunyuan_manager.py + POST /hunyuan/image_to_3d, modeled directly on the TRELLIS pattern. Split its two-stage pipeline (shape DiT, then PBR paint) behind a single endpoint with a paint: bool flag. Output to exp_results/hunyuan/{job_id}/.

Rationale: Hunyuan's PBR paint model produces noticeably more photoreal textures than TRELLIS.2 today on a non-trivial subset of inputs, and it accepts multi-view input (2mv variant) which TRELLIS does not.

Blocker to resolve first: confirm Tencent's license terms match the intended deployment — if this is served publicly to EU users or has >1 M MAU, we need a separate agreement or we skip it.

Phase 3 — part-aware pipeline (optional, 1–2 weeks, biggest research payoff). This is where PartField becomes genuinely differentiated. Add a two-stage chain:

image → PartCrafter (part-separated mesh) → HoloPart (amodal completion per part) → PartField (feature field / correspondence)

Nothing in the ComfyUI ecosystem currently does this cleanly — it's a real reason someone would come to PartField specifically rather than a generic image-to-3D service. Gate it behind POST /parts/image_to_parts.

Skip ComfyUI-3D-Pack (dep rot, overlaps with what we'd build), SF3D (license), TripoSR/InstantMesh (outclassed), PartPacker (non-commercial).

6. Integration sketch (for Phase 2)

Following the existing pattern (trellis_manager.py → api.py), a second backend is a ~200-line diff:

partfield/
├── trellis_manager.py        # existing
├── hunyuan_manager.py        # NEW — singleton, TTL offload, load/run/state API
├── api.py                    # + /hunyuan/image_to_3d, /hunyuan/status, /hunyuan/jobs
├── hunyuan_src/              # NEW — vendor the Tencent-Hunyuan/Hunyuan3D-2.1 tree (~same scale as trellis_src)
└── exp_results/hunyuan/      # per-job output dirs, same convention as trellis/

No change to PartField's core inference/clustering pipeline. The two backends run serially via their own locks — they can't share a GPU at 24 + 29 GB anyway without offload.

For Phase 3, PartCrafter+HoloPart feed directly into the existing /infer + /cluster endpoints, since those already accept meshes in data/.

7. Sources

ComfyUI ecosystem

Models

Comparisons / news

Uncertainty flagged

  • PartCrafter explicit license text was not directly verified — read the repo before production use.
  • Hunyuan3D 2.5/3.0 weight-release status could change (both closed as of April 2026).
  • TRELLIS.2 release date is December 2025 per multiple community sources; exact weights-drop day unverified.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment