Skip to content

reference.md

Latest commit

 

History

History
181 lines (137 loc) · 7.22 KB

reference.md

File metadata and controls

181 lines (137 loc) · 7.22 KB
Raw

remu — reference

Everything beyond the README quickstart: environment, data conventions, the four stages, and per-stage drivers for standalone use. The unified launcher (scripts/run.py) is the normal entry point; the per-stage drivers below are for debugging or running a single stage.

1. Environment & GPU toolchain

One unified env hosts every stage. Key pins:

  • Python 3.12, torch 2.5.1+cu121, torchvision 0.20.1, kaolin 0.18.0
  • numpy 2.x, trimesh 4.12.2, open3d 0.19.0, nvdiffrast 0.4
  • diffusers 0.31.0 / transformers 4.46.2 (UP2You + SD3 inpaint)
  • scikit-image >=0.24.0, PyMaxflow (neural-UDF)
  • no chumpy, no pytorch3d

The nvdiffrast / kaolin CUDA plugins build lazily at runtime:

export CUDA_HOME=/usr/local/cuda-12.1 CC=$(which gcc-12) CXX=$(which g++-12)
export PATH=$CUDA_HOME/bin:$PATH

nvcc 12.1 requires gcc ≤ 12 — set CC/CXX explicitly.

2. Weights & data

We ship no weights of our own — the pipeline is optimization-based.

What Source Path
SMPL-X body models smpl-x.is.tue.mpg.de body_models/smplx/SMPLX_*.npz
SiTH recon model files.ait.ethz.ch checkpoints/sith/recon_model.pth
SiTH diffusion HF hohs/SiTH-diffusion-1K (auto-downloaded) —
UP2You weights HF Co2y/UP2You checkpoints/up2you/{pretrained_models,human_models} 
# Download helpers
python scripts/download_data.py --subset examples
wget -O checkpoints/sith/recon_model.pth \
    https://files.ait.ethz.ch/projects/SiTH/recon_model.pth
huggingface-cli download Co2y/UP2You --local-dir checkpoints/up2you

Dataset layout

<dataset_root>/
  prompts.json                        per-subject segmentation prompts
  input_img/<sub>_<Layer>.png         per-layer RGB images
  smplx_params/
    <sub>_<Layer>_smplx.pkl           SMPL-X params (betas, body_pose, gender, scale)
    <sub>_<Layer>_smplx.obj           matching SMPL-X body mesh
  • Layers: Outer, Inner, Lower (3 layers) or Outer, InOuter, Inner, Lower (4 layers).
  • Gender: auto-detected from the pkl. Pass --gender only to override.
  • prompts.json: auto-discovered from dataset root or its parent. Each subject needs a per-layer prompt (garment noun for 3D segmentation). Use plain garment nouns (jacket, shirt) — never color-qualified (blue jacket) which causes LangSAM to return full-person masks.
  • Scale: SiTH-fit subjects carry scale in the pkl. 4D-Dress subjects use scale=1.0.
  • Frame convention: 4D-Dress params are pelvis-centered (global_orient=0, transl=-J0). SiTH-fit params carry lbs_frame=sith_fit with their original global_orient, transl, and scale.

3. The four stages

Each preset selects one backend per stage. The launcher uses a fixed 4-directory layout:

<work>/<preset>/<subject>/
  data/            → recon/           → registration/      → refine/
  (input_img,        (*_reco.obj,       (centered_meshes/,   (*.obj + *.mtl +
   output_img,        smplx_params/)     M_displaced/,        *_albedo.png +
   back_images)                          G_displaced/,        *_normal.png,
                                         label/, smplx/)      lowres/)

Step 1 — Data (src/data/)

  • layers (bmvc): per-layer images already on disk.
  • single_image (remu): one Outer image → SD3 inpaint cascade generates inner layers (Outer→InOuter→Inner→Lower). Each step: segment source garment → generate variants → select best (default d002_g000) → use as source for next layer.

The SD3 model (stabilityai/stable-diffusion-3-medium-diffusers) is gated — accept its HF terms and run huggingface-cli login before using generate: true.

python scripts/data/run_data.py --config configs/remu.yaml \
    --dataset-root <dir> --out-dir <out> --subjects <sub> --materialize

Step 2 — Reconstruction (src/reconstruction/)

Per-layer image + SMPL-X → textured 3D mesh.

  • SiTH (bmvc): back-view hallucination → neural reconstruction. ~1.5 min/layer on A100. Uses 1K diffusion model for quality.
  • UP2You (remu): DINOv2 + SD pipelines, GT SMPL-X β injection. Outputs A-pose meshes with gender=neutral and scale=1.0. Canonicalization is skipped for UP2You since all layers are already aligned.
python scripts/recon/run_sith.py   --image-dir <dir> --checkpoint <ckpt>
python scripts/recon/run_up2you.py --image-dir <dir> --gt-beta <sub>.pt \
    --weights-dir <weights> --human-models <models> --out-dir <out>

Step 3 — Registration & penetration removal (src/registration/)

For each layer: 3D garment segmentation (render → LangSAM → multiview vote) → canonicalize (inverse-LBS repose to outer layer's pose) → penetration removal (GPU-accelerated layerwise solver).

  • Canonicalize backends: knn_lbs (bmvc) | diffuse_lbs (remu, visibly cleaner).
  • Gender is auto-detected from the reconstruction pkl.
  • Penetration profile auto-detected from layers (InOuter → teaser4, else default3).
  • UP2You recon skips canonicalization — meshes are already aligned in A-pose.
python scripts/register/run_4ddress_registration.py \
    --recon sith --backend knn_lbs --subject 00127 \
    --recon-dir <recons> --params-dir <params> \
    --body-model-root body_models --out-dir <out> --run-penetration

Step 4 — Refinement (src/refinement/)

Restore texture + normal detail that penetration removal destroys.

  • neural_udf (bmvc): UDF fit → DCUDF single-layer extraction → render-bake. ~12 min on A100 (joint fit).
  • mesh (remu): Laplacian smooth → decimate → iso-contour cut → UV + render-bake. ~3 min on A100.

Both backends produce hi-res + lowres (~8K verts for simulation) output:

refine/
  <sub>-<Layer>.obj + .mtl + _albedo.png + _normal.png   (hi-res)
  lowres/                                                  (~8K verts)
  smplx/                                                   (forwarded body)

python scripts/refine/run_mesh.py       --config configs/remu.yaml --subject 00127 --reg-dir <reg> --out-dir <out>
python scripts/refine/run_neural_udf.py --config configs/bmvc.yaml --subject 00127 --reg-dir <reg> --out-dir <out>

Optional — Cloth simulation

Refined lowres garments can be driven through ContourCraft for physics simulation. See contourcraft_demo.md.

4. Customizing the pipeline

The pipeline is config-driven: a preset (configs/<preset>.yaml) names one backend per stage. To change behaviour, edit the preset, not the code.

  • Swap a backend — change backend in the preset:
    • reconstruction: sith | up2you
    • registration: knn_lbs | diffuse_lbs
    • refinement: neural_udf | mesh
  • Tune a stage — knobs are documented inline in the preset YAML.
  • Add your own backend — implement the stage interface and register in build():
    • src/reconstruction/base.py (ReconstructionBackend)
    • src/registration/canonicalize/base.py
    • src/refinement/base.py (RefinementBackend)
  • Mix presets — --steps runs a subset; --recon-dir / --reg-dir feed outputs from another run.