PAJAMA / RELIGHT

06 / 08 · RELIGHTING

Learn materials without learning the wrong thing

Trace inverse rendering from observations to a relightable asset.

One pixel has many explanations

Diffuse observations contain the product ρE. Doubling reflectance and halving illumination leaves a pixel unchanged, within valid material ranges. More camera views constrain geometry but do not automatically resolve all material-light ambiguity under the same illumination.

At cosine 1, Lambert radiance 0.24 can come from ρ=0.3 and E=0.8π, or ρ=0.6 and E=0.4π. Zero loss on that pixel proves neither material estimate correct.

ρEπ=(kρ)(E/k)π\frac{\rho E}{\pi}=\frac{(k\rho)(E/k)}{\pi}

One training iteration, end to end

1. Pick a calibrated camera, time and reference image; linearize the image with a known or modeled camera response. 2. Evaluate canonical Gaussians and deformation. 3. Project geometry and obtain normals/visibility. 4. Evaluate BRDF under estimated light and composite the image.

5. Compare predicted pixels to observations; add geometry, material and temporal constraints. 6. Backpropagate through differentiable projection, shading and compositing; update selected parameters. 7. Repeat across views/lights; test held-out lighting. Densification must initialize new material attributes consistently.

min⁡Θg,Θm,ΘL,Θd∑c,t∥R(Θg,Θm,ΘL,Θd;c,t)−Ic,t∥1+λnLn+λmLm+λtLt\min_{\Theta_g,\Theta_m,\Theta_L,\Theta_d}\sum_{c,t}\|R(\Theta_g,\Theta_m,\Theta_L,\Theta_d;c,t)-I_{c,t}\|_1+\lambda_n\mathcal L_n+\lambda_m\mathcal L_m+\lambda_t\mathcal L_t

Constrain the problem before scaling it up

Known lights, calibrated exposure, multiple illumination conditions, reliable geometry/normals and material priors constrain different ambiguities. Begin with a synthetic object whose cameras, lights and materials are known. Then try real scenes.

GS-IR and Relightable 3D Gaussian provide released training code. Their CUDA rasterizers, datasets, checkpoints and exports are separate from this viewer. This site links them but does not claim a reproduced training run. Our implementation is the known-material forward renderer.

Predict it, then run it

Open this experiment →

The formula meets actual code

This excerpt is extracted from the repository during the site build. SH environment lighting and inverse training are future extensions; their chapters point to the current direct-light/data boundary, not an implemented trainer.

src/relight.rs · L19–28
Full source ↗
#[repr(C)]
#[derive(Clone, Copy, Pod, Zeroable)]
struct Splat {
    center: [f32; 4],
    u: [f32; 4],
    v: [f32; 4],
    w: [f32; 4],
    normal: [f32; 4],
    base: [f32; 4],
}

Check your understanding

Why is high novel-view PSNR insufficient evidence of successful relighting?

Reveal the reasoning

The model may reproduce old lighting with baked shadows. Validate unseen lights, material/normal consistency and temporal stability, not only new cameras under the old light.

Primary references for this chapter

GS-IR · CVPR 2024 ↗

Relightable 3D Gaussian · ECCV 2024 ↗