PAJAMA / RELIGHT

08 / 08 · RELIGHTING

Bring relighting into the engine

Separate working capabilities from the next implementation milestones.

Define the asset contract first

Declare version, units, coordinates, center/covariance/opacity, oriented normals, linear material fields, time/deformation binding and visibility convention. State whether color means base color, captured radiance or a learned residual. Never silently reinterpret an old RGB field.

The teaching layout is 96 bytes: six vec4 values. Roughness and metallic are frame uniforms for our single-material object. A real importer validates material attributes, rejects nonfinite values and declares missing capabilities.

Choose where shading happens

Here we shade each Gaussian then blend radiance. Deferred shading instead splats a G-buffer and shades pixels later. It may reduce repeated BRDF work, but averaging normals/roughness before nonlinear shading is not equivalent to blending shaded surfaces.

The host owns the surface, clock and input. RelightPass owns GPU resources and records commands into a host-provided encoder. It runs in browser WebGPU and native offscreen DX12. Separate passes can later handle shadows and environment prefiltering.

Five verifiable milestones

A, implemented: synthetic known materials/normals, direct light, five inspection modes, deformation and HDR composition. B, next: reproduce a released static method, build an explicit asset converter and match its reference renderer under a fixed light.

C: environment diffuse SH and specular IBL; compare a white Lambert sphere to numerical integration. D: visibility with a moving blocker. E: deform an imported asset with consistent material/normal binding; measure flicker under a fixed light. B–E are planned, not features of this lab.

Prepare a convincing explanation before the 28th

First reproduce the viewer: covariance, sorting and alpha. Then use one session for Lambert/GGX, one for this lab and its two passes, and one comparing inverse rendering with SH transfer. Be ready to explain material-light ambiguity and dynamic normal/visibility updates.

Reproducing every recent paper is a much larger project. For this deadline, build evidence: a running core, pixel checks, accurate capability boundaries and a concrete next step.

Predict it, then run it

Open this experiment →

The formula meets actual code

This excerpt is extracted from the repository during the site build. SH environment lighting and inverse training are future extensions; their chapters point to the current direct-light/data boundary, not an implemented trainer.

src/relight.rs · L298–397
Full source ↗
    pub fn render(
        &mut self,
        device: &wgpu::Device,
        queue: &wgpu::Queue,
        encoder: &mut wgpu::CommandEncoder,
        target: &wgpu::TextureView,
        size: [u32; 2],
        frame: RelightFrame,
    ) -> Result<(), String> {
        frame.validate()?;
        if size.contains(&0)
            || size
                .iter()
                .any(|&v| v > device.limits().max_texture_dimension_2d)
        {
            return Err("Invalid relight target size".into());
        }
        if self.hdr.as_ref().is_none_or(|v| v.2 != size) {
            let texture = device.create_texture(&wgpu::TextureDescriptor {
                label: Some("linear-radiance"),
                size: wgpu::Extent3d {
                    width: size[0],
                    height: size[1],
                    depth_or_array_layers: 1,
                },
                mip_level_count: 1,
                sample_count: 1,
                dimension: wgpu::TextureDimension::D2,
                format: wgpu::TextureFormat::Rgba16Float,
                usage: wgpu::TextureUsages::RENDER_ATTACHMENT
                    | wgpu::TextureUsages::TEXTURE_BINDING,
                view_formats: &[],
            });
            let view = texture.create_view(&Default::default());
            let bind = device.create_bind_group(&wgpu::BindGroupDescriptor {
                label: None,
                layout: &self.hdr_layout,
                entries: &[wgpu::BindGroupEntry {
                    binding: 0,
                    resource: wgpu::BindingResource::TextureView(&view),
                }],
            });
            self.hdr = Some((view, bind, size));
        }
        let view = Mat4::look_at_rh(frame.eye(), Vec3::ZERO, Vec3::Y);
        let projection = Mat4::perspective_rh(
            45_f32.to_radians(),
            size[0] as f32 / size[1] as f32,
            0.05,
            50.0,
        );
        queue.write_buffer(
            &self.globals,
            0,
            bytemuck::bytes_of(&Globals {
                vp: (projection * view).to_cols_array_2d(),
                eye: frame.eye().extend(0.0).to_array(),
                light: frame.light().extend(frame.intensity).to_array(),
                material: [frame.roughness, frame.metallic, frame.mode as f32, 0.0],
                viewport: [
                    size[0] as f32,
                    size[1] as f32,
                    if self.srgb { 1.0 } else { 0.0 },
                    0.0,
                ],
            }),
        );
        queue.write_buffer(&self.splats, 0, bytemuck::cast_slice(&scene(frame, view)));
        let (hdr, texture_binding, _) = self.hdr.as_ref().unwrap();
        for (view, pipeline, display) in [
            (hdr, &self.splat_pipeline, false),
            (target, &self.display_pipeline, true),
        ] {
            let mut pass = encoder.begin_render_pass(&wgpu::RenderPassDescriptor {
                label: Some("relight-pass"),
                color_attachments: &[Some(wgpu::RenderPassColorAttachment {
                    view,
                    resolve_target: None,
                    depth_slice: None,
                    ops: wgpu::Operations {
                        load: wgpu::LoadOp::Clear(wgpu::Color::TRANSPARENT),
                        store: wgpu::StoreOp::Store,
                    },
                })],
                depth_stencil_attachment: None,
                timestamp_writes: None,
                occlusion_query_set: None,
                multiview_mask: None,
            });
            pass.set_pipeline(pipeline);
            pass.set_bind_group(0, &self.bindings, &[]);
            if display {
                pass.set_bind_group(1, texture_binding, &[]);
                pass.draw(0..3, 0..1);
            } else {
                pass.draw(0..6, 0..COUNT as u32);
            }
        }
        Ok(())
    }

Check your understanding

What if an imported model has only appearance SH and no materials?

Reveal the reasoning

Use its original appearance renderer and declare relighting unavailable, or run an explicitly identified offline estimation/transfer pipeline. Arbitrary RGB multiplication is not recovered physical relighting.