PAJAMA / RELIGHT

04 / 08 · RELIGHTING

Send light through the splat pipeline

Follow two GPU passes and their color spaces.

Shade the surfel, project the footprint

The asset has 1,536 thin Gaussian ellipsoids on an analytic sphere. Each stores a center, three covariance axes, an outward normal and linear base color. Roughness and metallic are uniform across this object. WGSL shades each center, holding that radiance constant across its footprint.

This approximates a surface using 3D Gaussian footprints; it is not the perspective-correct 2DGS algorithm. CPU code transforms and sorts centers far-to-near. WGSL projects covariance, expands a three-sigma quad and evaluates Gaussian weight. Overlap makes it nearly opaque, with soft silhouettes.

Σ=UUT,U=[u v w],Σ2D=JΣJT+0.3I,α=oexp⁡(−12∥q∥2)\Sigma=UU^T,\quad U=[\mathbf u\ \mathbf v\ \mathbf w],\quad \Sigma_{2D}=J\Sigma J^T+0.3I,\quad \alpha=o\exp(-\tfrac12\|\mathbf q\|^2)

Add linear light, then tone-map

Fragments write (αL,α) into RGBA16Float. In far-to-near order, premultiplied blending computes new=αL+(1−α)old. Because α is already included, the source blend factor is One. Using SourceAlpha again would incorrectly square α.

A fullscreen pass adds visible background, applies per-channel Reinhard x/(1+x), then encodes to sRGB once. sRGB attachments encode automatically; a UNORM canvas needs explicit encoding. Normal/base-color debug modes skip Reinhard.

Cnew=αL+(1−α)Cold,T(x)=x1+x,s(x)={12.92xx≤0.00313081.055x1/2.4−0.055x>0.0031308C_{\rm new}=\alpha L+(1-\alpha)C_{\rm old},\quad T(x)=\frac{x}{1+x},\quad s(x)=\begin{cases}12.92x&x\le0.0031308\\1.055x^{1/2.4}-0.055&x>0.0031308\end{cases}

What is and is not implemented

The convex shape has one distant white light. There is no cast-shadow pass, environment map, indirect light, inverse-material estimator or training optimizer. Backface rejection is not a general light-visibility solution. Concave scenes need visibility data or ray/shadow queries.

Native offscreen checks verify light/material response, roughness, deformation and camera changes. Windows/DX12 and browser behavior are tested here; sharing WGSL does not certify untested Metal/Vulkan devices.

Predict it, then run it

Open this experiment →

The formula meets actual code

This excerpt is extracted from the repository during the site build. SH environment lighting and inverse training are future extensions; their chapters point to the current direct-light/data boundary, not an implemented trainer.

src/relight.rs · L298–397
Full source ↗
    pub fn render(
        &mut self,
        device: &wgpu::Device,
        queue: &wgpu::Queue,
        encoder: &mut wgpu::CommandEncoder,
        target: &wgpu::TextureView,
        size: [u32; 2],
        frame: RelightFrame,
    ) -> Result<(), String> {
        frame.validate()?;
        if size.contains(&0)
            || size
                .iter()
                .any(|&v| v > device.limits().max_texture_dimension_2d)
        {
            return Err("Invalid relight target size".into());
        }
        if self.hdr.as_ref().is_none_or(|v| v.2 != size) {
            let texture = device.create_texture(&wgpu::TextureDescriptor {
                label: Some("linear-radiance"),
                size: wgpu::Extent3d {
                    width: size[0],
                    height: size[1],
                    depth_or_array_layers: 1,
                },
                mip_level_count: 1,
                sample_count: 1,
                dimension: wgpu::TextureDimension::D2,
                format: wgpu::TextureFormat::Rgba16Float,
                usage: wgpu::TextureUsages::RENDER_ATTACHMENT
                    | wgpu::TextureUsages::TEXTURE_BINDING,
                view_formats: &[],
            });
            let view = texture.create_view(&Default::default());
            let bind = device.create_bind_group(&wgpu::BindGroupDescriptor {
                label: None,
                layout: &self.hdr_layout,
                entries: &[wgpu::BindGroupEntry {
                    binding: 0,
                    resource: wgpu::BindingResource::TextureView(&view),
                }],
            });
            self.hdr = Some((view, bind, size));
        }
        let view = Mat4::look_at_rh(frame.eye(), Vec3::ZERO, Vec3::Y);
        let projection = Mat4::perspective_rh(
            45_f32.to_radians(),
            size[0] as f32 / size[1] as f32,
            0.05,
            50.0,
        );
        queue.write_buffer(
            &self.globals,
            0,
            bytemuck::bytes_of(&Globals {
                vp: (projection * view).to_cols_array_2d(),
                eye: frame.eye().extend(0.0).to_array(),
                light: frame.light().extend(frame.intensity).to_array(),
                material: [frame.roughness, frame.metallic, frame.mode as f32, 0.0],
                viewport: [
                    size[0] as f32,
                    size[1] as f32,
                    if self.srgb { 1.0 } else { 0.0 },
                    0.0,
                ],
            }),
        );
        queue.write_buffer(&self.splats, 0, bytemuck::cast_slice(&scene(frame, view)));
        let (hdr, texture_binding, _) = self.hdr.as_ref().unwrap();
        for (view, pipeline, display) in [
            (hdr, &self.splat_pipeline, false),
            (target, &self.display_pipeline, true),
        ] {
            let mut pass = encoder.begin_render_pass(&wgpu::RenderPassDescriptor {
                label: Some("relight-pass"),
                color_attachments: &[Some(wgpu::RenderPassColorAttachment {
                    view,
                    resolve_target: None,
                    depth_slice: None,
                    ops: wgpu::Operations {
                        load: wgpu::LoadOp::Clear(wgpu::Color::TRANSPARENT),
                        store: wgpu::StoreOp::Store,
                    },
                })],
                depth_stencil_attachment: None,
                timestamp_writes: None,
                occlusion_query_set: None,
                multiview_mask: None,
            });
            pass.set_pipeline(pipeline);
            pass.set_bind_group(0, &self.bindings, &[]);
            if display {
                pass.set_bind_group(1, texture_binding, &[]);
                pass.draw(0..3, 0..1);
            } else {
                pass.draw(0..6, 0..COUNT as u32);
            }
        }
        Ok(())
    }

Check your understanding

Why not tone-map every Gaussian before blending?

Reveal the reasoning

Tone mapping is nonlinear: T(αa+(1−α)b) generally differs from αT(a)+(1−α)T(b). Add light in linear HDR first.