Give the splat a camera and a 3D shape

1. Move the world into the camera’s coordinate system

The view matrix answers “where is this point relative to the camera?” The projection matrix makes distant objects smaller. The code applies projection × view × point, using column-major matrices. A point carries w = 1 so translation affects it; an axis direction carries w = 0 so translation does not affect it.

The toy camera looks along its local −Z with +Y up. The research capture has a different world orientation, so the stage 7 camera uses −Y up. This is a dataset convention, not a different projection equation. Keep the conversion at the data boundary.

2. Divide only after reaching clip space

The GPU divides clip xyz by clip w to obtain NDC. For a small world axis a, compute delta = VP × (a, 0). The quotient rule tells us how clip.xy / clip.w changes along that axis. Multiply by half the viewport dimensions to convert that small NDC movement to pixels.

You can read the derivative as “how many pixels would this tiny movement move the point?” This is a first-order approximation around the Gaussian center. It is accurate for small footprints away from the camera plane, not an exact projection of an arbitrarily large volume.

3. Add three projected axes into a covariance

Project each scaled world axis into 2D, giving d₀, d₁, d₂. For each axis take its outer product, then add the three matrices. This yields [[a,b],[b,c]]. The diagonal describes spread in x and y; b describes their correlation. Adding 0.3 to the diagonal is the baseline’s pixel-space footprint regularization.

4. Turn covariance into two visible axes

Eigenvectors give the ellipse directions; square roots of eigenvalues give one-standard-deviation lengths. Extend three standard deviations along each direction to place the quad’s corners. Convert the pixel offset back to NDC and multiply by center.w before adding to clip.xy. Omitting that last multiplication makes size change incorrectly with depth.

The toy uses axis-aligned 3D scales so the camera effect is isolated. The real shader first builds rotated axes from a quaternion. The toy camera range keeps both centers safely in front of the camera; a general viewer additionally needs robust near-plane handling and conservative bounds.

IDEA → IMPLEMENTATION

Follow the running code

Excerpts are extracted from the repository at site build time. Expand only the part you are studying; full files include context and line numbers.

pub fn camera(frame: Frame, width: u32, height: u32) -> (Mat4, Mat4) {
    // The research capture uses a different world orientation from the toy scene.
    let (target, up, sign, distance) = if frame.stage == 7 {
        (
            Vec3::new(0.0, 3.5, 14.0),
            Vec3::NEG_Y,
            -1.0,
            frame.distance * 3.0,
        )
    } else {
        (Vec3::ZERO, Vec3::Y, 1.0, frame.distance)
    };
    let orbit = Vec3::new(
        frame.yaw.sin() * frame.pitch.cos(),
        frame.pitch.sin(),
        sign * frame.yaw.cos() * frame.pitch.cos(),
    );
    let view = Mat4::look_at_rh(target + orbit * distance, target, up);
    let projection = Mat4::perspective_rh(
        50_f32.to_radians(),
        width as f32 / height as f32,
        0.05,
        200.0,
    );
    (view, projection)
}

fn project_axis(center: vec4<f32>, axis: vec3<f32>) -> vec2<f32> {
    let delta = globals.view_projection * vec4(axis, 0.0);
    // Quotient rule for (clip.xy / clip.w), then NDC -> pixels.
    return (delta.xy * center.w - center.xy * delta.w) / (center.w * center.w) * globals.viewport_stage.xy * 0.5;
}

    let center = globals.view_projection * vec4(splat.center_alpha.xyz, 1.0);
    let scale = splat.scale_pad.xyz;
    // Axis-aligned world ellipsoids here; the real pass also rotates these axes.
    let a0 = project_axis(center, vec3(scale.x, 0., 0.));
    let a1 = project_axis(center, vec3(0., scale.y, 0.));
    let a2 = project_axis(center, vec3(0., 0., scale.z));
    let a = a0.x*a0.x + a1.x*a1.x + a2.x*a2.x + 0.3;
    let b = a0.x*a0.y + a1.x*a1.y + a2.x*a2.y;
    let c = a0.y*a0.y + a1.y*a1.y + a2.y*a2.y + 0.3;
    let root = sqrt(max(0.0, (a-c)*(a-c) + 4.0*b*b));
    let large = max(0.1, 0.5*(a+c+root));
    let small = max(0.1, 0.5*(a+c-root));
    var axis = vec2(1.0, 0.0);
    if abs(b) > 0.00001 { axis = normalize(vec2(b, large-a)); }
    else if c > a { axis = vec2(0.0, 1.0); }
    let perpendicular = vec2(-axis.y, axis.x);
    let pixels = 3.0 * (q.x * sqrt(large) * axis + q.y * sqrt(small) * perpendicular);
    let ndc = pixels * 2.0 / globals.viewport_stage.xy;
    out.clip = center + vec4(ndc * center.w, 0.0, 0.0);

Real-model render pass ↗ · Real-model shader ↗

Keep the theory beside the code

Open the related derivation or lab ↗

3D Gaussian Splatting · original research ↗

COLMAP · camera conventions ↗