Gaussian Splatting Explained: Real-Time Photorealism From Photographs

Gaussian splatting reconstructs a three-dimensional scene from a set of ordinary photographs, then renders new viewpoints of that scene at real-time frame rates. The technique was introduced in the 2023 paper “3D Gaussian Splatting for Real-Time Radiance Field Rendering,” which reported rendering well above 30 frames per second at 1080p resolution.^[1] That speed is the entire point. The neural radiance fields that came before it produced similar image quality but rendered far too slowly for interactive use. Gaussian splatting closed that gap by changing how the scene is stored and drawn.

What a Gaussian Splat Actually Is

A scene in this method is not a mesh of polygons and it is not a neural network. It is a collection of three-dimensional blobs, each one a Gaussian — the same bell-curve shape used in probability and signal processing. A single scene can contain millions of them.^[1]

Every Gaussian carries a few values. A position in three-dimensional space. A covariance matrix that controls its shape and orientation, so it can be a wide flat disc or a narrow elongated spike. An opacity. And a colour that changes depending on the angle from which it is viewed, stored as a compact set of spherical harmonic coefficients.

The important property is that this representation is explicit. There is no network to evaluate, no ray to march through a volume. The scene is fully described by the list of blobs and their parameters.

Why It Renders in Real Time When NeRF Could Not

A neural radiance field, or NeRF, stores the scene implicitly inside a small neural network.^[2] To render a pixel, you cast a ray into the scene, sample hundreds of points along it, and query the network at each sample to get a colour and density. Multiply that across all the pixels in a 1080p frame and you are evaluating a neural network millions of times per frame. That is why NeRF typically renders a single image in seconds or minutes, not milliseconds.

Gaussian splatting replaces ray marching with rasterisation. Each blob is projected onto the screen as a 2D ellipse, sorted by depth, and alpha-blended from back to front. This is the same kind of operation a GPU was designed to execute for triangles, and it runs at hardware speed.

The difference in cost is large. The 2023 paper reported real-time rendering at high frame rates while matching the visual quality of the strongest NeRF methods of the time, and it reached that quality after minutes of training rather than the hours or days some neural methods required.^[1]

How a Scene Gets Built

The input is a set of photographs taken from many angles around a scene. The first step is structure from motion, commonly run with the COLMAP tool, which estimates where each camera was and produces a sparse cloud of points.^[3] Those points seed the initial Gaussians.

From there the system optimises. It renders the current Gaussians from each known camera position, compares the result against the original photograph, and adjusts every blob by gradient descent to reduce the difference. Gaussians that cover regions with too little detail are split into smaller blobs. Gaussians that become nearly transparent are pruned. After thousands of iterations the blobs settle into positions and shapes that faithfully reproduce the training views and generalise reasonably well to novel viewpoints in between.

Looking Ahead

The speed comes with costs. A finished scene of millions of Gaussians is large: file sizes of several hundred megabytes are typical, and streaming them over a network adds latency. The static-capture problem is also real. A Gaussian scene records a place as it was at one moment. Moving objects, changing lights, or a cloudy day followed by sun are all outside what the basic model handles.

There are structural limits too. Reflective surfaces — mirrors, wet floors, polished metal — produce inconsistent colours from different angles, and the spherical-harmonic colour model approximates this rather than resolving it cleanly. Regions that the camera never saw are filled by interpolation and often contain floaters, stray blobs hanging in empty space.

The active research now targets exactly these gaps: compressing the representation, handling motion over time, and importing splats into game engines so they sit beside conventional geometry. For any application that needs a real location rendered interactively, photorealistic and at frame rate, this approach now competes directly with both photogrammetry, which yields heavier meshes, and neural fields, which render too slowly. The open question is no longer whether captured reality can run in real time. It is how much of it can be edited, lit, and animated once it is there.