Few-shot Gaussian Splatting

The question

I tested whether synthetic training views generated with Stable Diffusion help 3D Gaussian Splatting when only a handful of real photographs exist. I compared inpainting, outpainting and pose-guided augmentation on the Tanks & Temples Truck scene across 3 subset sizes (5, 10 and 20 real views) and 4 synthetic-data ratios, in 242 training runs over 3 seeds.

The 32 held-out photographs are identical in every run and synthetic images are added to training only, so any difference in the held-out score belongs to the synthetic images.

Pipeline: 251 photographs, COLMAP camera poses, a frozen split with 32 held-out views, farthest-point subset selection, then either k real images or k real plus diffusion-augmented images go into 3DGS training, evaluated on the held-out views with PSNR, SSIM and LPIPS. — Two lanes into one trainer: k real photographs on their own, or the same k plus a diffusion-generated set. Everything downstream is identical.

Three ways to make a synthetic view

Each strategy is defined by how it obtains a camera pose for the synthetic image, and that turns out to decide whether it helps or hurts.

Inpainting: the pose is copied exactly from a real view. A random 8–22 % region is regenerated and composited back, so only the masked pixels are synthetic.
Outpainting: the same camera centre with widened intrinsics. Diffusion invents the border ring around the real frame.
Pose-guided: a new pose interpolated between two real cameras. Monocular depth (Depth Anything V2) anchored to sparse COLMAP points is forward-warped, and diffusion fills the disocclusion holes.

Results

ΔPSNR in dB against the same seed's own baseline at the same subset size, averaged over 3 seeds. k is the number of real views and the ratio is synthetic to real images. * marks effects whose mean exceeds the between-seed standard deviation.

Outpainting: crosses from helpful to harmful

Ratio	k = 5	k = 10	k = 20
20–25 %	+0.140 *	+0.172 *	−0.283 *
40–50 %	+0.140 *	+0.013	−0.542 *
100 %	+0.182 *	−0.423 *	−0.983 *
200 %	+0.285 *	−0.003	−0.618 *

Inpainting: flat, and independent of subset size

Ratio	k = 5	k = 10	k = 20
100 %	−0.166 *	−0.213 *	−0.161 *
200 %	+0.029	−0.141	−0.102 *

Pose-guided: harmful everywhere, and worse with more real data

Ratio	k = 5	k = 10	k = 20
20–25 %	−0.309 *	−0.451 *	−1.017 *
200 %	−1.013 *	−1.451 *	−1.884 *

The measured training noise floor is σ = 0.039 dB (≈ 0.055 dB paired), so these effects are real even where small.

Three plots of change in PSNR against the synthetic-to-real ratio for inpainting, outpainting and pose-guided augmentation, with one curve each for 5, 10 and 20 real views. — ΔPSNR against the synthetic-to-real ratio for each strategy and subset size. The grey band is the measured noise floor (±0.055 dB).

Left: PSNR rising from 15.20 dB at 5 real views to 19.74 dB at 20, with the full-data result at 219 views. Right: bar chart comparing gains from more real views with the best and worst synthetic conditions. — What real photographs buy (left), and real and synthetic gains on the same scale (right).

Why: pose novelty

Synthetic views help only at very few real views (up to +0.29 dB PSNR with outpainting at 5 views) and become harmful by 20 views. Ordered by how much new camera pose each strategy invents, inpainting (none) has almost no effect, outpainting (same centre, wider view) helps when real views are scarce and hurts when they are not, and pose-guided synthesis (fully novel viewpoints) always hurts.

A synthetic view supplies coverage and inconsistency in fixed proportion. Coverage loses value as real views accumulate, but inconsistency does not. One decaying term and one roughly constant term produce a sign change, which is what the data show.

Grid of held-out renders of the truck: ground truth, 5 real views, 5 real plus 200 percent synthetic, 20 real, and 20 real plus 200 percent synthetic, each labelled with its PSNR. — The same 200 % outpainting on three held-out views: it helps at 5 real views and hurts at 20.

Few-shot 3D Gaussian Splatting with diffusion-based data augmentation