Memorization vs Generation, Playable

EP 01

A generator you can train on 8 "images"

Pretend each gold square is a training image, flattened to a point in a 2D "image space". Our toy generator does what all generative models do in spirit: it learned a probability cloud around the data and samples from it. Hit sample — every white dot is a brand-new output, with a faint line to its nearest training neighbor. Click the canvas to add your own training images and watch the cloud follow.

Model spread σ 42

Honesty label: this is a kernel-density generator, not a neural net — but "smooth cloud around the training set, width set by training" is exactly the behavior we need to study memorization, and the failure modes transfer one-to-one.

The key move: from now on, "how original is this output?" has a number — its distance to the nearest training example. Diffusion models are audited the same way, just with perceptual embeddings instead of pixels.

EP 02

The novelty meter — copy or creation, measured live

One sample at a time now. Each output's nearest-training-neighbor distance slams onto the meter below the canvas: land in the red zone and it's effectively a copy; land beyond it and the model made something the dataset never contained. Drag σ down and watch the needle migrate into the red.

Model spread σ 34

The copy threshold (13px here) is a judgment call — real audits use "closer than the training set's own duplicates" or human raters. The meter and the running copy-rate are computed from actual samples, nothing staged.

Why it matters: "did the model memorize?" is not a vibe, it's this measurement. Carlini-style extraction attacks on image models are essentially this meter run in reverse: generate millions of samples and keep the ones that land in the red zone.

EP 03

Overfit it yourself — watch samples snap onto the data

Training on a tiny dataset keeps sharpening the model's cloud around the 8 points it has. Press train and keep pressing: early on, samples fill the space between the data (generalization). Past the sweet spot the cloud collapses — every "new" sample lands exactly on a training point, ringed in red. The model has become a lookup table with extra steps.

Each press shrinks the model's spread (σ = 58 / (1 + 0.55 × epochs)) and redraws 40 fresh samples — the copy count is measured, not scripted. Around 2–5 epochs is the honest sweet spot; by 30 it's a photocopier.

This is why "just train longer" is dangerous on small data: loss keeps going down the whole time — memorizing the training set is the best possible loss. Real image models fight this with early stopping, augmentation, and above all: not having tiny datasets.

EP 04

The duplication effect — one repeated image poisons the well

Here's the famous one. Twelve unique training images, balanced model, healthy novelty. Now duplicate a single image 20 times in the training set — as happens constantly in web-scraped data — and generate. The model piles probability mass on that one point and starts emitting it near-verbatim, over and over.

Duplication both raises how often that item is drawn (20 of 32 training slots) and sharpens the model around it (σ shrinks with √count) — the same double whammy measured in real diffusion models. The verbatim count is live.

The privacy/copyright punchline: the training images researchers extracted from Stable Diffusion were overwhelmingly ones duplicated many times in LAION. If your face or artwork appears once in a billion images, you're probably safe; if it appears 500 times, the model may hand it back verbatim. Deduplication is now standard for exactly this reason.

EP 05

The escape hatch — enough diverse data, and copying stops paying

True data in the real world lies on a manifold — here, a spiral. A good output should be ON the spiral (realistic) but not ON a training point (novel). With 8 training images you can't have both: wide σ falls off the manifold, narrow σ copies. Give it 400 and the trap opens — samples land on the curve between training points: genuinely new, genuinely plausible.

σ 10

Green = novel AND on-manifold (the win). Red = near-verbatim copy. Grey = novel but off-manifold (implausible mush). All three percentages are computed from the 120 samples against the live dataset.

The whole debate in one picture: generation and memorization are the same machine at different data densities. Billions of diverse images is why Stable Diffusion mostly creates; tiny, duplicated, or rare slices of that data are where it quietly copies. Both sides of the copyright argument are pointing at the same spiral.