Dropout Dance while Fitting y = x²

A tiny neural network, trained live in the browser with hand-written backpropagation, fitting a noisy y = x² curve — with every weight, every bias, and every dropout mask visible as it happens, so you can watch dropout do its job instead of just reading about it.

Why this exists

Dropout is usually taught as a single sentence: "randomly zero out some units during training to prevent overfitting." That sentence is true, but it describes something that normally happens invisibly, deep inside a training loop, on a network far too large to watch directly — the only evidence you ever see of it working is an indirect one, a validation loss curve that looks a little better days later.

The goal of this project is to close that gap: to make the actual, mechanical effect of dropout — and how useful it really is — something you can watch happen, live, on a network small enough to see in full. Every one of the network's 8 hidden units, all of its weights and biases, and the exact dropout mask in force at any moment are drawn on screen and updated every frame. When a unit is dropped, you see it fade in the diagram. When the mask changes, you see the fitted curve visibly react. Training a network with dropout stops being an abstract idea and becomes something with a shape you can point at.

Features

• A real 1 → 4 → 4 → 1 network (Tanh hidden units, linear output), trained by hand-written backpropagation and full-batch gradient descent — no ML library involved anywhere
• Toggleable dropout with an adjustable drop rate (0.1 – 0.7)
• A live SVG network diagram: every current weight and bias is drawn as text and updates every frame; units silenced by the current dropout mask visibly fade, along with every connection touching them
• The dance trail: a fading afterimage of the model's output curve over recent frames, so the moment-to-moment movement of the fit — not just its current position — is visible at a glance
• Two dataset modes — randomly-sampled, noisier data ("Reset") versus evenly-spaced, low-noise data ("Uniform Data") — for comparing how much dropout's effect actually depends on how noisy the data is
• Start / Stop the live training loop at any time, and Reset to regenerate the dataset and reinitialize the network from scratch

How it works

The network and the training loop

The network is a small fully-connected regression model:

Input (x)  →  Hidden 1 (4 × Tanh)  →  Hidden 2 (4 × Tanh)  →  Output (1 × linear)

Every training run is full-batch gradient descent: each "epoch" computes the forward pass and the gradient for every point in the current dataset, sums them, and takes one weight-update step using plain mean-squared-error loss. There's no library here — the forward pass, the loss, and every partial derivative in the backward pass are written out by hand.

Dropout, specifically

Dropout is implemented with the standard inverted dropout convention: during training, each hidden unit is kept with probability 1 - rate and dropped with probability rate; a kept unit's activation is divided by the keep probability (a1d[j] = a1[j] * mask[j] / keep), so no separate rescaling is needed at inference time — predict(), the function that draws the plotted curve, is simply the clean forward pass with no masking at all.

The mask itself isn't redrawn every single epoch — it's redrawn every 5 epochs (MASK_EVERY), and stays fixed while a short run of training happens under it. That's a deliberate choice: it turns dropout's effect into something you can actually watch as discrete, readable steps — train for a few epochs under one random subnetwork, watch the fit settle a little toward it, then see the mask swap and the fit visibly react to a different subnetwork — rather than a blur of per-step noise too fast to follow.

That mask has a direct, exact consequence for the gradient math: a dropped unit's scaled activation is exactly 0, so every gradient that flows through it is also exactly 0. Concretely, on any epoch a unit is dropped:

• none of its own incoming weights (and bias) receive any gradient that epoch, and
• none of its outgoing weights to the next layer receive any gradient either, because its contribution to every downstream sum is zero.

So each masked window isn't just "some noise added to training" — it genuinely trains a different subnetwork of the same shared pool of weights, leaving whichever weights belong only to dropped units frozen in place until they're active again.

The dance trail

The trail on the right-hand plot is drawn with a simple compositing trick: an off-screen canvas is faded very slightly every frame (destination-out at low alpha) and then has a faint copy of the current predicted curve stamped onto it, before being drawn underneath the live curve. The result is a fading afterimage of every recent shape the model's output has taken. It's the single most direct way this project has of showing not just where the fit is right now, but how much it has been moving to get there.

Understanding dropout through the dance trail

What you're actually looking at

Because of how the masking works (explained above), enabling dropout doesn't just add random jitter to the output — it changes which weights get updated on any given step. Over a short run, that shows up as visibly different behavior from training without dropout:

Dropout ON (rate 0.4), 84 epochs	Dropout OFF, same 84 epochs

A few things are worth reading carefully in that comparison. With dropout off, every epoch updates the entire network at once, so the fit converges faster in raw epoch count — the trail is tighter, and the curve has already visibly started bending toward the bottom of the parabola. With dropout on, only a fraction of the network updates on any given masked window, so progress is slower and noisier — the trail is wide, showing the fit genuinely wobbling between the handful of different subnetworks it has trained under so far, and the network diagram shows exactly which units (here, h1_1 and h2_2) are sitting out the current window.

That trade-off — slower, noisier, more conservative progress in exchange for never letting any single unit (or narrow combination of units) become solely responsible for the fit — is the entire mechanism behind why dropout regularizes a network. It doesn't just add "noise" for its own sake; it prevents units from co-adapting: from a network's units quietly agreeing among themselves to specialize around each other's specific quirks (including the quirks of noisy training points), which is one of the concrete ways small networks overfit small, noisy datasets.

An honest caveat

This project trains on the data it's shown and only ever plots that same data — there's no held-out validation or test set anywhere in it. So strictly speaking, what you're watching here is dropout's effect on training dynamics (how the fit moves, how fast the loss on the training points drops, which weights are frozen when), not a direct, measured before/after comparison of generalization to unseen data. That distinction matters: don't read the lower training loss in the dropout-off screenshot above as "dropout is worse" — a fair generalization comparison would need to score both networks on points they never trained on, which this tool intentionally doesn't do, to keep everything on screen at once and easy to follow. What this tool is built to show is the mechanism, directly, so that the usual textbook claims about why dropout helps generalization stop being abstract.

When it matters more — and when it barely shows

The two dataset modes make that same point concretely. "Reset" generates scattered points with a fairly large amount of noise; "Uniform Data" generates evenly-spaced points with much less of it:

Noisy data (Reset)	Uniform, low-noise data (Uniform Data)

Try training with dropout on under each mode. On the noisy dataset, a network without dropout has real room to bend itself around individual outlier points — exactly the behavior dropout exists to discourage. On the clean, evenly-spaced dataset there's much less noise available to overfit to in the first place, so dropout's regularizing effect on the shape of the fit is far less visually dramatic — though the dance trail will still show the same underlying training-dynamics wobble, since that part comes from the masking mechanism itself, independent of the data.

Controls

Control	Action
Dropout enabled checkbox	Turns dropout on or off. Changes apply from the next mask refresh (every 5 epochs), not necessarily instantly
Drop rate slider	Sets the probability of dropping each hidden unit (0.1 – 0.7)
Start / Stop	Runs (or pauses) the live training loop — two epochs per visible frame
Reset (noisy)	Regenerates a fresh, randomly-sampled noisy dataset, reinitializes the network, and clears the dance trail
Uniform Data (no outliers)	Same as Reset, but with an evenly-spaced, low-noise dataset instead

Getting started

No installation, no package manager, no build step, and no internet connection required — everything here is plain HTML, CSS and JavaScript with zero external dependencies.

1. Clone or download this repository.
2. Open index.html in any modern browser.

That's it. If your browser restricts local script loading when opening files directly, serve the folder with any static file server instead:

python3 -m http.server 8000
# then open http://localhost:8000

Deploying with GitHub Pages

Since this is a fully static site, GitHub Pages works out of the box:

1. Push this repository to GitHub.
2. Go to Settings → Pages.
3. Under Build and deployment, choose Deploy from a branch, pick your default branch and the / (root) folder.
4. Save — your live demo will be at https://<your-username>.github.io/<repository-name>/.

Code walkthrough

This project has no build step, no framework and no package manager: it's plain HTML, CSS and JavaScript, loaded directly by the browser, split into small files purely for readability.

Markup and styling

index.html contains structure only — the control bar and the two empty panels. Both the SVG network diagram and the canvas plot are populated entirely by JavaScript at runtime; nothing about their content is hard-coded in the HTML.

css/style.css defines the dark theme as a small set of CSS custom properties under :root (--bg, --panel, --fit, and so on), so the whole palette can be re-themed from one place. Every corner in the UI — buttons, panels, swatches — is deliberately square, enforced with one global border-radius: 0 !important rule, as a consistent flat visual style rather than an oversight.

JavaScript: five files, one dependency order

There's no module bundler here, so the five script files are loaded as plain, classic <script> tags in a specific order and share the browser's single global scope — each file simply continues where the previous one left off. The load order in index.html is the dependency order:

1. data.js defines the target function (trueCurve, y = x²) and the two dataset generators, makeData() (noisy, random x positions) and makeDataStd() (evenly spaced, low noise). The currently active dataset lives in a single data variable that later files read from and the Reset / Uniform Data buttons reassign.
2. network.js is the model: the weight and bias matrices (W1/b1/W2/b2/W3/b3), initNet(), the dropout mask state (curMask1, curMask2, bernoulli()), and the two core functions — trainEpoch(), which runs one full-batch forward-and-backward step with dropout applied, and predict(), the clean forward pass with no masking, used everywhere the model's actual output needs to be drawn.
3. plot.js owns the right-hand canvas: converting data coordinates to pixels (toPx), drawing the axes and the two reference curves, and render(), which also maintains the fading off-screen trail canvas described above.
4. network-diagram.js builds the SVG diagram once (nodes, edges, and their labels, generated from the cols layout definition), then exposes updateNet(), which is called every frame to refresh only the text and opacity of the existing elements — the diagram's structure never changes, only what's drawn on it.
5. main.js is the entry point: it defines step() (the function the Start/Stop button's setInterval calls repeatedly), wires up every control in the top bar, and makes the first render() / updateNet() call so the page isn't blank before training starts.

The pattern across all five files is deliberately simple: a handful of plain variables hold the current state (the weight matrices, the current dataset, the current dropout masks), and a small number of functions read that state and redraw the SVG or canvas. There's no virtual DOM, no reactive framework and no data-binding library involved.

Project structure

.
├── index.html                       # Page structure and layout only
├── css/
│   └── style.css                    # Dark theme, control bar, panel and legend styling
├── js/
│   ├── data.js                      # Target function + the two dataset generators
│   ├── network.js                   # Weights/biases, dropout masking, train step, predict
│   ├── plot.js                      # Canvas plot: axes, curves, data points, dance trail
│   ├── network-diagram.js           # Live SVG diagram of every node, weight and bias
│   └── main.js                      # Entry point: play/pause loop, control wiring
└── assets/
    └── screenshots/                 # Images used in this README

Technology

Vanilla HTML, CSS and JavaScript only — no framework, no build tooling, no package manager, and no external libraries or CDN dependencies of any kind. Every line of math (the network's forward pass, backpropagation, and gradient descent) is written by hand.

Browser support

Any modern browser with Canvas and SVG support (Chrome, Edge, Firefox, Safari) — no WebGL or other special capability is required.

License

Released under the MIT License — see the LICENSE file for the full text.

Author

Hadi Sarhangi Fard GitHub: github.com/Hadifard