Neurons & Forward Pass
- Compute a neuron's output by hand: weighted sum, bias, nonlinearity
- Explain why stacking linear layers without nonlinearities collapses to one linear layer
- Implement a full forward pass for a 2-2-1 network in pure NumPy and verify it against hand math
- Count the parameters of any dense network from its layer sizes
- Describe how depth lets a network bend decision boundaries that a line cannot
| Spaced-rep warm-up: due cards from Weeks 11–12 | 10 min |
| ELI5 + tech read; watch 3B1B chapter 1; study the nn-forward visualizer | 25 min |
| Guided: hand-computed forward pass + XOR boundary | 40 min |
| Practice: generic forward + parameter counting | 20 min |
| Project: nn_forward.py committed | 15 min |
| Quiz + write flashcards | 10 min |
Builds on: Day 51 — Matrices as transformations · Day 53 — Derivatives & gradients · Day 64 — NumPy vectorization & broadcasting · Day 72 — Logistic regression & sigmoid
Picture a sound engineer's mixing desk: rows and rows of dials. Each dial takes one incoming signal and decides how loudly it contributes to the mix — turn it up, turn it down, or flip it negative to subtract. One "neuron" is exactly one channel strip: it takes all its inputs, scales each by its own dial (a weight), adds a base level (the bias), and then runs the sum through a gate that only lets strong signals pass (the activation function). A layer is a whole row of these channel strips listening to the same inputs; a network is several rows chained, so each row remixes the mix of the row before.
Here is the trick that makes it more than a fancy mixer: the gate between rows is *nonlinear*. Without it, ten rows of dials would collapse into one big row — mixing a mix is still just a mix. With the gates, each layer can carve, fold, and bend the signal, and stacked bends can trace shapes no single straight cut ever could. Today you turn every dial by hand once, so the machinery is never mystical again.
Every model you will touch from here to Day 180 — MNIST classifiers, word2vec, GPT, the embedding models behind your Day-115 vector database — is layers of weighted sums and nonlinearities. When a customer asks "but what is the model actually doing?", the FDE who can sketch a neuron on a whiteboard in ninety seconds owns the room. And when a PyTorch shape error explodes on Day 87, you will debug it by replaying today's matrix dimensions in your head: (batch, in) times (in, out) or nothing works.
Layers of dials — one input flows through a 2-3-1 network
step 1 / 5A tiny network: 2 inputs, 3 hidden neurons, 1 output. Every line is a weight — a dial the training will turn. Right now, data flows LEFT to RIGHT.
Guided practice
One forward pass by hand, then let NumPy check you
20 min- Take input x = [1, 2] into a 2-2-1 net with ReLU on the hidden layer.
- Hidden neuron 1: weights [1, -1], bias 0. Hidden neuron 2: weights [0.5, 1], bias -1. Output neuron: weights [2, 1], bias 0.5.
- On paper: compute z1 = 1*1 + (-1)*2 + 0 = -1 and z2 = 0.5*1 + 1*2 - 1 = 1.5. Apply ReLU: a1 = 0, a2 = 1.5.
- Output: y = 2*0 + 1*1.5 + 0.5 = 2.0. Every arrow in the network now has a number on it.
- Run the starter code and confirm NumPy agrees with your paper. Then change ReLU to the identity function and recompute — note that the network becomes a single linear formula in x.
Watch the boundary bend
20 min- XOR is the classic function no straight line can separate: (0,0)->0, (1,1)->0, (0,1)->1, (1,0)->1.
- Run the starter: it evaluates a hand-set 2-2-1 tanh network on a grid of points and prints an ASCII map of where the output is above 0.5.
- Confirm the four XOR corners are classified correctly — the '#' region must cover (0,1) and (1,0) but not (0,0) or (1,1). A straight line cannot do that; two bent half-planes can.
- Set the hidden activation to identity (linear) and rerun. The map degenerates into a single straight split and XOR breaks. Write one sentence: which ingredient did you just remove?
On your own
A forward pass for any architecture
20 minWrite forward(x, layers) where layers is a list of (W, b) tuples, applying ReLU between layers but not after the last. It must work for ANY chain of compatible shapes — 2-2-1, 4-8-8-3, whatever. Then write count_params(layers) and verify it gives 9 for the 2-2-1 net and 101,770 for 784-128-10 (create the matrices with np.random.randn at the right shapes).
Constraints: no loops over individual neurons — matrix operations only; handle a batch X of shape (batch, n_in) as well as a single vector.
Hints (only if stuck): store W as (out, in) and compute X @ W.T + b for batches; parameter count per layer is W.size + b.size.
Your forward-pass library, v0
Create nn_forward.py in your practice repo: a Layer class holding W and b (initialized with np.random.randn(out, in) * 0.1), a forward method, and an MLP class chaining layers with ReLU. Include a __main__ block that (a) reproduces the hand-computed 2-2-1 example exactly by setting the weights manually, (b) prints the parameter count for 784-128-10, and (c) prints the XOR ASCII map from guided exercise 2. This file is not throwaway — Day 86 adds gradients to it, and Day 87 rebuilds it in PyTorch to show what you earned.
Common mistakes & misconceptions
- Thinking the activation is optional polish. Without it the whole stack collapses to one linear map — depth buys you literally nothing.
- Confusing weight shape conventions. If W is (out, in), a vector goes through as W @ x but a batch goes through as X @ W.T. Mixing these is the #1 shape bug on Day 87.
- Forgetting the bias. Without b every layer's output must pass through the origin; the XOR solution above is impossible.
- Believing "universal approximation" means any net trains easily. It says a wide-enough net CAN represent the function, not that gradient descent will find it.
- Counting parameters as just weights. Each layer adds out_features biases too — interviewers check.
- Treating sigmoid as the default hidden activation. It saturates; modern default is ReLU in hidden layers, sigmoid/softmax only at the output when you need probabilities.
Q1. A 10-layer network uses no activation functions anywhere. What can it represent?
Q2. How many parameters does a dense 3-4-2 network have (weights + biases)?
Q3. Hidden pre-activations come out as z = [-2.0, 0.0, 3.5]. After ReLU, the activations are…
Go deeper — curated resources
- Why initialization scale matters — Multiply randn weights by 1.0 instead of 0.1 in a 10-layer net and print activation magnitudes per layer — they explode. This foreshadows Day 89's training-stability toolkit.
- Hand computation matches NumPy on the 2-2-1 example (y = 2.0)
- XOR boundary demo run with nonlinear AND linear activation, difference explained in one sentence
- nn_forward.py committed with all three __main__ demos passing
- Quiz ≥ 2/3
← Back: A layer is Day 51's matrix transformation wearing a new hat, the dot product inside each neuron is Day 50's alignment score, and the sigmoid output is Day 72's logistic regression — a neural net's last layer IS logistic regression on learned features.
Forward →: Tomorrow (Day 86) blame flows backwards through exactly the graph you built today. Day 87 rebuilds it in PyTorch, Day 90 scales it to MNIST, and Day 94's attention scores are the same dot products with a starring role.
Unlocks: D86 Backpropagation from Scratch · D87 PyTorch — Tensors & Autograd · D91 Week 13 Checkpoint — The First Neural Check · D92 Embeddings — Meaning as Geometry