PyTorch — Tensors & Autograd
- Create and manipulate tensors: shapes, dtypes, devices, and views
- Use requires_grad, backward(), and .grad to reproduce Day 86's gradients exactly
- Explain when and why to use torch.no_grad()
- Rebuild the Day-86 neuron as an nn.Module in ~20 lines
- Diagnose the three classic PyTorch error messages: shape mismatch, dtype mismatch, device mismatch
| Spaced-rep warm-up: Day 86 gradient-rule cards | 10 min |
| ELI5 + tech read; PyTorch Learn-the-Basics tensor + autograd pages | 30 min |
| Guided: Day-86 reproduction, nn.Module neuron, error clinic | 50 min |
| Practice: port the MLP | 20 min |
| Quiz + flashcards | 10 min |
Builds on: Day 86 — Backprop & the micrograd engine · Day 64 — NumPy arrays & broadcasting · Day 85 — Forward pass & layer shapes
Yesterday you built a hand-cranked calculator that routes blame backwards through a graph. It worked — and it was sixty lines you had to write, test, and maintain, and it only handled single numbers. PyTorch is that same machine rebuilt by a professional factory: it handles whole grids of numbers at once (tensors), runs on GPUs, ships a thousand pre-differentiated operations, and never forgets a backward rule. The crank is now a button labeled .backward().
The mental model transfers one-to-one. A tensor with requires_grad=True is your Value object. Every operation you apply quietly records itself onto a graph, exactly like your _prev sets. Calling .backward() on the loss runs your topological sort and fills every parameter's .grad — same +=, same accumulation, same need to zero it. Nothing new is happening conceptually; you already built this. Today is about learning the factory's controls: how to make tensors, reshape them, keep them on the same device, and read the error messages the machine prints when you feed it mismatched parts.
PyTorch is the lingua franca of modern AI: research papers ship in it, Hugging Face models load into it, and your Day-97 GPT and Day-128 LoRA fine-tune are written in it. For an AI engineer the daily reality is less "invent architectures" and more "load a model, move tensors to the right device, and fix the shape error at 5 pm before the customer demo" — fluency in tensors and error messages IS the job skill. Understanding autograd from Day 86 means PyTorch is never magic to you, which is exactly the confidence that survives production incidents.
Guided practice
Reproduce Day 86 in PyTorch — same graph, same numbers
15 min- (Local run.) Install PyTorch if needed:
pip install torch. Verify withpython -c "import torch; print(torch.__version__)". - Rebuild yesterday's graph: a=2, b=-3, c=10, L = (a*b + c) * -2, with requires_grad on a, b, c.
- Call L.backward() and print a.grad, b.grad, c.grad. They must be exactly 6, -4, -2 — your engine's answers.
- Call L.backward() a second time (inside a fresh forward) WITHOUT zeroing and print a.grad again: 12. You are watching the same accumulation you implemented with +=.
- Wrap a forward pass in
with torch.no_grad():and confirm the result has requires_grad=False.
The 20-line neuron, then the nn.Module version
20 min- Port guided exercise 3 from Day 86: one tanh neuron on the OR-like dataset, but with tensors. Weights are
torch.randn(2, requires_grad=True); the update loop usestorch.no_grad()around the parameter step and zeros grads with.grad = None. - Run it: loss must fall below 0.1 within 50 steps, just like yesterday.
- Now the professional version in the starter: an
nn.Modulewith onenn.Linear(2, 1). Note what disappeared: manual weight creation, manual parameter listing. - Print
list(model.parameters())and match each tensor to Day 85's W and b. Checkmodel.linear.weight.shapeis (1, 2) — out×in, your convention. - Train it with the same manual update loop and confirm the loss falls.
Error-message clinic — break it on purpose
15 min- Take the working model from exercise 2 and cause each classic error ON PURPOSE, reading the full traceback each time before fixing it.
- Shape: feed
X.T(shape 2×4) into the model. Read the "mat1 and mat2 shapes cannot be multiplied" message and decode which two shapes it names and why. - Dtype: build X from NumPy with
torch.from_numpy(np.array([[0,0],[1,0]], dtype=np.float64))and feed it in. Fix with.float(). - Device (if you have a GPU or Apple Silicon; otherwise read along): move the model with
.to(device)but not the data. Fix by moving both. - In your notes, write the three error signatures and their one-line fixes — this trio is 80% of beginner PyTorch debugging.
On your own
Port your MLP forward pass
20 minRebuild Day 85's MLP as an nn.Module: constructor takes a list of layer sizes like [2, 8, 8, 1], builds nn.Linear layers with nn.ModuleList, applies ReLU between layers but not after the last. Then: (1) verify the parameter count for [784, 128, 10] is 101,770 by summing p.numel(); (2) copy your Day-85 hand-set 2-2-1 weights into the module's tensors (under no_grad) and confirm the output is exactly 2.0 for x = [1, 2].
Constraints: architecture fully driven by the sizes list; no hardcoded layer count.
Hints: iterate zip(sizes, sizes[1:]); assign with layer.weight.copy_(...) inside torch.no_grad().
torch_basics.py — your Rosetta stone
Commit torch_basics.py to your practice repo: a runnable file with four labeled sections — (1) the Day-86 graph reproduced with autograd, asserting grads 6/-4/-2; (2) the accumulation demo with a comment explaining the += heritage; (3) the configurable MLP module with the 101,770 parameter assertion and the hand-set 2-2-1 check; (4) an ERRORS section with the three broken snippets commented out, each followed by its error signature and fix as comments. This file is your personal PyTorch phrasebook — Day 88 imports the MLP from it, and you will paste from it for months.
Common mistakes & misconceptions
- Calling loss.backward() twice on the same graph — PyTorch frees the graph after backward by default; rebuild the forward pass (or understand retain_graph) instead.
- Updating parameters outside torch.no_grad(). The update op gets recorded into the graph, and the next backward differentiates through your optimizer step — subtle, wrong, and slow.
- Forgetting that .grad accumulates. It is a feature (gradient accumulation for big batches) but a bug if you never zero — same lesson as Day 86, now with a footgun-shaped API.
- Mixing float64 NumPy arrays with float32 models. torch.from_numpy preserves dtype; call .float() at the boundary.
- Using .data or .numpy() on a graph tensor to "escape" autograd. Use .detach() — it is explicit, safe, and what code reviewers expect.
- Calling model.forward(x) directly. It skips hooks; always call model(x).
Q1. You run loss.backward() every step but never zero gradients. What happens?
Q2. Why wrap the parameter update in torch.no_grad()?
Q3. "mat1 and mat2 shapes cannot be multiplied (4x2 and 4x2)" — the most likely fix is…
Go deeper — curated resources
- PyTorch autograd mechanics (official notes) ↗ — Find the "Autograd mechanics" note in the docs: how views, in-place ops, and grad_fn chains interact — the details behind today's mental model.
- Autograd reproduces Day 86's gradients exactly (6, -4, -2)
- nn.Module neuron trained; parameters mapped to Day 85's W and b
- All three error messages triggered, read, and fixed deliberately
- torch_basics.py committed with passing assertions
- Quiz ≥ 2/3
← Back: requires_grad is your Day-86 Value; backward() is your topological sort; the accumulation footgun is your own += rule. nn.Linear is Day 85's (out, in) weight matrix with professional initialization.
Forward →: Day 88 adds the missing pieces — DataLoaders, optimizers, and the canonical loop. Day 90 scales this to MNIST, Day 97 builds GPT from these same Modules, and Day 103 loads Hugging Face models that are nothing but big nn.Modules.
Unlocks: D88 Training Loops & Data · D103 The Model Landscape & Local Inference