pandas I — DataFrames
- Explain what a DataFrame is (labeled columns over NumPy arrays sharing an index)
- Select data correctly with [], loc, iloc, and boolean masks — and say which is which
- Add derived columns with assign and build readable method chains
- Control dtypes on load, use categoricals, and measure memory savings
- Avoid the chained-indexing trap when writing values
| Spaced-rep warm-up: Day 64 cards (broadcasting, views, axes) | 10 min |
| ELI5 + tech read: Series, index, loc/iloc, dtypes | 20 min |
| Guided: build & interrogate + selection drills | 40 min |
| Practice: five stakeholder questions | 20 min |
| Project: ticket_report.py round-trip | 20 min |
| Quiz + flashcards | 10 min |
Builds on: Day 64 — NumPy arrays, masks & dtypes · Day 36 — SQL — tables, rows & columns
A spreadsheet is wonderful until you have to do the same thing twice. You cleaned March's data by hand — forty clicks — and now April's file arrives. A DataFrame is a spreadsheet where every action is a line of code instead of a click: filter, sort, add a column, summarize. Run the script again on April's file and the forty clicks replay themselves in a millisecond. And because the steps are written down, a teammate can review them, and future-you can figure out what past-you actually did — the spreadsheet that scripts, and remembers.
Under the hood there is no magic: each column is one of yesterday's NumPy arrays, and a shared index — row labels — keeps the columns glued together. That one design fact explains most of pandas: why columns are fast and typed, why filtering is a boolean mask like Day 64's, and why there are TWO ways to point at rows — by label (loc, like "the row named 42") and by position (iloc, like "the 42nd row") — which become different questions the moment sorting or filtering shuffles the rows.
Something like 80% of applied ML is data plumbing, and pandas is the pipe wrench. Every eval log you will analyze (Day 137), every cost report (Day 146), and every customer CSV an FDE receives arrives as a table; the person who can answer "which category of tickets breaches SLA most?" in ninety seconds of chained pandas is the person the room listens to. This week builds to Day 70's EDA report and Day 83's churn project — both are pandas from the first line.
Guided practice
Build and interrogate your first DataFrame
20 min- Paste the starter — it generates a seeded 500-row support-ticket table (no files needed; every run is identical).
- Run the opening ritual you will use on every dataset forever:
df.head(),df.shape,df.info(),df.describe(), anddf["category"].value_counts(). - Read
df.info()closely: which columns are int64/float64, which are object? Note the memory estimate. - Convert
categoryandpriorityto thecategorydtype and comparememory_usage(deep=True).sum()before and after. Record the ratio. - Sanity-check the data like an engineer: any negative response times?
(df.response_min < 0).sum(). CSAT within 1–5? min/max. This 30-second habit catches half of all data bugs at the door. - Answer your first question: what fraction of tickets are urgent? (value_counts with normalize=True).
Selection without superstition
20 min- Loc vs iloc, felt directly: sort by
response_mindescending intoslow = df.sort_values("response_min", ascending=False). Compareslow.iloc[0](slowest ticket) withslow.loc[0](the ticket whose LABEL is 0 — a totally different row). Write one sentence on why both behaviors are correct. - Boolean drills — predict row counts before running: (a) urgent tickets; (b) urgent AND response over 120 min (remember the parentheses around each condition); (c) billing OR outage via
.isin(["billing", "outage"]); (d) NOT csat 5 via~. - The trap, on purpose: run
df[df.priority == "urgent"]["escalated"] = Trueand then check whether anescalatedcolumn exists — chained indexing did nothing (watch for the warning). Now do it right with a single.loc[mask, "escalated"] = Trueand verify. - Build a chained pipeline in parentheses: assign an
sla_breachedcolumn (response over 60 min), filter to breaches, sort by response descending, take the 10 worst, select three display columns. One statement, one operation per line. - Re-run the whole script top to bottom and confirm identical output — reproducibility is the product.
On your own
Five stakeholder questions, five chains
20 minUsing the seeded ticket DataFrame, answer each question with ONE parenthesized chain (no intermediate variables, no groupby — that is tomorrow):
- What share of tickets falls in each priority level? (value_counts with normalize)
- What is the median response time for urgent tickets vs non-urgent? (two masked medians, one line each)
- Which categories do the 20 slowest tickets come from? (sort, head, value_counts)
- What fraction of SLA breaches (response over 60) got a CSAT of 1 or 2? (mask, then mean of a condition — Day 57's trick on a Series)
- Which agent handled the most urgent tickets?
Then write each answer as a sentence with the number in it — "38% of breached tickets scored CSAT ≤ 2" — because a number without a sentence is not yet a finding.
Hints: .value_counts(normalize=True); boolean masks compose with &; (series <= 2).mean() turns a condition into a rate. Keep this file — Day 67 upgrades these questions into a full EDA.
ticket_report.py — load, type, verify, report
Build the round-trip you will perform on every real dataset. ticket_report.py should: (1) generate the seeded ticket data and write it to tickets.csv AND tickets.parquet; (2) reload the CSV with explicit dtype= mapping (low-cardinality strings as "category") and compare dtypes and memory against a naive pd.read_csv with no arguments; (3) run five assert-based sanity checks (row count, no negative response times, csat within 1–5, no duplicate ticket_ids, expected columns present); (4) print a five-line text report: ticket count, breach rate, urgent share, median response for urgent vs not, and the three slowest tickets. Commit it — Day 68 turns these asserts into a real validation contract.
Common mistakes & misconceptions
- Writing via chained indexing: df[mask]["col"] = value. It may hit a temporary copy and vanish. One indexer: df.loc[mask, "col"] = value.
- Confusing loc and iloc after a sort or filter. loc answers "the row labeled 3"; iloc answers "the 3rd row" — different rows once order changes.
- Forgetting parentheses in masks: df[df.a == 1 & df.b == 2] — & binds before ==, so this crashes or lies. Wrap every condition.
- Iterating with iterrows() to compute something per row. It is a Python loop in disguise (Day 64's 100× tax); look for the vectorized column expression.
- Loading everything as default dtypes: strings become object blobs, dates stay strings, memory balloons. Pass dtype= and parse_dates= at read time.
- Testing for missing values with == np.nan, which is never True. Use .isna() / .notna().
Q1. After s = df.sort_values("x"), what is the difference between s.iloc[0] and s.loc[0]?
Q2. What is wrong with df[df.priority == "urgent"]["escalated"] = True?
Q3. A string column has 4 distinct values across 1M rows. The best dtype choice is…
Go deeper — curated resources
- Copy-on-write in modern pandas — pandas 2+ makes copy-on-write standard: chained assignment never modifies the original, turning a silent maybe-bug into a consistent rule. Read the short docs page so the warnings you see make sense.
- Opening ritual run and memory savings from categoricals recorded
- loc/iloc divergence demonstrated and explained in one sentence
- Five stakeholder questions answered as sentences with numbers
- ticket_report.py committed with all asserts passing; quiz ≥ 2/3
← Back: Every fast pandas operation is Day 64's NumPy underneath — boolean filtering IS the boolean mask, and dtype discipline is the same contract. The rows-and-columns mental model comes from Day 36's SQL tables.
Forward →: Tomorrow adds the power tools — groupby, merge, reshape — and Day 67 turns interrogation into full EDA. The typed-load + sanity-assert habit becomes Day 68's data contracts, and eval logs get exactly this treatment on Day 137.
Unlocks: D66 pandas II — Wrangling · D67 Exploratory Data Analysis