How to Handle Missing Data in Pandas

Handle missing data in pandas in three steps: find it with isna(), then either fill it (fillna, ffill, bfill, interpolate) or drop it (dropna). Which one is right depends on whether a missing value means "zero," "unchanged," or "unusable."

How Do You Find Missing Values?

isna() returns a boolean mask of missing cells, and summing it gives a per-column count — the standard first look at any new dataset. notna() is its inverse.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "temp":  [20.1, np.nan, 22.4, np.nan, 23.0],
    "city":  ["NYC", "NYC", None, "LA", "LA"],
})

df.isna().sum()
# temp    2
# city    1

How Does fillna Work?

fillna replaces missing values with a constant, and accepts a dict to use a different fill per column — usually better than one blanket value.

df.fillna(0)                     # every NaN becomes 0 (rarely right)

df.fillna({"temp": df["temp"].mean(), "city": "unknown"})
#    temp     city
# 0  20.1      NYC
# 1  21.8      NYC   <- column mean
# 2  22.4  unknown
# 3  21.8       LA
# 4  23.0       LA

What Are ffill, bfill, and interpolate?

ffill() carries the last valid value forward, bfill() pulls the next valid value backward, and interpolate() estimates numeric gaps linearly. These fit ordered data like time series, where "missing" usually means "unchanged since last reading."

df["temp"].ffill()          # 20.1, 20.1, 22.4, 22.4, 23.0
df["temp"].bfill()          # 20.1, 22.4, 22.4, 23.0, 23.0
df["temp"].interpolate()    # 20.1, 21.25, 22.4, 22.7, 23.0

# df.fillna(method="ffill") is deprecated — call ffill() directly.

When Should You Use dropna?

dropna removes rows (or columns with axis=1) containing missing values; use it when a row is unusable without the value, not as a reflex. subset= limits which columns count, and thresh= keeps rows with at least N valid values.

df.dropna()                        # drop rows with ANY NaN
df.dropna(subset=["temp"])         # only require temp to be present
df.dropna(thresh=2)                # keep rows with >= 2 non-null values

Why Avoid inplace=True?

Pandas is moving away from inplace=True: it rarely saves memory, breaks method chaining, and is deprecated for most methods on the path to pandas 3.0. Assign the result instead.

# Aging pattern:
df.fillna(0, inplace=True)

# Preferred:
df = df.fillna(0)
clean = df.dropna(subset=["temp"]).assign(city=lambda d: d["city"].fillna("unknown"))

Common Pitfalls

• NaN != NaN: df[df["temp"] == np.nan] always returns nothing. Use isna().
• NaN is a float: an integer column with missing values silently becomes float64. Use the nullable Int64 dtype to keep integers.
• Leading NaN survives ffill: there's nothing before it to carry forward. Chain .ffill().bfill() if edges must be filled.
• Filling before train/test split leaks data: compute fill values (means, medians) on training data only.

Missing values often enter at load time — the na_values options covered in reading CSVs in Python let you catch placeholders like "N/A" or "-" as real NaN from the start.