How to Handle Missing Data in Pandas

Handle missing data in pandas in three steps: find it with isna(), then either fill it (fillna, ffill, bfill, interpolate) or drop it (dropna). Which one is right depends on whether a missing value means "zero," "unchanged," or "unusable."

How Do You Find Missing Values?

isna() returns a boolean mask of missing cells, and summing it gives a per-column count — the standard first look at any new dataset. notna() is its inverse.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "temp":  [20.1, np.nan, 22.4, np.nan, 23.0],
    "city":  ["NYC", "NYC", None, "LA", "LA"],
})

df.isna().sum()
# temp    2
# city    1

Why Does to_numeric(errors="coerce") Create New NaN?

pd.to_numeric(..., errors="coerce") turns every value it can't parse as a number into NaN, so fixing a text column's data type can add missing values that weren't there before. Strip formatting like commas first so real numbers survive, and convert types before you count, fill, or drop — a dropna() that runs before the conversion misses the NaN it creates.

prices = pd.Series(["999", "1,299", "n/a", "849.99"])

pd.to_numeric(prices, errors="coerce")
# 999.00, NaN, NaN, 849.99     <- "1,299" lost to the comma

pd.to_numeric(prices.str.replace(",", ""), errors="coerce")
# 999.00, 1299.00, NaN, 849.99 <- only "n/a" is missing

How Does fillna Work?

fillna replaces missing values with a constant, and accepts a dict to use a different fill per column — usually better than one blanket value.

df.fillna(0)                     # every NaN becomes 0 (rarely right)

df.fillna({"temp": df["temp"].mean(), "city": "unknown"})
#         temp     city
# 0  20.100000      NYC
# 1  21.833333      NYC   <- column mean
# 2  22.400000  unknown
# 3  21.833333       LA
# 4  23.000000       LA

What Are ffill, bfill, and interpolate?

ffill() carries the last valid value forward, bfill() pulls the next valid value backward, and interpolate() estimates numeric gaps linearly. These fit ordered data like time series, where "missing" usually means "unchanged since last reading."

df["temp"].ffill()          # 20.1, 20.1, 22.4, 22.4, 23.0
df["temp"].bfill()          # 20.1, 22.4, 22.4, 23.0, 23.0
df["temp"].interpolate()    # 20.1, 21.25, 22.4, 22.7, 23.0

# df.fillna(method="ffill") raises a TypeError in pandas 3 — call ffill() directly.

When Should You Use dropna?

dropna removes rows (or columns with axis=1) containing missing values; use it when a row is unusable without the value, not as a reflex. subset= limits which columns count, and thresh= keeps rows with at least N valid values.

df.dropna()                        # drop rows with ANY NaN
df.dropna(subset=["temp"])         # only require temp to be present
df.dropna(thresh=2)                # keep rows with >= 2 non-null values

What's the Difference Between dropna and fillna?

dropna deletes every row with a missing value, taking that row's valid data in other columns with it; fillna keeps every row and substitutes a value you choose, such as a constant, the column mean or median, or the minimum. Drop when only a few rows are affected and they're unusable anyway; fill when losing rows would bias or shrink your data and you can justify the replacement value.

len(df)             # 5
len(df.dropna())    # 2  -- 3 of 5 rows gone, including their valid values

filled = df.fillna({"temp": df["temp"].median(), "city": "unknown"})
len(filled)         # 5  -- every row kept

df["temp"].fillna(df["temp"].min())   # fill with the column minimum instead

Why Avoid inplace=True?

inplace=True modifies the DataFrame directly instead of returning a new one, and dropna(inplace=True) returns None, so df = df.dropna(inplace=True) replaces your data with None. It rarely saves memory and breaks method chaining, and pandas 3.1 deprecates inplace on dropna, so it now emits a warning. Assign the result instead.

# Aging pattern:
df.dropna(inplace=True)          # modifies df, returns None
df = df.dropna(inplace=True)     # bug: df is now None

# Preferred:
df = df.dropna()
clean = df.dropna(subset=["temp"]).assign(city=lambda d: d["city"].fillna("unknown"))

Common Pitfalls

Most missing-data bugs come from five habits: comparing to NaN with ==, ignoring the float conversion NaN forces on integer columns, passing column names to dropna positionally, expecting ffill to fill leading gaps, and computing fill values on the full dataset before a train/test split.

• NaN != NaN: df[df["temp"] == np.nan] always returns nothing. Use isna().
• NaN is a float: an integer column with missing values silently becomes float64. Use the nullable Int64 dtype to keep integers.
• Column names go in subset=: df.dropna("temp") raises a TypeError in current pandas because dropna's arguments are keyword-only. Write df.dropna(subset=["temp"]).
• Leading NaN survives ffill: there's nothing before it to carry forward. Chain .ffill().bfill() if edges must be filled.
• Filling before train/test split leaks data: compute fill values (means, medians) on training data only.

Missing values often enter at load time — the na_values options covered in reading CSVs in Python let you catch placeholders like "N/A" or "-" as real NaN from the start. Once missing values are handled, the usual next cleaning step is removing duplicate rows with drop_duplicates.