pandas drop_duplicates(): Remove Duplicate Rows Properly

drop_duplicates() removes duplicate rows from a DataFrame, but using it properly means answering two questions first: which columns define a duplicate (subset), and which copy should survive (keep). Skip those and pandas silently keeps the first occurrence of rows that match on every column — which is often not the deduplication you actually wanted.

How do I remove duplicate rows in pandas?

Call df.drop_duplicates(). With no arguments it compares entire rows and keeps the first occurrence of each. It returns a new DataFrame — your original is untouched unless you assign the result back. This is the pandas equivalent of de-duplicating with SELECT DISTINCT in SQL.

import pandas as pd

df = pd.DataFrame({
    "order_id": [101, 102, 102, 103, 103, 103],
    "customer": ["amy", "ben", "ben", "cara", "cara", "cara"],
    "status":   ["paid", "paid", "paid", "open", "open", "paid"],
})

deduped = df.drop_duplicates()
print(deduped)
#    order_id customer status
# 0       101      amy   paid
# 1       102      ben   paid
# 3       103     cara   open
# 5       103     cara   paid   <-- kept: status differs, so not a full-row dupe

Notice row 5 survives: order 103 appears with two different statuses, so the rows are not identical. That is exactly why subset exists.

How do the subset and keep parameters work?

subset narrows the duplicate check to specific columns — rows count as duplicates when they match on just those columns. keep chooses the survivor: "first" keeps the earliest occurrence, "last" the latest, and False keeps none. Together they express "one row per key, and here's the tie-break."

# One row per order_id, keeping the last version seen
latest = df.drop_duplicates(subset=["order_id"], keep="last")
print(latest)
#    order_id customer status
# 0       101      amy   paid
# 2       102      ben   paid
# 5       103     cara   paid

# Duplicate check on a two-column business key
df.drop_duplicates(subset=["order_id", "customer"])

How do I see all duplicate rows instead of dropping them?

Two tools: keep=False shows every member of each duplicate group (great for eyeballing what went wrong), and duplicated() returns a boolean mask that flags duplicates without removing anything — ideal for counting them or filtering.

# Every row involved in a duplicate group
dupes = df[df.duplicated(subset=["order_id"], keep=False)]
print(dupes)
#    order_id customer status
# 1       102      ben   paid
# 2       102      ben   paid
# 3       103     cara   open
# 4       103     cara   open
# 5       103     cara   paid

# How many duplicate rows (beyond the first of each group)?
print(df.duplicated(subset=["order_id"]).sum())   # 3

Audit before you delete — the same advice we give for counting duplicates in SQL and for removing duplicates in Excel. A quick value_counts() on the key column tells you which keys are inflated and by how much; see pandas value_counts().

How do I control which duplicate row survives?

keep="first" and keep="last" only refer to current row order, so the reliable pattern is: sort so your preferred row lands first (or last), then drop. Want the most recent record per customer? Sort by timestamp descending and keep the first.

events = pd.DataFrame({
    "customer": ["amy", "ben", "amy", "ben"],
    "plan":     ["free", "free", "pro", "team"],
    "updated":  pd.to_datetime(["2026-01-05", "2026-02-01",
                                "2026-03-10", "2026-01-20"]),
})

latest = (events
          .sort_values("updated", ascending=False)
          .drop_duplicates(subset=["customer"], keep="first"))
print(latest)
#   customer  plan    updated
# 2      amy   pro 2026-03-10
# 1      ben  free 2026-02-01

This sort-then-drop chain is the pandas answer to SQL's ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...) = 1 pattern, and it is far cheaper than a groupby-apply.

What are the common drop_duplicates() gotchas?

Three trip people up repeatedly. First, the index keeps its old labels after dropping — chain .reset_index(drop=True) if you need clean positions. Second, NaN values are treated as equal to each other, so two rows with NaN in the key column do count as duplicates. Third, assign the result back; forgetting to is the classic "it didn't work" bug.

# Gotcha 1: gap-toothed index
clean = df.drop_duplicates(subset=["order_id"]).reset_index(drop=True)

# Gotcha 2: NaNs match each other
d = pd.DataFrame({"k": [None, None], "v": [1, 1]})
print(len(d.drop_duplicates()))   # 1 — the NaN rows were deduped

# Gotcha 3: no assignment, no effect
df.drop_duplicates(subset=["order_id"])          # result discarded!
df = df.drop_duplicates(subset=["order_id"])     # correct