Pandas Removing Duplicates

How it works

Two drop-offs can legitimately share a point and weight. A repeated drop_id is a stronger signal of duplicate identity, but its records might disagree. duplicated identifies candidates and drop_duplicates follows a chosen keep policy; neither decides which competing record is true. This example removes one exact repeated record only after showing the duplicated identity list.

Run the code

Select Try in playground to load the complete editable source. Nothing executes until you choose Run. The browser uses Pyodide 0.27.7, Pandas 2.2.3 and NumPy 2.0.2. Each ordinary execution starts a fresh isolated runtime. The tables are fictional and do not use personal files.

For local work, use a separate virtual environment and run python duplicates.py. Download the complete source. The reviewed local environment uses Python 3.12.14, Pandas 3.0.6 and NumPy 2.5.4; the browser uses Pyodide 0.27.7 with Pandas 2.2.3 and NumPy 2.0.2. Compatible examples are checked in both. See the environment setup. No Apache or database server is required.

Example

import pandas as pd
import numpy as np

table = pd.DataFrame({"drop_id":[101, 102, 101], "kg":[12., 8., 12.]})
print(table.loc[table.duplicated("drop_id", keep=False), "drop_id"].tolist())
cleaned = table.drop_duplicates(subset=["drop_id", "kg"], keep="first")
print(cleaned["drop_id"].tolist())

Expected result

[101, 101]
[101, 102]

The duplicate rows agree on every field in this tiny record shape. If the repeated id had a different weight, this exact-record removal would preserve both and require a conflict-resolution policy before key-based reporting.