How to use it
- Paste your data or upload the file. CSV, TSV, a JSON array of objects, and JSON Lines all work.
- Tick the columns that define a duplicate. For a contact list that's often just email. Leave all ticked to only drop rows that are identical everywhere.
- Pick exact or near duplicates, and which copy to keep.
- Check the cleaning options, then hit the button.
- Look at Removed duplicates before you trust the result. Every removed row shows which row it duplicated.
Clean first, then compare
Most "duplicates" that slip through are the same value with junk around it: a trailing space, a non-breaking space pasted in from a web page, a zero-width character nobody can see. If you compare before cleaning, those rows look different. So the cleaning runs first, on every cell, and the duplicate check runs on the cleaned values.
Invisible characters deserve a mention. Zero-width spaces, soft hyphens and byte order marks come in through copy and paste and break joins, lookups and filters without showing up anywhere. The remove-invisible option strips them and turns non-breaking spaces into normal ones.
Exact or near duplicates
Exact compares the ticked columns after cleaning, ignoring case and punctuation if you leave those options on. It's fast and predictable, and it's the right choice when you have a reliable key like an email or an ID.
Near duplicates uses the same matching engine as the fuzzy matcher to find rows that are similar enough, then groups them, so if A matches B and B matches C, all three are one group and only one survives. Start at 92 and read the removed rows. Near-duplicate removal is where people lose real records, so the tool shows you every decision.
Tip. If you're not sure what's wrong with a file yet, run it through the CSV profiler first. It shows placeholders, stray whitespace and near-unique ID columns before you start deleting anything.
Questions
What counts as a duplicate?
Two rows are duplicates when the columns you ticked hold the same values. By default case and punctuation are ignored, so PRIYA.RAMAN@example.com and priya.raman@example.com are the same email. Near-duplicate mode goes further and catches rows that are similar but not identical, like "Acme Corp, Chennai" and "Acme Corporation, Chennai".
Which copy is kept?
The first one in the file, unless you switch to keeping the last. Everything removed is listed with the row it duplicated, and you can download that list to check it.
Does cleaning change my data?
Only the options you leave ticked. Trimming and stripping invisible characters are on by default because they almost never hurt. Blanking out placeholders like N/A and null is off by default, because sometimes N/A means something.
How big a file can it take?
Files up to 200 MB load. Exact duplicate removal on a few hundred thousand rows takes a second or two. Near-duplicate mode does far more work per row, and I'd keep it under about 50,000 rows.
Is my file uploaded?
No. The page reads and cleans it in your browser. There's no server involved.