Remove Duplicates

Deduplicate a table by the column that identifies a record, or a plain list line by line.

Runs on your deviceDataWorkingNo account needed

What this tool does

Remove Duplicates walks your data once and builds a comparison key for each row. A row whose key has been seen already is a duplicate; the rest are the unique rows, and they come back in the order they sat in the file. Which copy survives is yours to choose. The first occurrence discards everything that follows it; The last occurrence keeps the final copy instead. Survivors are written at the position of whichever copy won, so switching to the last one puts that record where the later copy sat rather than where the first one did.

The key is built from the whole row unless you name columns, and that default matters more than it sounds. An export carrying an id, a timestamp or a signup date differs in at least one cell on every line, so a whole-row comparison finds nothing to remove and the summary reports no duplicates. Naming the field that actually identifies a record is usually the difference between a clean result and no result at all — email as a header name, 3 as a position, 3-5 as a span, and any mixture of those in one list. A name is matched case-insensitively against the header row, which means that row has to be there: untick First row is a header and only numbers will resolve.

Two checkboxes decide what counts as the same value, and one of them is stricter here than you might expect. Match case is switched on by default, so [email protected] and [email protected] survive as two separate records until you turn it off — worth doing for addresses, which mail servers do not treat as case-sensitive. Ignore leading and trailing spaces is on, so a cell carrying a stray tab keys the same as the tidy copy of itself. Neither setting touches your data: folding and trimming happen on the key alone, every surviving row is emitted from the cells that were parsed, and nothing is rounded, re-dated, sorted or merged.

Everything happens inside the page, so the rows are never sent anywhere and unplugging the network takes nothing away from it. The shape of the workspace is the real constraint: text arrives through the input box and leaves through Copy output, with no upload step and no saved file, so the ceiling is whatever you can paste and the tab can hold — a few megabytes of pasted CSV is fine, and an export running to hundreds of thousands of rows wants a script instead. Automatic delimiter detection deserves one look as well. It reads the first five lines, counts commas, semicolons, tabs and pipes wherever they fall, quoted text included, and takes the highest count, with a tie going to whichever of those four comes first in that order. A semicolon-separated export whose address column is full of commas can therefore be read as comma-separated, so set the input delimiter yourself when the result comes back shredded.

How to use it

  1. Paste the table or the list. Put it in the Data box. Press Load sample and you get seven customer rows in which one person is listed twice, alongside a name that holds a quoted comma. A leading byte-order mark is stripped.
  2. Choose CSV or plain lines. CSV rows compares cells and can work on named columns. Plain lines treats every line as one value, which suits a list of emails, IDs or keywords with no structure.
  3. Name the columns that decide. Leave the box empty to compare whole rows, or list the identifying fields — email, or 1, 3-5. Untick First row is a header if the file starts straight into data.
  4. Decide what counts as a match. Turn Match case off for anything where capitalisation is noise, leave trimming on, and choose whether the first or the last occurrence is the one that stays.
  5. Choose the output, then run. Deduplicated data returns the survivors; Only the removed duplicates returns what was cut. The line under the output pane counts the duplicates that went and the unique rows left, and Copy output takes whichever text is showing.

Processing & privacy

Runs on your device. Your file is processed directly in this browser.

Common use cases

Stitching two exports into one list

Two overlapping CRM downloads repeat the same people. Dedupe by the email column and every record is counted once, regardless of what the id or signup date columns say.

Checking what a dedupe would delete before you do it

Switch the output to Only the removed duplicates and the tool hands back exactly the rows it would have dropped, so the deletion can be reviewed rather than assumed.

Keeping the most recent version of a record

Where the same customer appears twice with different plan or date values, compare by email and keep the last occurrence, so the later row is the one that survives.

Cleaning a bare list of values

Plain lines mode takes a column pasted out of a spreadsheet — order numbers, SKUs, keywords — and returns unique rows without any delimiter or header to configure.

Questions

Do the rows I paste leave my machine?
No. The parse, the comparison and the rebuilt output all happen in the page in front of you. No request carries the rows anywhere, nothing is kept once the tab closes, and the tool still runs after you disconnect the network.
Why did it say no duplicates when I can see them?
Almost always because the comparison is still on whole rows and the repeated records differ in some other cell — an id, a timestamp, a note. Name the column that identifies the record and the same file usually produces the result you expected.
Which copy of a duplicate is kept, and where does it end up?
The first, unless you switch the dropdown to The last occurrence. Either way the winner is written at the position it held in the input, so a last-occurrence run reports the same counts but places that record further into the file.
Can I get the duplicates instead of losing them?
Yes, through the Only the removed duplicates output. It lists every discarded copy rather than one row per repeated value, so a record appearing three times contributes two rows, and the header row is carried over only when something was actually removed.
Does capitalisation count?
By default yes, because Match case starts switched on, which keeps ACME and acme apart. Untick it and both fold to the same key, while the row that survives is still written out with the capitalisation it arrived with.
What happens to blank rows and rows with missing cells?
Blank and whitespace-only rows are dropped before any comparison, unless you untick Drop empty rows and lines. Ragged rows are squared off first — each one widened to the column count of the widest row — so a row that stops early compares with empty cells at the end instead of pulling later values into the wrong fields.
How does this differ from Remove Duplicate Lines?
That tool compares entire lines and can also report a count table or isolate values that occurred only once. This one understands columns and delimiters, which is what tabular data needs. If the repeats only differ by stray spacing or casing inside cells, normalise them with Data Cleaner first, then dedupe.
Does the output keep my delimiter?
It uses the input's delimiter unless Output delimiter says otherwise. Cells are re-quoted for whichever character ends up separating them, so a value holding that character, a double quote or a line break comes back in quotes. CSV records are separated with CRLF; plain-lines output is joined with ordinary newlines.

Related tools

  • JSON to CSV — Flatten a JSON array into a CSV table.
  • CSV to JSON — Turn a CSV file into structured JSON.
  • CSV Viewer — Open and browse a large CSV without a spreadsheet app.
  • CSV to Excel — Convert CSV files into an XLSX workbook.
  • Excel to CSV — Export spreadsheet sheets to CSV.
  • CSV Merge — Combine multiple CSV files with matching columns.