Remove Duplicate Lines

Drop the repeats from a list, isolate them, or keep only the lines that occurred once.

Runs on your deviceTextWorkingNo account needed

What this tool does

Remove Duplicate Lines reads the input one line at a time and decides what survives. One of each line keeps the first occurrence and discards every later repeat. Only the repeated lines throws away the singletons and returns one copy of each value that appeared more than once. Only the lines that appear once does the reverse, deleting anything that was ever repeated. When you keep one of each, the Occurrence to keep dropdown chooses whether the survivor is the first copy or the last, which also moves where it sits in the output.

What counts as the same line is decided by two checkboxes, and both start in the forgiving position. Ignore leading and trailing spaces compares the trimmed line, so an item pushed over by a stray tab still matches its twin. Case sensitive is off, so Apple and apple are one value. Drop blank lines is on as well, so empty and whitespace-only lines are removed before anything is compared. The comparison is only ever used for matching: the line that survives is written out exactly as you typed it, indentation and capitalisation intact.

Matching is done on the whole line. There is no notion of a field or a column, so two rows differing only in a trailing comment are two different values, and deduplicating tabular data by one column is a job for Remove Duplicates instead. Show a count table is on by default and changes what the output pane holds: whichever of the three modes you picked, the moment the input contains a repeat that pane fills with each repeated value against the number of times it was seen, ordered by frequency and capped at 200 rows, and Copy output then copies that table as CSV rather than the surviving lines. Untick it to get the lines themselves back.

The work happens in this browser tab using the page's own JavaScript, so nothing you paste is transmitted or stored, and the tool still runs with the network disconnected. The list is held in memory while it is compared, which makes a very large paste sluggish long before it fails. Output stays in input order unless you tick Sort the result, which sorts case-insensitively and reads digits inside a line as numbers, so item2 lands before item10 rather than after it.

How to use it

  1. Paste the list. Drop it into the Lines box, one item per line, or press Load sample for a short list that already contains repeats, mixed capitalisation and a blank line.
  2. Choose what to keep. One of each line for a plain dedupe, Only the repeated lines when you need to report on the duplicates, Only the lines that appear once to isolate the values that occurred a single time.
  3. Set the matching rules. Decide whether capitalisation and surrounding spaces count, and whether blank lines are dropped. Keeping one of each also lets you pick the first or the last occurrence.
  4. Read the tally under the output. The hint reports how many lines were compared once any blank ones were dropped, how many came out and how many distinct values were repeated. Copy output puts whatever the pane holds on the clipboard.

Processing & privacy

Runs on your device. Your file is processed directly in this browser.

Common use cases

Cleaning a merged mailing list

Two exports stitched together repeat addresses. With case sensitivity off, [email protected] and [email protected] collapse into a single entry rather than surviving as two.

Reporting on what is duplicated

Switch to Only the repeated lines and the count table names each repeated value and how often it occurred — the answer you need when the duplicates have to be explained rather than deleted.

Comparing two lists for odd ones out

Paste both lists together and choose Only the lines that appear once. Untick Show a count table too, or the pane fills with the tally of what matched instead of the odd ones out.

Tidying a dump of URLs or keywords

Crawl exports and keyword reports repeat the same string dozens of times. Dedupe text first, then tick Sort the result so what is left reads in a sensible order.

Questions

Is my list uploaded anywhere?
No. Every comparison runs in this browser tab, so nothing is transmitted, logged or stored and the tool still works offline. That matters when the list is customer addresses, licence keys or internal hostnames.
Why is the output a table instead of my lines?
Because Show a count table is switched on and your input contained at least one repeat. The table takes over the output pane, and Copy output then copies it as CSV. Untick that setting and run again to get the deduplicated text back.
Are Apple and apple treated as the same line?
By default yes, because Case sensitive starts switched off. Tick it when capitalisation carries meaning — code identifiers, case-sensitive filenames, secrets — and the two are then counted as separate values.
Which copy of a duplicate survives?
The first one, unless you set Occurrence to keep to Last. The survivor stays in its own position, so keeping the last copy moves that value further down the result.
Can I deduplicate by a single column?
Not with this tool. It compares entire lines, so any difference anywhere in the line makes it a distinct value. Remove Duplicates works on tabular data and lets you name the column that decides.
What happens to blank lines?
They are discarded before matching, because Drop blank lines is on by default. Untick it and a blank line becomes an ordinary value — and with trimming still on, every blank line matches every other, so one of them survives.
Does it change my remaining lines in any way?
No. Trimming and case folding are used to compare lines, never to rewrite them, so the unique lines come out with their original spacing and capitalisation. The only optional change to the result is the order, and only when you tick Sort the result. The one place the trimmed form does show is the count table, which lists each repeated value as the first copy was spelled, minus its outer spacing.

Related tools