Remove Duplicate Lines

Remove duplicate lines from a list in one step and see how many lines were kept and removed. You can also check how often each entry was repeated.

0Original lines
0Lines kept
0Lines removed
iPhone app ยท Free to downloadLED Marquee - Scrolling TextTurn your iPhone into a scrolling LED sign for concerts, airports and events.View on the App Store

About the Remove Duplicate Lines

This tool keeps the first occurrence of every line and deletes the repeats. Use it to clean up email or phone lists, keyword lists, attendee lists or a column copied from a spreadsheet. The original order is preserved.

Typical jobs: merging sign-up sheets from two events where some people registered twice, combining keyword lists from several ad groups before uploading them, or finding SKUs entered twice in a product feed. The Duplicates view lists every repeated line with how many times it appeared, so you can see what overlapped before you delete anything.

How to use

  1. Paste your list into the box, one item per line. A column copied from Excel or Google Sheets works too.
  2. Turn Ignore leading/trailing spaces, Ignore case or Keep empty lines on or off as needed.
  3. Check the deduplicated list and tap Copy.
  4. Tap Duplicates to see which lines appeared more than once and how often.

Worked examples

A newsletter sign-up list (default options)

Input
john@example.com
sara@example.com
john@example.com
Mike@Example.com
mike@example.comโฃ (trailing space)
sara@example.com
Result
john@example.com / sara@example.com / Mike@Example.com / mike@example.com
6 original lines โ†’ 4 kept, 2 removed ยท Duplicates: john@example.com 2ร—, sara@example.com 2ร—

The trailing space is ignored by default, but Mike@Example.com and mike@example.com still count as different lines because case matters.

The same list with Ignore case on

Input
Same 6 lines + Ignore case ticked
Result
john@example.com / sara@example.com / Mike@Example.com
3 kept, 3 removed ยท Mike@Example.com 2ร—

Email addresses are effectively case-insensitive, so this is the right setting for mailing lists. The spelling that appears first is the one kept. Leave it off for case-sensitive data such as coupon codes.

Ignore spaces turned off (common mistake)

Input
Same 6 lines + Ignore leading/trailing spaces unticked
Result
4 kept: โ€ฆ Mike@Example.com / mike@example.comโฃ (still has the space)

The count looks the same, but the value with the stray space survives and can break a VLOOKUP or an email import later. Keep this option on for data copied from spreadsheets.

What the options do

OptionWhen onExample
Ignore leading/trailing spacesSpaces at the start and end of a line are removed before comparing"apple " and "apple" count as the same
Ignore caseUpper- and lowercase are treated alikeApple = apple
Keep empty linesBlank lines stay where they areKeeps paragraph breaks

When case is ignored, the spelling of the first occurrence is kept.

Using it with Excel or Google Sheets

Select the column, copy it (Ctrl+C or โŒ˜C) and paste it into the box. Copy the result and paste it back into the sheet: each line goes into its own cell. Inside the apps you can also use Data โ†’ Remove Duplicates in Excel or the =UNIQUE(A:A) function in Google Sheets and Excel 365.

FAQ

Does the order change?

No. Each line stays where it first appeared. To sort the list alphabetically, use Sort Lines afterwards.

How many lines can it handle?

Everything runs in your browser, so even tens of thousands of lines are processed instantly. Nothing is uploaded.

Does it work with comma-separated lists?

Lines are compared one by one, so replace the commas with line breaks first.

Can I export how many times each line appeared?

Yes. Switch to the Duplicates view and tap Copy: each repeated line is copied as "line[tab]count", which pastes into two columns in Excel or Google Sheets, sorted from most to least frequent.

What happens to empty lines?

By default all empty lines are removed. Tick Keep empty lines to leave them where they are, for example between groups; empty lines are never treated as duplicates of each other.