Remove Duplicate Lines

What Does Remove Duplicate Lines Do?

It deletes repeated lines from a list, so each item appears once. You can ignore differences in capital letters and extra spaces, drop empty lines, keep the first or last copy, and sort what is left. It works on any list with one item per line - email addresses, keywords, URLs, product codes or names.

When to Use It

• Cleaning a mailing list before importing it.
• Merging keyword lists from several research tools.
• Finding which entries appear more than once in an export.

How to Use It

1. Paste your list, one item per line.
2. Tick Ignore upper/lower case if Asha@Example.com and asha@example.com should count as the same.
3. Choose what to keep: the first copy, the last copy, only lines that appear once, or only the duplicated ones.
4. Copy or download the result.

Example

This list of seven lines contains asha@example.com twice (once with capitals), ben@example.com twice (once with a trailing space) and an empty line. With case and spaces ignored, the result is four lines - asha@example.com, ben@example.com, chen@example.com and diego@example.com - and the status line reports that two lines had duplicates.

Keeping the Order

Unlike a spreadsheet's "remove duplicates" after sorting, the tool keeps your lines in their original order unless you tick Sort. Choosing Only the duplicated lines is a quick way to find out which entries were repeated.

Limitations

Lines are compared as whole lines, so Asha Khan and Khan, Asha are different. Two lines that look the same can still differ because of an invisible character such as a zero-width space - check them with the Hidden Character Detector. To find repeated sentences inside a paragraph, use the Duplicate Sentence Finder.