Every list you paste from somewhere else arrives with repeats. A merged spreadsheet doubles the same customers. A keyword export repeats the same terms in different capitalisation. A scraped link list contains the same URL five times. A survey export repeats the same answers. Cleaning that by hand means scanning thousands of lines and hoping you catch them all — and the moment the list is long enough to matter, hand-checking stops being an option. Our free Remove Duplicate Lines tool does the whole job in one click: paste any list, and it strips out the repeats, tells you exactly how many it removed, and hands back a clean list ready to copy. Everything runs in your browser — your list never leaves your device, so customer lists and private data stay private. This guide explains how deduplication works, the two options that change the result, and the mistakes that quietly undo a clean list.
Step one: paste your list into the input box — one item per line. Step two: choose your options. Tick Ignore case if entries like "Boston" and "boston" should be treated as the same line; leave it off if casing is meaningful, for example in passwords or codes. Tick Sort lines if you want the cleaned list in alphabetical order; otherwise the original order is preserved and the first occurrence of each line is kept. Step three: press Remove Duplicates. The result box reports the headline figure — "3 duplicates removed · 7 unique lines kept" — and shows the cleaned list, ready to copy with the Copy Result button. Two small behaviours are worth knowing before you paste. Trailing spaces are trimmed before comparing, so "Boston " and "Boston" count as the same line. And repeated blank lines collapse into one, since stray empty lines are the most common accidental "duplicates" in pasted lists — lines that contain only spaces are treated as blank.
The tool walks through your lines from top to bottom, remembering each line it has seen. The first time a line appears it is kept; every later repeat is counted as a duplicate and dropped. The arithmetic is simple: kept lines = first occurrence of each unique line, and duplicates removed = total lines − kept lines. With ignore-case on, lines are compared in lowercase, so "SEO Tips" and "seo tips" match — but the first version's original casing is what survives, not a normalised one. Sorting happens after deduplication, using alphabetical order. The whole operation is linear in the size of your list, so even tens of thousands of lines process instantly. The worked example on the tool page shows why the options matter. Take a 10-line list: boston, Denver, boston, Austin, denver, Boston, Seattle, austin, Denver, Miami. Case-sensitive with no sort, it removes 2 duplicates — the second "boston" and the third "Denver" — and keeps 8 lines. Turn ignore-case on and the same list gives 5 duplicates removed, because "boston", "Boston" and "denver", "Denver" and "austin", "Austin" now collapse onto each other. Add sorting and the result is the same 5 unique lines in alphabetical order: Austin, Denver, boston, Seattle, Miami. The same list gives 2 or 5 duplicates depending on one checkbox — which is exactly why the option exists, and why you should decide before you paste, not after.
Cleaning customer or email lists. Merged spreadsheets always contain repeat entries; dedup before importing to avoid double emails — ignore case on. SEO keyword lists. Keyword research exports repeat the same terms with different casing; a clean list uploads correctly to ad platforms — ignore case on, sort on. Removing repeated URLs. Scraped or exported link lists contain the same URLs many times — case-sensitive, because URLs can be case-sensitive. Survey and form responses. Open-text answers repeat; dedup to see the distinct responses — ignore case on. Log file cleanup. Repeated error lines hide the unique failures — case-sensitive, no sort, so the order of first appearance, which is usually chronological, is preserved. The SEO keyword case is the classic: export 5,000 keywords from a research tool, run them through here with ignore-case and sort, and you get a clean alphabetical file in seconds — a job that takes half an hour by hand in a spreadsheet.
Deduping before fixing the paste. Copying from a PDF or a web page often introduces stray line breaks and double spaces, which create false duplicates and false uniques; fix the paste first with PasteFix, which repairs the spacing and stray line breaks copying introduces, then deduplicate the lines here. Using ignore-case on case-sensitive data. Passwords, codes and URLs can differ only by casing — turn the option off for those, or you will delete entries that were genuinely different. Sorting when order matters. The original order is the order of first appearance, which for logs and exports is usually chronological; ticking sort destroys that information. Expecting fuzzy matching. "Boston" and "Bostn" are different lines to a deduper, and near-duplicates with typos are not detected. Forgetting the blank-line rule. A list that looks clean can hide a dozen empty lines that collapse into one — harmless, but it explains a count that looks one or two higher than you expected. Pasting a list that is already sorted and then sorting again. Harmless, but it hides the first-occurrence order you might have wanted to keep.
The honest limits are four. First, the tool compares whole lines only: "Boston " (with a trailing space) matches "Boston", but "New Boston" and "Boston" are different lines, and near-duplicates with typos are not detected — fuzzy matching is beyond a line deduper. Second, sorting is plain alphabetical, not natural sort, so "item10" comes before "item2". Third, extremely large pastes (hundreds of thousands of lines) may be slow on old phones. Fourth, remember the tool only shows you the count after running — it cannot warn you before you paste. Within those limits, the tool does one job completely: it turns a messy pasted list into a clean one in a click, privately, with an exact count of what changed. Lists are one kind of transcript; recordings are another. The Silence Chapters tool splits podcasts and meetings at their pauses into clean, exportable clips — the audio equivalent of deduplicating a text file, one honest cut at a time. The companion guides are our text case guide, for fixing the casing before you dedup, our word count guide, for measuring what the cleaned list is worth, and our character counter guide, for checking the cleaned lines against tweet, SMS and meta limits.
Fix the casing before you dedup — the case guide, with a worked example.
Count what the cleaned list is worth — words, characters and reading time.
Check the cleaned lines against tweet, SMS and meta limits.