Clean text without losing information
By Delete Duplicate Lines Online · Published · Updated
Keep the original file before starting. A useful cleanup changes the specific problem you have identified; it does not assume that every difference is a mistake.
Decide what counts as a duplicate
Exact matching treats Apple and apple as different. Ignore case only when capitalization has no meaning in the list. Trimming for comparison can match lines with extra spaces without rewriting the selected original line.
Choose which occurrence stays
Keep first retains the earliest entry. Keep last retains the latest occurrence and its position. Remove all duplicates is different: a repeated group disappears entirely, leaving only lines that occurred once.
Handle blank lines separately
Blank lines may separate paragraphs or groups. Remove them only when your output is meant to be a continuous list. The empty-line tool preserves repeats.
Check the output before replacing data
- Try a small representative sample.
- Compare the original and output counts.
- Inspect a few expected matches and a few expected differences.
- Use the repeated-line report when you need occurrence counts.
- Save the result under a new filename.