Define what makes two entries the same
Are “Apple,” “apple,” and “APPLE” duplicates? Is “José” the same as “Jose”? Is “Item 02” equivalent to “Item 2”? The answer depends on the data. Choose comparison rules before deleting anything and keep the original list.
Identify the real separator
Lists may use newlines, commas, tabs, semicolons, or custom delimiters. Comma-separated text is not automatically a simple list because quoted values can contain commas and line breaks. Parse genuine CSV rather than splitting blindly.
A visually simple paste may contain multi-column records or embedded delimiters.
Normalize only what is irrelevant
- Trim leading and trailing whitespace.
- Collapse repeated spaces when spacing has no meaning.
- Normalize line endings.
- Choose case-sensitive or case-insensitive comparison.
- Use Unicode normalization only when appropriate.
- Remove empty records deliberately.
Codes, usernames, file paths, and cryptographic values may require exact comparison.
Natural and locale-aware sorting
Character sorting can place “item 10” before “item 2.” Natural sorting compares numeric segments by value. Locale-aware collation changes how accents and case compare. JavaScript’s Intl.Collator supports locale-sensitive comparison, and modern array sorting is stable, preserving the original order of equal values.
| Mode | Best for | Risk |
|---|---|---|
| Exact character | Technical identifiers | Unnatural numeric order |
| Case-insensitive | General names | Case-sensitive values may collide |
| Natural numeric | Filenames and numbered items | Leading-zero distinctions |
| Locale-aware | Human-language names | Results vary by locale/options |
Use Jivaro’s Text Sorter and Cleaner
- Open Text Sorter & Cleaner.
- Paste text or import TXT, CSV, or Markdown.
- Select the true separator.
- Apply whitespace and empty-line cleanup.
- Choose duplicate and case behavior.
- Select alphabetical or natural sorting.
- Review counts and a sample of collisions.
- Copy or download the result.
Keep a reversible duplicate report
Record input count, empty records removed, duplicate count, normalization rules, sort mode, locale, and output count. For customer or product data, preserve the canonical value and all variants that mapped to it.
Reformat for the consumer
Changing lines to CSV requires quoting values that contain commas, quotes, or line breaks. Markdown tables require escaping pipes. Case conversion can corrupt identifiers and URLs. Test the final format with its real importer.
Large-list limits
Sorting and deduplication create additional in-memory structures. Very large inputs may freeze a browser tab or exceed clipboard limits. Partition only when duplicates cannot cross partitions; otherwise use a streaming or database workflow.
Quality-control checklist
Input minus empties and duplicates equals output.
Review case, accent, space, and numeric variants.
Inspect 1, 2, 9, 10, and leading zeros.
Confirm accents, symbols, emoji, and scripts.
Test quoting, delimiter, and line endings.
Never make the cleaned output the only copy.
Domain-specific duplicate rules
Email addresses may be compared case-insensitively in the domain but should not be rewritten casually in the local part. Phone numbers need country context. Company names may require a maintained alias map. File paths can be case-sensitive on one system and insensitive on another. Human names cannot be reliably deduplicated by spelling alone.
When the record represents a person, account, payment, or legal entity, list cleanup is not identity resolution. Preserve candidate matches and require review rather than deleting automatically.
Stable sorting supports multi-key workflows
A stable sort allows a second sort to preserve the first order among equal values. For example, sort records by name, then stably sort by status to group status while keeping names ordered inside each group. Document the sequence because reversing the sort order changes the result.
Frequently asked questions
Only when case is irrelevant. Usernames, paths, codes, and identifiers may be case-sensitive.
Character sorting compares text positions; natural sorting compares numeric segments by value.
No. CSV fields can contain quoted commas and line breaks.
Record counts, inspect removed variants, test Unicode/case collisions, and retain the original.
Related Jivaro apps
Sort, clean, deduplicate, transform, format, copy, and download text lists with locale-aware ordering and optional preserved headers privately in your browser.
Open appSources and references
- Unicode Collation AlgorithmUnicode Consortium · reference
- Unicode Text SegmentationUnicode Consortium · reference
- Intl.CollatorMDN Web Docs · reference
- Array.prototype.sort()MDN Web Docs · reference
- Text Sorter & CleanerJivaro · first-party