Get help with installation and running here.
Moderators: DataMystic Support, Moderators
-
lngzlz
- Posts: 41
- Joined: Wed Sep 21, 2005 10:43 pm
Post
by lngzlz » Wed Feb 22, 2006 6:39 am
Is it possible for TextPipe to remove duplicate lines without having to sort all the lines first?
If sorted, the original order of lines has been crashed. This is usually not desirable.

-
DataMystic Support
- Site Admin
- Posts: 2229
- Joined: Mon Jun 30, 2003 12:32 pm
- Location: Melbourne, Australia
-
Contact:
Post
by DataMystic Support » Wed Feb 22, 2006 8:02 am
We've added a new filter to the next release - 'etext\remove duplicates without sorting.fll'.
Here is the filter export:
|--Comment...
| Add line numbers which we will use to later restore the original line order
|
|--Add line numbers
| Start line 0
| Increment 1
| [ ] Don't increment on blank lines
| [ ] Don't number blank lines
| Format %7.7d
| [ ] Don't reset on end of file
|
|--Comment...
| Compare lines (ignoring line numbers) and discard duplicates
|
|--Ascending ANSI sort (case insensitive), remove duplicates, start column 9, length 50
|
|--Comment...
| Return the lines to the original order
|
|--Ascending numeric sort, length 7
|
|--Comment...
| Now remove the line numbers
|
|--Remove column:8 columns starting at column 1
|
-
lngzlz
- Posts: 41
- Joined: Wed Sep 21, 2005 10:43 pm
Post
by lngzlz » Wed Feb 22, 2006 2:55 pm
Compare lines (ignoring line numbers) and discard duplicates
How can I ignore line numbers when comparing lines?
-
DataMystic Support
- Site Admin
- Posts: 2229
- Joined: Mon Jun 30, 2003 12:32 pm
- Location: Melbourne, Australia
-
Contact:
Post
by DataMystic Support » Wed Feb 22, 2006 3:17 pm
In
Ascending ANSI sort (case insensitive), remove duplicates, start column 9, length 50
set the Start Column (here set to 9).