Remove duplicate lines without sorting?

Get help with installation and running here.

Moderators: DataMystic Support, Moderators

lngzlz
Posts: 41
Joined: Wed Sep 21, 2005 10:43 pm

Remove duplicate lines without sorting?

Postby lngzlz » Wed Feb 22, 2006 6:39 am

Is it possible for TextPipe to remove duplicate lines without having to sort all the lines first?

If sorted, the original order of lines has been crashed. This is usually not desirable. :oops:

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Postby DataMystic Support » Wed Feb 22, 2006 8:02 am

We've added a new filter to the next release - 'etext\remove duplicates without sorting.fll'.

Here is the filter export:

|--Comment...
| Add line numbers which we will use to later restore the original line order
|
|--Add line numbers
| Start line 0
| Increment 1
| [ ] Don't increment on blank lines
| [ ] Don't number blank lines
| Format %7.7d
| [ ] Don't reset on end of file
|
|--Comment...
| Compare lines (ignoring line numbers) and discard duplicates
|
|--Ascending ANSI sort (case insensitive), remove duplicates, start column 9, length 50
|
|--Comment...
| Return the lines to the original order
|
|--Ascending numeric sort, length 7
|
|--Comment...
| Now remove the line numbers
|
|--Remove column:8 columns starting at column 1
|
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

lngzlz
Posts: 41
Joined: Wed Sep 21, 2005 10:43 pm

Postby lngzlz » Wed Feb 22, 2006 2:55 pm

Compare lines (ignoring line numbers) and discard duplicates


How can I ignore line numbers when comparing lines?

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Postby DataMystic Support » Wed Feb 22, 2006 3:17 pm

In

Ascending ANSI sort (case insensitive), remove duplicates, start column 9, length 50

set the Start Column (here set to 9).
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments


Return to “TextPipe Tips and Tricks, Questions and Support”

Who is online

Users browsing this forum: Yahoo [Bot] and 2 guests