Note: To remove non-adjacent duplicate lines, use the filter etext\remove duplicates without sorting.fll.

Removes successive duplicate lines. The duplicate lines must be adjacent for this filter to remove them. This filter is most commonly used after a sort - where duplicate lines will be placed together. From a set of duplicate lines, the line that gets output is the last duplicate line of the set, unless the set starts on the first line of the file, in which case the first line gets output.

Ignore case

If ignore case is checked, lines do not need to be cased identically to be considered duplicates. Two identical lines, one in upper case, and one in lower case, would be considered duplicates and removed by this filter. If ignore case is unchecked, the lines must be identical to be considered duplicates. The case checking routines are ANSI aware, so their behaviour may change depending on your locale.

Start column

The Start Column field allows you to ignore leading characters before a comparison is made. This can be very useful for ignoring line numbers is source code etc.

It can also be used (in conjunction with an external sort program) to remove duplicates from an unsorted file. First add line numbers to every line. Then use an external sort program on the file, ignoring the line numbers (the first 6 columns). Then use this filter (with Ignore Leading Characters set to 6) to remove duplicate lines. Finally, remove the line numbers using the Remove Columns (remove first 6 columns) filter. 

Length

The comparison can also ignore trailing characters if desired, by setting the length to less than the length of the line.

See also

Count duplicate lines
Remove menu

 Contact Us   Support   Community   Tutorials and User Guides (online)
 Copyright © 1999-2006 DataMystic. All rights reserved.