The Count Duplicate Lines filter outputs a count of the number of times a line has been repeated (only if there are 2 or more occurrences). The file should be sorted prior to this filter so that duplicate lines are placed next to each other. A single copy of each duplicated line is output, with a count of the number of duplicates at the start of each line. From a set of duplicate lines, the line that gets output is the last duplicate line of the set, unless the set starts on the first line of the file, in which case the first line gets output (when Ignore Case is checked, the duplicate lines can be different).

Ignore case

If ignore case is checked, lines do not need to be cased identically to be considered duplicates. Two identical lines, one in upper case, and one in lower case, would be considered duplicates and removed by this filter. If ignore case is unchecked, the lines must be identical to be considered duplicates. The case checking routines are ANSI aware, so their behaviour may change depending on your locale.

Start column

The comparison can also ignore leading characters if desired by setting the start column higher than 1. This can be used to skip line numbers, which can be used to find duplicates that are not adjacent. To skip line numbers, set the Start Column to 6 (or so), and set the length to 4096, or a length greater than your maximum line length.

Length

The comparison can also ignore trailing characters if desired, by setting the length to less than the length of the line.

Include counts of 1

Normally this filter only outputs lines with counts of 2 or more (ie, they are duplicates). When this box is checked,

See also

Remove duplicate lines
Remove lines found in file
Retain lines found in file

 Contact Us   Support   Community   Tutorials and User Guides (online)
 Copyright © 1999-2006 DataMystic. All rights reserved.