Count duplicate lines filter and non-Roman script

Get help with installation and running here.

Moderators: DataMystic Support, Moderators

Post Reply
DFH
Posts: 944
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Count duplicate lines filter and non-Roman script

Post by DFH » Wed Sep 27, 2017 6:33 pm

When the Count duplicate lines filter is used on UTF-8 encoded input data that is in a particular non-Roman script, such as the Gurmukhi writing system used for Eastern Punjabi, what collation algorithm determines the sort order of the output lines?

I successfully made a filter to extract and count all the words from a Punjabi Bible translation, and I am puzzled by the order of the counted words.

cf. For a language that uses Roman script such as Shona, it's clear that the output is sorted on the words as if they were ANSI data, even though the file being processed is UTF-8.

It get's more complicated even with Roman script data when the alphabet makes use of codepoints beyond \xFF.

Best regards,

David

DFH
Posts: 944
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Count duplicate lines filter and non-Roman script

Post by DFH » Wed May 22, 2019 3:23 am

This post has not yet had a response!

DFH
Posts: 944
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Count duplicate lines filter and non-Roman script

Post by DFH » Mon Mar 02, 2020 9:49 pm

Any update to report on this?

Aside: Is the same collation algorithm used in the Sort filter?

Might it be feasible to enhance both filters by including in the UI a drop-down list control for selecting a writing system specific collation order?

David

DFH
Posts: 944
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Count duplicate lines filter and non-Roman script

Post by DFH » Mon May 04, 2020 5:22 am

Help for this filter includes:
The file need NOT be sorted prior to this filter.
That may well be, but the output of using this filter on an input stream with a mixture of Unicode blocks comes out in a very strange order!

The output of this filter then needs to be piped into the Sort filter and UTF-8 sorted on the text column corresponding to the start of the lines that were counted.

IMHO, including a dropdown in UI for this filter to select the sort method should be implemented.
This would obviate the need for using the sort filter afterwards.

David

Post Reply