Suggestion: Provide filter Remove all diacritics

Get help with installation and running here.

Moderators: DataMystic Support, Moderators

User avatar
DataMystic Support
Site Admin
Posts: 2203
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Suggestion: Provide filter Remove all diacritics

Post by DataMystic Support » Fri May 08, 2020 8:23 am

Hi David,

I understand the value, and it might be possible with come caveats.

My concern is that TextPipe has tried hard never to assume the format of incoming files, because auto-detection is generally inprecise. But knowing internally what one filter expects vs one another filter is giving it is do-able.

I can see some potential issues. If two filters were separated by a search/replace, then there is no way of knowing if the search/replace modified the file format. But equally, this might be the express intention of the user, to modify the file format.
Regards,

Simon Carter, https://www.DataMystic.com
https://www.JadeDiabetes.com - Insulin dose calculator for Type 1 diabetes
https://www.DownloadPipe.com - 250,000 free software downloads

DFH
Posts: 950
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Suggestion: Provide filter Remove all diacritics

Post by DFH » Sat May 09, 2020 6:15 pm

See also my recent email (sent yesterday) about grouping the Unicode filters that are UTF-16LE only.

Aside: hasn't the official Unicode terrminology for UTF-16LE been changed to UCS-2 BOM ?

David

User avatar
DataMystic Support
Site Admin
Posts: 2203
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Suggestion: Provide filter Remove all diacritics

Post by DataMystic Support » Mon May 11, 2020 8:26 am

Thanks David - had a look but did not find any reference to this terminology change. Do you have a reference?
Regards,

Simon Carter, https://www.DataMystic.com
https://www.JadeDiabetes.com - Insulin dose calculator for Type 1 diabetes
https://www.DownloadPipe.com - 250,000 free software downloads

DFH
Posts: 950
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Suggestion: Provide filter Remove all diacritics

Post by DFH » Tue May 12, 2020 7:24 am

I was mistaken. UCS-2 was an earlier form that preceded UTF-16.

I was confused by Notepad++ having changed menu options from UTF-16 LE to UCS-2.

It may be because it doesn't fully support the former.

Ah well.

https://en.wikipedia.org/wiki/Universal_Coded_Character_Set

User avatar
DataMystic Support
Site Admin
Posts: 2203
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Suggestion: Provide filter Remove all diacritics

Post by DataMystic Support » Tue May 12, 2020 7:49 am

Ok, thanks for the clarification!
Regards,

Simon Carter, https://www.DataMystic.com
https://www.JadeDiabetes.com - Insulin dose calculator for Type 1 diabetes
https://www.DownloadPipe.com - 250,000 free software downloads

DFH
Posts: 950
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Suggestion: Provide filter Remove all diacritics

Post by DFH » Sat May 16, 2020 4:54 am

Please add Help page for the new Remove diacritics filter.

Please add Help page to explain Filter Library\Unicode\UTF-16LE only

Please add See also links in these existing filter help pages to the new page.

David

User avatar
DataMystic Support
Site Admin
Posts: 2203
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Suggestion: Provide filter Remove all diacritics

Post by DataMystic Support » Sat May 16, 2020 8:02 pm

Remove diacritics is ready for TP 11.6. It will handle UTF16-LE - any other format can be converted using other TP filters.
Regards,

Simon Carter, https://www.DataMystic.com
https://www.JadeDiabetes.com - Insulin dose calculator for Type 1 diabetes
https://www.DownloadPipe.com - 250,000 free software downloads

Post Reply