Subtle bug in Convert Numeric HTML/XML Entities to text

Get help with installation and running here.

Moderators: DataMystic Support, Moderators

Post Reply
DFH
Posts: 871
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Subtle bug in Convert Numeric HTML/XML Entities to text

Post by DFH » Mon Dec 10, 2018 4:28 am

Hi Simon,

I have just encountered some very rare errors in the output of the filter Convert Numeric HTML/XML Entities to text.

My input file contains 31,286 NCRs that are to be converted.
These are scattered through out a UTF-8 text file that has 36,888 lines and a total of 7,219,971 characters.

The vast majority of the NCRs are converted correctly.

However, there were 5 locations where the conversion was incorrect.

I can only conclude that there's a subtle software bug in TextPipe Standard 10.7.2.

The problem is in understanding the root cause, because the same NCRs are converted correctly in other locations in the file.

The errors were detected by comparing the output file with one obtained by using BabelPad version 1.0.0.4 to Convert NCRs to Unicode.

This issue is critical, seeing as the TextPipe conversion is producing these rare but wrong results.

The attached .diff file was generated using WinMerge. Not ideal, but it does provide the context.

NB. I can readily send you a copy of the input file by email so that you might investigate further in detail.

Best regards,

David
Attachments
Reina1569.Bible.UTF-8.diff.zip
Zip contains .diff generated by WinMerge 0.86
(950 Bytes) Downloaded 107 times

DFH
Posts: 871
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Subtle bug in Convert Numeric HTML/XML Entities to text

Post by DFH » Mon Dec 10, 2018 4:44 am

Since posting the issue, I have found a workaround.

If the filter Convert Numeric HTML/XML Entities to text is plac1ed under a Restrict to each line in turn filter, the output file is without error.

I thereby conclude that the bug must have some relation to data caching or memory management.

David

DFH
Posts: 871
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Subtle bug in Convert Numeric HTML/XML Entities to text

Post by DFH » Tue Feb 05, 2019 10:29 pm

A workaround is one thing, but were you able to confirm whether or not my conjecture about data caching was valid?

If so, is there anything that can be done to prevent the bad output that I observed without such a workaround?

David

User avatar
DataMystic Support
Site Admin
Posts: 2328
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Subtle bug in Convert Numeric HTML/XML Entities to text

Post by DataMystic Support » Mon May 06, 2019 10:51 am

Hi David - can you please send the original file and the TextPipe filter?

The filter is a simple byte-by-byte state machine, expecting

&#<decimal digits>; or
&#x<hex digits>; or

It handles 2-byte unicode characters properly.

I don't understand why *any* changes are being made to your file as the non-&# data should pass straight through.

Simon
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

DFH
Posts: 871
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Subtle bug in Convert Numeric HTML/XML Entities to text

Post by DFH » Tue May 07, 2019 6:09 pm

Email sent just.

David

User avatar
DataMystic Support
Site Admin
Posts: 2328
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Subtle bug in Convert Numeric HTML/XML Entities to text

Post by DataMystic Support » Tue May 21, 2019 6:22 am

Hi David,

This definitely is an issue with TextPipe 10.7.2.

But testing with TextPipe 10.8, the issue is no longer there.
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

DFH
Posts: 871
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Subtle bug in Convert Numeric HTML/XML Entities to text

Post by DFH » Wed May 22, 2019 12:03 am

That's good news!

I await TextPipe 10.8 with bated breath. :)

David

Post Reply