conversion from unknown encoding to utf-8

Get help with installation and running here.

Moderators: DataMystic Support, Moderators

Richar007
Posts: 2
Joined: Wed Feb 03, 2010 1:45 am

conversion from unknown encoding to utf-8

Postby Richar007 » Wed Feb 03, 2010 1:59 am

Is it possible to create scripting using textpipe to convert from unknown encoding to utf-8?
I'm trying to process batches of .sql files (of unknown encoding) which are then concatenated to be one big sql file.
So far, all my investigation suggests the input files need to be the same encoding and I need to use that as a parameter for the conversion - can anyone do better?!
Thanks.

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: conversion from unknown encoding to utf-8

Postby DataMystic Support » Wed Feb 03, 2010 7:36 am

If the input files are only utf-16, utf-32 and utf-8, then you can use Filters\Restrict\Restrict to Unicode UTF16 files and
Filters\Restrict\Restrict to Unicode UTF32 files
to force conversion to utf-8.

Aside from that, there are heuristic approaches to detecting an file encoding, but as yet not included in TextPipe.
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

Richar007
Posts: 2
Joined: Wed Feb 03, 2010 1:45 am

Re: conversion from unknown encoding to utf-8

Postby Richar007 » Mon Feb 15, 2010 9:08 pm

I've managed to find a new tool that will process files with any encoding and change all files into UTF-8, and I've got a textpipe filter working to join these into one long file with various headers and footers, but I'd like to run it at the command line and I cannot find the command line command for "Remove BOM (Byte Order Mark)" - am I being daft or does it not exist?

Thanks in advance for any help.

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: conversion from unknown encoding to utf-8

Postby DataMystic Support » Tue Feb 16, 2010 8:11 am

Quite a few filters cannot be added from the command line - but you can save it to a .fll filter file, and then add filters together on the command line.

Which program did you use to convert files?
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments


Return to “TextPipe Tips and Tricks, Questions and Support”

Who is online

Users browsing this forum: No registered users and 1 guest