How big a file?

Get help with installation and running here.

Moderators: DataMystic Support, Moderators

sheridany
Posts: 39
Joined: Thu Nov 15, 2007 4:20 am

How big a file?

Postby sheridany » Wed Apr 27, 2011 2:07 pm

I have a file that has a 1MM rows (22 columns csv format )daily that needs to be cleaned extensively. How many rows can TP handle on a single cpu workstation or is that even an option. Whats the best way to utilize TP for a job this big.

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: How big a file?

Postby DataMystic Support » Thu Apr 28, 2011 12:21 pm

TP can handle billions of rows of CSV data. Just point it at the file with a list of filters.

What cleansing does it need?
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

sheridany
Posts: 39
Joined: Thu Nov 15, 2007 4:20 am

Re: How big a file?

Postby sheridany » Thu Apr 28, 2011 10:55 pm

The usual cleanup some search and replace remove blanks trim leading and trailing etc. The usual TP stuff. From a deployment standpoint and a ETL perspective we would like to load the clean data into a database after TP has processed the file. How might we do that?

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: How big a file?

Postby DataMystic Support » Fri Apr 29, 2011 10:14 am

I assume you want to trim blanks on each field in turn rather than with entire lines, so use Filters\Restrict\Delimited fields (CSV, Tab, Pipe, etc) to restrict to each field in turn, and inside this filter add the trim filters.

You will then need to modify the CSV to add Filters\Add\Left margin of

Code: Select all

insert into tablename () values (

and a Filters\Add\Right margin of

Code: Select all

);


Then add a Filters\Special\Database connection as the last step.
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments


Return to “TextPipe Tips and Tricks, Questions and Support”

Who is online

Users browsing this forum: No registered users and 1 guest