speeding up search and replace

Discuss DataPipe. Get help with installation and running it, notify us of bugs, request new features and enhancements.

Moderators: DataMystic Support, Moderators

Post Reply

speeding up search and replace

Post by nixonrn » Sat Jan 24, 2004 4:42 am

I'm trying to do a bunch of search and replaces on a dataset with nearly 1 million records in it and its taking datapipe forever to do it. In three days it only did 70,000 plus records. Is there a way to speed this up.

Posts: 18
Joined: Tue Sep 23, 2003 3:13 am


Post by jring » Fri Mar 12, 2004 4:56 pm

hmmm that's interesting. I ran a datapipe program for 3 days against a sql db and got 5 million rows with about 15 columns through.... I think the routine used a combination of 40 nested TPP filters... So your case sounds a bit perplexing....

I did learn a few tricks that may help:

1. Make sure the columns in your database table you're processing are not indexed. If they are, you're database will hate you for it.

2. Smart groupings with where clauses in sql statements make smaller more manageable views than good'ol "SELECT * from..." Since datapipe opens a cursor, you should only pull the records into memory that are going to be processed.

If you give me a little bit more to work with, I may be able to give a little bit more back.

Post Reply