Search/Replace List memory issue

Get help with installation and running here.

Moderators: DataMystic Support, Moderators

alnico
Posts: 74
Joined: Fri Oct 12, 2007 11:57 pm

Search/Replace List memory issue

Postby alnico » Mon Apr 11, 2011 10:58 am

If a run a Search/Replace List of 10,000 lines (3.5Mb) against 3 (100 line 45kb) input files...problems occur...

Immediately TextPipe will hang and the header bar may or may not say "Not Responding".
After about 4 minutes TextPipe will come back to life and throw errors:

"Out of memory"
"Not enough storage available to process this command"

I always get the first error but not always the second one.

I have tested this on XP and Win7(64bit) with the same results.

I have never run a search replace list this big before...but am planning on it being much larger (50x larger).

Is something buggy here or are there limitations that I need to be aware of?

Running TextPipe 8.8

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Search/Replace List memory issue

Postby DataMystic Support » Tue Apr 12, 2011 8:26 am

We did a lot of performance work on that filter for one user - so it should be fine, however, it may need to check for user input periodically.

Coulld you email me sample data?
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Search/Replace List memory issue

Postby DataMystic Support » Tue Apr 12, 2011 4:51 pm

We can fix the unresponsiveness easily, but I need to get an idea of what you are doing.

Sample files will be really helpful - especially to see what kind of search/replace you are doing, if you are sorting the list (ie longest first) etc.
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

alnico
Posts: 74
Joined: Fri Oct 12, 2007 11:57 pm

Re: Search/Replace List memory issue

Postby alnico » Wed Apr 13, 2011 1:50 am

Attached is a filter with three input files and a tab deliminated Replace list.

If a smaller Replace list is used...it works fine; otherwise TextPipe throws errors or simply stops.

However, I have since figured out a more efficient way of getting what I need here...so fixing this is not an issue; but maybe you will still find it useful 'IF' there is indeed a bug...(maybe it is simply to large for my system resources).

Thanks,

Brent

===UPDATE===============================

I had to remove about 10,000 lines from the Replace List tab file (included in the zip archive)...as the file upload is limited to 256kb.
I just ran the filter...and to my surprise it does work with a 70,000 Replace List...so initially I guess I went over the limit ;-)
Last edited by alnico on Thu Apr 14, 2011 11:43 pm, edited 1 time in total.

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Search/Replace List memory issue

Postby DataMystic Support » Wed Apr 13, 2011 1:19 pm

Thanks Brent,

70,000 replaces is a lot, but if you check the simultaneous replace it will combine them into a much smaller number of patterns which uses less memory.

Could you email me the original data (not upload?) The forum limit is designed for filters rather than sample data.
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

alnico
Posts: 74
Joined: Fri Oct 12, 2007 11:57 pm

Re: Search/Replace List memory issue

Postby alnico » Thu Apr 14, 2011 11:50 pm

Simon, I emailed you the original data to test.

Simultaneous replace does not work as they contain patterns; not 'exact' matches.

But again, I figured out a more efficient solution...

Thanks,

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Search/Replace List memory issue

Postby DataMystic Support » Thu Apr 21, 2011 1:58 pm

You need to check the 'Simultaneous Search' option - otherwise it creates one filter for every single search/replace row.
It only took 5 seconds on my machine for > 65535 search/replaces in your file.
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments


Return to “TextPipe Tips and Tricks, Questions and Support”

Who is online

Users browsing this forum: No registered users and 2 guests