Out of memory error using mainframe copybook as a subfilter

Get help with installation and running here.

Moderators: DataMystic Support, Moderators

novakevin
Posts: 7
Joined: Wed Nov 30, 2011 8:54 am

Out of memory error using mainframe copybook as a subfilter

Postby novakevin » Wed Nov 30, 2011 9:31 am

I have a number of mainframe files that I want to convert in a single batch using the AWESOME "Mainframe Copybook Filter" in TextPipe Pro. The files have a bunch of different formats, but I can easily determine the format from the filename so I can use the right copybook filter. My filter list look like this (I have more than 3 filetypes, but you get the idea):

* Restrict to filenames matching [filename pattern A]
* -- Mainframe Copybook Filter [copybook A]
* Restrict to filenames matching [filename pattern B]
* -- Mainframe Copybook Filter [copybook B]
* Restrict to filenames matching [filename pattern C]
* -- Mainframe Copybook Filter [copybook C]

This works exactly as expected for small files (less than ~1 GB). However, for larger files, I get a "out of memory" from TextPipe at or near the end of the Mainframe copybook filter, even though windows performance monitor shows that I still have around 2 GBs of physical RAM free (the memory usage does slowly increase though as TPP processes the file). If I create a filter list that contains just a single Mainframe Copybook filter (NOT as a subfilter), I can process multi-GB files with no problem, and the memory usage stays virtually constant the entire time. I'm running Windows 7.

I don't understand why TPP is running out of error in this situation. Is there another/better way to accomplish this? I have tried both TPP 8.9.6 and 8.9.9 and the "out of memory" error occurs in both versions. Thanks!

Kevin

User avatar
DataMystic Support
Site Admin
Posts: 2164
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Out of memory error using mainframe copybook as a subfil

Postby DataMystic Support » Thu Dec 01, 2011 12:41 pm

Hi Kevin,'

I suspect that these filters store the entire file before passing it to the subfilters. I will check and get back to you - these can be rewritten if required.
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

novakevin
Posts: 7
Joined: Wed Nov 30, 2011 8:54 am

Re: Out of memory error using mainframe copybook as a subfil

Postby novakevin » Sat Dec 03, 2011 8:50 am

Thanks Simon; let me know what you find.

Kevin

User avatar
DataMystic Support
Site Admin
Posts: 2164
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Out of memory error using mainframe copybook as a subfil

Postby DataMystic Support » Mon Jan 23, 2012 10:12 am

Hi Kevin,

We have changed this ready for the v9 release, due out shortly - it will now work without file size limitations. We tested with a 7GB file, restricted to a filename pattern, and TextPipe's memory stayed constant at 18,484 MB.
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments


Return to “TextPipe Tips and Tricks, Questions and Support”

Who is online

Users browsing this forum: No registered users and 10 guests