Convert Mainframe EBCDIC files to PC format - TextPipe Pro
There are 5 main types of Mainframe EBCDIC copybook that TextPipe will help you with:
- Simple EBCDIC files with no structure. These can be converted using the Mainframe \ EBCDIC to ASCII filter.
- Fixed length mainframe files. Use the Mainframe \ EBCDIC to ASCII filter followed by the Convert \ End of Line Characters filter. For 132 column mainframe reports, set the fixed length to 133
- Mainframe CMS Format (variable line length files). Use the Convert \ End of Line Characters filter.
- Fixed record size (ie one segment) with a copybook (e.g. with packed and zoned decimals). Use the Mainframe Copybook filter and simply paste in your copybook.
- Multi-record (multiple segments) format. Use the structure below.
When a Mainframe EBCDIC copybook consists of multiple records, you need to separate them out into a general structure, with each record type going to a new file for clarity and ease of loading/processing. TextPipe 11 makes this easy.
|
Sample Multi-Segment Mainframe Cooybook
In our sample copybook below, each record starts with a 3 byte field (T-TRAA-LEN), followed by a 5 letter record type id, T-REC_ID, which has values of #REC#, #TRAA, #TRAD etc. There are other fields beyond this that are unique to this segment. Note: Often, the record type field's VALUES are not included in the copybook, but has to be read from the comments or documentation. In these cases the record type field usually has a constant name like REC-ID.
05 T-REC.
10 T-REC-LEN PIC S9(4) COMP VALUE 68.
10 T-REC-ID PIC X(5) VALUE '#REC#'.
10 T-REC-STAT PIC X VALUE 'N'.
...
05 T-TRAA.
10 T-TRAA-LEN PIC S9(4) COMP VALUE 69.
10 T-REC-ID PIC X(5) VALUE '#TRAA'.
10 T-TRAA-STAT PIC X VALUE 'N'.
...
05 T-TRAC.
10 T-TRAC-LEN PIC S9(4) COMP VALUE 68.
10 T-REC-ID PIC X(5) VALUE '#TRAC'.
10 T-TRAC-STAT PIC X VALUE 'N'.
...
05 T-TRAD.
10 T-TRAD-LEN PIC S9(4) COMP VALUE 199.
10 T-REC-ID PIC X(5) VALUE '#TRAD'.
10 T-TRAD-STAT PIC X VALUE 'N'.
...
Multi-segment Mainframe EBCDIC copybook conversion
With TextPipe 11+ (download), the steps to convert a multi-segment cobol file are:
- Open TextPipe, and drag and drop your binary file that you want to convert (this file is in EBCDIC, not ASCII)
- From the Filters to Apply tab, Add a Filter Library\Mainframe\Mainframe copybook master record, and set the general options such as the output format (CSV, Tab, Pipe, XML, ;-delimited, JSON, SQL Insert script)
- Set the name of the field that contains the Record Type e.g. T-REC-ID, RECORD-TYPE. This field in each segment tells TextPipe which segment type is being processed
- Set the base output filename - this is the base folder and file that will be used for each segment. The name of the file will be base output filename plus the Segment Type
- Add one or more Filter Library\Mainframe\Mainframe copybook child record as subfilters (ie inside) the Mainframe copybook master record. You can drag child filters inside the master filter. These child filters inherit settings from their parent.
- In each child record), set the child record type (e.g. #REC#, #TRAA, #TRAD, A, B, C, 01, 02, 03) and paste in the associated copybook. If TextPipe complains about the format of your copybook, please contact support.
Here is how it looks in TextPipe:
When a Mainframe EBCDIC copybook consists of multiple records, you need to separate them out into a general structure, with each record type going to a new file for clarity and ease of loading/processing. TextPipe makes this easy.
|
|
TextPipe v10 and earlier - Mainframe EBCDIC copybook filter design pattern
With TextPipe (download), the steps are
- Use an EasyPattern match to identify each record type we have in the file – using the record header
- Pass this record header with its associated data to a subfilter
- The subfilter handles just that record type, and puts each record type in a new file.
In our sample file, each record starts with a 3 byte field which we will ignore, followed by a 5 letter record type id, #REC#, #TRAA, #TRAD etc. There is also a field following this record id, but we don't need it, and the value might not always be 'N'. Note: Often, the record type field's VALUES are not included in the copybook, but has to be read from the comments or documentation. In these cases the record type field usually has a constant name like REC-ID.
Sample Mainframe Cooybook
05 T-REC.
10 T-REC-LEN PIC S9(4) COMP VALUE 68.
10 T-REC-ID PIC X(5) VALUE '#REC#'.
10 T-REC-STAT PIC X VALUE 'N'.
...
05 T-TRAA.
10 T-TRAA-LEN PIC S9(4) COMP VALUE 69.
10 T-TRAA-ID PIC X(5) VALUE '#TRAA'.
10 T-TRAA-STAT PIC X VALUE 'N'.
...
05 T-TRAC.
10 T-TRAC-LEN PIC S9(4) COMP VALUE 68.
10 T-TRAC-ID PIC X(5) VALUE '#TRAC'.
10 T-TRAC-STAT PIC X VALUE 'N'.
...
05 T-TRAD.
10 T-TRAD-LEN PIC S9(4) COMP VALUE 199.
10 T-TRAD-ID PIC X(5) VALUE '#TRAD'.
10 T-TRAD-STAT PIC X VALUE 'N'.
...
Records
The general TextPipe pattern for each record type within the file looks like this:
Let's dig into this.
The first step is an EasyPattern (a user-friendly type of pattern match). It looks for
- The start of the text, followed by
- Any 3 characters (the PIC S9(4) COMP field we are ignoring), followed by
- the EBCDIC characters #TRTS. (Remembering that PC's use ASCII, we need to specify the literal is EBCDIC), followed by
- 65 other characters – the rest of the record
When something matches this pattern, it must be a record of type #TRTS, and it gets passed unchanged to the subfilter:
- The Mainframe Copybook filter applies just the copybook segment that relates to the #TRTS record
- The result goes to the file TRTS-records.txt on your desktop
- The text is then prevented from continuing in the filter – it stops dead here.
Pro Tip: We can easily copy this structure by selecting the top filter, and choosing the [x2] button in the footer.
Working out the copybook size
When we paste each copybook fragment into a mainframe copybook filter, we can click the [Show Parse Tree] button to find out the record length. e.g.
-01 COPYBOOK 1-73 Length:73
05 T-TRAA 1-73 Length:73
10 T-TRAA-LEN S9(4) COMP-3. 1-3 Length:3
10 T-TRAA-ID X(5). 4-8 Length:5
10 T-TRAA-STAT X(1). 9 Length:1
10 T-TRAA-COMPL X(1). 10 Length:1
10 T-TRAA-COM S9(4) COMP-3. 11-13 Length:3
10 T-TRAA-DATA 14-73 Length:60
…
Once we know the length is 73, we just subtract 8 from it, (the first 3 characters followed by the 5 character record type = 8) to find out the remaining characters we need, in this case, 65:
[ TextStart, 3 chars, EBCDIC('#TRAA'),
65 chars ]
We also have to grab the record type TRAA from the copybook and put it into the pattern match, and also into the output filename.
%USERPROFILE%\Desktop\TRAA-Records.txt
Then we move onto the next record type.
Master record match
Once we have setup all the record types, we need to gather ALL the record patterns into one MASTER pattern match at the top of the list.
The master copybook pattern match looks like this:
[ TextStart,
3 chars,
EBCDIC('#'),
(EBCDIC('TRTS'), 65 chars)
or
(EBCDIC('TRAA'), 65 chars)
or
..one line for each record type..
(EBCDIC('TRTS'), 29 chars) ]
Ready to Rock 'n' Roll!
Then you're ready to drag and drop your mainframe file into TextPipe and click [Go] or press [F9]!
Errors?
If the error log reports that there were extra bytes in the file that could not be matched, this indicates that
- the pattern matches are incorrect
- the copybooks are incorrect. If you are converting a single record type, check that the file type is an exact multiple of the record size from the [Show Parse Tree] option. If you know you have just a single record type, you can use a website to get the prime factors of the file size to work out a likely copybook size.
- the settings used by the mainframe copybook filter are wrong. Try experimenting with the settings 'Assume adjacent COMP fields are overlapped' and 'Allow COMP-3 to be unsigned' of the most common or initial record type. Unfortunately the copybook does not tell us how exactly the data has been stored.
- the original data or original copybook is wrong
- the original data might have been converted to ASCII when it was transferred off the mainframe, and this is the most common problem we see. This leads to corrupted packed decimal values, so always ensure that the file you receive is FTPed off the mainframe in binary mode.
May the force be with you!
Automation
Once tested, the TextPipe filter can be run automatically or on a schedule from the command line. It can also be controlled by other software using COM.
Need Help With Your Copybook?
Please drop us an email with your copybook support2@datamystic.com
|