Delete duplicate lines in a delimeted file

Get help with installation and running here.

Moderators: DataMystic Support, Moderators

leorr11
Posts: 2
Joined: Sun May 15, 2016 5:15 am

Delete duplicate lines in a delimeted file

Postby leorr11 » Sun May 15, 2016 5:29 am

Hi all,

I have a files with pipe delimited fields.
I`d like to delete lines which are duplicated in one field only.

I`ve been trying in several ways without result.

It seems that "delete duplicate lines" and "count duplicate lines" don`t work as a subfilter of "restrict fields"

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Delete duplicate lines in a delimeted file

Postby DataMystic Support » Mon May 16, 2016 5:10 pm

No, as each text fragment passed to Delete duplicate lines is considered a new 'file'.

Are the lines with duplicate fields next to each other? If so, use an EasyPattern like this to identify them:

Code: Select all

[ capture( pipefield), pipe, capture( pipefield), pipe, capture( pipefield), pipe, cr, lf
  group1, pipe ]


This will find lines where field1 is repeated on the next line.

This next EasyPattern finds lines where field 3 is repeated on the next line:

Code: Select all

[ capture( pipefield), pipe, capture( pipefield), pipe, capture( pipefield), pipe, cr, lf
  capture( pipefield), pipe, capture( pipefield), pipe, group3, pipe ]
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

leorr11
Posts: 2
Joined: Sun May 15, 2016 5:15 am

Re: Delete duplicate lines in a delimeted file

Postby leorr11 » Tue May 17, 2016 6:23 pm

thanks, it works

JamesB
Posts: 1
Joined: Sun Aug 21, 2016 8:31 pm

Re: Delete duplicate lines in a delimeted file

Postby JamesB » Mon Aug 22, 2016 3:46 pm

I have a similar question, is there a way to delete duplicate lines but leave the original line? When I do this it seems to remove even the original.

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Delete duplicate lines in a delimeted file

Postby DataMystic Support » Mon Aug 22, 2016 4:20 pm

Which version of TP do you have?

Is Filter Library\Remove\Duplicate lines your only filter?
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments


Return to “TextPipe Tips and Tricks, Questions and Support”

Who is online

Users browsing this forum: No registered users and 5 guests