Randomize lines is too slow! - worked out a faster way.

Get help with installation and running here.

Moderators: DataMystic Support, Moderators

Post Reply
Posts: 11
Joined: Sun Oct 09, 2005 8:15 pm

Randomize lines is too slow! - worked out a faster way.

Post by Moz » Sun Oct 09, 2005 8:21 pm


I've worked out a faster randomise now. I was trying to randomise millions of lines of numbers, but the "Randomize Lines" filter took 9 hours to process each file (and I have a few to go through).

My solution is:
Simply grab a load of "random" numbers from somewhere like random.org (two lists, one longer than the other)- add these in a "add repeating text side by side" filter to add a random number to the end of each line. As one of the lists is longer than the other you'll end up with pretty much random numbers all the way down the right hand column of your file.

Then copy these to the beginning of the line, and sort numerically whilst restricting the sort to the number of digits you have placed at the beginning.

After sorting numerically remove the digits from the beginn ing and end of each file and you end up with your lines randomised in a MUCH quicker way.

I ran this against 5 files, each with about 2 million lines in and it took 16 minutes to process - much less than the 9 hours per file when using the "randomize lines" filter.

It would be beneficial for TextPipe Pro to have a facility to generate a random digit from 0-9 - this could be used as an "insert special character" function.



User avatar
DataMystic Support
Site Admin
Posts: 2206
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia

Post by DataMystic Support » Mon Oct 10, 2005 10:32 am

Good idea Moz,

Look for this in the next release (out this week)! It will be the macro @randomdigit, which will return a single digit, so @randomdigit@randomdigit will return 2 random digits. These can be used in the Add header/footer/left/right margin filters, and insert column filters.

We'll also add @randomletter.


Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests