Delete fast all lines containing string

cephalon · Mar 19, 2010#12010-03-19T15:50+00:00

Hi all:

I'm new to the forums and new to UltraEdit. I'm looking for a way to delete lines that contain a string. I've seen plenty of discussion on this, and I've created a script to do this based on the tutorial here: http://goo.gl/JpUK .

Problem is, it's incredibly slow. My goal is to be able to cut out repetitive lines from log files to get down to the important stuff, and in a large log, it takes way too long to run the looping tutorial script to remove all lines.

In vim, I just use :g/<pattern>/d and this is far, far faster than running the UltraEdit script. It doesn't seem like such a simple thing should take quite so long.

Is there a more efficient script out there somewhere that I could take a look at? Or am I overlooking something incredibly simple?

Thanks in advance,
Brandon

Mofi · Mar 19, 2010#22010-03-19T17:08+00:00

What about using a regular expression replace all executed from top of the file.

For example using the Perl regexp engine search for ^.*your string.*\r*\n and the replace string is just an empty string.

cephalon · Mar 19, 2010#32010-03-19T17:12+00:00

Beautiful. Exactly what I was looking for. Did not think to use a regEx to match a line w/ string... Duh. I was just leaving blank spaces in all my lines that had the search string.

Thanks for the help. I modified the tutorial script I referenced in my first post to look like this, and it's much faster.

It works, but does it look right to everyone? I am overlooking a shortcut?

Code: Select all

// Start at the beginning of the file
UltraEdit.activeDocument.top();

// Search string variable used for deletion of lines
var delString = UltraEdit.getString("Delete lines containing what?",1);

// Turn on regular expressions to find entire line.
UltraEdit.activeDocument.findReplace.regExp=true;
UltraEdit.activeDocument.findReplace.replaceAll=true;
UltraEdit.activeDocument.findReplace.find("^.*" + delString + ".*\r*\n");

while (UltraEdit.activeDocument.isFound()) {
   UltraEdit.activeDocument.findReplace.replace("^.*" + delString + ".*\r*\n","");
}

So, it works... But still not as fast as I'd like... Shouldn't I be able to use a script to delete all lines containing a string as fast as I can.. say... list all lines containing those strings?

The while loop is the delay culprit here, but I couldn't get the replace all to work because it leaves too much stuff without looping to check-- When it deletes line 8, for example, line 9 becomes line 8, and it starts replacing again on the next line... So if I have a lot of instances of a string in sequential lines, it will skip every other one.

Any ideas?

Thank you.

Mofi · Mar 21, 2010#42010-03-21T15:02+00:00

The following script deletes all lines containing an entered string.

Code: Select all

if (UltraEdit.document.length) {

   UltraEdit.insertMode();
   if (typeof(UltraEdit.columnModeOff) == "function") UltraEdit.columnModeOff();
   else if (typeof(UltraEdit.activeDocument.columnModeOff) == "function") UltraEdit.activeDocument.columnModeOff();
   UltraEdit.perlReOn();

   // Start at the beginning of the file
   UltraEdit.activeDocument.top();

   // Get string variable used for deletion of lines.
   var sUserString = UltraEdit.getString("Delete lines containing what?",1);
   // Escape every Perl regular expression character of the string with
   // a backslash to make sure that none of the characters in the entered
   // string is interpreted as regular expression character.
   var sDelString = sUserString.replace(/([$()*+.?\[\]\\{|}\^])/g,"\\$1");

   // Delete all lines containing the entered string using a Perl regexp replace.
   UltraEdit.activeDocument.findReplace.mode=0;
   UltraEdit.activeDocument.findReplace.matchCase=false;
   UltraEdit.activeDocument.findReplace.matchWord=false;
   UltraEdit.activeDocument.findReplace.regExp=true;
   UltraEdit.activeDocument.findReplace.searchDown=true;
   UltraEdit.activeDocument.findReplace.searchInColumn=false;
   UltraEdit.activeDocument.findReplace.preserveCase=false;
   UltraEdit.activeDocument.findReplace.replaceAll=true;
   UltraEdit.activeDocument.findReplace.replaceInAllOpen=false;
   var sSearchString = "^.*" + sDelString + ".*(?:\\r?\\n|\\r)";
   UltraEdit.activeDocument.findReplace.replace(sSearchString,"");
}

Interesting is that using var sSearchString = "^.*" + sDelString + ".*\r*\n"; results in not deleting all consecutive lines containing the string. When the JavaScript engine converts \r to 0x0D and \n to 0x0A and passes the string with these 2 bytes to the Perl engine, every second line of a block containing the search string is not deleted. But escaping \r and \n with an additional backslash to really pass to the Perl engine \r*\n works. I reported by email to IDM support the issue with not all lines deleted on first run when consecutive lines contain the searched string and just \r*\n is used instead of \\r*\\n.

Update 1: Since UE v19.00.0.1022, first public version of UE 19.00, the search string "^.*" + sDelString + ".*\r*\n" works also for deleting consecutive lines. However, it is nevertheless better to use \\r and \\n in JavaScript strings which are search/replace strings for a Unix/Perl regular expression find/replace.

Update 2: Even better is using var sSearchString = "^.*" + sDelString + ".*(?:\\r?\\n|\\r)"; as with this OR expression the script works for files with DOS or UNIX or MAC terminated lines.

naaronne · May 24, 2010#52010-05-24T15:23+00:00

I've been using the included script for a while now, it's actually the example provided by UE website. Every since switching to the Linux version of UE, I've been unable to run it. I keep getting the following error message:

Code: Select all

An error occurred on line: 17
Script failed.

Could anyone provide insight as to why this is happening.

Thanks in advance.
Nick

Code: Select all

// Start at the beginning of the file
UltraEdit.activeDocument.top();

// Search string variable used for deletion of lines
var delString = UltraEdit.getString("Delete lines containing what?",1);

// Establish our search string for the loop condition
UltraEdit.activeDocument.findReplace.find(delString);

// Variable to count found items
i = 0;

// Start with nothing on the clipboard
UltraEdit.clearClipboard();

     // Loop through the file, count and delete found lines
     while (UltraEdit.activeDocument.isFound()){  //<------- line 17 (failure)
     UltraEdit.activeDocument.top();
     UltraEdit.activeDocument.findReplace.find(delString);
     
     // We use an if statement here, because when the above "isFound" is evaluated,
     // the script will use the last Find operation for re-evaluation instead of initiating
     // a new one, causing one extra unintended deletion.
     if (UltraEdit.activeDocument.isFound()) {
          UltraEdit.activeDocument.selectLine();
          UltraEdit.activeDocument.cutAppend();
          i++;
     }
}

if (i == 0) {
     UltraEdit.messageBox("No occurrences of \"" + delString + "\" found in this file.\r\n No lines were deleted.");
} else {
     // Create a new file for the report
     UltraEdit.newFile();
     UltraEdit.activeDocument.paste();
     UltraEdit.activeDocument.top();
     UltraEdit.activeDocument.write(i + " lines containing \"" + delString + "\" were found and deleted in this file.\r\n\r\n");
}

Mofi · May 24, 2010#62010-05-24T17:30+00:00

I don't have UEX and therefore can't give you a verified answer. What happens when you change the while loop to following?

Code: Select all

while (1) {
     UltraEdit.activeDocument.top();
     UltraEdit.activeDocument.findReplace.find(delString);
     
     // We use an if statement here, because when the above "isFound" is evaluated,
     // the script will use the last Find operation for re-evaluation instead of initiating
     // a new one, causing one extra unintended deletion.
     if (UltraEdit.activeDocument.isFound()) {
          UltraEdit.activeDocument.selectLine();
          UltraEdit.activeDocument.cutAppend();
          i++;
     }
     else break;
}

If the error is now on line 24, the command isFound() is not available in UEX. As workaround following script could be used which is surely much faster than your version.

Code: Select all

// Start at the beginning of the file
UltraEdit.activeDocument.top();

// Search string variable used for deletion of lines
var delString = UltraEdit.getString("Delete lines containing what?",1);

// Variable to count found items
i = 0;

// Start with nothing on the clipboard
UltraEdit.clearClipboard();

// Loop through the file, count and delete found lines
while( UltraEdit.activeDocument.findReplace.find(delString) ) {
     // We use an if statement here, because when the above "isFound" is evaluated,
     // the script will use the last Find operation for re-evaluation instead of initiating
     // a new one, causing one extra unintended deletion.
     UltraEdit.activeDocument.selectLine();
     UltraEdit.activeDocument.cutAppend();
     i++;
}

if (i == 0) {
     UltraEdit.messageBox("No occurrences of \"" + delString + "\" found in this file.\r\n No lines were deleted.");
} else {
     // Create a new file for the report
     UltraEdit.newFile();
     UltraEdit.activeDocument.paste();
     UltraEdit.activeDocument.top();
     UltraEdit.activeDocument.write(i + " lines containing \"" + delString + "\" were found and deleted in this file.\r\n\r\n");
}

Above script tested with UltraEdit for Windows.