Delete fast all lines containing string

Delete fast all lines containing string

5
NewbieNewbie
5

    Mar 19, 2010#1

    Hi all:

    I'm new to the forums and new to UltraEdit. I'm looking for a way to delete lines that contain a string. I've seen plenty of discussion on this, and I've created a script to do this based on the tutorial here: http://goo.gl/JpUK .

    Problem is, it's incredibly slow. My goal is to be able to cut out repetitive lines from log files to get down to the important stuff, and in a large log, it takes way too long to run the looping tutorial script to remove all lines.

    In vim, I just use :g/<pattern>/d and this is far, far faster than running the UltraEdit script. It doesn't seem like such a simple thing should take quite so long.

    Is there a more efficient script out there somewhere that I could take a look at? Or am I overlooking something incredibly simple?

    Thanks in advance,
    Brandon

    6,603548
    Grand MasterGrand Master
    6,603548

      Mar 19, 2010#2

      What about using a regular expression replace all executed from top of the file.

      For example using the Perl regexp engine search for ^.*your string.*\r*\n and the replace string is just an empty string.
      Best regards from an UC/UE/UES for Windows user from Austria

      5
      NewbieNewbie
      5

        Mar 19, 2010#3

        Beautiful. Exactly what I was looking for. Did not think to use a regEx to match a line w/ string... Duh. I was just leaving blank spaces in all my lines that had the search string.

        Thanks for the help. I modified the tutorial script I referenced in my first post to look like this, and it's much faster.

        It works, but does it look right to everyone? I am overlooking a shortcut?

        Code: Select all

        // Start at the beginning of the file
        UltraEdit.activeDocument.top();
        
        // Search string variable used for deletion of lines
        var delString = UltraEdit.getString("Delete lines containing what?",1);
        
        // Turn on regular expressions to find entire line.
        UltraEdit.activeDocument.findReplace.regExp=true;
        UltraEdit.activeDocument.findReplace.replaceAll=true;
        UltraEdit.activeDocument.findReplace.find("^.*" + delString + ".*\r*\n");
        
        while (UltraEdit.activeDocument.isFound()) {
           UltraEdit.activeDocument.findReplace.replace("^.*" + delString + ".*\r*\n","");
        }
        So, it works... But still not as fast as I'd like... Shouldn't I be able to use a script to delete all lines containing a string as fast as I can.. say... list all lines containing those strings?

        The while loop is the delay culprit here, but I couldn't get the replace all to work because it leaves too much stuff without looping to check-- When it deletes line 8, for example, line 9 becomes line 8, and it starts replacing again on the next line... So if I have a lot of instances of a string in sequential lines, it will skip every other one.

        Any ideas?

        Thank you.

        6,603548
        Grand MasterGrand Master
        6,603548

          Mar 21, 2010#4

          The following script deletes all lines containing an entered string.

          Code: Select all

          if (UltraEdit.document.length) {
          
             UltraEdit.insertMode();
             if (typeof(UltraEdit.columnModeOff) == "function") UltraEdit.columnModeOff();
             else if (typeof(UltraEdit.activeDocument.columnModeOff) == "function") UltraEdit.activeDocument.columnModeOff();
             UltraEdit.perlReOn();
          
             // Start at the beginning of the file
             UltraEdit.activeDocument.top();
          
             // Get string variable used for deletion of lines.
             var sUserString = UltraEdit.getString("Delete lines containing what?",1);
             // Escape every Perl regular expression character of the string with
             // a backslash to make sure that none of the characters in the entered
             // string is interpreted as regular expression character.
             var sDelString = sUserString.replace(/([$()*+.?\[\]\\{|}\^])/g,"\\$1");
          
             // Delete all lines containing the entered string using a Perl regexp replace.
             UltraEdit.activeDocument.findReplace.mode=0;
             UltraEdit.activeDocument.findReplace.matchCase=false;
             UltraEdit.activeDocument.findReplace.matchWord=false;
             UltraEdit.activeDocument.findReplace.regExp=true;
             UltraEdit.activeDocument.findReplace.searchDown=true;
             UltraEdit.activeDocument.findReplace.searchInColumn=false;
             UltraEdit.activeDocument.findReplace.preserveCase=false;
             UltraEdit.activeDocument.findReplace.replaceAll=true;
             UltraEdit.activeDocument.findReplace.replaceInAllOpen=false;
             var sSearchString = "^.*" + sDelString + ".*(?:\\r?\\n|\\r)";
             UltraEdit.activeDocument.findReplace.replace(sSearchString,"");
          }
          Interesting is that using var sSearchString = "^.*" + sDelString + ".*\r*\n"; results in not deleting all consecutive lines containing the string. When the JavaScript engine converts \r to 0x0D and \n to 0x0A and passes the string with these 2 bytes to the Perl engine, every second line of a block containing the search string is not deleted. But escaping \r and \n with an additional backslash to really pass to the Perl engine \r*\n works. I reported by email to IDM support the issue with not all lines deleted on first run when consecutive lines contain the searched string and just \r*\n is used instead of \\r*\\n.

          Update 1: Since UE v19.00.0.1022, first public version of UE 19.00, the search string "^.*" + sDelString + ".*\r*\n" works also for deleting consecutive lines. However, it is nevertheless better to use \\r and \\n in JavaScript strings which are search/replace strings for a Unix/Perl regular expression find/replace.

          Update 2: Even better is using var sSearchString = "^.*" + sDelString + ".*(?:\\r?\\n|\\r)"; as with this OR expression the script works for files with DOS or UNIX or MAC terminated lines.
          Best regards from an UC/UE/UES for Windows user from Austria

          1
          NewbieNewbie
          1

            May 24, 2010#5

            I've been using the included script for a while now, it's actually the example provided by UE website. Every since switching to the Linux version of UE, I've been unable to run it. I keep getting the following error message:

            Code: Select all

            An error occurred on line: 17
            Script failed.
            Could anyone provide insight as to why this is happening.

            Thanks in advance.
            Nick

            Code: Select all

            // Start at the beginning of the file
            UltraEdit.activeDocument.top();
            
            // Search string variable used for deletion of lines
            var delString = UltraEdit.getString("Delete lines containing what?",1);
            
            // Establish our search string for the loop condition
            UltraEdit.activeDocument.findReplace.find(delString);
            
            // Variable to count found items
            i = 0;
            
            // Start with nothing on the clipboard
            UltraEdit.clearClipboard();
            
                 // Loop through the file, count and delete found lines
                 while (UltraEdit.activeDocument.isFound()){  //<------- line 17 (failure)
                 UltraEdit.activeDocument.top();
                 UltraEdit.activeDocument.findReplace.find(delString);
                 
                 // We use an if statement here, because when the above "isFound" is evaluated,
                 // the script will use the last Find operation for re-evaluation instead of initiating
                 // a new one, causing one extra unintended deletion.
                 if (UltraEdit.activeDocument.isFound()) {
                      UltraEdit.activeDocument.selectLine();
                      UltraEdit.activeDocument.cutAppend();
                      i++;
                 }
            }
            
            if (i == 0) {
                 UltraEdit.messageBox("No occurrences of \"" + delString + "\" found in this file.\r\n No lines were deleted.");
            } else {
                 // Create a new file for the report
                 UltraEdit.newFile();
                 UltraEdit.activeDocument.paste();
                 UltraEdit.activeDocument.top();
                 UltraEdit.activeDocument.write(i + " lines containing \"" + delString + "\" were found and deleted in this file.\r\n\r\n");
            }
            

            6,603548
            Grand MasterGrand Master
            6,603548

              May 24, 2010#6

              I don't have UEX and therefore can't give you a verified answer. What happens when you change the while loop to following?

              Code: Select all

              while (1) {
                   UltraEdit.activeDocument.top();
                   UltraEdit.activeDocument.findReplace.find(delString);
                   
                   // We use an if statement here, because when the above "isFound" is evaluated,
                   // the script will use the last Find operation for re-evaluation instead of initiating
                   // a new one, causing one extra unintended deletion.
                   if (UltraEdit.activeDocument.isFound()) {
                        UltraEdit.activeDocument.selectLine();
                        UltraEdit.activeDocument.cutAppend();
                        i++;
                   }
                   else break;
              }
              If the error is now on line 24, the command isFound() is not available in UEX. As workaround following script could be used which is surely much faster than your version.

              Code: Select all

              // Start at the beginning of the file
              UltraEdit.activeDocument.top();
              
              // Search string variable used for deletion of lines
              var delString = UltraEdit.getString("Delete lines containing what?",1);
              
              // Variable to count found items
              i = 0;
              
              // Start with nothing on the clipboard
              UltraEdit.clearClipboard();
              
              // Loop through the file, count and delete found lines
              while( UltraEdit.activeDocument.findReplace.find(delString) ) {
                   // We use an if statement here, because when the above "isFound" is evaluated,
                   // the script will use the last Find operation for re-evaluation instead of initiating
                   // a new one, causing one extra unintended deletion.
                   UltraEdit.activeDocument.selectLine();
                   UltraEdit.activeDocument.cutAppend();
                   i++;
              }
              
              if (i == 0) {
                   UltraEdit.messageBox("No occurrences of \"" + delString + "\" found in this file.\r\n No lines were deleted.");
              } else {
                   // Create a new file for the report
                   UltraEdit.newFile();
                   UltraEdit.activeDocument.paste();
                   UltraEdit.activeDocument.top();
                   UltraEdit.activeDocument.write(i + " lines containing \"" + delString + "\" were found and deleted in this file.\r\n\r\n");
              }
              Above script tested with UltraEdit for Windows.
              Best regards from an UC/UE/UES for Windows user from Austria