How to delete all lines NOT containing specific word or string or expression?

How to delete all lines NOT containing specific word or string or expression?

9
NewbieNewbie
9

    Jan 14, 2005#1

    How to delete all lines not containing a specific word or a specific string or where a specific regular expression cannot find a string in the line?

    Unix style if possible.
    Thanks

    206
    MasterMaster
    206

      Jan 14, 2005#2

      This has been discussed before, see Find one word and other not present in the line.

      9
      NewbieNewbie
      9

        Jan 16, 2005#3

        Thanks

        12
        Basic UserBasic User
        12

          Jun 21, 2006#4

          And where is code for macro ?

          I need example delete all lines which do not contain [images]

          Code: Select all

          [images]http://...[/images]
          Reply: [quotes]bla bla[/quotes]
          [images]http://...[/images]
          [quotes]bla bla[/quotes]
          = in this example I need to remove line 2 and 4

          6,603548
          Grand MasterGrand Master
          6,603548

            Jun 21, 2006#5

            The second macro I have posted yesterday at Delete all lines not starting with a special phrase will also work for your example without modification.

            It will work also if [images] is not at start of the line with a little modification of the first macro posted at the referenced forum topic.

            Code: Select all

            InsertMode
            ColumnModeOff
            HexOff
            Clipboard 9
            ClearClipboard
            Bottom
            IfColNum 1
            Else
            "
            "
            EndIf
            Top
            Loop
            Find "[images]"
            IfFound
            SelectLine
            CopyAppend
            Else
            ExitLoop
            EndIf
            EndLoop
            SelectAll
            Paste
            Top
            ClearClipboard
            Clipboard 0

            12
            Basic UserBasic User
            12

              Jun 21, 2006#6

              Thanks :)

              6
              NewbieNewbie
              6

                Apr 21, 2009#7

                I know how to use the Find/Replace with %*string*^p to find or replace entire lines (replace with ""), but how would I go about removing every line that doesn't contain a certain string?

                I'm trying to parse through multiple chat logs and only keep the lines with a very particular system message.

                6,603548
                Grand MasterGrand Master
                6,603548

                  Apr 21, 2009#8

                  A macro solution is posted above.

                  A quick manual solution is to simply search for the string of interest and check the advanced search option List Lines Containing String before executing the search. Then press button Clipboard to copy the lines of interest to the clipboard and close the dialog. Open a new file with Ctrl+N or select everything in current file with Ctrl+A and paste the clipboard content with Ctrl+V.

                  You can also use Find In Files to get into an edit window or the output window all lines of interest of all your logs.

                  236
                  MasterMaster
                  236

                    Apr 21, 2009#9

                    A Perl regular expression (courtesy of https://www.regular-expressions.info/completelines.html) that does the job is:

                    ^(?:(?!%REGEX%).)*$\r\n

                    Substitute %REGEX% with the string you want to find.

                    Replace all with nothing.

                    6
                    NewbieNewbie
                    6

                      May 01, 2009#10

                      Mofi wrote:A quick manual solution is to simply search for the string of interest and check the advanced search option List Lines Containing String before executing the search. Then press button Clipboard to copy the lines of interest to the clipboard and close the dialog. Open a new file with Ctrl+N or select everything in current file with Ctrl+A and paste the clipboard content with Ctrl+V.
                      This worked perfect for what I needed. Thanks.
                      pietzcker wrote:A Perl regular expression (courtesy of https://www.regular-expressions.info/completelines.html) that does the job is:

                      ^(?:(?!%REGEX%).)*$\r\n

                      Replace all with nothing.
                      Also works. Two awesome ways to parse out the information I want. Thanks guys.

                      3
                      NewbieNewbie
                      3

                        Feb 27, 2011#11

                        I've been thinking about this problem for a couple of weeks now and the other day, the solution finally came to me!!! :D GrandMaster Mofi, your macro is the best and fastest solution if the number of lines containing the keyword, phrase or symbol is less than the number of lines that are missing the keyword, phrase or symbol. However, in most cases I've dealt with so far, unfortunately the lines containing the keyword, phrase or symbol far outnumber the lines that are missing the keyword, phrase or symbol.

                        That's the reason why I've been trying to figure out how to target and remove lines that do not have a colon. In my example, I had thousands of lines that had colons and less than 20 that were missing colons. But the big problem for me was that there were no specific symbols for me to search for.

                        For example, for the problem in this thread: Find one word and other not present in the line

                        If I was making a macro, I would include find and replace all td tags with rt class and then Find td to get the desired result, but I've been using a word list for input and there is no specific symbol or word that I can use so then one day, all of a sudden, I had the thought, why not make my own symbol by using column insert, and then use a find and replace RegExp to remove that particular symbol from the lines that do contain colons, then take and slightly modify the code that you posted above and cut and append all lines with that particular symbol!!!?  :wink:

                        So this is the macro I've created. It's designed to cut and append all lines that don't have a colon in them and when it's done running, just in case you wanted to use the data that was cut out, I left it on clipboard 9 and did NOT clear the clipboard so if you want to use that data, you simply paste and presto, all the words that were cut out in one neat list! ;) So I hope everyone interested in this subject, will feel free to try my macro and if you want to use it for something other than cutting out lines without colons, all you have to do is go to:

                        Find RegExp "½^(*^):"
                        Replace All "^1:"

                        and replace the colons with something else like the word "images" And it should be the fastest method for removing the lines that don't contain this word, provided that the number of lines that are missing this word is less than the number of lines containing the word! ;)

                        To be more specific, change it to:

                        Find RegExp "½^(*^)images"
                        Replace All "^1images"

                        I've tested this on my UE and it's working great!!! :D My version of UE is 14.10.0.1024

                        If this macro works for you, please let me know and even more importantly, if it does NOT work for you, please be sure to post a message in this thread. Thanks for listening!  :)

                        MM

                        Edit: Macro code replaced by a new version, see my next post. The example was also removed.

                        6,603548
                        Grand MasterGrand Master
                        6,603548

                          Feb 27, 2011#12

                          The best method is the usage of the Perl regular expression replace all as posted by pietzcker to delete all lines not containing a colon.

                          InsertMode
                          ColumnModeOff
                          HexOff
                          PerlReOn
                          Bottom
                          IfColNumGt 1
                          InsertLine
                          IfColNumGt 1
                          DeleteToStartofLine
                          EndIf
                          EndIf
                          Top
                          Find MatchCase RegExp "^(?:(?!:).)*$(?:\r?\n|\r)?"
                          Replace All ""

                          Of course that macro does not record the deleted lines. But it is very easy to create a copy of the source file and replace in the copy all lines containing a colon to get the opposite result.

                          And users of UE v16.00 or later don't need a macro for that job at all. Simply running a non regular expression Find searching for : executed with advanced option button Show Lines (show just lines with a colon, hide all other), or Hide Lines (hide all lines with a colon, only show all other) and using Edit - Delete - Delete All Hidden Lines produce the same result without the need to use a macro which must be modified for every search, or using a Perl regular expression replace without knowing how it works.

                          Your macro could be improved a little bit, see below (removed later). But macro above is much faster.

                          3
                          NewbieNewbie
                          3

                            Mar 05, 2011#13

                            First of all, I apologize for taking so long to reply. :oops: Real life got very hectic all of a sudden and once again, you gave me quite a lot to think about, Mofi. Thank you very much for reposting pietzckers' Perl code for my convenience, and of course, thank you pietzcker for posting it originally. It's AMAZING!!!!!!!!!! :D :D :D I tested it on 10,000 lines and it was finished running in less than a second!!!!!!!!!!!!!!! 8O Mofi, I took your advice and hopefully the code that I posted in the 1st code box will accomplish what you suggested. It seems to be working on my end! ;)

                            It's lightning fast and it is superior to my original macro. However, my macro has one advantage to it. If you want to change your search from lines that don't contain colons to lines that don't contain something else like a @ symbol, all you have to do is change the code

                            Find RegExp "½^(*^):"
                            Replace All "^1:"

                            to

                            Find RegExp "½^(*^)@"
                            Replace All "^1@"

                            I tried that with the Perl Expression and got a bunch of errors and it was still trying to find lines without colons. :( That's why I have continued work on the original macro I made and posted it in the 2nd code box.

                            I thought about it and decided that my original macro would not work well on a doc that had a lot of blank lines so I added some code to remove the blank lines first. I also noticed that when I did modify the code to find lines not containing an @ symbol, after 5000 lines it seemed to miss some of those lines so I added some code that would perform the operation one more time in order to take care of the lines that were missed the first time.

                            I really appreciate you improving my macro! :D I've never used Perl expressions until now so in my opinion, your biggest improvement was adding the UnixReOff. After running pietzckers' Perl macro, a lot of the macro's I've made would not work anymore :oops: , so I guess I need to add that UnixReOff to all of my macros, just in case I or someone else was using something like Perl Expressions previously.

                            I also noticed you had a 0 after the word Loop. :? Why did you do that? I pasted your code in my UE and after I saved it, the 0 just dissappeared. I ran your version of the macro and worked well in my UE with one exception.

                            I know you're probably wondering why it was, everytime I used NewFile, I followed it with PreviousDocument and then NextDocument. Well the reason is, that little bit of extra code appears to fix the double click bug that my version of UE has. If I don't include it, I'm almost assured that I will have problems if I double click the macro. :( I was thrilled when I made that discovery a couple of weeks ago! :D

                            I also noticed you added ColumnModeOn over ColumnInsert. Could this be another reason why a lot of my macros work for me but not other people? I definitely want to make my macro's compatible with most versions of UE and all versions if possible so I appreciate the advice! :)

                            I'd have to say my most embarassing mistake was when I used:
                            Find RegExp "*½*^p"

                            I just remember thinking that part would be easy to code and then I ended up pulling my hair out trying to find the proper code for it and by the time I was finally finished, I was just thrilled that it was working and I wasn't about to mess with it anymore! :lol:

                            Well I think that's about it. I would very much appreciate it if you would test both macros, and let me know what you think. Thanks again for all your help, Mofi. I look forward to your reply! :D
                            MM


                            pietzckers' perl macro, slightly modified

                            Code: Select all

                            InsertMode
                            ColumnModeOff
                            HexOff
                            SelectAll
                            Copy
                            NewFile
                            PreviousDocument
                            NextDocument
                            Paste
                            EndSelect
                            UnixReOff
                            Top
                            Find RegExp "%
                            "
                            Replace All ""
                            Bottom
                            Key END
                            IfColNum 1
                            Key BACKSPACE
                            EndIf
                            Bottom
                            IfColNumGt 1
                            InsertLine
                            IfColNumGt 1
                            DeleteToStartofLine
                            EndIf
                            EndIf
                            Top
                            Find RegExp "%*:*
                            "
                            Replace All ""
                            SelectAll
                            Clipboard 8
                            Copy
                            PreviousDocument
                            PerlReOn
                            Bottom
                            IfColNumGt 1
                            InsertLine
                            IfColNumGt 1
                            DeleteToStartofLine
                            EndIf
                            EndIf
                            Top
                            Find RegExp "^(?:(?!:).)*$\r\n"
                            Replace All ""
                            NextDocument
                            CloseFile NoSave
                            My original macro, slightly modified

                            Code: Select all

                            InsertMode
                            ColumnModeOff
                            HexOff
                            UnixReOff
                            Top
                            Find RegExp "%
                            "
                            Replace All ""
                            Bottom
                            Key END
                            IfColNum 1
                            Key BACKSPACE
                            EndIf
                            Bottom
                            IfColNumGt 1
                            InsertLine
                            IfColNumGt 1
                            DeleteToStartofLine
                            EndIf
                            EndIf
                            Top
                            ColumnModeOn
                            ColumnInsert "½"
                            ColumnModeOff
                            Find RegExp "½^(*^):"
                            Replace All "^1:"
                            TrimTrailingSpaces
                            Top
                            Clipboard 9
                            ClearClipboard
                            Loop
                            Find RegExp "%½*^p"
                            IfFound
                            SelectLine
                            CutAppend
                            Else
                            ExitLoop
                            EndIf
                            EndLoop
                            NewFile
                            PreviousDocument
                            NextDocument
                            Paste
                            Top
                            Find "½"
                            Replace All ""
                            SelectAll
                            Cut
                            CloseFile NoSave
                            Else
                            Bottom
                            NextDocument
                            CloseFile NoSave
                            EndIf
                            Top
                            Clipboard 8
                            ClearClipboard
                            Loop
                            Find RegExp "%½*^p"
                            IfFound
                            SelectLine
                            CutAppend
                            Else
                            ExitLoop
                            EndIf
                            EndLoop
                            NewFile
                            PreviousDocument
                            NextDocument
                            Clipboard 8
                            Paste
                            Top
                            "
                            "
                            Top
                            Clipboard 9
                            Paste
                            Top
                            Find "½"
                            Replace All ""
                            SelectAll
                            Cut
                            CloseFile NoSave

                            6,603548
                            Grand MasterGrand Master
                            6,603548

                              Mar 05, 2011#14

                              M@Cr0MaN wrote:I tried that with the Perl Expression and got a bunch of errors and it was still trying to find lines without colons.
                              You have replaced obviously both colons in the Perl regular expression string by @ instead of just the second formatted bold red in my previous post. The first colon has a special meaning in combination with the question mark. ?: immediately after and opening round bracket means: Do not mark (tag) the string found by the expression inside the round brackets. If you have replaced that colon also by @, you have created an invalid Perl regular expression.

                              Find RegExp "^(?:(?!@).)*$(?:\r?\n\r)?"

                              works fine for deleting all DOS/Windows/UNIX/MAC terminated lines (including blank lines) not containing character @ even the last line of the file not containing @ and having no line termination at all.
                              M@Cr0MaN wrote:After running pietzckers' Perl macro, a lot of the macro's I've made would not work anymore.
                              That's the reason why all my macros in Mofis_Macro_Examples.uem contain at end (or above ExitMacro command) the comment lines.

                              Code: Select all

                              // Activate UnixReOn or PerlReOn (v12+ of UE) here at the end of the macro
                              // if you do not use UltraEdit style regular expressions by default - see
                              // search configuration. Macro command UnixReOff sets the regular expression
                              // option to UltraEdit style.
                              //  UnixReOn
                              //  PerlReOn
                              Users of newer versions of UltraEdit than you don't have to take care about setting regular expression engine on macro exit because UltraEdit remembers current active regular expression engine on start of a macro and re-selects this engine automatically on exit of a macro.
                              M@Cr0MaN wrote:I also noticed you had a 0 after the word Loop. Why did you do that?
                              From help of UltraEdit about macro command Loop: A value of 0 indicates Loop forever.

                              Your version of UltraEdit always omits loop count value zero on decoding of a macro into the macro editor dialog. Newer versions of UltraEdit always decode infinite loops with value zero. All versions of UltraEdit support infinite loops broken by the commands ExitLoop or ExitMacro either with value zero after command Loop or without any value on compilation a macro into its binary form.

                              There is no difference whether in compiled macro code nor on compilation of a macro source code in macro editor dialog to the binary macro code in any version of UltraEdit. It simply does not matter if the text version of a macro code contains Loop 0 or just Loop without a loop count value. Older versions of UltraEdit just decode the binary macro code of an infinite loop to just Loop while newer versions of UltraEdit decode the same binary macro code to Loop 0. But in the reverse direction for compilation of text source to binary code it does not make a difference in any version of UltraEdit.
                              M@Cr0MaN wrote:I also noticed you added ColumnModeOn over ColumnInsert
                              As you can see in the Column menu the menu item Insert/Fill Columns is disabled if Column Mode is not active as most other commands in the column menu. Therefore no special column command in a macro should be used without making sure that column mode is turned on during macro execution before using one of the special column commands.

                              From your macro with the Perl regular expression:

                              SelectAll
                              Copy
                              NewFile
                              PreviousDocument
                              NextDocument
                              Paste

                              EndSelect

                              EndSelect after pasting something into a new file is unnecessary. Pasting something always results in disabling the selection mode. It looks like you have recorded that macro. Well, you see here that a paste results in disabling the selection mode. Better would be to use the command EndSelect at top of the macro after the commands SelectAll and Copy before opening the new file. That would stop the selection mode in the right file where this mode was enabled by command SelectAll.

                              3
                              NewbieNewbie
                              3

                                Mar 07, 2011#15

                                Mofi wrote:You have obviously replaced both colons in the Perl regular expression string by @ instead of just the second formatted bold red in my previous post.
                                You're absolutely right, Mofi. That's exactly what I did! :oops:
                                Mofi wrote:Find RegExp "^(?:(?!@).)*$\r\n"
                                Right again this works GREAT! It will end up being a big time saver for quite a few of my slow macros! :D Many thanks for explaining this to me and giving me this sample! :) I feel kind of silly since I didn't continue to experiment and try just changing only one of the colons to @. Or actually I think I did try that but I changed the wrong one!
                                Mofi wrote:That's the reason why all my macros in Mofis_Macro_Examples.uem contain at end (or above ExitMacro command) the comment lines ...
                                Oh, that's good to know. Thanks for the info, Mofi! :)
                                Mofi wrote:From help of UltraEdit about macro command Loop: A value of 0 indicates Loop forever ...
                                Oh, I see. That's kind of a relief to know.
                                Mofi wrote:As you can see in the Column menu the menu item Insert/Fill Columns is disabled if Column Mode is not active ...
                                More good info! Thanks for that! :0 ) I guess it's easy to neglect stuff like this when it still works for you. But like I already mentioned, I want to make my macros compatible with all versions of UE, so I'll try to make sure I add column mode on to all the macros I have that use column insert, from now on! ;) I think I already did that with most of my macros, but when I noticed it didn't really make any difference for my version of UE, I guess I got lazy.:oops: I know one thing for certain, if you forget to use columnModeOff, it can really screw up your Macro! :lol: I've made that mistake plenty of times in the past! :( but then again I've learned a lot through trial and error! ;)
                                Mofi wrote:EndSelect after pasting something into a new file is unnecessary...
                                This is good to know for certain. I have been unsure about this till now because of the double click bug that I have. It would automatically select lines even if I had used the command EndSelect and I guess that made me a little paranoid about making sure there was no selection after the paste command. But now that I know for certain that this is unnecessary, I won't use that command after the paste command anymore. And I totally agree with you about where I should have used EndSelect! :D Thanks Mofi, for taking the time to explain all this to me. You're lessons are much appreciated! :D

                                MM