I have log files that have multiple blank lines before and after all text lines. These documents have multiple lines for same information. I would like to remove duplicate lines, I tried to run this script removeLinesWithDuplicates.js from your script list. But this script did not do anything. Then I created new document and duplicated some lines. And ran this script but it did not remove duplicate lines. Can you please advise what I am doing wrong here?
There is no script needed to delete successive duplicate lines.
A Perl regular expression Replace All using a back-reference in search string executed from top of file can be used for this task.
The search string is ^(.*(:?\r?\n|\r))\K\1+ and the replace string is an empty string.
Each line with 0 or more characters ending with CR+LF, just LF or just CR is first marked (copied into memory), then the start location is reset by \K which means the matched and selected line is no longer selected, and next it is searched if the marked line (in memory) is found again from current position in file one or more times for a positive match resulting in execution of the replace deleting the selected successive duplicate lines.
The last line in the file must have also a line ending as it is ignored otherwise by the search expression.
A Perl regular expression Replace All using a back-reference in search string executed from top of file can be used for this task.
The search string is ^(.*(:?\r?\n|\r))\K\1+ and the replace string is an empty string.
Each line with 0 or more characters ending with CR+LF, just LF or just CR is first marked (copied into memory), then the start location is reset by \K which means the matched and selected line is no longer selected, and next it is searched if the marked line (in memory) is found again from current position in file one or more times for a positive match resulting in execution of the replace deleting the selected successive duplicate lines.
The last line in the file must have also a line ending as it is ignored otherwise by the search expression.
Best regards from an UC/UE/UES for Windows user from Austria
