How to remove successive duplicate lines?

How to remove successive duplicate lines?

2
NewbieNewbie
2

    Jan 24, 2018#1

    I have log files that have multiple blank lines before and after all text lines. These documents have multiple lines for same information. I would like to remove duplicate lines, I tried to run this script  removeLinesWithDuplicates.js from your script list. But this script did not do anything. Then I created new document and duplicated some lines. And ran this script but it did not remove duplicate lines. Can you please advise what I am doing wrong here?

    6,648558
    Grand MasterGrand Master
    6,648558

      Jan 24, 2018#2

      There is no script needed to delete successive duplicate lines.

      A Perl regular expression Replace All using a back-reference in search string executed from top of file can be used for this task.

      The search string is ^(.*(:?\r?\n|\r))\K\1+ and the replace string is an empty string.

      Each line with 0 or more characters ending with CR+LF, just LF or just CR is first marked (copied into memory), then the start location is reset by \K which means the matched and selected line is no longer selected, and next it is searched if the marked line (in memory) is found again from current position in file one or more times for a positive match resulting in execution of the replace deleting the selected successive duplicate lines.

      The last line in the file must have also a line ending as it is ignored otherwise by the search expression.
      Best regards from an UC/UE/UES for Windows user from Austria

      2
      NewbieNewbie
      2

        Jan 24, 2018#3

        Thanks for your quick response. It worked perfectly.