Search/Replace based on list

Search/Replace based on list

3
NewbieNewbie
3

    Nov 24, 2020#1

    Hi,
    I'm sure this is simple but I can't find the answer looking through the docs...
    I'm looking to use UltraEdit as a log file redaction tool, the search and replace strings are generally fixed for this so ideally I'd want to use a CSV file with two columns in it (original & replacement string) and for UltraEdit to search/replace the contents of a folder based on that. I only ever need to to a direct string replace (e.g. replace "actualhostname" with "server")
    I don't see using an external file as a documented option but I'm wondering if I could use the Favourites feature in a similar way and just build up a set search/replace list within the Favourites? If so where does that list get written to and is it editable/viewable outside of UltraEdit?
    If Favourites can be used in this way is there a limit to the number of entries that can be stored? We shouldn't need many - probably around 50 but a max of around 100 would be preferable to allow some head room.

    Thanks,
    Nick

    6,603548
    Grand MasterGrand Master
    6,603548

      Nov 24, 2020#2

      An UltraEdit macro or UltraEdit script is used for such a task in general. A macro could be created by simply recording the replaces done once manually. A script must be really coded.

      In general it is most efficient that the file to modify with a series of finds and replaces is not opened at all in UltraEdit, but command Replace in Files is used to modify this single file as in this case no undo steps are recorded, no line change indications, etc. which are all not needed in this case.

      I would suggest nowadays an UltraEdit script for such a task instead of an UltraEdit macro if the find/replace strings need to be changed or extended or removed from time to time. It is easier to edit a CSV file with find/replace strings than a macro in macro editor. It is also possible that a macro works with find/replace strings in a text file, but UltraEdit macros do not support variables which makes the process less efficient on execution in comparison to an UltraEdit script which can load all finds/replaces into memory at once and then run in a loop the replaces on the file.

      I wrote already several macros (many years ago) and scripts (in the last years) for that task which can be found in macros and scripts forums.

      Look for example on the scripts posted at:
      One of the older macro solutions is posted at How to Perform a Find & Replace via Macro using a source file.

      Please let me know if one of the existing script or macro solutions can be used by you for your task with small modifications by yourself or if I should write a new script for you. That would be no problem for me and according to what you posted already, it could write the script quite quickly. But I would need more information like the full qualified file name of the CSV file, the full qualified file name of log file to modify, which character is used as separator in the CSV file, is it necessary to handle also find/replace strings containing the separator, run the replaces case-sensitive or case-insensitive.
      Best regards from an UC/UE/UES for Windows user from Austria

      3
      NewbieNewbie
      3

        Nov 25, 2020#3

        Many thanks for the reply and the examples look pretty clear and I might be able to figure it out myself, that said if you did have the time to create a custom script that would be greatly appreciated.

        To answer your questions:
        • CSV name can be whatever really (probably something like D:\redaction\redactionlist.csv) but would be static so preferable if it's hardcoded rather than prompted for
        • Tab as the separator & no need to handle tab as a search/replace string
        • Should be case-insensitive for the search string matching, replacement string I guess would preserve the case as per csv file
        • Would also want to point it at a folder and have it 'replace in files' anything within that folder inc. within all sub-folders (regardless if they're hidden or not), happy to edit the script each time as the top level folder name will change but a prompt for this location would also work well
        • As a bonus if it could also create a log of files scanned and ideally a count of replacements in each file that would be great for an audit trail

        6,603548
        Grand MasterGrand Master
        6,603548

          Nov 25, 2020#4

          Here is a customized script which I think fulfills all your requirements.

          The strings assigned to the variables g_sListFile and g_sFolderDefault must be defined by you. The variables g_sFolderReplace and sLogFileName can be defined with a non-empty string to avoid the prompt for the base folder path respectively automatically save and close the log file.

          Code: Select all

          // Define fixed strings used globally.
          var g_asFindReplaceList;   // Array of find and replace strings.
          var g_sLogFileContents;
          var g_sListFile = "D:\\redaction\\redactionlist.csv";
          var g_sFolderDefault = "D:\\Base\\Folder\\";
          // Assign g_sFolderDefault or a different folder path instead of
          // an empty string to avoid the prompt for the base folder path.
          var g_sFolderReplace = "";    // Path to folder with files to modify.
          
          // Define fixed strings used only in main code.
          var sSeparator = "\t";
          // Specify here a valid log file name with valid
          // full path for an automatic save and close.
          var sLogFileName = "";
          
          // === GetFindReplaceList ====================================================
          
          // This function reads the list of find and replace strings from CSV file.
          // It prompts the user also for the folder path if not defined at top.
          
          function GetFindReplaceList()
          {
             var nColumnNumber;
             var nLineNumber = 0;
             var nListFile = -1;
          
             // Is any file opened?
             if (UltraEdit.document.length > 0)
             {
                // Search in list of opened files for the list file.
                // Compare the full qualified file names case-insensitive.
                var sListFile = g_sListFile.toLowerCase();
                for (var nDocIndex = 0; nDocIndex < UltraEdit.document.length; nDocIndex++)
                {
                   if (UltraEdit.document[nDocIndex].path.length != sListFile.length) continue;
                   if (UltraEdit.document[nDocIndex].path.toLowerCase() != sListFile) continue;
                   nLineNumber = UltraEdit.document[nDocIndex].currentLineNum;
                   nColumnNumber = UltraEdit.document[nDocIndex].currentColumnNum;
                   // This line is just for UltraEdit < v16.00 and UES < v10.00.
                   if (typeof(UltraEdit.activeDocumentIdx) == "undefined") nColumnNumber++;
                   nListFile = nDocIndex;
                   break;
                }
             }
          
             // Is the list file not already opened?
             if (nListFile < 0)
             {
                nListFile = UltraEdit.document.length;
                UltraEdit.open(g_sListFile);
                // Is the list file still not opened?
                if (nListFile == UltraEdit.document.length)
                {
                   g_asFindReplaceList = [];
                   UltraEdit.outputWindow.write("ERROR: Failed to open file: " + g_sListFile);
                   UltraEdit.outputWindow.showWindow(true);
                   return;
                }
             }
          
             // Get all lines in the list file as an array of strings
             // independent on the type of line endings (DOS/Unix/Mac).
             UltraEdit.document[nListFile].selectAll();
             if (UltraEdit.document[nListFile].isSel())
             {
                g_asFindReplaceList = UltraEdit.document[nListFile].selection.replace(/\r\n|\r/g,"\n").split("\n");
             }
             else
             {
                g_asFindReplaceList = [];
                UltraEdit.outputWindow.write("ERROR: There is nothing in file: " + g_sListFile);
                UltraEdit.outputWindow.showWindow(true);
             }
          
             // Prompt the user with at least one file opened now
             // for the folder path if not already defined at top.
             if (g_asFindReplaceList.length && !g_sFolderReplace.length)
             {
                g_sFolderReplace = UltraEdit.getString("The default base folder path is:\n"+g_sFolderDefault+"\nPlease enter base folder path:",1);
                if (g_sFolderReplace.length)
                {
                   // Replace all / by \ in user input folder path.
                   g_sFolderReplace = g_sFolderReplace.replace(/\x2F/g,"\\");
                }
                else
                {
                   g_sFolderReplace = g_sFolderDefault;
                }
             }
          
             if (nLineNumber)
             {
                // Restore initial caret position in already opened list file.
                UltraEdit.document[nListFile].gotoLine(nLineNumber,nColumnNumber);
             }
             else
             {
                // Close the list file just opened before.
                UltraEdit.closeFile(UltraEdit.document[nListFile].path,2);
             }
          }
          
          // === GetFilesList ==========================================================
          
          // This function gets the list of files on which the replaces are done.
          
          function GetFilesList()
          {
             // The folder path must end with a backslash.
             if (g_sFolderReplace[g_sFolderReplace.length-1] != "\\")
             {
                g_sFolderReplace += "\\";
             }
          
             // Run a Find in Files with an empty search string to get the
             // list of files in specified directory and all its subdirectories.
             UltraEdit.outputWindow.clear();
             UltraEdit.frInFiles.useOutputWindow=true;
             UltraEdit.frInFiles.filesToSearch=0;
             UltraEdit.frInFiles.searchInFilesTypes="*";
             UltraEdit.frInFiles.directoryStart=g_sFolderReplace;
             UltraEdit.frInFiles.ignoreHiddenSubs=false;
             UltraEdit.frInFiles.displayLinesDoNotMatch=false;
             UltraEdit.frInFiles.openMatchingFiles=false;
             UltraEdit.frInFiles.reverseSearch=false;
             UltraEdit.frInFiles.useEncoding=false;
             UltraEdit.frInFiles.searchSubs=true;
             UltraEdit.frInFiles.matchCase=false;
             UltraEdit.frInFiles.matchWord=false;
             UltraEdit.frInFiles.regExp=true;
             UltraEdit.frInFiles.find("");
          
             // Copy the files list written to output window to the
             // user clipboard 9 of UltraEdit and clear output window.
             UltraEdit.selectClipboard(9);
             UltraEdit.outputWindow.copy();
             UltraEdit.outputWindow.clear();
          
             // Get the files list from user clipboard 9 without the
             // summary on last line. The string must be adapted on
             // using not English version of UltraEdit or UEStudio.
             var sFilesList = UltraEdit.clipboardContent.replace(/Search complete[\s\S]+$/,"");
          
             // If there was nothing output than the summary by command
             // Find in Files, the folder does not exist or there is no file.
             if (!sFilesList.length)
             {
                g_asFindReplaceList = [];
                UltraEdit.clearClipboard();
                UltraEdit.selectClipboard(0);
                UltraEdit.outputWindow.write("ERROR: No file found in folder: " + g_sFolderReplace);
                UltraEdit.outputWindow.showWindow(true);
             }
             else
             {
                g_sLogFileContents = "Files processed:\n\n" + sFilesList;
             }
          }
          
          // === Main ==================================================================
          
          // Define environment for this script.
          UltraEdit.insertMode();
          if (typeof(UltraEdit.columnModeOff) == "function") UltraEdit.columnModeOff();
          else if (typeof(UltraEdit.activeDocument.columnModeOff) == "function") UltraEdit.activeDocument.columnModeOff();
          
          GetFindReplaceList();
          if (g_asFindReplaceList.length) GetFilesList();
          if (g_asFindReplaceList.length)
          {
             // Define the parameters for Replace in Files which were
             // not already defined for Find in Files in GetFilesList.
             UltraEdit.frInFiles.logChanges=true;
             UltraEdit.frInFiles.preserveCase=false;
          
             // Declare the variables used in the loop below.
             var nSeparator;   // Character index position of first separator occurrence
             var sFind;        // String to find by the command Replace in Files
             var sFindReplace; // Next line from list file with next find and replace
             var sReplace;     // Replace string to use by the command Replace in Files
          
             for (var nFindReplace = 0; nFindReplace < g_asFindReplaceList.length; nFindReplace++)
             {
                sFindReplace = g_asFindReplaceList[nFindReplace];
          
                // Ignore empty lines in list file. Empty lines can be used in
                // list file to create blocks to structure somehow the list.
                if (!sFindReplace.length) continue;
          
                nSeparator = sFindReplace.indexOf(sSeparator);
                // Is there a separator not at beginning of the line?
                if (nSeparator > 0)
                {
                   sFind = sFindReplace.substr(0,nSeparator);
                   // The replace string can be an empty string!
                   sReplace = sFindReplace.substr(nSeparator+1);
                }
                // Is there no separator in the line?
                else if (nSeparator < 0)
                {
                   sFind = sFindReplace;
                   // The replace string is an empty string.
                   sReplace = "";
                }
                else continue; // Ignore lines starting with the separator. Such
                               // lines can be used for comments in list file.
          
                // Run the case-insensitive Replace in Files.
                UltraEdit.frInFiles.replace(sFind,sReplace);
          
                // Append the find and replace strings to log file contents.
                g_sLogFileContents += '\nReplace "' + sFind + '" by "' + sReplace + '":\n';
                // Get the log of command Replace in Files and append it, too.
                UltraEdit.outputWindow.copy();
                UltraEdit.outputWindow.clear();
                g_sLogFileContents += UltraEdit.clipboardContent;
             }
          
             // Is there anything to write into the log file?
             if (g_sLogFileContents.length)
             {
                // Copy the log file contents to user clipboard 9.
                UltraEdit.clipboardContent = g_sLogFileContents;
                // Create a new file, make sure it is a file with DOS/Windows
                // line endings and paste the log file contents into the file
                // with an automatic conversion of the line endings to DOS
                // if not disabled in configuration of UltraEdit / UEStudio.
                UltraEdit.newFile();
                UltraEdit.activeDocument.unixMacToDos();
                UltraEdit.activeDocument.paste();
          
                // Is there a log file name defined at top of the script.
                if (sLogFileName.length)
                {
                   // Save the new file with that file name.
                   UltraEdit.saveAs(sLogFileName);
                   // Was the file saving really successful?
                   if (sLogFileName.toLowerCase() == UltraEdit.activeDocument.path.toLowerCase())
                   {
                      // Close the log file and inform the user in output
                      // window about successful saving of log file.
                      UltraEdit.closeFile(UltraEdit.activeDocument.path,2);
                      UltraEdit.outputWindow.write("Replace results written to: " + UltraEdit.activeDocument.path);
                      UltraEdit.outputWindow.showWindow(true);
                   }
                }  // Otherwise the new file remains open for user verification.
             }
          
             // Clear the user clipboard 9 and select Windows clipboard.
             UltraEdit.clearClipboard();
             UltraEdit.selectClipboard(0);
          }
          
          The CSV file can contain empty lines which are ignored by the code, for example to structure the find and replace strings a little bit in blocks.

          The CSV file can have also lines with first character being a horizontal tab which are also ignored making it possible to comment a line in CSV file.

          Please note that case-insensitive non-regular expression replaces are executed which means the symbols of special character summary on help page about Find command or Replace command can be also used in the find and replaces strings stored in the tab-separated values file with the exception of ^c and ^s which make no sense in this case.
          Best regards from an UC/UE/UES for Windows user from Austria

          3
          NewbieNewbie
          3

            Dec 01, 2020#5

            Sorry for the delay in responding, have been on leave. Again - many thanks for producing this, certainly will have saved me a lot of work.