Parse EDI file and retain selected transactions by ISA string

Parse EDI file and retain selected transactions by ISA string

3
NewbieNewbie
3

    May 27, 2015#1

    I would like to use a script on a set of EDI files that contain multiple transactions for different trading partners. The EDI files need to be parsed to remove all transactions that do not have a ISA envelope containing the string specified. I have tried several Macros and Scripts designed to split files based on record strings, blocks etc., but I cannot seem to put together what I need. Performing this function manually is possible, but errors can be introduced by the manual manipulation of the EDI files. The set of EDI files can be streamed or unwrapped, for clarity, I created an unwrapped example to clarify what is needed.

    For example, I would like to be able to enter the string needed to retain as below.

    Can the script find the string like "RECVPARTNER003" and select the file from the beginning of the ISA envelope to the end of the transaction IEA ~ segment terminator?

    I am not sure the most efficient approach to resolve the issue, retain all of the transactions in the EDI file, capturing in a different file or excluding the non-selected transactions and keeping the original file name.

    Either way would work.

    I appreciate any and all suggestions.

    Thank you in advance for your help!
    UE21.3_SCRIPT.PNG (38.55KiB)

    6,686585
    Grand MasterGrand Master
    6,686585

      May 27, 2015#2

      It is not really clear for me what the script should do, especially as I don't know anything about EDI files.

      Please copy a block from EDI file with several entries into your next post as code block by clicking on button Code above post edit area. And copy as code block also the desired output in a new file for this input when running the script with string RECVPARTNER003 entered by the user of the script. That would make it easier for us to understand what the script should do and we have an example to test the script. (UE scripts can't be executed on screen shots, just on text data.)
      Best regards from an UC/UE/UES for Windows user from Austria

      3
      NewbieNewbie
      3

        May 27, 2015#3

        Thank you very much for your reply!

        This is the EDI file example, it contains (12) sets of transactions, by selecting the ISA string RECVPARTNER0003 we should get back just (2) sets of data ISA...RECVPARTNER0003 thru IEA...~.

        I hope this helps clarify my question.

        Code: Select all

        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0003*150501*0900*U*00401*000000628*0*P*:~
        GS*FA*SENDER*RECVPARTNER0003*20150501*0900*628*X*004010~
        ST*997*6280001~
        AK1*PR*207924998~
        AK2*855*20921184~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*6280001~
        ST*997*6280002~
        AK1*PR*207924999~
        AK2*855*20921185~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*6280002~
        GE*2*628~
        IEA*1*000000628~
        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0001*150501*1420*U*00401*000002486*0*P*:~
        GS*FA*SENDER*RECVPARTNER0001*20150501*1420*2486*X*004010~
        ST*997*24860001~
        AK1*PR*207928561~
        AK2*855*20924720~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*24860001~
        GE*1*2486~
        IEA*1*000002486~
        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0003*150501*1310*U*00401*000000646*0*P*:~
        GS*FA*SENDER*RECVPARTNER0003*20150501*1310*646*X*004010~
        ST*997*6460001~
        AK1*PR*207927799~
        AK2*855*20923967~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*6460001~
        ST*997*6460002~
        AK1*PR*207927801~
        AK2*855*20923969~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*6460002~
        ST*997*6460003~
        AK1*PR*207927848~
        AK2*855*20924016~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*6460003~
        GE*3*646~
        IEA*1*000000646~
        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0002*150501*1110*U*00401*000008540*0*P*:~
        GS*FA*SENDER*RECVPARTNER0002*20150501*1110*8540*X*004010~
        ST*997*85400001~
        AK1*PR*207926621~
        AK2*855*20922800~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*85400001~
        ST*997*85400002~
        AK1*PR*207926619~
        AK2*855*20922798~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*85400002~
        GE*2*8540~
        IEA*1*000008540~
        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0004*150501*1630*U*00401*000006203*0*P*:~
        GS*FA*SENDER*RECVPARTNER0004*20150501*1630*6203*X*004010~
        ST*997*62030001~
        AK1*PR*207929884~
        AK2*855*20926032~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*62030001~
        GE*1*6203~
        IEA*1*000006203~
        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0006*150501*1400*U*00401*000021665*0*P*:~
        GS*FA*SENDER*RECVPARTNER0006*20150501*1400*21665*X*004010~
        ST*997*216650001~
        AK1*PR*207928379~
        AK2*855*20924538~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*216650001~
        GE*1*21665~
        IEA*1*000021665~
        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0005*150501*1450*U*00401*000007339*0*P*:~
        GS*FA*SENDER*RECVPARTNER0005*20150501*1450*7339*X*004010~
        ST*997*73390001~
        AK1*PR*207928847~
        AK2*855*20925006~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*73390001~
        GE*1*7339~
        IEA*1*000007339~
        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0007*150501*1050*U*00401*000000312*0*P*:~
        GS*FA*SENDER*RECVPARTNER0007*20150501*1050*312*X*004010~
        ST*997*3120001~
        AK1*PR*207926103~
        AK2*855*20922287~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*3120001~
        ST*997*3120002~
        AK1*PR*207926076~
        AK2*855*20922260~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*3120002~
        ST*997*3120003~
        AK1*PR*207926075~
        AK2*855*20922259~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*3120003~
        ST*997*3120004~
        AK1*PR*207926102~
        AK2*855*20922286~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*3120004~
        ST*997*3120005~
        AK1*PR*207926065~
        AK2*855*20922249~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*3120005~
        GE*5*312~
        IEA*1*000000312~
        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0009*150501*1520*U*00401*000000923*0*P*:~
        GS*FA*SENDER*RECVPARTNER0009*20150501*1520*923*X*004010~
        ST*997*9230001~
        AK1*PR*207929292~
        AK2*855*20925446~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*9230001~
        GE*1*923~
        IEA*1*000000923~
        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0011*150501*0820*U*00401*000005966*0*P*:~
        GS*FA*SENDER*RECVPARTNER0011*20150501*0820*5966*X*004010~
        ST*997*59660001~
        AK1*PR*207924460~
        AK2*855*20920687~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*59660001~
        GE*1*5966~
        IEA*1*000005966~
        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0010*150501*0810*U*00401*000005887*0*P*:~
        GS*FA*SENDER*RECVPARTNER0010*20150501*0810*5887*X*004010~
        ST*997*58870001~
        AK1*PR*207924387~
        AK2*855*20920614~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*58870001~
        GE*1*5887~
        IEA*1*000005887~
        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0012*150501*1100*U*00401*000005060*0*P*:~
        GS*FA*SENDER*RECVPARTNER0012*20150501*1100*5060*X*004010~
        ST*997*50600001~
        AK1*PR*207926399~
        AK2*855*20922582~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*50600001~
        ST*997*50600002~
        AK1*PR*207926403~
        AK2*855*20922585~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*50600002~
        ST*997*50600003~
        AK1*PR*207926363~
        AK2*855*20922546~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*50600003~
        GE*3*5060~
        IEA*1*000005060~
        This is the desired result

        Code: Select all

        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0003*150501*0900*U*00401*000000628*0*P*:~
        GS*FA*SENDER*RECVPARTNER0003*20150501*0900*628*X*004010~
        ST*997*6280001~
        AK1*PR*207924998~
        AK2*855*20921184~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*6280001~
        ST*997*6280002~
        AK1*PR*207924999~
        AK2*855*20921185~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*6280002~
        GE*2*628~
        IEA*1*000000628~
        ISA*00*          *00*          *ZZ*SENDER      *08*RECVPARTNER0003*150501*1310*U*00401*000000646*0*P*:~
        GS*FA*SENDER*RECVPARTNER0003*20150501*1310*646*X*004010~
        ST*997*6460001~
        AK1*PR*207927799~
        AK2*855*20923967~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*6460001~
        ST*997*6460002~
        AK1*PR*207927801~
        AK2*855*20923969~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*6460002~
        ST*997*6460003~
        AK1*PR*207927848~
        AK2*855*20924016~
        AK5*A~
        AK9*A*1*1*1~
        SE*6*6460003~
        GE*3*646~
        IEA*1*000000646~

        6,686585
        Grand MasterGrand Master
        6,686585

          May 28, 2015#4

          With the input and output example it was clear what to do. I wrote 2 scripts.

          The first one below runs a Perl regular expression Replace in Files on all *.edi files in C:\Temp\ to remove all transactions from the files not having the RECVPARTNER number which is
          • predefined in script at top, or
          • determined from a selection in active file on script execution starting with RECVPARTNER and having 1 or more digits, or
          • is entered by the user of the script.
          You can of course define a different directory for the script and a different file type specification.

          How many transactions have been removed from which files is written into the output window which is not automatically opened by the script.

          Code: Select all

          var sNumber = "";
          
          // Must the script user enter the search string (or is it predefined)?
          if (!sNumber.length)
          {
             if (UltraEdit.document.length > 0)  // Is any file opened?
             {
                UltraEdit.insertMode();
                if (typeof(UltraEdit.columnModeOff) == "function") UltraEdit.columnModeOff();
                else if (typeof(UltraEdit.activeDocument.columnModeOff) == "function") UltraEdit.activeDocument.columnModeOff();;
                // Is there a selection in active file?
                if (UltraEdit.activeDocument.isSel())
                {
                   // Get the selected text and use it if it starts with "RECVPARTNER".
                   sNumber = UltraEdit.activeDocument.selection;
                   // Does the selected text start with string "RECVPARTNER" and
                   // has 1 or more digits and nothing else to end of selection?
                   if (sNumber.search(/^RECVPARTNER\d+$/) != 0)
                   {
                      sNumber = "";  // No, ignore selected text.
                   }
                   else
                   {
                      sNumber = sNumber.replace(/RECVPARTNER/,"");
                   }
                }
             }
          }
          
          // Is the number whether predefined nor no correct selection?
          if (!sNumber.length)
          {
             // Ask user of script for the number.
             sNumber = UltraEdit.getString("Enter RECVPARTNER number with all digits:",1);
             if (sNumber.length)
             {
                // The number must consist of digits and no other character.
                if (sNumber.search(/^\d+$/) != 0)
                {
                   sNumber = "";
                }
             }
          }
          
          // Is a number now defined?
          if (sNumber.length)
          {
             // Define the parameters for the Perl regular expression Replace in Files.
             UltraEdit.perlReOn();
             UltraEdit.frInFiles.filesToSearch=0;
             UltraEdit.frInFiles.searchSubs=false;
             UltraEdit.frInFiles.searchInFilesTypes="*.edi";
             UltraEdit.frInFiles.directoryStart="C:\\Temp\\";
             UltraEdit.frInFiles.ignoreHiddenSubs=false;
             UltraEdit.frInFiles.unicodeSearch=false;
             UltraEdit.frInFiles.preserveCase=false;
             UltraEdit.frInFiles.logChanges=true;
             UltraEdit.frInFiles.matchWord=false;
             UltraEdit.frInFiles.matchCase=true;
             UltraEdit.frInFiles.regExp=true;
          
             // Build the entire Perl regular expression search string to delete
             // all transactions except those with entered RECVPARTNER number.
             var sSearchString = "^ISA\\*.+?RECVPARTNER(?!" + sNumber + ")[\\s\\S]+?IEA\\*\\d.+\\s*";
             UltraEdit.frInFiles.replace(sSearchString,"");
          }
          
          I wrote a second script based on FindStringsToNewFile.js which copies all found transactions with predefined, selected or entered RECVPARTNER number from active file only into a new file.

          It would be of course possible to extended this script to run on a list of files, see GetListOfFiles, or all open files.

          Code: Select all

          var sSearchString = "";
          
          
          //== OutputMessage =========================================================
          
          // This function displays a message in a message box or write the message
          // into a new file for users of UltraEdit v13.00 or UEStudio v6.20.
          function OutputMessage (sMessageText)
          {
             if (UltraEdit.messageBox) UltraEdit.messageBox(sMessageText);
             else
             {
                UltraEdit.newFile();
                UltraEdit.activeDocument.unixMacToDos();
                UltraEdit.activeDocument.unicodeToASCII();
                UltraEdit.activeDocument.write(sMessageText.replace(/\n/g,"\r\n"));
                UltraEdit.activeDocument.insertLine();
             }
          }
          
          
          //== FindTransactionsToNewFile =============================================
          
          if (UltraEdit.document.length > 0)     // Is any file currently opened?
          {
             UltraEdit.insertMode();             // Define environment for the script.
             if (typeof(UltraEdit.columnModeOff) == "function") UltraEdit.columnModeOff();
             else if (typeof(UltraEdit.activeDocument.columnModeOff) == "function") UltraEdit.activeDocument.columnModeOff();
             UltraEdit.activeDocument.hexOff();
          
             // Must the script user enter the search string (or is it predefined)?
             if (!sSearchString.length)
             {
                // Is there a selection in active file?
                if (UltraEdit.activeDocument.isSel())
                {
                   // Get the selected text and use it if it starts with "RECVPARTNER".
                   sSearchString = UltraEdit.activeDocument.selection;
                   // Does the selected text start with string "RECVPARTNER" and
                   // has 1 or more digits and nothing else to end of selection?
                   if (sSearchString.search(/^RECVPARTNER\d+$/) != 0)
                   {
                      sSearchString = "";  // No, ignore selected text.
                   }
                }
             }
             if (!sSearchString.length)
             {
                // Otherwise ignore the selection and ask user of script for the number.
                sSearchString = UltraEdit.getString("Enter RECVPARTNER number with all digits:",1);
                if (sSearchString.length)
                {
                   // The number must consist of digits and no other character.
                   if (sSearchString.search(/^\d+$/) == 0)
                   {
                      sSearchString = "RECVPARTNER" + sSearchString;
                   }
                   else
                   {
                      sSearchString = "";
                   }
                }
             }
             if (sSearchString.length)
             {
                // Make sure that last line of file has a line termination.
                UltraEdit.activeDocument.bottom();
                if (UltraEdit.activeDocument.isColNumGt(1))
                {
                   UltraEdit.activeDocument.insertLine();
                }
          
                // Build the entire regular expression search string.
                sSearchString = "ISA\\*.+?" + sSearchString + "[\\s\\S]+?IEA\\*\\d.+\\s+";
          
                // Search entire file with a case sensitive regular expression.
                // As the script user can define also an invalid regular expression
                // via wrong selection, catch this error and inform the script user
                // about this mistake.
                var bValidExpression = true;
                try
                {
                   var rRegSearch = new RegExp(sSearchString,"g");
                }
                catch (Error)
                {
                   OutputMessage("The entered search expression is an invalid regular expression.\n\n"+Error);
                   bValidExpression = false;
                }
          
                if (bValidExpression)
                {
                   // Search entire file with a not case sensitive regular expression
                   // search which puts all found strings into an array of strings.
                   UltraEdit.activeDocument.selectAll();
          
                   // An error in the line below on script execution is caused usually by
                   // a too large file for this script, use FindStringsToNewFileExtended.js.
                   var asFoundTransactions = UltraEdit.activeDocument.selection.match(rRegSearch);
          
                   UltraEdit.activeDocument.top();  // Cancel selection.
          
                   if (asFoundTransactions != null) // Was any string found?
                   {
                      UltraEdit.newFile();          // Open a new file.
          
                      // If version of UE/UES supports direct write to clipboard,
                      // use user clipboard 9 to paste the found strings into the
                      // new file as this is much faster than using write command.
                      if (typeof(UltraEdit.clipboardContent) == "string")
                      {
                         var nActiveClipboard = UltraEdit.clipboardIdx;
                         UltraEdit.selectClipboard(9);
                         UltraEdit.clipboardContent = asFoundTransactions.join("");
                         UltraEdit.activeDocument.paste();
                         UltraEdit.clearClipboard();
                         UltraEdit.selectClipboard(nActiveClipboard);
                      }
                      else UltraEdit.activeDocument.write(asFoundTransactions.join(""));
          
                      // Set caret to top of file with found strings and inform user
                      // about number of strings found with the entered expression.
                      UltraEdit.activeDocument.top();
                      var nCount = asFoundTransactions.length;
                      OutputMessage("Found "+nCount+" transaction"+(nCount==1?".":"s."));
                   }
                   else OutputMessage("No transaction found!");
                }
             }
             else OutputMessage("No RECVPARTNER number entered!");
          }
          else OutputMessage("You should have a file opened when you run this script!");
          
          Best regards from an UC/UE/UES for Windows user from Austria

          3
          NewbieNewbie
          3

            May 28, 2015#5

            Mofi,

            I cannot thank you enough for your expertise.

            The second script worked, I needed to change the actual file ISA partner to a value that would work with the script.

            This is just a temporary workaround until I can change the script to allow an alphanumeric string to be used.

            I also appreciate the very well documented code that explains each process in the script, very useful for me.

            Thank you also for providing a solution that allows the script to select the search string as outlined your explanation.

            Cheers,

            Mark

            6,686585
            Grand MasterGrand Master
            6,686585

              May 29, 2015#6

              Fine. There are the lines

              Code: Select all

              if (sSearchString.search(/^RECVPARTNER\d+$/) != 0)
              and

              Code: Select all

              if (sSearchString.search(/^\d+$/) == 0)
              in the second script.

              Replacing in those 2 lines \d by \w (any word character, i.e. A-Za-z0-9_ and language specific word characters like German äöüÄÖÜß) or [A-Z0-9] (just upper case letters A-Z and digits) or [A-Za-z\d] (just upper and lower case letters A-Z and digits), the script would except alphanumeric characters and not only digits. Of course the comments in the script and the prompt text should be also modified after this modification.

              BTW: MergeTextFiles.js might be of interest for you if you first want to merge the data of all EDI files together into a single file on which the second script is executed to extract data of interest.
              Best regards from an UC/UE/UES for Windows user from Austria