Odd issue using SplitFile.js

Odd issue using SplitFile.js

7
NewbieNewbie
7

    Jul 13, 2022#1

    I have been using SplitFile.js for sometime now; however I just got a new i9 Dell 3650 Tower and I get an "Cannot allocate memory" error when trying to run the script and it sort of works; see attached. FWIW, it works on my i9 Dell 7550 Precision laptop, so I am using it as the work around.

    So my silly question is, is there some configuration that I need to do for the new computer?

    My UE version: 26.10.0.72
    Ultra Edit Split File Results.jpg (122.44KiB)
    Ultra Edit Split File Error.jpg (258.33KiB)

    6,686586
    Grand MasterGrand Master
    6,686586

      Jul 13, 2022#2

      Well, the question cannot be really answered without seeing the code of the UltraEdit script.

      Do you have used on Dell 7550 Precision laptop the command Backup settings to backup all your UltraEdit configuration into a file and used on Dell 3650 Tower the same command to Restore the configuration from the backup file created on laptop?

      The error message itself just tells us that the JavaScript core engine or UltraEdit itself (depends on code) cannot allocate a large enough block from RAM.
      Best regards from an UC/UE/UES for Windows user from Austria

      7
      NewbieNewbie
      7

        Jul 13, 2022#3

        I can run the "Backup Settings" and share them with you if you desire, but before I do that, I have to know if there is anything in the file that would need to be sanitized.

        FWIW the configuration is simple and is the same on both computers; the only configuration changes I made are ... "work with No backup", "Do not reload previous files" and "Open files without temp file and no prompt"; aside from this, the installation should be default.

        6,686586
        Grand MasterGrand Master
        6,686586

          Jul 14, 2022#4

          Okay, so the configuration is the same on both computers. That means a configuration settings cannot be responsible for the difference on script execution. Therefore sharing your configuration with you is also not necessary.

          It would be good if you would share with us the script file SplitFile.js to look on the code which results on execution in the error message.
          Best regards from an UC/UE/UES for Windows user from Austria

          7
          NewbieNewbie
          7

            Jul 14, 2022#5

            Thanks and here you go ...

            Code: Select all

            // First file (most left on file tabs bar) must be the file to split.     
                                                                                      
            if (UltraEdit.document.length > 0) {  // Is any file opened?              
                                                                                      
               var nLinesPerFile = 100000;                                           
               var nNextLineNum = nLinesPerFile + 1;                                  
               var nFileCount = 0;                                                    
               var bCopyFirstLine = false;   // Copy first line into every file.       
                                                                                      
               // Define the environment for the script.                              
               UltraEdit.insertMode();                                                
               UltraEdit.columnModeOff();                                             
               UltraEdit.document[0].hexOff();                                        
               // Move caret to top of the file.                                      
               UltraEdit.document[0].top();                                           
                                                                                      
               if (bCopyFirstLine) {                                                  
                  // Code to copy first line of file into user clipboard 8 for pasting
                  // it later into every file created on splitting the large file.    
                  UltraEdit.selectClipboard(8);                                       
                  UltraEdit.document[0].selectLine();                                 
                  UltraEdit.document[0].copy();                                       
                  UltraEdit.document[0].cancelSelect();                               
                  UltraEdit.document[0].gotoLine(2,1);                                
               }                                                                      
               UltraEdit.selectClipboard(9);                                          
                                                                                      
               // Quick and dirty solution to get file name without extension         
               // and the file extension. Does not work for all file names.           
               var nLastPoint = UltraEdit.document[0].path.lastIndexOf('.');          
               if (nLastPoint < 0) nLastPoint = UltraEdit.document[0].path.length;    
               var sFileName = UltraEdit.document[0].path.substr(0,nLastPoint) + '_'; 
               var sFileExt = UltraEdit.document[0].path.substr(nLastPoint);          
                                                                                      
               while (1) {                                                            
                  UltraEdit.document[0].gotoLineSelect(nNextLineNum,1);               
                  if (!UltraEdit.document[0].isSel()) break;                          
                                                                                      
                  // Copy the selected block to user clipboard 9.                     
                  UltraEdit.document[0].copy();                                       
                  UltraEdit.document[0].cancelSelect();                               
                                                                                      
                  UltraEdit.newFile();                                                
                                                                                      
                  if (bCopyFirstLine) {                                               
                     // Paste the first line first into the new file.                 
                     UltraEdit.selectClipboard(8);                                    
                     UltraEdit.activeDocument.paste();                                
                     UltraEdit.selectClipboard(9);                                    
                  }                                                                   
                  // Paste the file copied block into the new file.                   
                  UltraEdit.activeDocument.paste();                                   
                  nFileCount++;                                                       
                  UltraEdit.saveAs(sFileName + nFileCount + sFileExt);                
                  UltraEdit.closeFile(UltraEdit.activeDocument.path,2);               
                  nNextLineNum += nLinesPerFile;                                      
               }                                                                      
               UltraEdit.document[0].top();                                           
                                                                                      
               // Clear the used user clipboards and reselect system clipboard.       
               UltraEdit.clearClipboard();                                            
               if (bCopyFirstLine) {                                                  
                  UltraEdit.selectClipboard(8);                                       
                  UltraEdit.clearClipboard();                                         
               }                                                                      
               UltraEdit.selectClipboard(0);                                          
               UltraEdit.messageBox(nFileCount + " files created.");                  
            }                                                                         
            

            6,686586
            Grand MasterGrand Master
            6,686586

              Jul 14, 2022#6

              The script was written obviously by me, but without all the trailing spaces as in posted code. It is written for large and huge files where counting the lines must be enabled in configuration.

              I restored 32-bit UltraEdit v26.10.0.72 from my archives. Then I created beginning from a small file which just 78,122 bytes opened with creation of a temporary file a large file with 680,808, 588 bytes by coping the small file contents and pasting it again and again and again. The large file has had finally 12,033,718 lines. There was several times displayed the message Cannot allocate ... (don't remember anymore the exact message text) while creating the large file with using a temporary file. Then I exited UltraEdit.

              I started UltraEdit again, opened again the now large file named VHBC_17-19.txt, this time without the usage of a temporary file, and run the script. It created the files VHBC_17-19_1.txt to VHBC_17-19_121.txt without showing any error message during script execution. The files created by the script had a file size between 5,519 KB and 5,530 KB. The first 120 files had exactly 100,000 lines as defined in the script. So I am unable to reproduce the issue.

              It is really odd that the sizes of the created files are in your case mainly 0 KB, one has 91,985 KB, another one 92,463 and the last one 10,421 KB. The lines must be really long when 100,000 lines produce two files with about 89 MB. It would be perhaps a good idea on so long lines to reduce the value of nLinesPerFile to 20000 to reduce the memory required as one block to copy the lines via a user clipboard from the large file into all the new files.

              The only other idea I have is to insert below

              Code: Select all

                   // Paste the file copied block into the new file.
                    UltraEdit.activeDocument.paste();
              additionally the line

              Code: Select all

                   UltraEdit.clearClipboard();
              That line explicitly clears the clipboard after the paste to free memory. It should not make a difference, but who knows which other processes are running on your Dell 3650 Tower which also require memory at the same time like an anti-virus application immediately scanning the file just created by UltraEdit during the script execution.

              PS: I wanted to see the message Cannot allocate ... and therefore did again start with the same small file as before creating a large file with pasting 100,000 lines at end of the file again and again. But this time no message was displayed by UltraEdit v26.10.0.72. I have the impression that background processes had caused the messages on first creation of the large file as that was done short after starting Windows on which Windows starts usually several tasks in background. The second creation of a large file was more than 30 minutes later.
              Best regards from an UC/UE/UES for Windows user from Austria

              7
              NewbieNewbie
              7

                Jul 14, 2022#7

                Thanks for trying and if it means anything I am using x64, not x32. It still runs on the old Ryzen 1950X as it always had.

                I have never timed it, as I would start it and go do something else and it was usually finished by the time I got back to it. This error occurred seconds after I started it and the files noted in the screenshot took merely a few minutes.

                I will implement your code updates and keep you in the loop for any other occurrences, provided there are new results to share. FWIW, here are some average raw data file specs ...

                Data Rows: 3,158,061
                File Size: 2,939,756 KB
                Text File, Tab Delimited, 180 Data Columns

                Column Header (some data rows are nearly as long as this column header, but the average is 1410 characters)

                Code: Select all

                XXX_PPPPPPPP_FFFF_NNNN    XXX_PPPPPPPP_SSSSSS_II    FFFF_YYYY_MMMMM    III    III_MMM    TTTTT_DDD_NNN    DDD_NNN    CCCCC_TTTT    CCCCC_SSSSSS_HHH    CCCCC_SSSSSS_DDD    PPP_CCC_DDDDD    CCCCC_KKK    LLLLLL_CCCCC_III    PPPPPPP_DDDD    BBBBBB_DDDD    RRRRRRR_DDDD    FFFF_HHH    TTTT_HHH    FFFF_DDD    TTTT_DDD    AAAAAAAAA_DDDD    DDDDDDDDD_DDDD    DDDDDD_HHHH    DDD_CCCC    DDD_DDDDD    IIII_AAA_HHH    IIII_AAA_DDD    BBBBBB_AAA_HHH    BBBBBB_AAA_DDD    AAAAAAA_AAA_DDD    AAAAAAA_AAA_HHH    RRRRR_AAA    QQQ_BBBBBB_DDD    QQQ_AAAAAAA_DDD    MMMMMMM_AAAAAAA    PPPP_CCCC    PPPP_CCCC_DDDD    PPPP_MMM_1    PPPP_MMM_2    PPPP_MMM_3    PPPP_MMM_4    PPPPPPP_DDDD    PPPPPPP_DDDD_DDDD    DDDD_1    DDDD_1_DDDD    DDDD_2    DDDD_2_DDDD    DDDD_3    DDDD_3_DDDD    DDDD_4    DDDD_4_DDDD    DDDD_5    DDDD_5_DDDD    DDDD_6    DDDD_6_DDDD    DDDD_7    DDDD_7_DDDD    DDDD_8    DDDD_8_DDDD    AAAAAAAAA_DDDD_CCCC    SSSS_PPPP_CCCC_1    SSSS_PPPP_CCCC_2    SSSS_PPPP_CCCC_3    SSSS_PPPP_CCCC_4    SSSS_PPPP_CCCC_5    SSSS_PPPP_CCCC_6    RRR_CCC    PPPPPPP_SSSSSS_CCCC    RRRRR_SSS    II_MMMMMMII_SSSSSSTED    II_MMMMMMII    RRRRR_DDD    RRRRR_DDD    RRRRR_LLLL_NNNN    RRRRR_FFFFF_NNNN    RRRRR_MM    RRRRR_AAAAAAA    RRRRR_AAAAAAA_LLLL_2    RRRRR_CCCC    RRRRR_SSSSS    RRRRR_ZZZ    RRRRR_ZZZ_4    RRRRR_AAA    RRRRR_AAA_II_MMMMMS    RRRRR_GGGGGG    XXX_II_MMMMMMII    XXX_RRRRRRRRR_NNNN    XXX_RRRRRRRRR_AAAAAAA    XXX_RRRRRRRRR_CCCC_SS_ZZZ    XXX_BBBB_PPPP_NNNN    XXX_BBBB_PPPP_MMMMM_II    XXX_BBBB_PPPP_MMMMM_II    XXX_BBBB_PPPP_SSS    XXX_BBBB_PPPP_TTT_II    XXX_BBBB_PPPP_NNN    XXX_BBBB_PPPP_TTT_CCCC    XXX_BBBB_PPPP_TTT_QQQ_II    XXX_BBBB_PPPP_AAAA    XXX_BBBB_PPPP_CCCC_SS_ZZZ    BBBB_PPPPPPPP_II    BBBB_PPPPPPPP_MMMMM_II    OOO_BBBB_PPPP_NNN    BBBB_PPPP_TTTT_CCC    BBBB_PPPP_SSSS_CCC    XXX_PPP_TT_PPPP_NNNN    XXX_PPP_TT_PPPP_MMMMM_II    XXX_PPP_TT_PPPP_MMMMM_II    XXX_PPP_TT_PPPP_SSS    XXX_PPP_TT_PPPP_TTT_II    XXX_PPP_TT_PPPP_NNN    XXX_PPP_TT_PPPP_TTT_CCCC    XXX_PPP_TT_PPPP_TTT_QQQ_II    XXX_PPP_TT_PPPP_AAAA    XXX_PPP_TT_PPPP_CCCC_SS_ZZZ    XXX_SSSS_FFF_PPPP_NNNN    XXX_SSSS_FFF_PPPP_MMMMM_II    XXX_SSSS_FFF_PPPP_MMMMM_II    XXX_SSSS_FFF_PPPP_SSS    XXX_SSSS_FFF_PPPP_TTT_II    XXX_SSSS_FFF_PPPP_NNN    XXX_SSSS_FFF_PPPP_TTT_CCCC    XXX_SSSS_FFF_PPPP_QQQ_II    XXX_SSSS_FFF_PPPP_AAAA    XXX_SSSS_FFF_PPPP_CCCC_SS_ZZZ    XXX_RRRR_PPPP_NNNN    XXX_RRRR_PPPP_MMMMM_II    XXX_RRRR_PPPP_MMMMM_II    XXX_RRRR_PPPP_SSS    XXX_RRRR_PPPP_TTT_II    XXX_RRRR_PPPP_NNN    XXX_RRRR_PPPP_TTT_CCCC    XXX_RRRR_PPPP_QQQ_II    XXX_RRRR_PPPP_AAAA    XXX_RRRR_PPPP_CCCC_SS_ZZZ    XXX_AAAA_PPPP_NNNN    XXX_AAAA_PPPP_MMMMM_II    XXX_AAAA_PPPP_MMMMM_II    XXX_AAAA_PPPP_SSS    XXX_AAAA_PPPP_TTT_II    XXX_AAAA_PPPP_NNN    XXX_AAAA_PPPP_TTT_CCCC    XXX_AAAA_PPPP_QQQ_II    XXX_AAAA_PPPP_AAAA    XXX_AAAA_PPPP_CCCC_SS_ZZZ    XXX_RRRRR_PPPP_NNNN    XXX_RRRRR_PPPP_MMMMM_II    XXX_RRRRR_PPPP_MMMMM_II    XXX_RRRRR_PPPP_SSS    XXX_RRRRR_PPPP_TTT_II    XXX_RRRRR_PPPP_NNN    XXX_RRRRR_PPPP_TTT_CCCC    XXX_RRRRR_PPPP_QQQ_II    XXX_RRRRR_PPPP_AAAA    XXX_RRRRR_PPPP_CCCC_SS_ZZZ    XXX_OOOOO_PPPP_NNNN    XXX_OOOOO_PPPP_MMMMM_II    XXX_OOOOO_PPPP_MMMMM_II    XXX_OOOOO_PPPP_SSS    XXX_OOOOO_PPPP_TTT_II    XXX_OOOOO_PPPP_NNN    XXX_OOOOO_PPPP_TTT_CCCC    XXX_OOOOO_PPPP_QQQ_II    XXX_OOOOO_PPPP_AAAA    XXX_OOOOO_PPPP_CCCC_SS_ZZZ    XXX_SSSSS_PPPP_NNNN    XXX_SSSSS_PPPP_MMMMM_II    XXX_SSSSS_PPPP_MMMMM_II    XXX_SSSSS_PPPP_SSS    XXX_SSSSS_PPPP_TTT_II    XXX_SSSSS_PPPP_NNN    XXX_SSSSS_PPPP_TTT_CCCC    XXX_SSSSS_PPPP_QQQ_II    XXX_SSSSS_PPPP_AAAA    XXX_SSSSS_PPPP_CCCC_SS_ZZZ    EEE_CCCC_1    EEE_CCCC_2    CCCCC_EEE_1    CCCCC_EEE_2    CCCCC_EEE_3    II_PPPP_AAAAAA