Batch Script Encoding Problem

Batch Script Encoding Problem

2
NewbieNewbie
2

    Mar 29, 2018#1

    Hi everyone,

    I'm using "UltraEdit Text/Hex Editor (x64) Version 24.20.0.40".

    Problem I'm having is with batch scripts not executing correctly. This happens when I copy/paste and save batch script code.

    I have included few screenshots and one of the batch script code that I have a problem with.

    Code: Select all

    @echo off & setlocal enableDelayedExpansion
    
    mode 50,50
    
    
    
    set ^"LF=^
    
    
    
    ^" Above empty line is required - do not remove
    
    set ^"\n=^^^%LF%%LF%^%LF%%LF%^^"
    
    for /F %%a in ('echo prompt $E^| cmd') do set "ESC=%%a"
    
    
    
    set plot=for %%# in (1 2) do if %%#==2 ( for /f "tokens=1-3 delims=:" %%1 in ("^!args^!:^!ESC^!") do (%\n%
    
        for /f "tokens=1-4" %%a in ("%%~1") do ^<nul set /p "=%%3[%%b;%%aH%%3[38;5;%%cm%%3[48;5;%%dm%%~2%%3[0m"%\n%
    
    )) else set args=
    
    
    
    set circle=for %%# in (1 2) do if %%#==2 ( for /f "tokens=1-6" %%1 in ("^!args^!") do (%\n%
    
            if "%%~5" equ "" ( set "hue=30" ) else ( set "hue=%%~5")%\n%
    
            for /l %%y in (-%%3,1,%%3) do for /l %%x in (-%%3,1,%%3) do (%\n%
    
                set /a "S=(%%x * %%x) + (%%y * %%y) - (%%3 * %%3)", "_3div2=%%3 / 2", "tx=%%x+%%1", "ty=%%y+%%2"%\n%
    
                for /f "tokens=1-5" %%a in ("^!S^! ^!_3div2^! ^!tx^! ^!ty^! ^!hue^!") do (%\n%
    
                           if "%%6" equ "/f" ( if %%a leq 1    ( ^!plot^! %%c %%d %%e 0 :%%~4)%\n%
    
                    ) else if "%%6" equ "/n" ( if %%a geq -%%3 ( ^!plot^! %%c %%d %%e 0 :%%~4)%\n%
    
                    ) else if %%a geq -%%3 if %%a leq %%b      ( ^!plot^! %%c %%d %%e 0 :%%~4)%\n%
    
            ))%\n%
    
            set "s="%\n%
    
    )) else set args=
    
    
    
    for /l %%b in (0,1,4) do (
    
        set /a "x%%b=!random! %% 40 + 10"
    
        set /a "y%%b=!random! %% 40 + 10"
    
        set /a "i%%b=!random! %% 3 + 1"
    
        set /a "j%%b=!random! %% 2 + 1"
    
        set /a "s%%b=!random! %% 3 + 2"
    
        set /a "b%%b=50 - s%%b"
    
        set /a "c%%b=!random! %% 255"
    
    )
    
    
    
    for /l %%# in () do ( for /l %%b in (0,1,4) do (
    
        
    
            set /a "x%%b+=i%%b", "y%%b+=j%%b"
    
            
    
            if !x%%b! geq !b%%b! set /a "x%%b=b%%b", "i%%b*=-1"
    
            if !y%%b! geq !b%%b! set /a "y%%b=b%%b", "j%%b*=-1"
    
            if !x%%b! leq !s%%b! set /a "x%%b=s%%b", "i%%b*=-1"
    
            if !y%%b! leq !s%%b! set /a "y%%b=s%%b", "j%%b*=-1"
    
            
    
            %circle% !x%%b! !y%%b! !s%%b! . !c%%b!
    
        )
    
        cls
    
    )
    Above batch script code when saved using Notepad execute correctly as you can see it from below link.

    https://gyazo.com/0b9e699c26b7088c983879d8d897238e

    Above batch script code when saved using UltraEdit execute as shown below screenshot.


    Can I please get a advice for resolving this problem. Thank you!

    6,686585
    Grand MasterGrand Master
    6,686585

      Mar 30, 2018#2

      It looks like you don't know anything about character encoding and why all the encoding settings in UltraEdit configuration mean which is the problem here.

      Quick fix for your problem:
      1. Open Advanced - Settings or Configuration - File handling - Save and uncheck the two settings:
        • Write UTF-8 BOM header to all UTF-8 files when saved
        • Write UTF-8 BOM on new files created within this program (if above is not set)
      2. On next time using Save As make sure Default is set for option Encoding and take a look on this option in future on saving a new file.
      Real solution for your problem is understanding character encoding and configure UltraEdit accordingly as explained below.

      I posted lots about right character encoding and don't want to repeat them all here. Please read carefully the following topics and all pages linked from these topics to get knowledge about character encoding. Every programmer should understand how characters are encoded and which application expects or supports which character encoding.
      You have configured that a new file is created as UTF-8 encoded using 1 to 4 (6) bytes per character depending on character which is not good as Windows command interpreter expects batch files as OEM encoded which means 1 byte per character with an OEM code page according to region setting for current user account. Open a command prompt window and run the command chcp (change code page) without any parameter to get displayed which OEM code page Windows command interpreter works with according to the region you have configured for your account in Windows Region and Language settings.

      Depending on various criteria like version of Windows, font set for console, ... it is possible to use UTF-8 encoding for batch files, but the batch file must change with first line the "code page" to UTF-8 for getting the rest of the batch file correct executed. For details on UTF-8 encoded batch files see Stack Overflow answer on Using another language (code page) in a batch file made for others.

      The two settings Write UTF-8 BOM header to all UTF-8 files when saved and Write UTF-8 BOM on new files created within this program (if above is not set) at Advanced - Settings or Configuration - File handling - Save determine if a new file created as UTF-8 encoded is saved without BOM (byte order mark) or with BOM by default with option Encoding in Save As dialog set to Default. If UTF-8 encoded is used for a batch file although not recommended, the UTF-8 encoded batch file must be saved without BOM. The Windows command interpreter interprets the BOM as first 3 characters of the command line to process and execute resulting always in an error message for the first line in batch file. Most applications don't support UTF-8 encoded files with BOM. For that reason it is best to have both UTF-8 related save settings not checked in configuration of UltraEdit.

      So if you are using UltraEdit mainly for batch file editing, I strongly recommend to select ANSI (1 byte per character) as Default encoding (for new files and file open when auto-detect fails). And I recommend to configure UltraEdit as written in the topics referenced above to edit *.bat and *.cmd files by default using OEM character set (code page) according to your region settings instead of Windows-1254 code page in case of you ever write non ASCII characters into a batch file.

      But if UltraEdit is mainly used to edit UTF-8 encoded HTML and XML files, keep UTF-8 set for Default encoding (for new files and file open when auto-detect fails), but make sure that the settings Write UTF-8 BOM header to all UTF-8 files when saved and Write UTF-8 BOM on new files created within this program (if above is not set) are both not checked. UTF-8 encoded HTML and XML files must be also saved without BOM as there is the charset (HTML, XHTML) meta tag respectively the encoding attribute (XML) in the file which informs all applications interpreting the data about the character encoding in the file. See also my post about supported HTML/XHTML UTF-8 charset and XML UTF-8 encoding declarations.

      As long as a batch file does not contain any character with a code value greater 127 decimal, there is binary no difference between UTF-8 encoded batch file without BOM and an ASCII/ANSI/OEM encoded batch file. The batch file uses in this case just ASCII characters which are encoded identical in all character encodings.

      But if a new file is created by default UTF-8 encoded and the batch file should be OEM encoded as expected by Windows command interpreter on your computer with your region settings because of containing a non ASCII character with a code value greater 127 decimal and being available in OEM code page at all, you have to convert the new UTF-8 encoded batch file with code/text from UTF-8 to ASCII (multi-byte to single-byte per character) with selecting the correct OEM code page.

      It is of course also possible to use a macro stored in a macro file being automatically loaded on startup of UltraEdit which has a hotkey assigned for fast execution by key to create a new batch file with encoding required for batch files. I use such macro to create a new batch file by pressing Ctrl+Shift+B which is always a non Unicode file independent on configuration setting for new files and which is automatically edited with OEM character set enabled once having it saved as *.bat or *.cmd file. The macro code is simple:

      Code: Select all

      InsertMode
      ColumnModeOff
      NewFile
      UnicodeToASCII
      UTF8ToASCII
      UnixMacToDos
      "@echo off
      "
      
      The file created by this macro is always encoded with 1 byte per character with DOS/Windows line endings as long as option Encoding in Save As dialog is set to Default on saving next the new batch file with file extension .bat or .cmd (differs in exit code (errorlevel) handling) to keep the encoding of the new file as set by the macro on save.
      Best regards from an UC/UE/UES for Windows user from Austria

      2
      NewbieNewbie
      2

        Mar 30, 2018#3

        Hi Mofi,

        Thank you for your extensive explanation, you are correct I don't much understand about file encoding schemes but I will check your link to read about the file encoding schemes.

        I will also check out the other links you have provided.

        The two option you have ask me to uncheck was already unchecked. I have changed the Default encoding (for new files and file open when auto-detect fails) to ANSI encoding and this have resolve the problem.

        Thank you very much for your help!