I do not understand why to convert in a UTF-8 encoded HTML/XHTML/XML file those characters to HTML entities in hexadecimal notation.

I suggest reading the forum topic Script to convert special characters to HTML code. There is also a ZIP file with two scripts to convert all characters in a file or just in a selection to named HTML entities. You can replace the named entities by the corresponding entities in hexadecimal notation.

For the reverse conversion just exchange the search and replace strings, i. e. asHtml5Entities[nEntityIndex+1],asHtml5Entities[nEntityIndex] instead of asHtml5Entities[nEntityIndex],asHtml5Entities[nEntityIndex+1] with just moved +1 from the replace to the search string.

The attached ZIP file contains the revised script working with UltraEdit for Windows v2024.0.0.35 on the input test file for both conversions.

The corrections are:

1. The definitions of the functions hexDecimalToUnicode and unicodeToHexDecimal are moved from inside the main code to above main code. That makes the code better readable in my opinion. The two replace functions are still defined as anonymous functions inside these two functions. I would define them in my script also outside of the two functions as I do not like anonymous functions and function definitions inside functions, but it is your script.
2. The function parameter text of these two functions is renamed to textToConvert for not getting this function parameter false positive syntax highlighted as known keyword. The name is also better in my opinion as it describes what is done with the string passed to the functions.
3. The list of entity strings to ignore is reduced to those strings from your list which can be matched by the regular expression in the replace function at all.
4. The regular expression for finding entities in hexadecimal notation was wrong for the further processing in the anonymous replace function. There was passed to the anonymous replace function just the four hexadecimal digits because of the marking round brackets if an entity in hexadecimal notation with exactly four digits was found by the replace function. The entire entity with &#x at the beginning, the hexadecimal digits, and the semicolon at the end must be passed to the anonymous replace function for being correct processed further.
5. The code of the anonymous replace function for converting an entity in hexadecimal notation to the appropriate Unicode encoded character is reduced to a minimum. More is not necessary.
6. char was defined in the ECMAScript specifications 1 till 3 as reserved keyword for future use as it can be read at Lexical grammar - Keywords. That is the reason for being syntax highlighted as keyword by the wordfile javascript.uew of UltraEdit. It is still recommended not using the word char as variable name or function parameter name, especially if this is a completely wrong name for what is passed to the anonymous replace function of function hexDecimalToUnicode.
   BTW: You see here why anonymous functions are ugly as it is hard to describe in which anonymous function is something to change or was changed by another developer.
   The names hexEntity and nonAsciiChar are definitely better for every reader of the code.
7. Lines in main code not needed at all are removed from main code.
8. There is used now UltraEdit.getString instead of UltraEdit.getValue for prompting the user for the type of conversion as in this case nothing must be first deleted like the 0 on using  UltraEdit.getValue.
9. There is checked first if the user has really entered 1 or 2 before making anything with active file which gives the script user the possibility to cancel the script execution on having it started by mistake by entering nothing or anything other than 1 or 2.
10. A text file being not UTF-8, UTF-16 LE or UTF-16 BE encoded must be explicitly converted from non Unicode to UTF-8. The encoding property of an UltraEdit document object is a read-only property as described in help of UltraEdit on page Scripting commands. It is not possible to assign a different integer value to this property and expect that this results in a conversion of the characters in the entire file.

UEStudio v12.20.0.1004 is not full Unicode aware. This version of UEStudio does not use wide character arrays for JavaScript strings (= UTF-16 encoded strings) like UltraEdit for Windows v24.00 and UEStudio v17.00 and newer versions. The conversion with a version of UEStudio from 2012 must be done as my script AllToHtmlEntities.js do it by running Perl regular expressions on active file instead of loading the entire file contents as JavaScript string into memory of the JavaScript engine embedded in UEStudio v12.20.0.1004. The topic UltraEdit.clipboardContent not supporting Chinese characters? has more detailed information about working with scripts on Unicode encoded files with a version of UltraEdit for Windows < 24.00 and a version of UEStudio < v17.00.