UE v24.00 and v24.10 changes source code in US-ASCII files interpreted false positive as UESC files (fixed)

hoppalla · Jun 18, 2017#12017-06-18T07:22+00:00

I've a lot of .js (jscript) files, stored as US-ASCII only, which contains lines with "\uxxxx" like

var CleanFileName = FileName.replace(/<|>|:|\u0022|\/|\\|\||\?|\*/g, "");

When I open the file in UE 24.10, status bar shows now UESC and changes the source code to

Code: Select all

var CleanFileName = FileName.replace(/<|>|:|"|\/|\\|\||\?|\*/g, "");

and of course saves it so. (The file size of this example is -5 as before.)

This syntax can't be read from the jscript interpreter anymore and results in an error.

To fix the problem I had to disable the setting "File Handling / Encoding / Automatically detect encoding", but then UTF-8 files will not be detected anymore (e.g. my Perl scripts) which is also very bad. (BTW: UTF-16 files will be detected.)

This is a big problem!

UE 24 MUST NOT change the source code. (Not detecting escape-sequences to show encoding!) The programmers must see the exact characters for the interpreter/compiler - \u00022 and not "!

UE 24 must make a difference between the "technical" (Bytes, UTF 8/16/32 - with/without BOM) and "functional" (escape-sequences) detection. Later shouldn't be used in any way of course to avoid errors.

Or did I miss any other setting(s)?

An urgent reply is required because I'm responsible for UE in our company and we just planned the roll-out UE24 (over UE23) to 260 co-workers this week - which it looks like I have to cancel to avoid troubles.

Tested with non public build 24.10.0.28.

Mofi · Jun 18, 2017#22017-06-18T18:12+00:00

I looked on this issue and indeed, UltraEdit loads the file as ASCII Escaped Unicode file.

Auto-detection of ASCII Escaped Unicode files could be disabled in configuration.

In UltraEdit for Windows < v24.00 by unchecking Detect ASCII/ANSI files with escaped Unicode at File Handling - Unicode/UTF-8 Detection in configuration.

In UltraEdit for Windows v24.00.0.42, v24.00.0.45 and v24.00.0.49 by unchecking Detect ANSI files with escaped Unicode at File Handling - Encoding in configuration.

In UltraEdit for Windows v24.00.0.53 and v24.00.0.56 the File Handling - Encoding configuration dialog does not contain anymore that setting, but those versions of UltraEdit still supported Auto Detect UASCII=0 in section [Settings] in INI file of UltraEdit.

But the setting does whether exist in any configuration dialog nor is it supported anymore since UltraEdit for Windows v24.00.0.76. The UltraEdit executables do not contain anymore Auto Detect UASCII since v24.00.0.76.

I don't have any idea why this configuration setting was removed while really rarely needed auto-detection of ASCII Escaped Unicode files is still enabled internally with now no possibility for an UltraEdit user to disable it.

The only possibility to load an ASCII/ANSI file containing Unicode escape sequences among disabling auto-detection of Unicode files without BOM completely is using File - Open command and select ASCII (ansi code page auto detection) for option Encoding for files like this one. This is of course not practicable on daily usage for programmers and script writers working with lots of files containing Unicode escape sequences in search/replace strings in program/script code.

So at the moment I can only recommend not to roll-out UltraEdit v24.10.0.24 this week and send a bug report email to IDM support. I do that too because I'm also interested in why this configuration setting is not available anymore although auto-detection of ASCII Escaped Unicode files is still enabled by default.

hoppalla · Jun 18, 2017#32017-06-18T18:58+00:00

Hi Mofi,

I would need a decision until tomorrow morning, which I have now: Not rolling out this UE version. Thanks. (I've also written to IDM support of course.)

I noticed the problem on my private PC with 24.10.0.28 yesterday - I didn't change my .js files for some month - but yesterday.

I just checked my .ini file and I still have the entry "Auto Detect UASCII=0" (which I always had).

At least UE 24.10.0.28 does not use "Auto Detect UASCII" anymore.

I'm sure they have to bring this setting back.

I got this build ~2 weeks ago from IDM support because of a bug in Restore a backup.

Auch beste Grüße aus Wien - Danke

Mofi · Jul 20, 2017#42017-07-20T16:28+00:00

This issue is fixed with UltraEdit for Windows v24.10.0.32 and UEStudio v17.10.

Some more details about this fix received by email from IDM support 8 days after sending a report by email to IDM support with a quick reply on same day.

The configuration setting Detect ASCII/ANSI files with escaped Unicode respectively Detect ANSI files with escaped Unicode is removed from configuration and there are no plans to add it again to configuration.

But the setting Auto Detect UASCII=0/1 in section [Settings] in INI file is recognized again and disables/enables auto-detection of ASCII Escaped Unicode files. So this configuration setting with default value 0 is supported only as special setting which must be manually added/edited directly in the INI file.