There's a bug with the built-in Base64 encoding/decoding routines in UltraEdit 16.0.1038. I assume the bug also exists in previous versions, but I haven't tested it.

The encoding/decoding routines work fine when performed on ASCII plain text. But if you try encoding some Unicode text, and then try to decode it, the bug becomes apparent.

Mofi wrote:Can you explain the problem more detailed with a step by step list how to reproduce it because I can't reproduce it using UE v16.00.0.1038.

1) Create a new file in UltraEdit. Then go to the File Menu ---> Conversions ---> ASCII to Unicode
2) Type the word "hello" into the editor
3) Highlight the word with your keyboard or mouse
4) Edit Menu ---> Encode Base64
5) Now highlight the encoded text
5) Edit Menu ---> Decode Base64

The decoded text will be exactly the same as what you started with. So as you can see, normal ASCII characters are encoded correctly.


Now let's try encoding something with non-ASCII characters.

1) Erase all the text in your editor window
2) Copy and paste the word "привет" into your editor window
3) Highlight the word with your keyboard or mouse
4) Edit Menu ---> Encode Base64
5) Now highlight the encoded text
5) Edit Menu ---> Decode Base64

As you can see, the decoded text is not what you started with, and therefore we can see that the problem is with the handling of Base64 encoding of non-ASCII text.

Done. And I received a response from IDM to let me know that they're able to reproduce the problem, and they're working on fixing it.

I looked on this issue with UE v17.00 and word привет is still not got back after encoding/decoding it with Base64.

UTF-16 encoded привет is in hexadecimal 3F 04 40 04 38 04 32 04 35 04 42 04. Omitting the high bytes with value 04, those bytes would be in ASCII ?@825B. Using Encode Base64 on UTF-16 string привет results in P0A4MjVC. Using Encode Base64 on ASCII string ?@825B results also in P0A4MjVC.

So I wanted to know more about Base64 encoding and read (not entirely) the wikipedia article about Base64. According to this article Base64 encoding is only for ASCII strings (single byte strings). The UTF-7 encoding is needed for encoding UTF-16 characters which is called also modified Base64.

Now the question is, which Base64 encoding is implemented in UltraEdit at all, the standard Base64 or the modified Base64? It looks like standard Base64.

Of course, standard Base64 encoding can be used for binary files. Therefore the UTF-16 characters could be read as binary array like in hex edit mode and therefore encoding UTF-16 characters with standard Base64 encoding is also possible. But why was UTF-7 encoding introduced if standard Base64 can be used also by reading the UTF-16 strings as binary data stream?

I don't want to read all the RFCs to get an answer on that question because for my work with UltraEdit that is totally unimportant. But perhaps my post is an explanation why Base64 encoding is not working for UTF-16 characters in UltraEdit.

Mofi wrote: I don't want to read all the RFCs to get an answer on that question because for my work with UltraEdit that is totally unimportant. But perhaps my post is an explanation why Base64 encoding is not working for UTF-16 characters in UltraEdit.

You're over-complicating this issue. It has nothing to do with "standard" or "modified" Base64 encoding, and reading every RFC in the world will not help.
The bug in UltraEdit is the result of an improper pointer operation in the string handling routine. The bug is present not only for UTF-16 strings, but also when you specifically tell UltraEdit to produce UTF-8 (on the File menu).

I use Base64 encoding in some of my applications which process string values, and the original unmodified Base64 routines from 20 years ago work perfectly on both UTF-8 and UTF-16 strings. The key thing to making it work properly is to account for the correct character size/length (and therefore the correct pointer operation) when referencing them. When I started encoding Unicode text in my programs a couple of years ago, I had the exact same problem which is currently in UltraEdit. I fixed it without reading any RFCs. The only thing I did was some good old-fashioned debugging and some trial-and-error.

I had hoped it would be fixed in UltraEdit by now, but I guess the programmers are too busy doing much more important things like replacing icons and implementing a new serial number registration system.

I'd like to include PNG files in an HTML file according to the data URI scheme. To do this, I need to convert binary PNG files as base64 strings. This is how I try do do this (using UltraEdit 20 on WinXP SP3 32-bit):
1. Open the file reddot.png with UE.
2. Switch from hex mode to text mode.
3. Mark the text, use encode to base64.

It should yield:
iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAGUlEQVQY02NgAIL/QATDDOgCcAniBbGZCQCVRCnXEyWCYwAAAABJRU5ErkJggg==

While it is:
iVBORw0KGgogICANSUhEUiAgIAUgICAFCAYgICCNbyblICAgHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBCAO9TXL0Y4OHyAgICBJRU5ErkJggg==

Can anyone help me with the process of correctly encoding to base64? Or is the bug still unfixed?

Thanks in advance
Mike

Aw, shucks! Thanks for clarifying. I've been assuming that inlining binary files into text like files such as HTML was the major application of base64 conversions, so UltraEdit could surely do this and I was simply missing the crucial step or something.

Thanks again
Mike