I do not understand why you should get any "out of memory" error on running a Replace in Files.
For testing I copied directory %windir%\SoftwareDistribution to a temporary folder, started UltraEdit and executed a Perl regular expression Replace in Files with search string \x00+, an empty replace string, with in files/types *.*, with directory being the copied directory, with search subdirectories enabled, and with option List changed files not enabled. Of course I did not have any of the binary or text files opened in UltraEdit as this is not necessary for a Replace in Files. After a few seconds a message box was displayed with showing the text "24320025 items replaced in 17 files".
I copied the directory once more, changed search string to \x00 to really remove each null byte separately from each file and enabled option List changed files. After a few seconds the output window opened with information on how many replaces were done on which files and finally showing "33833176 items replaced in 17 files". (Yes, it makes sense to apply the multiplier + on removing null bytes from binary files.)
I used 32-bit UE v220.127.116.11. I did not see any out of memory error message as expected because a Replace in Files does not use much memory at all (no undo recording, no display updates during execution).
If you can reproduce the occurrence of any out of memory error message with 64-bit version of UltraEdit 18.104.22.168 on running a Replace in Files on not opened files, I would suggest to report this as an issue by email to IDM support.
Don't use Find in Files command with a Perl regular expression search string to find binary data bytes. Although the find itself works, the display of found "strings" in output window or results window won't as Find in Files is designed for text searches and not for binary data searches.