Wow... I thought this little excercise would be no problem for "ULTRA"edit. I am using UE 15.
I had to retrieve approx 1 million rows of data (only approx 120 char per record{row}) into a flat file (or files) and email it. It was data by month for a whole year.
Ugh!... but I had to get it done, bosses didn't want to hear why this was stupid.

Data output from SQL2008 created a 65MB csv file.
Zipping the file created an 11MB file. We have a 10MB limit on sending email attachments. So, I thought simple... Suck a copy of the file into Ultraedit find the halfway mark in the file (first 6 month mark) and zap the remainder of the file (last 6 months of data) and save the truncated file to a new file name. This would give me a file for half the year that would zip to less then 10MB. I would do this again for the other half of the data and then do two separate emails of the zipped files. (Not pretty but I was in a just "get'er done" mode with a deadline)
The big 65mb file loaded into Ultraedit. It was pretty slow but it did all load. (Running XP SP3 with 3GB). I did a seek to the bottom of the file (to see the last rows in the file) and everything appeared normal. Approx 1.1 million rows of data were in the file I had loaded in Ultraedit. I found the halfway mark and then selected the 500,000 rows to the bottom of the file. I then deleted them. I then saved this adjusted file to a new file name. The new file was created and the file size of the new file seemed reasonable (it was approx 1/2 the number of bytes of the orginal file). I closed UltraEdit and then re-opened the original 65mb file to re-do the process again but this time creating a file with the 2nd half of the year.
Well... the 65mb original file (which was never saved or adjusted in UE) now only had the half of the data in it. The first half of the year was all that was in the file. I closed UE and tried again... same thing. I opened the orginal file in notepad (that's a long wait) and it only had half the data. So I pulled a fresh copy of the data from the SQL server again to a new CSV file and tried the whole process all over again. Same results.

Conclusion... Ultraedit seems to really choke on files this big.
It doesn't make sense to me.
I seem to remember be able to do this type of work with V11. I may reload an old copy of UE to test it out.
Anyone else have problems editing very large files.

Well, I checked my settings and I did have "Use temporary file for editing (normal operation)" selected under "File Handling - Temporary Files". I tried the whole file editing process again, but this time I looked for the 'temp' files that should have been created when I opened the large csv ile. The temp file was no where to be found. I had the same results as before editing and saving the large csv file. (The orginal file was modifiedl). So I went into UE and went to "File Handling -temporary Files" and selected "Open without temp file and NO PROMPT". I clicked apply and closed UE. I then opened UE went back to "File Handling -temporary Files" and changed it back to "Use temporary file for editing (normal operation)". Clicked apply. Closed UE and then opened it again the tried the whole process again. Things worked but in a weird way. As I said, this time it did work (it created a backup "temp" copy of my BIG file in my temp file location) and it let me edit the BIG file and save the edited (reduced in size version) it to a new name while the original file file remained intact.
Here is what is weird... while I was opening the orginal big csv file I was presented with a message box that said... "The file you're opening is 62.51 MB in size. The exceeds the limit specifed in configuration for temporary file threshold and may reduce editing performance of this file. Please choose one of the options below." Three radio button options were "1) Disable temporary files when opening large files (greater than 47MB) for this edit session only", "2) Disable temporary files when opening large files (greater than 47MB) permanently and don't ask again" or "3) Don't change anything; continue to use temporary files" This last option was pre-selected

Here's my question(s).... (and comments)

It's strange that the "Use temporary file for editing (normal operation)" option was not working, even though it was selected.
It seems that I had to 'reset' it to make it work properly.
I upgraded to UE15 (from UE14) a couple of months ago (I've had UE since V9). Could the upgrade and installation over my previous install could have caused this break?
I haven't done any 'large' file editing in that time.

Next, Where does the 47MB value come from? I have looked at all my configuration settings and I don't see anything that would set that particular value. Nothing is mentioned in Help about this 47MB limit either. If this is a built in warning / constraint, why can't I find any reference to it?

IMHO, these are still unexpected results from UE.

Thanks Mofi... great info...

I guess that power tip link explains the 47MB value.
Interesting though... the message box shown in the power tip (for UE14) is very different from the one I see in UE15

The UE14 power tip message box is phrased as "Do you want to disable the creation of temporary files..." with you having to select 'No' to keep the temporary file option turned on (otherwise they are turned off for the file you are about to open). The new message box has a default value of "Don't change anything; continue to use temporary files" which is a smarter approach.

My configuration preference is to just leave the "Use temporary file for editing (normal operation)" option set.
And not bother with a threshold value.

I don't click on files to open them (in any application) unless I have a clear understanding of what I am opening and what I expect to do with it. In other words, I don't open really big files by mistake or without knowing the impact.

As it turns out, the only hiccup in all this was my UE15 not recognizing my pre-exisiting configuration of "Use temporary file for editing (normal operation)" until I reset it.

Hi Gregsoc,

I had the same thing on 14.20 and reported this to Support: after long trials, I could trace back that I had a corrupted configuration file.

• with that config, UE would simply open large files without temp files without asking (despite the default settings being used). So it would simply overwrite the file on disk without warning.
• with a new, clean config, UE would behave as described by Mofi above

I could never trace back why I had a corrupted config. I just used the various dialogs to change config settings and apparently, along the way, corrupted my config files. So there must be a hidden bug someplace.

I find it also interesting to note that IDM chooses to bring up the "Temporary file handling" dialog box if the user uses the default (Use temporary files). This is VISTA behavior: Are you sure you want to move the mouse cursor to the left?.

Support explained that they did this because they wanted to make customers aware that there are faster ways of dealing with large files, ie., with temp files disabled. So they are targeting people who haven't read the manual. As a punishment, the dialog box does not explain the consequences of opening without temp files.

As if they were concerned about performance: If one opens a file read only, the temp file behavior is the same as if one opens a file for editing. I've yet to understand what the purpose of a temp file is in the case of read only access.

Hi Stefan_E

Good information...

I suspect my configuration file was somehow corrupt... because I had to re-apply the "Use temporary file for editing (normal operation)" option to make it work as expected. That's my only point.

To me, it seems that Mofi is convinced there is some "pure and justified logic" to the how IDM has set this configuration up, but I agree with you, the prompting does seem backwards.

I set the option to 'always use temporary files'... so just do it (and don't bug me again!!). And I am also not going to try and pick some 'magical' threshold number that will determine if I should or shouldn't use temp files. My use of UE is wide and varied, so that number would just become meaningless.

That's interesting about the 'read only' file open feature still trying to use temp files. They must have missed that part of their review on 'performance options' for users. If they are going to have a dialog box, maybe they should make one of the options "open as read only", which would forgot the temp files and just let the user troll through the file. That's mostly what I use UE for when I do open really large files. I am not usally interesting in editing these files. Its mostly for just finding some pattern or string (e.g. in big log files or data dumps from external systems).

Thanks again...

The performance in 15.10 seems to be slower than 14.x by a large margin.

I am using temp files (as I did previously), but now it takes about 40-45 sec to open a 77MB Text file. With 14.x it was noticably faster.

It takes twice as long on a 400 MB file...I've worked with larger files in previous version of UE and it was never this slow.

Any options I can use to speed it up? I don't want to permanently edit the file (too much change of an OOPS).

I've set in the config to use large memory buffers (I am running 64-bit XP with 4GB ram on Dual Core)...

I was just considering updating from 10.10 to 15+, but after reading this and several other recent posts, there doesn't seem to be much point to do so. I occasionally work with large 100+ mb data files to clean up for data analysis programs, and 10.10 may still be a better tool.

I'm still at version 14 and the function list being in a separate thread makes a huge difference for large files. Of course that assumes the large files are in a known language and not plain text, in which case there is no function list. So 14 and 15 should be much faster than 10 for files that aren't just plain text.

Hi Guys,

I am reusing this old discussion as I observed that v16.00.0.1040 (latest) seems not to have a separate thread for the function list. On some (complex) C files I am using, I need to wait the function list to have finish the parsing before being able to edit the file. As the parsing occurs each time I am moving to a different file, it is kind of annoying. I was using v12.10b on my previous machine, which was much faster.

Do you get the same behavior? Any tips?

Thanks a lot

Thanks Mofi for the prompt reply.

No, I can't edit the file while the scan is occuring. I get the hourglass pointer, and the entire UE is locked until the list is populated.

I can also report the slide on the right of the function list window is starting small on the top right corner, and slowly grows while the list is getting populated. Not sure you understand my poor English, but anyway, this is slow and annoying

Thanks