Hi everyone,
This is my first post on an UltraEdit forum. I'm writing a script to take a long excerpt of Unicode text (in Japanese), break it into sentences, and load each sentence as a string into an array. I will process the array later.
Everything works just fine, up until the point when I check the contents of the array. It seems that some of the characters are getting corrupted. I'm guessing there's some sort of formatting issue, but I really don't know how to solve the problem.
First, here's my JavaScript code:
Everything seems to work just fine. However, some characters are corrupted in the process.
For example, it my input text is this:
... the output window shows this:
Any ideas?
This is my first post on an UltraEdit forum. I'm writing a script to take a long excerpt of Unicode text (in Japanese), break it into sentences, and load each sentence as a string into an array. I will process the array later.
Everything works just fine, up until the point when I check the contents of the array. It seems that some of the characters are getting corrupted. I'm guessing there's some sort of formatting issue, but I really don't know how to solve the problem.
First, here's my JavaScript code:
Code: Select all
// Ask the user how each sentence entry ends (typically, this is the Japanese period character)
var strEntryTerminator = UltraEdit.getString("What ends each sentence?",1);
// Report what the user inputted to the debug window
UltraEdit.outputWindow.write("Entry terminator is" + strEntryTerminator);
// Use DOS-style line terminator for Windows Notepad Unicode .txt files
var lineTerminator = "\r\n";
// Establish our search string for the loop condition
UltraEdit.activeDocument.top();
UltraEdit.activeDocument.findReplace.mode=0; //Replace all in current file
UltraEdit.activeDocument.findReplace.replaceAll=true; //Replace all instances
// Remove all line terminators in the file, making the data one continuous line of text
UltraEdit.activeDocument.findReplace.replace(lineTerminator, "");
// Remove multiple spaces (up to ten) so that there is a maximum of one space between text
var SpaceDeletion = 1;
while (SpaceDeletion < 10) {
UltraEdit.activeDocument.findReplace.replace(" ", " ");
SpaceDeletion ++;
}
// Replace a period plus a space with a period (removing leading spaces from entries)
UltraEdit.activeDocument.findReplace.replace(strEntryTerminator + " ", strEntryTerminator);
// Replace a period with a period plus a terminator, which will put each sentence on its own line
UltraEdit.activeDocument.findReplace.replace(strEntryTerminator, strEntryTerminator + lineTerminator);
// Select all data in the document
UltraEdit.activeDocument.selectAll();
// Selection becomes variable
var mySelection = UltraEdit.activeDocument.selection;
// Split lines at lineTerminator and load them into an array
var resultArr = new Array();
resultArr = mySelection.split(lineTerminator);
// Display total number of records in debug window
resultLength = resultArr.length;
UltraEdit.outputWindow.write(resultLength + " total entries");
// Write array values in debug window
for (var i = 0; i < resultArr.length; i++) {
UltraEdit.outputWindow.write("Value: " + i + " \"" + resultArr[i]);
}
For example, it my input text is this:
Code: Select all
ブラックホール（英語：black hole）とは、きわめて高密度で大質量で、きわめて強い重力のために、物質だけでなく光さえも脱出できない天体のこと[1]。
きわめて強い重力のために光さえも抜け出せなくなった時空の領域、とされている。
「ブラック・ホール」（黒い穴）という名は、アメリカの物理学者ジョン・ホイーラーが1967年にこうした天体を呼ぶために編み出した[2]。
それ以前は「collapsar[3] コラプサー」（崩壊した星）などと呼ばれていた。Code: Select all
Running script: C:\Program Files\IDM Computer Solutions\UltraEdit\scripts\JapaneseDocumentToSRS.js
========================================================================================================
Entry terminator is縲・
5 total entries
Value: 0 "・ｽu・ｽ・ｽ・ｽb・ｽN・ｽz・ｽ[・ｽ・ｽ・ｽi・ｽp・ｽ・ｽFblack hole・ｽj・ｽﾆは、・ｽ・ｽ・ｽ・ｽﾟて搾ｿｽ・ｽ・ｽ・ｽx・ｽﾅ大質・ｽﾊで、・ｽ・ｽ・ｽ・ｽﾟて具ｿｽ・ｽ・ｽ・ｽd・ｽﾍのゑｿｽ・ｽﾟに、・ｽ・ｽ・ｽ・ｽ・ｽ・ｽ・ｽ・ｽ・ｽﾅなゑｿｽ・ｽ・ｽ・ｽ・ｽ・ｽ・ｽ・ｽ・ｽE・ｽo・ｽﾅゑｿｽ・ｽﾈゑｿｽ・ｽV・ｽﾌのゑｿｽ・ｽ・ｽ[1]・ｽB
Value: 1 "・ｽ・ｽ・ｽ・ｽﾟて具ｿｽ・ｽ・ｽ・ｽd・ｽﾍのゑｿｽ・ｽﾟに鯉ｿｽ・ｽ・ｽ・ｽ・ｽ・ｽ・ｽ・ｽ・ｽ・ｽ・ｽo・ｽ・ｽ・ｽﾈゑｿｽ・ｽﾈゑｿｽ・ｽ・ｽ・ｽ・ｽ・ｽ・ｽﾌ領茨ｿｽA・ｽﾆゑｿｽ・ｽ・ｽﾄゑｿｽ・ｽ・ｽB
Value: 2 "縲後ヶ繝ｩ繝・け繝ｻ繝帙・繝ｫ縲搾ｼ磯ｻ偵＞遨ｴ・峨→縺・≧蜷阪・縲√い繝｡繝ｪ繧ｫ縺ｮ迚ｩ逅・ｭｦ閠・ず繝ｧ繝ｳ繝ｻ繝帙う繝ｼ繝ｩ繝ｼ縺・967蟷ｴ縺ｫ縺薙≧縺励◆螟ｩ菴薙ｒ蜻ｼ縺ｶ縺溘ａ縺ｫ邱ｨ縺ｿ蜃ｺ縺励◆[2]縲・
Value: 3 "・ｽ・ｽ・ｽ・ｽﾈ前・ｽﾍ「collapsar[3] ・ｽR・ｽ・ｽ・ｽv・ｽT・ｽ[・ｽv・ｽi・ｽ・ｽ・ｽｵゑｿｽ・ｽ・ｽ・ｽj・ｽﾈどと呼ばゑｿｽﾄゑｿｽ・ｽ・ｽ・ｽB
Value: 4 "
