This is not a configuration issue. It is a bug of UEStudio v17.20 as I have just found out after reading your post. UEStudio < v17.20 don't have this issue. The ANSI encoded output of JavaScript Lint is interpreted by UES v17.20.0.xx UTF-16 LE encoded which results in unreadable text in output window. I reported this issue by email to IDM support.

There is a workaround for this issue of UES v17.20 by configuring a user tool doing the same as UEStudio does on executing Javascript Lint.

On ribbon tab Advanced click in first group Tools on item User tools and click in opened popup menu on Configure tools. Click on button Insert and configure the user tool as follows:

Tab Command:

Menu item name: JS Lint
Command line: "C:\Program Files\IDM Computer Solutions\UEStudio\tools\jsl\jsl.exe" -conf "%Env:APPDATA\IDMComp\UEStudio\tools\jsl\jsl.default.conf" -process "%f"
Working directory: %p
Toolbar bitmap/icon (file path): empty

Tab Options:

Program type: DOS program selected
Save active file: checked
Save all files first: not checked

Tab Output:

Command output: Output list box selected
Show DOS box: not checked
Capture output: checked
Replace selected text with: No replace selected
Handle output as Unicode: not checked

Click on button OK.

The red formatted path to program files folder of UEStudio must be adapted to your installation.

The user tool JS Lint can be executed now for active JavaScript file by pressing Ctrl+Shift+1 (default hotkey for executing first user tool) or clicking on ribbon tab Advanced on item User tools and next on JS Lint.

Update: This bug was fixed with UES v18.00.0.04. So the workaround using a user tool is necessary only for UEStudio v17.20.