Opening a text file that may be ASCII *or* Unicode

[Andrew Aronoff]

I'm using VBS to open a text file that may be ASCII or Unicode. I
don't know in advance the format of the file. I need to compare the
file, line by line, with an ASCII equivalent stored in the script. If
differences are found in any line, I want to output to a text file the
ASCII version of the line in the text file. Unicode characters without
ASCII equivalents can be represented as a question mark. There will be
very very few such characters in the Unicode version of the text file.

Here's my simple code to open the text file for reading:

Const ForReading = 1
Const TriStateFalse = 0, TriStateTrue = -1, TriStateUseDefault = -2
Set oTextFile = Fso.OpenTextFile (strFileName, ForReading, _
False,TriState_value_here)

If I use TriStateUseDefault, both ASCII and Unicode files can be read
and compared to ASCII, with any differences correctly output to a text
file. I don't understand why. How is TriStateUseDefault able to
correctly distinguish between ASCII and Unicode files when the files
are opened with OpenTextFile?

regards, Andy

[Miyahn]

Post by Andrew Aronoff
How is TriStateUseDefault able to
correctly distinguish between ASCII and Unicode files when the files
are opened with OpenTextFile?

Try this script and see the head 2 bytes of Unicode file.

Dim aBuf(), bBuf(), Size, I
If WScript.Arguments.Count = 0 Then Reg_UnReg: WScript.Quit
If WScript.Arguments.Count > 1 Then WScript.Quit
With CreateObject("Scripting.FileSystemObject")
If Not .FileExists(WScript.Arguments(0)) Then WScript.Quit
End With
With CreateObject("ADODB.Stream")
.Mode = 3: .Type = 1: .Open
.LoadFromFile WScript.Arguments(0)
Size = .Size: ReDim aBuf(Size), bBuf(Size)
For I = 1 To .Size: bBuf(I) = .Read(1): Next
.Close
End With
With CreateObject("InternetExplorer.Application")
.Navigate "about:blank"
' .document.ParentWindow.ResizeTo 500, 700
.Visible = True
aBuf(0) = "<pre>" & vbCrLf
For I = 1 To Size
aBuf(I) = Right("0" & Hex(AscB(bBuf(I))), 2) & " "
If I Mod 16 = 0 Then aBuf(I) = aBuf(I) & vbCrLf
If I Mod 256 = 0 Then _
aBuf(I) = aBuf(I) & "</pre>" & vbCrLf & "<pre>" & vbCrLf
Next
aBuf(Size) = aBuf(Size) & "</pre>"
.document.Write CStr(Join(aBuf, ""))
End With
Sub Reg_UnReg
Const TKey = "HKCR\*\shell\", SKey = "HexDump\"
Const Menu = "HexdecimalDump(&H)"
With CreateObject("WScript.Shell")
On Error Resume Next
.RegRead TKey & SKey
If Err Then
On Error GoTo 0
.RegWrite Tkey & SKey, Menu
.RegWrite Tkey & SKey & "command\", _
"wscript """ & WScript.ScriptFullName & """ ""%L"""
.PopUp "Added to context menu", 1
Else
On Error GoTo 0
.RegDelete Tkey & SKey & "command\"
.RegDelete Tkey & SKey
.PopUp "Deleted from context menu", 1
End If
End With
End Sub

[Andrew Aronoff]

Thanks, Miyahn, for that information. I did not know that the top two
bytes of a Unicode text file serve as a format code.

I still have a number of questions.

1. My strategy is to open the text file (which may be in ASCII or
Unicode format) with TriStateUseDefault and then compare the opened
file, line by line, with an ASCII version stored in the script. Can
you confirm that opening with TriStateUseDefault will convert the
Unicode to ASCII whenever possible?

That is:

open text file with TriStateUseDefault
read Unicode line
compare to stored ASCII line
if ASCII equivalent of Unicode = stored ASCII, compare will succeed

2. If my strategy will work, *why* does it work? Just how does
TriStateUseDefault work?

3. What's the disadvantage, if any, of using TriStateUseDefault?

4. Why isn't TriStateUseDefault used exclusively whenever opening a
text file?

5. How does TriStateUseDefault handle a Unicode character that has no
ASCII equivalent? (I admit I don't even know how to "pollute" a
Unicode text file with such a character to test it.)

regards, Andy

[Miyahn]

1. Can you confirm that opening with TriStateUseDefault will convert the
Unicode to ASCII whenever possible?

No. Not at all.

2. If my strategy will work, *why* does it work? Just how does
TriStateUseDefault work?

NA.

3. What's the disadvantage, if any, of using TriStateUseDefault?

I don't know.
There might be some overhead at reading process.

4. Why isn't TriStateUseDefault used exclusively whenever opening a
text file?

Ask to MS.

5. How does TriStateUseDefault handle a Unicode character that has no
ASCII equivalent?

I think, only read in to the string buffer without any conversion.

[Andrew Aronoff]

Post by Miyahn

1. Can you confirm that opening with TriStateUseDefault will convert the
Unicode to ASCII whenever possible?

No. Not at all.

Except that's what I'm seeing. The Unicode file opened with
TriStateUseDefault is comparing perfectly with plain ASCII stored in
the script.

Which brings up my second question again:

2. If my strategy will work, *why* does it work? Just how does
TriStateUseDefault work?

Post by Miyahn

4. Why isn't TriStateUseDefault used exclusively whenever opening a
text file?

Ask to MS.

I know you don't work for MS, but by posting here, I was hoping that
MS might answer. I'm still hoping... ;-)

I'd still like to know how TriStateUseDefault works and the best way
to use OpenTextFile when the format of the text file (Unicode or
ASCII) is not known in advance:
- Open as TriStateUseDefault and handle as ASCII?
- Open as Unicode and read the first two bytes, then close and reopen
as ASCII if the file is not Unicode?
- Something else?

BTW, I googled this and came up ignorant. If this is documented
somewhere, please provide a link.

regards, Andy

[Miyahn]

Post by Andrew Aronoff

Post by Miyahn

1. Can you confirm that opening with TriStateUseDefault will convert the
Unicode to ASCII whenever possible?

No. Not at all.

Except that's what I'm seeing. The Unicode file opened with
TriStateUseDefault is comparing perfectly with plain ASCII stored in
the script.

I think that 'plain ASCII stored in the script' isn't 'plain ASCII'.

Try this.

strASCII = "abc"
For I = 1 To LenB(strASCII)
Buf = Buf & " " & Right("0" & Hex(AscB(MidB(strASCII, I, 1))), 2)
Next
MsgBox Mid(Buf, 2)

[Andrew Aronoff]

Excellent observation! VBS appears to do manage internal strings in
Unicode. I didn't know that.

My question, then, should be rephrased as follows:

1. Can you confirm that opening with TriStateUseDefault will convert
ASCII to Unicode whenever possible?

2. If my strategy will work, *why* does it work? Just how does
TriStateUseDefault work?

Do you agree now, at least, with my questions? ;-)

Thanks for your help.

regards, Andy

Post by Miyahn

Post by Andrew Aronoff

Post by Miyahn

1. Can you confirm that opening with TriStateUseDefault will convert the
Unicode to ASCII whenever possible?

No. Not at all.

Except that's what I'm seeing. The Unicode file opened with
TriStateUseDefault is comparing perfectly with plain ASCII stored in
the script.

I think that 'plain ASCII stored in the script' isn't 'plain ASCII'.
Try this.
strASCII = "abc"
For I = 1 To LenB(strASCII)
Buf = Buf & " " & Right("0" & Hex(AscB(MidB(strASCII, I, 1))), 2)
Next
MsgBox Mid(Buf, 2)

[Andrew Aronoff]

Since I can't find any documentation about TriStateUseDefault, I
decided to open the file in ASCII; read the first two characters;
close the file; compare those characters to 255 & 254; if true, open
in Unicode, otherwise open in ASCII.


Const ForReading = 1
Const TriStateFalse_ASCII = 0, TriStateTrue_Unicode = -1

'strFileName points to a text file in ASCII or Unicode
Set oTextFile = Fso.OpenTextFile (strFileName, ForReading, _
False,TriStateFalse_ASCII)

'read 1st 2 chrs, find Asc chr code
intAsc1Chr = Asc(oTextFile.Read(1))
intAsc2Chr = Asc(oTextFile.Read(1))

oTextFile.Close

If intAsc1Chr = 255 And intAsc2Chr = 254 Then

'open the file in Unicode
Set oTextFile = Fso.OpenTextFile (strFileName,ForReading, _
False,TriStateTrue_Unicode)

Else

'open the file in ASCII
Set oTextFile = Fso.OpenTextFile (strFileName,ForReading, _
False,TriStateFalse_ASCII)

End If


It's not elegant, but it seems to work.

regards, Andy

[Joe Earnest]

Hi Andrew,

[top post]

Michael Harris posted something very similar back in 2001. AFAIK, this is
is the only way to determine status prior to opening.

Here's MH's post from 2001

http://groups-beta.google.com/group/microsoft.public.scripting.vbscript/browse_frm/thread/628f93f8430000a5/66d14306ff6c925c?q=unicode+255+254+group:microsoft.public.scripting.*+author:Michael+author:Harris&rnum=1&hl=en#66d14306ff6c925c

There is a fly in the ointment, however, even in MH's post. There are at
least 5 different unicode BOMs, that signal how the file is interpreted --

UTF-8: EF BB BF
UTF-16, Big-Endian: FE FF
UTF-16, Little-Endian: FF FE
UTF-32, Big-Endian: 00 00 FE FF
UTF-32, Little-Endian: FF FE 00 00

Your technique catches the two most common for Western users. Once you've
opened your file, however, it's neither time-consuming nor much additional
code, to read the first four bytes and test for all of these. I have a WSC
routine that's called by a file open-for-reading method to do that. (A
small suggestion for your posted code, either error-trap or get the file
size first, to insure that file contains the appropriate number of bytes
that you're reading. It could well be ASCII empty -- no bytes.)

You might want to take a look at these --

UTF & BOM
http://www.unicode.org/faq/utf_bom.html
(for the BOM table, scroll down to the Byte Order Mark heading)

Joel Spolsky, The Absolute Minimum Every Software Developer Absolutely,
Positively Must Know About Unicode and Character Sets (No Excuses!)
http://www.joelonsoftware.com/articles/Unicode.html
(thanks to mayayana for this one)

Regards,
Joe Earnest

Post by Andrew Aronoff
Since I can't find any documentation about TriStateUseDefault, I
decided to open the file in ASCII; read the first two characters;
close the file; compare those characters to 255 & 254; if true, open
in Unicode, otherwise open in ASCII.
Const ForReading = 1
Const TriStateFalse_ASCII = 0, TriStateTrue_Unicode = -1
'strFileName points to a text file in ASCII or Unicode
Set oTextFile = Fso.OpenTextFile (strFileName, ForReading, _
False,TriStateFalse_ASCII)
'read 1st 2 chrs, find Asc chr code
intAsc1Chr = Asc(oTextFile.Read(1))
intAsc2Chr = Asc(oTextFile.Read(1))
oTextFile.Close
If intAsc1Chr = 255 And intAsc2Chr = 254 Then
'open the file in Unicode
Set oTextFile = Fso.OpenTextFile (strFileName,ForReading, _
False,TriStateTrue_Unicode)
Else
'open the file in ASCII
Set oTextFile = Fso.OpenTextFile (strFileName,ForReading, _
False,TriStateFalse_ASCII)
End If
It's not elegant, but it seems to work.
regards, Andy
--
**********
Please send e-mail to: usenet (dot) post (at) aaronoff (dot) com
To identify everything that starts up with Windows, download
"Silent Runners.vbs" at www.silentrunners.org
**********

[Andrew Aronoff]

Thanks, Joe, for your response and the link to Michael's post. I
appreciate your suggestions.

I still have an underlying question -- why does TriStateUseDefault
work when opening ASCII and Unicode files? What, exactly, does
TriStateUseDefault *do*? I've only read that it follows the system
default, but how does one set that in Windows? What is the
disadvantage, if any, of using TriStateUseDefault to open a text file?

regards, Andy

Post by Joe Earnest
Hi Andrew,
[top post]
Michael Harris posted something very similar back in 2001. AFAIK, this is
is the only way to determine status prior to opening.
Here's MH's post from 2001
http://tinyurl.com/ctzho
There is a fly in the ointment, however, even in MH's post. There are at
least 5 different unicode BOMs, that signal how the file is interpreted --
UTF-8: EF BB BF
UTF-16, Big-Endian: FE FF
UTF-16, Little-Endian: FF FE
UTF-32, Big-Endian: 00 00 FE FF
UTF-32, Little-Endian: FF FE 00 00
Your technique catches the two most common for Western users. Once you've
opened your file, however, it's neither time-consuming nor much additional
code, to read the first four bytes and test for all of these. I have a WSC
routine that's called by a file open-for-reading method to do that. (A
small suggestion for your posted code, either error-trap or get the file
size first, to insure that file contains the appropriate number of bytes
that you're reading. It could well be ASCII empty -- no bytes.)
You might want to take a look at these --
UTF & BOM
http://www.unicode.org/faq/utf_bom.html
(for the BOM table, scroll down to the Byte Order Mark heading)
Joel Spolsky, The Absolute Minimum Every Software Developer Absolutely,
Positively Must Know About Unicode and Character Sets (No Excuses!)
http://www.joelonsoftware.com/articles/Unicode.html
(thanks to mayayana for this one)
Regards,
Joe Earnest

Post by Andrew Aronoff
Since I can't find any documentation about TriStateUseDefault, I
decided to open the file in ASCII; read the first two characters;
close the file; compare those characters to 255 & 254; if true, open
in Unicode, otherwise open in ASCII.
Const ForReading = 1
Const TriStateFalse_ASCII = 0, TriStateTrue_Unicode = -1
'strFileName points to a text file in ASCII or Unicode
Set oTextFile = Fso.OpenTextFile (strFileName, ForReading, _
False,TriStateFalse_ASCII)
'read 1st 2 chrs, find Asc chr code
intAsc1Chr = Asc(oTextFile.Read(1))
intAsc2Chr = Asc(oTextFile.Read(1))
oTextFile.Close
If intAsc1Chr = 255 And intAsc2Chr = 254 Then
'open the file in Unicode
Set oTextFile = Fso.OpenTextFile (strFileName,ForReading, _
False,TriStateTrue_Unicode)
Else
'open the file in ASCII
Set oTextFile = Fso.OpenTextFile (strFileName,ForReading, _
False,TriStateFalse_ASCII)
End If
It's not elegant, but it seems to work.
regards, Andy

[Joe Earnest]

Hi,

[top post]

I have always heard that 2k/Me/XP/2k3 default to unicode and other OSs
default to ANSI. I've always set the flag specifically, to avoid trying to
figure out exactly what the devil System Default means.

If you open Control Panel/Regional and Language Options and look under the
Advanced tab, you'll find a number of obscure checkboxes that few of us
(certainly not me) are competent to set. Among them are UTF checkboxes, as
well page code checkboxes. I have always assumed that these somehow figure
into both the default text/unicode flag and the actual unicode BOM that is
written by the system.

There, that should give you not only no real additional information, but
conflicting supposition. What a deal. :-{

MH seems to be around today, maybe he can give a more definite answer.

Regards,
Joe Earnest

Post by Andrew Aronoff
Thanks, Joe, for your response and the link to Michael's post. I
appreciate your suggestions.
I still have an underlying question -- why does TriStateUseDefault
work when opening ASCII and Unicode files? What, exactly, does
TriStateUseDefault *do*? I've only read that it follows the system
default, but how does one set that in Windows? What is the
disadvantage, if any, of using TriStateUseDefault to open a text file?
regards, Andy

Post by Joe Earnest
Hi Andrew,
[top post]
Michael Harris posted something very similar back in 2001. AFAIK, this is
is the only way to determine status prior to opening.
Here's MH's post from 2001
http://tinyurl.com/ctzho
There is a fly in the ointment, however, even in MH's post. There are at
least 5 different unicode BOMs, that signal how the file is interpreted --
UTF-8: EF BB BF
UTF-16, Big-Endian: FE FF
UTF-16, Little-Endian: FF FE
UTF-32, Big-Endian: 00 00 FE FF
UTF-32, Little-Endian: FF FE 00 00
Your technique catches the two most common for Western users. Once you've
opened your file, however, it's neither time-consuming nor much additional
code, to read the first four bytes and test for all of these. I have a WSC
routine that's called by a file open-for-reading method to do that. (A
small suggestion for your posted code, either error-trap or get the file
size first, to insure that file contains the appropriate number of bytes
that you're reading. It could well be ASCII empty -- no bytes.)
You might want to take a look at these --
UTF & BOM
http://www.unicode.org/faq/utf_bom.html
(for the BOM table, scroll down to the Byte Order Mark heading)
Joel Spolsky, The Absolute Minimum Every Software Developer Absolutely,
Positively Must Know About Unicode and Character Sets (No Excuses!)
http://www.joelonsoftware.com/articles/Unicode.html
(thanks to mayayana for this one)
Regards,
Joe Earnest

Post by Andrew Aronoff
Since I can't find any documentation about TriStateUseDefault, I
decided to open the file in ASCII; read the first two characters;
close the file; compare those characters to 255 & 254; if true, open
in Unicode, otherwise open in ASCII.
Const ForReading = 1
Const TriStateFalse_ASCII = 0, TriStateTrue_Unicode = -1
'strFileName points to a text file in ASCII or Unicode
Set oTextFile = Fso.OpenTextFile (strFileName, ForReading, _
False,TriStateFalse_ASCII)
'read 1st 2 chrs, find Asc chr code
intAsc1Chr = Asc(oTextFile.Read(1))
intAsc2Chr = Asc(oTextFile.Read(1))
oTextFile.Close
If intAsc1Chr = 255 And intAsc2Chr = 254 Then
'open the file in Unicode
Set oTextFile = Fso.OpenTextFile (strFileName,ForReading, _
False,TriStateTrue_Unicode)
Else
'open the file in ASCII
Set oTextFile = Fso.OpenTextFile (strFileName,ForReading, _
False,TriStateFalse_ASCII)
End If
It's not elegant, but it seems to work.
regards, Andy

--
**********
Please send e-mail to: usenet (dot) post (at) aaronoff (dot) com
To identify everything that starts up with Windows, download
"Silent Runners.vbs" at www.silentrunners.org
**********

[Michael Harris (MVP)]

Post by Joe Earnest
MH seems to be around today, maybe he can give a more definite answer.

No really, you seem to be doing pretty good ;-)...

My *guess* is that TriStateDefault *may* cause the textstream open methods
to sniff the data when opening existing text files.

[Andrew Aronoff]

Thanks for clarifying that you really couldn't clarify. ;-)

Post by Joe Earnest
If you open Control Panel/Regional and Language Options and look under the
Advanced tab, you'll find a number of obscure checkboxes

Actually, I think the meaning of these checkboxes was explained in
Joel Spolsky's article. These are codepages that the system is
configured to interpret. I do not believe the checked boxes set the
"System Default". Of course, I may be wrong and actually believe this.
;-)

There is also a "Set Default" button. However, my default is English
and, as I stated previously, TriStateUseDefault successfully
differentiates here between an ANSII and Unicode text file when used
in an OpenTextFile statement (reinforcing Michael's definitively
hesitant pronouncement).

I'll have my scripts read the first two or four bytes and take
appropriate action in a subsequent OpenTextFile.

Thanks again.

regards, Andy

Post by Joe Earnest
Hi,
[top post]
I have always heard that 2k/Me/XP/2k3 default to unicode and other OSs
default to ANSI. I've always set the flag specifically, to avoid trying to
figure out exactly what the devil System Default means.
If you open Control Panel/Regional and Language Options and look under the
Advanced tab, you'll find a number of obscure checkboxes that few of us
(certainly not me) are competent to set. Among them are UTF checkboxes, as
well page code checkboxes. I have always assumed that these somehow figure
into both the default text/unicode flag and the actual unicode BOM that is
written by the system.
There, that should give you not only no real additional information, but
conflicting supposition. What a deal. :-{
MH seems to be around today, maybe he can give a more definite answer.
Regards,
Joe Earnest

Post by Andrew Aronoff
Thanks, Joe, for your response and the link to Michael's post. I
appreciate your suggestions.
I still have an underlying question -- why does TriStateUseDefault
work when opening ASCII and Unicode files? What, exactly, does
TriStateUseDefault *do*? I've only read that it follows the system
default, but how does one set that in Windows? What is the
disadvantage, if any, of using TriStateUseDefault to open a text file?
regards, Andy

Post by Joe Earnest
Hi Andrew,
[top post]
Michael Harris posted something very similar back in 2001. AFAIK, this is
is the only way to determine status prior to opening.
Here's MH's post from 2001
http://tinyurl.com/ctzho
There is a fly in the ointment, however, even in MH's post. There are at
least 5 different unicode BOMs, that signal how the file is interpreted --
UTF-8: EF BB BF
UTF-16, Big-Endian: FE FF
UTF-16, Little-Endian: FF FE
UTF-32, Big-Endian: 00 00 FE FF
UTF-32, Little-Endian: FF FE 00 00
Your technique catches the two most common for Western users. Once you've
opened your file, however, it's neither time-consuming nor much additional
code, to read the first four bytes and test for all of these. I have a WSC
routine that's called by a file open-for-reading method to do that. (A
small suggestion for your posted code, either error-trap or get the file
size first, to insure that file contains the appropriate number of bytes
that you're reading. It could well be ASCII empty -- no bytes.)
You might want to take a look at these --
UTF & BOM
http://www.unicode.org/faq/utf_bom.html
(for the BOM table, scroll down to the Byte Order Mark heading)
Joel Spolsky, The Absolute Minimum Every Software Developer Absolutely,
Positively Must Know About Unicode and Character Sets (No Excuses!)
http://www.joelonsoftware.com/articles/Unicode.html
(thanks to mayayana for this one)
Regards,
Joe Earnest

Post by Andrew Aronoff
Since I can't find any documentation about TriStateUseDefault, I
decided to open the file in ASCII; read the first two characters;
close the file; compare those characters to 255 & 254; if true, open
in Unicode, otherwise open in ASCII.
Const ForReading = 1
Const TriStateFalse_ASCII = 0, TriStateTrue_Unicode = -1
'strFileName points to a text file in ASCII or Unicode
Set oTextFile = Fso.OpenTextFile (strFileName, ForReading, _
False,TriStateFalse_ASCII)
'read 1st 2 chrs, find Asc chr code
intAsc1Chr = Asc(oTextFile.Read(1))
intAsc2Chr = Asc(oTextFile.Read(1))
oTextFile.Close
If intAsc1Chr = 255 And intAsc2Chr = 254 Then
'open the file in Unicode
Set oTextFile = Fso.OpenTextFile (strFileName,ForReading, _
False,TriStateTrue_Unicode)
Else
'open the file in ASCII
Set oTextFile = Fso.OpenTextFile (strFileName,ForReading, _
False,TriStateFalse_ASCII)
End If
It's not elegant, but it seems to work.
regards, Andy