Interesting Textstream limit

[Mayayana]

A tidbit of info for future reference:
I've just been dealing with parsing a CSV file that's 173 MB.
I needed to adjust each line and write a new file. The new file
is 82 MB. Everything worked fine. I split the CSV into a
string array, wrote each new line to a new array, then used
Join to make a string. But when I tried to write it to disk I
kept getting an out of memory error. Textstream just
couldn't do it. I ended up opening the file for appending, then
wrote 1 MB at a time from the string until it was all written
to disk. That worked fine, even though Textstream.Write the82MBstring
failed. If anyone knows what the actual limit is I'd be curious
to know.

[Auric__]

Post by Mayayana
I've just been dealing with parsing a CSV file that's 173 MB.
I needed to adjust each line and write a new file. The new file
is 82 MB. Everything worked fine. I split the CSV into a
string array, wrote each new line to a new array, then used
Join to make a string. But when I tried to write it to disk I
kept getting an out of memory error. Textstream just
couldn't do it. I ended up opening the file for appending, then
wrote 1 MB at a time from the string until it was all written
to disk. That worked fine, even though Textstream.Write the82MBstring
failed. If anyone knows what the actual limit is I'd be curious
to know.

I would guess 64MB. Try your script with the output size doubled on each run
and see where it fails.

[Mayayana]

|> If anyone knows what the actual limit is I'd be curious
| > to know.
|
| I would guess 64MB. Try your script with the output size doubled on each
run
| and see where it fails.
|

Yes, obviously I could spend the time to work that out,
but the point of asking is so that I don't have to. :)

I don't know any reason that 64 million should be a
particular limit. A 16-bit number goes up to 65000+.
A 32-bit number goes up to about 4 billion (or 2 billion
if it's a signed data type). It could have something to do
with WScript limits, I suppose. I looked online but so
far haven't found anything.

[Dave "Crash" Dummy]

|> If anyone knows what the actual limit is I'd be curious | > to
know. | | I would guess 64MB. Try your script with the output size
doubled on each run | and see where it fails. |
Yes, obviously I could spend the time to work that out, but the point
of asking is so that I don't have to. :)
I don't know any reason that 64 million should be a particular limit.
A 16-bit number goes up to 65000+. A 32-bit number goes up to about 4
billion (or 2 billion if it's a signed data type). It could have
something to do with WScript limits, I suppose. I looked online but
so far haven't found anything.

Since I don't have a life, I did some fooling around. I had no problem
creating 268,435,456 byte string and writing it to file with a single
operation.
'--------------test.vbs-----------
str="X"
for n=1 to 28
str=str&str
next
msgbox len(str)
set fso=CreateObject("Scripting.FileSystemObject")
set oFile=fso.CreateTextFile("string.txt")
oFile.write str
'-----------------------------------

[Dave "Crash" Dummy]

Footnote:
I have 4 GB of RAM. I notice that you are using Outlook Express and
therefore presumably running XP. You are maybe limited by the OS and
available RAM, not VBScript.

[Mayayana]

Yours gives me the same out of memory error. I have
4 GB RAM, which is really something like 3.2 GB, but still
far more than enough for the job. I am on XP32. Maybe
the limit is with 32-bit WScript. But I still can't think of
any reason for a limit under 2 GB.

|
| Since I don't have a life, I did some fooling around. I had no problem
| creating 268,435,456 byte string and writing it to file with a single
| operation.
| '--------------test.vbs-----------
| str="X"
| for n=1 to 28
| str=str&str
| next
| msgbox len(str)
| set fso=CreateObject("Scripting.FileSystemObject")
| set oFile=fso.CreateTextFile("string.txt")
| oFile.write str
| '-----------------------------------
| --
| Crash
|
| Today is the first day of the rest of your life,
| and there's not a damned thing you can do about it.

[Dave "Crash" Dummy]

Post by Mayayana
Yours gives me the same out of memory error. I have
4 GB RAM, which is really something like 3.2 GB, but still
far more than enough for the job. I am on XP32. Maybe
the limit is with 32-bit WScript. But I still can't think of
any reason for a limit under 2 GB.

That could be. I am running W7 x64.

[GS]

Post by Mayayana
I've just been dealing with parsing a CSV file that's 173 MB.
I needed to adjust each line and write a new file. The new file
is 82 MB. Everything worked fine. I split the CSV into a
string array, wrote each new line to a new array, then used
Join to make a string. But when I tried to write it to disk I
kept getting an out of memory error. Textstream just
couldn't do it. I ended up opening the file for appending, then
wrote 1 MB at a time from the string until it was all written
to disk. That worked fine, even though Textstream.Write the82MBstring
failed. If anyone knows what the actual limit is I'd be curious
to know.

I assume you mean you wrote back in 'blocks' or 'chunks' of text! I've
been trying to 'collect' code samples for doing exactly that in as
efficient a manner as possible. Would you be interested in contributing
your code? Thanks in advance...

[Mayayana]

| I assume you mean you wrote back in 'blocks' or 'chunks' of text! I've
| been trying to 'collect' code samples for doing exactly that in as
| efficient a manner as possible. Would you be interested in contributing
| your code? Thanks in advance...
|

You're welcome to it, but it's nothing special:

Sub WriteFile(Path, content)
Set TS = FSO.OpenTextFile(Path, 8, True)
Dim i6
i6 = 0
Do
If Len(content) > (i6 + 1000000) Then
TS.Write Mid(content, i6 + 1, 1000000) 'content
i6 = i6 + 1000000
Else
TS.Write right(content, (len(content) - i6))
Exit Do
End If
Loop
TS.Close
Set TS = Nothing
End Sub

[GS]

Post by Mayayana
| I assume you mean you wrote back in 'blocks' or 'chunks' of text! I've
| been trying to 'collect' code samples for doing exactly that in as
| efficient a manner as possible. Would you be interested in contributing
| your code? Thanks in advance...
|
Sub WriteFile(Path, content)
Set TS = FSO.OpenTextFile(Path, 8, True)
Dim i6
i6 = 0
Do
If Len(content) > (i6 + 1000000) Then
TS.Write Mid(content, i6 + 1, 1000000) 'content
i6 = i6 + 1000000
Else
TS.Write right(content, (len(content) - i6))
Exit Do
End If
Loop
TS.Close
Set TS = Nothing
End Sub

Thanks.., much appreciated!