Releasing memory from large strings

[fzabkar]

If I initialise a large string as follows (for reading a 100MB file into RAM) ...

Code: Select all

Dim sBigString As String
Dim lenfil As ULong
lenfil = 100000000
sBigString = Space( lenfil )
Get #infil, 1, sBigString

... will setting the string to null return its memory to the heap?

Code: Select all

sBigString = ""

Essentially what I'm asking is whether this is equivalent to Callocate (100000000) and then Deallocate.

The reason I use a large string is that it is easier to search (with Instr and InstrRev), even when searching for binary data.

[jj2007]

... will setting the string to null return its memory to the heap?

Yes, using msvcrt.free, which in turn uses RtlFreeHeap

[badidea]

It does release memory, see:

Code: Select all

Dim sBigString As String
Dim lenfil As ULong
lenfil = 100 * 1000 * 1000

print fre() \ (1000 * 1000)
sBigString = Space( lenfil )
print fre() \ (1000 * 1000)
sBigString = ""
print fre() \ (1000 * 1000)

Result here:
2990
2890
2989
So, not all memory is released it seems with = "".

[fxm]

The memory allocated in the heap for a string of N characters is greater than that allocated by Callocate(N) in order to optimize the execution time for string length changes (the memory is allocated by large step to minimize the number of re-allocations induced by small string length changes).

We can check it by monitoring the string descriptor data, as follows:

Code: Select all

Dim sBigString As String
Dim lenfil As ULong
lenfil = 100000000
sBigString = Space( lenfil )

Type StringDescriptor
    Dim As Zstring Ptr pz
    Dim As Uinteger NbUsedChr
    Dim As Uinteger NbavailableChr
End Type

Dim As StringDescriptor Ptr p = Cptr(Any Ptr, @sBigString)
Print "Addr.", "Chr. used", "Chr. available"

Print p->pz,p->NbUsedChr, p->NbavailableChr

sBigString = ""
Print p->pz,p->NbUsedChr, p->NbavailableChr

Sleep

/' output:
Addr.         Chr. used     Chr. available
34017312      100000000     112500000
0             0             0
'/

[fzabkar]

It does release memory, see:

Code: Select all

Dim sBigString As String
Dim lenfil As ULong
lenfil = 100 * 1000 * 1000

print fre() \ (1000 * 1000)
sBigString = Space( lenfil )
print fre() \ (1000 * 1000)
sBigString = ""
print fre() \ (1000 * 1000)

Result here:
2990
2890
2989
So, not all memory is released it seems with = "".

Doh, I was looking for a suitable command but didn't see FRE. Thanks.

[fzabkar]

These are my results :-?

Code: Select all

Dim sBigString As String
Dim lenfil As ULong
Dim As Ulong i,j,k 

lenfil = 100 * 1000 * 1000

i = fre(): print i / (1000 * 1000)
sBigString = Space( lenfil )
j = fre(): print j /  (1000 * 1000), (i - j) /  (1000 * 1000)
sBigString = ""
k = fre(): print k /  (1000 * 1000)

Sleep

Code: Select all

 943.63648
 863.707136    79.929344
 947.294208

[fxm]

The 'Fre()' FB keyword does not refresh the memory.
The 'Fre()' FB keyword returns the amount (in bytes) of available or unused dynamic memory which is allocated at this time by the OS.

I think that the quantity of dynamic memory allocated by the OS is not constant during the program execution. The OS try to regulate the available dynamic memory quantity for the program:
- When the program uses some quantity of allocated dynamic memory, the OS may allocate new memory quantity for the program, in asynchronous way.
- Inversely when the program frees some quantity of dynamic memory, the OS may reclaim some memory, always in asynchronous way.

This is why the variation of the value returned by the 'Fre()' FB keyword does not necessarily follow the variation of the dynamic memory quantity used by the program.

[jj2007]

... will setting the string to null return its memory to the heap?

Yes, using msvcrt.free, which in turn uses RtlFreeHeap

I know it's difficult to use an assembly level debugger, but what I wrote is the truth, and you won't get the truth from FRE(). However, if you insert a Sleep as shown below, you can get closer to the truth without using the debugger:

Code: Select all

Dim sBigString As String
Dim lenfil As ULong
lenfil = 100 * 1000 * 1000

print fre() \ (1000 * 1000)
sBigString = Space( lenfil )
print fre() \ (1000 * 1000)
sBigString = ""
Sleep	' wait a second
print fre() \ (1000 * 1000)
Sleep

Typical output:

Code: Select all

2993
2892
2996

Here is the relevant code under the hood, compiled with Gcc-32 (Gas and FB 64 are very similar):

Code: Select all

CPU Disasm
Address              Hex dump                   Command                      Comments
004020D0             Ú$  53                     push ebx                     ; TmpFb.004020D0(guessed Arg1)
004020D1             ³.  83EC 18                sub esp, 18
004020D4             ³.  8B5C24 20              mov ebx, [esp+20]
004020D8             ³.  85DB                   test ebx, ebx
004020DA             ³. 74 22                  jz short 004020FE
004020DC             ³.  8B03                   mov eax, [ebx]               ; get the pointer to the string memory
004020DE             ³.  85C0                   test eax, eax
004020E0             ³. 74 1C                  jz short 004020FE            ; if zero, don't free it
004020E2             ³.  890424                 mov [esp], eax               ; Úmemblock
004020E5             ³.  E8 DE270000            call <jmp.&msvcrt.free>      ; ÀMSVCRT.free, call RtlFreeHeap
004020EA             ³.  C703 00000000          mov dword ptr [ebx], 0       ; mark deleted
004020F0             ³.  C743 04 00000000       mov dword ptr [ebx+4], 0
004020F7             ³.  C743 08 00000000       mov dword ptr [ebx+8], 0
004020FE             ³>  83C4 18                add esp, 18
00402101             ³.  5B                     pop ebx
00402102             À.  C2 0400                retn 4

[MrSwiss]

So, not all memory is released it seems with = "".

That is correct. Only the memory that holds 'string data' allocated from heap is released.
The memory used by the 'string header' (12 bytes FBC 32, 24 bytes FBC 64) is kept.
(only destroyed at program end or when going 'out of scope')

[fxm]

Unless you define it as dynamic:
Dim psBigString As String Ptr = New String
Dim lenfil As ULong
lenfil = 100000000
*psBigString = Space( lenfil )
Get #infil, 1, *sBigString
.....
Delete psBigString

There will then only be one pointer (4/8 bytes) left in the local scope.