Releasing memory from large strings

New to FreeBASIC? Post your questions here.
fzabkar
Posts: 48
Joined: Sep 29, 2018 2:52

Releasing memory from large strings

Postby fzabkar » Sep 18, 2020 20:11

If I initialise a large string as follows (for reading a 100MB file into RAM) ...

Code: Select all

Dim sBigString As String
Dim lenfil As ULong
lenfil = 100000000
sBigString = Space( lenfil )
Get #infil, 1, sBigString

... will setting the string to null return its memory to the heap?

Code: Select all

sBigString = ""

Essentially what I'm asking is whether this is equivalent to Callocate (100000000) and then Deallocate.

The reason I use a large string is that it is easier to search (with Instr and InstrRev), even when searching for binary data.
jj2007
Posts: 1729
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: Releasing memory from large strings

Postby jj2007 » Sep 18, 2020 20:31

fzabkar wrote:... will setting the string to null return its memory to the heap?
Yes, using msvcrt.free, which in turn uses RtlFreeHeap
badidea
Posts: 2153
Joined: May 24, 2007 22:10
Location: The Netherlands

Re: Releasing memory from large strings

Postby badidea » Sep 18, 2020 20:40

It does release memory, see:

Code: Select all

Dim sBigString As String
Dim lenfil As ULong
lenfil = 100 * 1000 * 1000

print fre() \ (1000 * 1000)
sBigString = Space( lenfil )
print fre() \ (1000 * 1000)
sBigString = ""
print fre() \ (1000 * 1000)

Result here:
2990
2890
2989

So, not all memory is released it seems with = "".
fxm
Posts: 9994
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Releasing memory from large strings

Postby fxm » Sep 18, 2020 21:10

The memory allocated in the heap for a string of N characters is greater than that allocated by Callocate(N) in order to optimize the execution time for string length changes (the memory is allocated by large step to minimize the number of re-allocations induced by small string length changes).

We can check it by monitoring the string descriptor data, as follows:

Code: Select all

Dim sBigString As String
Dim lenfil As ULong
lenfil = 100000000
sBigString = Space( lenfil )

Type StringDescriptor
    Dim As Zstring Ptr pz
    Dim As Uinteger NbUsedChr
    Dim As Uinteger NbavailableChr
End Type

Dim As StringDescriptor Ptr p = Cptr(Any Ptr, @sBigString)
Print "Addr.", "Chr. used", "Chr. available"

Print p->pz,p->NbUsedChr, p->NbavailableChr

sBigString = ""
Print p->pz,p->NbUsedChr, p->NbavailableChr

Sleep

/' output:
Addr.         Chr. used     Chr. available
34017312      100000000     112500000
0             0             0
'/
fzabkar
Posts: 48
Joined: Sep 29, 2018 2:52

Re: Releasing memory from large strings

Postby fzabkar » Sep 18, 2020 21:13

badidea wrote:It does release memory, see:

Code: Select all

Dim sBigString As String
Dim lenfil As ULong
lenfil = 100 * 1000 * 1000

print fre() \ (1000 * 1000)
sBigString = Space( lenfil )
print fre() \ (1000 * 1000)
sBigString = ""
print fre() \ (1000 * 1000)

Result here:
2990
2890
2989

So, not all memory is released it seems with = "".

Doh, I was looking for a suitable command but didn't see FRE. Thanks.
fzabkar
Posts: 48
Joined: Sep 29, 2018 2:52

Re: Releasing memory from large strings

Postby fzabkar » Sep 18, 2020 21:34

These are my results :-?

Code: Select all

Dim sBigString As String
Dim lenfil As ULong
Dim As Ulong i,j,k

lenfil = 100 * 1000 * 1000

i = fre(): print i / (1000 * 1000)
sBigString = Space( lenfil )
j = fre(): print j /  (1000 * 1000), (i - j) /  (1000 * 1000)
sBigString = ""
k = fre(): print k /  (1000 * 1000)

Sleep

Code: Select all

 943.63648
 863.707136    79.929344
 947.294208
fxm
Posts: 9994
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Releasing memory from large strings

Postby fxm » Sep 18, 2020 21:42

fxm wrote:The 'Fre()' FB keyword does not refresh the memory.
The 'Fre()' FB keyword returns the amount (in bytes) of available or unused dynamic memory which is allocated at this time by the OS.

I think that the quantity of dynamic memory allocated by the OS is not constant during the program execution. The OS try to regulate the available dynamic memory quantity for the program:
- When the program uses some quantity of allocated dynamic memory, the OS may allocate new memory quantity for the program, in asynchronous way.
- Inversely when the program frees some quantity of dynamic memory, the OS may reclaim some memory, always in asynchronous way.

This is why the variation of the value returned by the 'Fre()' FB keyword does not necessarily follow the variation of the dynamic memory quantity used by the program.
jj2007
Posts: 1729
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: Releasing memory from large strings

Postby jj2007 » Sep 18, 2020 23:19

jj2007 wrote:
fzabkar wrote:... will setting the string to null return its memory to the heap?
Yes, using msvcrt.free, which in turn uses RtlFreeHeap

I know it's difficult to use an assembly level debugger, but what I wrote is the truth, and you won't get the truth from FRE(). However, if you insert a Sleep as shown below, you can get closer to the truth without using the debugger:

Code: Select all

Dim sBigString As String
Dim lenfil As ULong
lenfil = 100 * 1000 * 1000

print fre() \ (1000 * 1000)
sBigString = Space( lenfil )
print fre() \ (1000 * 1000)
sBigString = ""
Sleep   ' wait a second
print fre() \ (1000 * 1000)
Sleep

Typical output:

Code: Select all

2993
2892
2996

Here is the relevant code under the hood, compiled with Gcc-32 (Gas and FB 64 are very similar):

Code: Select all

CPU Disasm
Address              Hex dump                   Command                      Comments
004020D0             Ú$  53                     push ebx                     ; TmpFb.004020D0(guessed Arg1)
004020D1             ³.  83EC 18                sub esp, 18
004020D4             ³.  8B5C24 20              mov ebx, [esp+20]
004020D8             ³.  85DB                   test ebx, ebx
004020DA             ³. 74 22                  jz short 004020FE
004020DC             ³.  8B03                   mov eax, [ebx]               ; get the pointer to the string memory
004020DE             ³.  85C0                   test eax, eax
004020E0             ³. 74 1C                  jz short 004020FE            ; if zero, don't free it
004020E2             ³.  890424                 mov [esp], eax               ; Úmemblock
004020E5             ³.  E8 DE270000            call <jmp.&msvcrt.free>      ; ÀMSVCRT.free, call RtlFreeHeap
004020EA             ³.  C703 00000000          mov dword ptr [ebx], 0       ; mark deleted
004020F0             ³.  C743 04 00000000       mov dword ptr [ebx+4], 0
004020F7             ³.  C743 08 00000000       mov dword ptr [ebx+8], 0
004020FE             ³>  83C4 18                add esp, 18
00402101             ³.  5B                     pop ebx
00402102             À.  C2 0400                retn 4
MrSwiss
Posts: 3636
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: Releasing memory from large strings

Postby MrSwiss » Sep 20, 2020 19:37

badidea wrote:So, not all memory is released it seems with = "".

That is correct. Only the memory that holds 'string data' allocated from heap is released.
The memory used by the 'string header' (12 bytes FBC 32, 24 bytes FBC 64) is kept.
(only destroyed at program end or when going 'out of scope')
fxm
Posts: 9994
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Releasing memory from large strings

Postby fxm » Sep 20, 2020 19:54

Unless you define it as dynamic:
Dim psBigString As String Ptr = New String
Dim lenfil As ULong
lenfil = 100000000
*psBigString = Space( lenfil )
Get #infil, 1, *sBigString
.....
Delete psBigString


There will then only be one pointer (4/8 bytes) left in the local scope.

Return to “Beginners”

Who is online

Users browsing this forum: No registered users and 9 guests