Array size limit

General FreeBASIC programming questions.
creek23
Posts: 260
Joined: Sep 09, 2007 1:57
Location: Philippines
Contact:

Array size limit

Postby creek23 » Mar 26, 2012 5:34

Hi all,

From my understanding, an array size is only limited by the available memory. But somehow my app is is crashing if the array size is 52517 long... while it works fine if its 52516 long only. And since I have 4GB, I was expecting a bigger size than what I'm getting now.

Btw, my array is an array of this UDT:

Code: Select all

Type Foo
   aaa As String
   bbb As Double
   ccc As Integer
   ddd As UInteger
   eee As UInteger
   fff As Integer
   ggg As Integer
   hhh As Integer
   iii As Long
   jjj As Integer
End Type


I guess 52516 is as good as 52516(size)x10(type-members), which would mean I'm technically getting a whooping 525,160. o.O

I'm using fbc 0.23 btw.

~creek23
VANYA
Posts: 1311
Joined: Oct 24, 2010 15:16
Location: Ярославль
Contact:

Re: Array size limit

Postby VANYA » Mar 26, 2012 7:20

I also asked this question before :)

Use global arrays

Code: Select all

Type Foo
   aaa As String
   bbb As Double
   ccc As Integer
   ddd As UInteger
   eee As UInteger
   fff As Integer
   ggg As Integer
   hhh As Integer
   iii As Long
   jjj As Integer
End Type

Dim shared a(60000) As foo
pestery
Posts: 493
Joined: Jun 16, 2007 2:00
Location: Australia

Re: Array size limit

Postby pestery » Mar 26, 2012 7:45

Its the stack size. When you don't use Shared the memory is allocated from the stack. On Windows I think the stack size defaults too about 1 MB (on Linux it's different, but I forget how exactly). You can increase the stack size by using the -t compiler option (here).

To solve your problem either use (Dim Shared array(1 to 52517) As Foo) or use the -t compiler option to increase the stack size.
marcov
Posts: 2750
Joined: Jun 16, 2005 9:45
Location: Eindhoven, NL
Contact:

Re: Array size limit

Postby marcov » Mar 26, 2012 10:19

pestery wrote: On Windows I think the stack size defaults too about 1 MB (on Linux it's different, but I forget how exactly).


On Linux, stack sizes are more enforced by the system, though these limits are typically lax and non-enforced on single user systems.

On servers however this is more strict. Type "ulimit -a" to see the limits.
creek23
Posts: 260
Joined: Sep 09, 2007 1:57
Location: Philippines
Contact:

Re: Array size limit

Postby creek23 » Mar 26, 2012 10:46

Hi all,

Thank you for the replies.

With regards to declaration, I actually declared it like this:

Code: Select all

Extern Bar() As Foo

since it will be used across BAS files and ReDim'ed it from inside a function. Is that as good as Shared?

~creek23
petan
Posts: 683
Joined: Feb 16, 2010 15:34
Location: Europe
Contact:

Re: Array size limit

Postby petan » Mar 26, 2012 11:55

Hi,
I use only "Shared" command for global variables (multidim, UDT...), and after, redimmed from anywhere, as I need.

Pete
fxm
Posts: 8959
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Array size limit

Postby fxm » Mar 26, 2012 12:13

creek23 wrote:With regards to declaration, I actually declared it like this:

Code: Select all

Extern Bar() As Foo

since it will be used across BAS files and ReDim'ed it from inside a function. Is that as good as Shared?

~creek23
The array Bar() is so dynamic and it is not in the stack.

Remark:
Useful memory size of one element 'Bar(I)':
Sizeof(Foo) + Cast(Integer Ptr, @Bar(I).aaa)[2] = 56 + Cast(Integer Ptr, @Bar(I).aaa)[2] bytes
('Cast(Integer Ptr, @Bar(I).aaa)[2]' bytes being the useful memory size allocated for the string data, which is greater or equal to the string length)
('Cast(Integer Ptr, @Bar(I).aaa)[2]' being the third data of the array descriptor which is at '@Bar(I).aaa')
In case of non-nul string, one additional byte is allocated to the string data, corresponding to the non-useful terminal character 'Chr(0)'.

Code: Select all

Type Foo
   aaa As String
   bbb As Double
   ccc As Integer
   ddd As UInteger
   eee As UInteger
   fff As Integer
   ggg As Integer
   hhh As Integer
   iii As Long
   jjj As Integer
End Type

Print Sizeof(Foo)
Print
Print Offsetof(Foo, aaa)
Print Offsetof(Foo, bbb)
Print Offsetof(Foo, ccc)
Print Offsetof(Foo, ddd)
Print Offsetof(Foo, eee)
Print Offsetof(Foo, fff)
Print Offsetof(Foo, ggg)
Print Offsetof(Foo, hhh)
Print Offsetof(Foo, iii)
Print Offsetof(Foo, jjj)

Sleep
creek23
Posts: 260
Joined: Sep 09, 2007 1:57
Location: Philippines
Contact:

Re: Array size limit

Postby creek23 » Mar 27, 2012 4:58

Thanks fxm. With your snippet, I just realized that from 52516 array size multiplied by 56 bytes (as returned by SizeOf) I am actually using 2,940,896 bytes. o.O

I guess it's a dead end for me here unless I write a better algorithm for my program. :(

~creek23
Richard
Posts: 2926
Joined: Jan 15, 2007 20:44
Location: Australia

Re: Array size limit

Postby Richard » Mar 27, 2012 6:50

There will always be arrays that get too big for memory.
A file on disk does not have the same size limitation.
Let the HDD cache speed up the disk access.
fxm
Posts: 8959
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Array size limit

Postby fxm » Mar 27, 2012 18:31

creek23 wrote:I just realized that from 52516 array size multiplied by 56 bytes (as returned by SizeOf) I am actually using 2,940,896 bytes.

That corresponds to 2.8 MB versus 4 GB of memory!
Why your program crashes when the array is greater than this limit of 52516 elements?
pestery
Posts: 493
Joined: Jun 16, 2007 2:00
Location: Australia

Re: Array size limit

Postby pestery » Mar 28, 2012 3:39

fxm wrote:Why your program crashes when the array is greater than this limit of 52516 elements?
Because creek23 was allocating the memory from the stack, not the system memory. Or at least that was the impression I got ;-)

Also, regarding the maximum amount of allocated memory, I think its only 2 GB because the pointers use signed Integers (because FB is a 32bit compiler). Although you can't normally allocate 2 GB in a single big chunk anyway, I've tried :-) Instead you should use 2 * 1 GB chunks, or 4 * 512 MB chunks, and there also needs to be enough memory left over for the operating system and any other programs you have running.

creek23 wrote:I guess it's a dead end for me here unless I write a better algorithm for my program. :(
Maybe not. You might just need to change how you define the array initially. For example:

Code: Select all

' The memory for the array is allocated from
' the stack, but is too large for the stack
' You could use this if you increase the stack size
Dim As Foo array(1 To 52517)

' The memory is allocated from the system memory (RAM)
' but is also visible from inside subs and functions
Dim Shared As Foo array(1 To 52517)

' The memory is allocated from the system memory (RAM)
' and is not visible from inside subs and functions
' The array is also resizeable
ReDim As Foo array(1 To 52517)

' This causes a syntax error
Common As Foo array(1 To 52517)

' This also causes a syntax error, but only because of
' the String in Foo. Because Constructors/Destructors
' are not allowed in common arrays. If the String is
' removed then this works like the normal ReDim, but
' is accessable from any module that includes it
Common As Foo array()
ReDim As Foo array(1 To 52517) ' Only include in 1 module

' This causes a syntax error
Extern As Foo array()
Dim As Foo array(1 To 52517)

' This works like the ReDim Shared, but is accessable
' from any module that includes the Extern statement
' And unlike Common you can also use Strings,
' Constructors and Destructors
Extern As Foo array()
ReDim As Foo array(1 To 52517) ' Only include in 1 module
TJF
Posts: 3467
Joined: Dec 06, 2009 22:27
Location: N47°, E15°

Re: Array size limit

Postby TJF » Mar 28, 2012 4:52

pestery wrote:Also, regarding the maximum amount of allocated memory, I think its only 2 GB because the pointers use signed Integers (because FB is a 32bit compiler). Although you can't normally allocate 2 GB in a single big chunk anyway, I've tried :-) Instead you should use 2 * 1 GB chunks, or 4 * 512 MB chunks, and there also needs to be enough memory left over for the operating system and any other programs you have running.

The 2 GB limit and the restriction regarding signed INTEGER is relevant for STRING variables. Their length is stored in an signed INTEGER.

Pointers in general are UINTEGERs and you can allocate up to 4 GB in one chunk (some memory must be left for the OS).
MichaelW
Posts: 3500
Joined: May 16, 2006 22:34
Location: USA

Re: Array size limit

Postby MichaelW » Mar 28, 2012 6:52

Under Windows each 32-bit process normally has a user-mode virtual address space of 2 GB, and the other 2 GB of the 32-bit address space is reserved for the OS and not accessible from user mode:

http://technet.microsoft.com/en-us/query/aa366778

A process cannot use more memory than will fit in the address space. Since Windows implements a virtual memory system, backed by a paging file, and assuming that the paging file can accommodate the size, the size of the allocation is not limited to the size of the physical memory. This code allocates and then reads through 1.8 GB of memory, and except for a low virtual-memory warning it runs OK on my systems, both of which have only 512 MB of physical memory.

Code: Select all

''=============================================================================
#include "windows.bi"
''=============================================================================
dim as ubyte ptr p
dim as ubyte r

p = VirtualAlloc( NULL, 1.8*1024*1024*1024, MEM_COMMIT, PAGE_READWRITE)
print hex(p);"h"
if p = 0 then
    sleep
    end
end if
for i as integer = 0 to 1.8*1024*1024*1024-4096
    if i and (i mod 1024*1024 = 0) then
        locate 3,1,0
        print i / (1024*1024)
    end if
    r = p[i]
next
print "done"
sleep

I arrived at 1.8 GB by starting at 2 GB and working back until the allocation succeeded.
fxm
Posts: 8959
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Array size limit

Postby fxm » Mar 28, 2012 7:15

pestery wrote:
fxm wrote:Why your program crashes when the array is greater than this limit of 52516 elements?
Because creek23 was allocating the memory from the stack, not the system memory. Or at least that was the impression I got ;-)

fxm wrote:
creek23 wrote:With regards to declaration, I actually declared it like this:

Code: Select all

Extern Bar() As Foo

since it will be used across BAS files and ReDim'ed it from inside a function. Is that as good as Shared?

~creek23
The array Bar() is so dynamic and it is not in the stack.

The declaration:
Extern Bar() As Foo
is only compatible with dynamic array!
fxm
Posts: 8959
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Array size limit

Postby fxm » Mar 28, 2012 11:29

pestery wrote:
creek23 wrote:I guess it's a dead end for me here unless I write a better algorithm for my program. :(
Maybe not. You might just need to change how you define the array initially. For example:

Code: Select all

' The memory for the array is allocated from
' the stack, but is too large for the stack
' You could use this if you increase the stack size
Dim As Foo array(1 To 52517)

' The memory is allocated from the system memory (RAM)
' but is also visible from inside subs and functions
Dim Shared As Foo array(1 To 52517)

' The memory is allocated from the system memory (RAM)
' and is not visible from inside subs and functions
' The array is also resizeable
ReDim As Foo array(1 To 52517)

' This causes a syntax error
Common As Foo array(1 To 52517)

' This also causes a syntax error, but only because of
' the String in Foo. Because Constructors/Destructors
' are not allowed in common arrays. If the String is
' removed then this works like the normal ReDim, but
' is accessable from any module that includes it
Common As Foo array()
ReDim As Foo array(1 To 52517) ' Only include in 1 module

' This causes a syntax error
Extern As Foo array()
Dim As Foo array(1 To 52517)

' This works like the ReDim Shared, but is accessable
' from any module that includes the Extern statement
' And unlike Common you can also use Strings,
' Constructors and Destructors
Extern As Foo array()
ReDim As Foo array(1 To 52517) ' Only include in 1 module

Additional syntax:

Code: Select all

' The memory is allocated from the system memory (RAM)
' and is not visible from inside subs and functions
' The array is not resizeable
Static As Foo array(1 To 52517)
Remark:
Static Shared ...
is equivalent to
Dim Shared ...

Return to “General”

Who is online

Users browsing this forum: Bing [Bot] and 1 guest