error 51: User Defined Type too big

[caseih]

What do you needa 2GB UDT for? What problem are you trying to solve? I'm sure there's a better way to do it.

[coderJeff]

@coderJeff, How do I declare a structure that takes up more than 2 gigabytes?

It's not possible unless the UDT size check is removed/changed in the compiler. Even if >2GiB struct was allowed, it is quite unwieldy and couldn't be used without a run time memory allocation anyway (i.e. ALLOCATE or NEW) due to static variable and stack size limitations (platform dependent).

As caseih suggests, there is probably a better way, which depends on what problem you have to solve.

[erik]

I have a long file with a certain structure. Instead of reading it byte by byte into memory, I decided to mapping it in memory. This way I get a pointer to memory.
This memory is organized in a certain way, in the form of a data structure, so it is logical to cast a raw pointer to a pointer to the structure.

But I can't declare such a structure because the compiler forbids creating structures larger than 2 gigabytes. A structure declaration is not a memory allocation, it is not a static variable creation, it is not compiled into any processor instructions. It is just a declaration.

In C language, large structures can be declared. Just imagine: a C programmer will come to port his program to FreeBASIC. And then the compiler shows him an error. A C programmer will exclaim, "Is it still impossible to create large structures in FreeBASIC? Funny!"

Therefore, I do not see any reasonable reasons for the existence of such this restriction.

[fxm]

Another workaround:

Code: Select all

Type ZstringPointerMapping
    Dim As Zstring Ptr p(Any)
    Declare Constructor(Byval nbBlock As Integer, Byval sizeBlock As Integer, Byval address1stBlock As Any Ptr)
End Type

Constructor ZstringPointerMapping(Byval nbBlock As Integer, Byval sizeBlock As Integer, Byval address1stBlock As Any Ptr)
    Redim This.p(0 To nbBlock - 1)
    This.p(0) = address1stBlock
    For I As Integer = 1 to nbBlock - 1
        This.p(I) = This.p(I - 1) + sizeBlock
    Next I
End Constructor


Const capacity = &h0FFFFFFF '       0FFF_FFFF

Redim As Ubyte array(0 To 9 * capacity -1)
array(8 * capacity + 0) = Asc("F")
array(8 * capacity + 1) = Asc("r")
array(8 * capacity + 2) = Asc("e")
array(8 * capacity + 3) = Asc("e")
array(8 * capacity + 4) = Asc("B")
array(8 * capacity + 5) = Asc("A")
array(8 * capacity + 6) = Asc("S")
array(8 * capacity + 7) = Asc("I")
array(8 * capacity + 8) = Asc("C")

Dim As ZstringPointerMapping zpm = ZstringPointerMapping(9, capacity, @array(0))
Print zpm.p(8)[0]

Sleep

[erik]

@fxm, but I can write code much easier if the structures were allowed to be larger than 2 gigabytes:

Code: Select all

Type Database
	Field1 As Long
	Field2 As Long
	...
End Type

Dim p As Database Ptr = MapViewOfFile(...)

p->Field1 = SomeValue1
p->Field2 = SomeValue2

However, I am getting a compilation error.

[fxm]

What kind of compilation error?

[caseih]

I have a long file with a certain structure. Instead of reading it byte by byte into memory, I decided to mapping it in memory. This way I get a pointer to memory.
This memory is organized in a certain way, in the form of a data structure, so it is logical to cast a raw pointer to a pointer to the structure.

Sure that's a good idea. But are you telling me your individual data structures (records) are 2 GB in length? How many elements within each record are you trying to access? Just looking at your UDT that FB doesn't like, I can't think of any good reason to use a UDT for that. Why not just create functions for each of the a1 through a9 members and have that function calculate the pointer offset and return it as a zstring pointer. Or macros. Or just do the pointer arithmetic in code. For example

my_string_ptr = memory_pointer + 0x0fffffff * a_number

In C language, large structures can be declared. Just imagine: a C programmer will come to port his program to FreeBASIC. And then the compiler shows him an error. A C programmer will exclaim, "Is it still impossible to create large structures in FreeBASIC? Funny!"

Therefore, I do not see any reasonable reasons for the existence of such this restriction.

I'm sure C does support structs with element offsets in the GB range. But how often are they used? I'm sure it is a useful feature for some, including you. Is it worth modifying FB to support? Guess that depends on the devs.

[dodicat]

On this machine in 64 bits Win 10, I can set a pointer to the huge zstring, and even If I stay within that memory slot, I can only use a small part of it.
If I try to use too much
say
For n As Long=1 To 11
g+=g
Next
I get a crash.
The whole program is on a knife edge on this box with this chunk of memory in use.

Code: Select all

#cmdline "-exx"

Const Capacity As Ulongint = &h00000000FFFFFFFF

Dim Shared AAA As ZString * Capacity

Type Foo 
      As ubyte Const Ptr p=@AAA[0]
      Declare Operator Cast() As String
      Declare Sub set(As  String)
      Declare Sub copy(As ubyte Ptr,As Long)
End Type

Operator Foo.cast() As String
Return *cast(zstring ptr,p)
End Operator

Sub Foo.copy(src As ubyte Ptr,count As Long)
      For n As Long=0 To count
            p[n]=src[n]
      Next n
End Sub

Sub Foo.set(s As String)
      copy(@s[0],Len(s))
End Sub

Dim  As foo u
print "Adress of field ";u.p

u.set("FreeBASIC ")
Print u

Dim shared byref As zString g=AAA
print "Address of string ";@g[0]
For n As Long=1 To 10
      g+=g
Next

u.set(g)
var tmp=*cast(zstring ptr,u.p)

print left(tmp,50)
print "String length ";len(tmp)


u.set("Press any key to end .  .  .")
Print u

Sleep
 

[caseih]

Fortunately, the original poster's problem does not involve actual memory allocation. So nothing will be on a "knife edge." The addresses he is accessing refer to a memory-mapped file, so there's no memory allocation done. The file is swapped in and out of real memory as needed, as part of the virtual memory system. It's just like a swap file essentially. On a Linux system you would see the VSS memory use in top grow to reflect this but that would have not much to do with actual RAM pressure.

Off topic, but nerdy, file I/O on most operating systems is done using a similar mechanism. Upon opening, the file is mapped into the address space, and then as the memory locations are read, the VM pulls the file in. All of this allows the kernel to do intelligent caching and managing the physical ram. The difference here is when you read from a file, it copies the file's contents from one memory location to another (your target buffer string for example), whereas explicitly mapping a file to memory allows direct access without the copying done by the file i/o routines in the runtime library. Although maybe there're optimizations that just adjust the memory map when a copy is made, eliminating actual copying of memory pages. This sort of thing is done by network drivers.

[fxm]

Documentation updated:
- TblVarTypes → fxm [added maximum size for UDTs] (in 'Standard Data Type Limits')
- KeyPgType → fxm [added link to 'Standard Data Type Limits']

[fxm]

On this machine in 64 bits Win 10, I can set a pointer to the huge zstring, and even If I stay within that memory slot, I can only use a small part of it.

All your problems stem from the fact that you want to allocate a large memory (4GB - 1) in the .BSS section (bad behavior as Jeff and I previously reported: compiler/linker error or even runtime crash).

No problem if we allocate the memory in the heap (your above code just modified consequently):

Code: Select all

#cmdline "-exx"

Const Capacity As Ulongint = &h00000000FFFFFFFF

Dim Shared AAA As ZString Ptr
AAA = Callocate(Capacity)

Type Foo 
      As ubyte Const Ptr Ptr p=@AAA
      Declare Operator Cast() As String
      Declare Sub set(As  String)
      Declare Sub copy(As ubyte Ptr,As Long)
End Type

Operator Foo.cast() As String
Return **cast(zstring ptr ptr,p)
End Operator

Sub Foo.copy(src As ubyte Ptr,count As Long)
      For n As Long=0 To count
            (*p)[n]=src[n]
      Next n
End Sub

Sub Foo.set(s As String)
      copy(@s[0],Len(s))
End Sub

Dim  As foo u
print "Adress of field ";*u.p

u.set("FreeBASIC ")
Print u

Dim byref As zString g=*AAA
print "Address of string ";@g[0]
print g
For n As Long=1 To 28
      g+=g
Next

u.set(g)
var tmp=*cast(zstring ptr,*u.p)

print left(tmp,50)
print "String length ";len(tmp)


u.set("Press any key to end .  .  .")
Print u

Deallocate(AAA)

Sleep

[erik]

That's not what I need.
I just need to declare a large structure.
I think the right solution would be to remove the structure size check from the compiler.

[paul doe]

...
In C language, large structures can be declared. Just imagine: a C programmer will come to port his program to FreeBASIC. And then the compiler shows him an error. A C programmer will exclaim, "Is it still impossible to create large structures in FreeBASIC? Funny!"
...

Not according to this:

https://stackoverflow.com/questions/772 ... structures

Quoting: '...So the maximum size is between 2^30 and (2^31 - 1).'

Which is roughly 2 GB. Like other posters said in this thread, I see no point in having such a large structure. Probably a code smell more than anything else.

[caseih]

I just need to declare a large structure.

You never answered my questions before. How many members do you need inside this large structure? How big is each member? Are they all the same size? Are you dealing with multiple records?

[fxm]

Yes, I have the same similar question:
If there was not this 2GB limit for Type size, what would be the exact structure of the Type that you would declare for mapping in the particular case of your application ?