How accessing to the array's descriptor structure? (ONLY FOR FBC VERSION < 1.08)

General FreeBASIC programming questions.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

How accessing to the array's descriptor structure? (ONLY FOR FBC VERSION < 1.08)

Post by fxm »

The array descriptor has the following structure (each item is coded on an Integer):
- pointer to the real or virtual element: @array(0, 0,...)
- pointer to the first real element: @array(lbound1, lbound2,...)
- "global" size in bytes: (ubound1-lbound1+1)*(ubound2-lbound2+1).....*(size of 1 element)
- size of one element in bytes
- number of dimensions
then for each dimension:
- number of elements: (ubound-lbound+1)
- lbound
- ubound

For the local fix-len arrays:
- the descriptor is just ahead the array elements,
- so from the address of the first array element, we can traced back directly to the structure of the descriptor.
For the global or var-len arrays:
- the descriptor is elsewhere in memory,
- so there is no direct way to access the descriptor.

For all types of array, there is a workaround to get the address of the descriptor (for 64-bit, compile with '-exx'):
- when an array is passed to a procedure, its descriptor is passed by reference,
- Just divert this parameter passing to retrieve the address of the descriptor in a generic Function(As Any Ptr) and return this as function return value.
- one macro allows to cast the function pointer depending on the array datatype.
Generic Function: 'arrayDescriptorPtrFunction (Byval p As Any Ptr) As Any Ptr'
Macro: 'arrayDescriptorPtr(array)'

Code: Select all

Function arrayDescriptorPtrFunction (Byval p As Any Ptr) As Any Ptr
  Return p
End function
#Define arrayDescriptorPtr(array) _
Cast(Function (() As Typeof((array))) As Any Ptr, @arrayDescriptorPtrFunction)(array())
An example with a Sub to print the descriptor items and several types of arrays in order to validate the different cases (for 64-bit, compile with '-exx'):

Code: Select all

Function arrayDescriptorPtrFunction (Byval p As Any Ptr) As Any Ptr
  Return p
End function
#Define arrayDescriptorPtr(array) _
Cast(Function (() As Typeof((array))) As Any Ptr, @arrayDescriptorPtrFunction)(array())

'------------------------------------------------------------------------------------

Sub printArrayDescriptor (Byval p As Any Ptr)
  Dim As Integer Ptr pu = p
  Print "[@array descriptor: "; pu; "]"
  Print "  @array(all_null_indexes)   ="; pu[0]
  Print "  @array(all_min_indexes)    ="; pu[1]
  Print "  array_total_size_in_bytes  ="; pu[2]
  Print "  array_element_size_in_bytes="; pu[3]
  Print "  number_of_array_dimensions ="; pu[4]
  For i As Integer = 1 to pu[4]
    Print "  [dimension number:"; i; "]"
    Print "    number_of_elements="; pu[5 + (i - 1) * 3]
    Print "    min_index         ="; pu[6 + (i - 1) * 3]
    Print "    max_index         ="; pu[7 + (i - 1) * 3]
  Next i
End Sub

'------------------------------------------------------------------------------------

Dim As Longint test1(0 to 9, 1 to 100)
Dim Shared As Longint test2(0 to 9, 1 to 100)
Dim As Longint test3()
Dim Shared As Longint test4()
Type UDT
  Dim As Longint test1(0 to 9, 1 to 100)
  Dim As Longint test2(Any, Any)
End Type
Dim As UDT u

Screen 0
Width , 30

printArrayDescriptor(arrayDescriptorPtr(test1))
Sleep
Cls
printArrayDescriptor(arrayDescriptorPtr(test2))
Sleep
Cls
printArrayDescriptor(arrayDescriptorPtr(test3))
Print
Redim test3(0 to 9, 1 to 100)
printArrayDescriptor(arrayDescriptorPtr(test3))
Sleep
Cls
printArrayDescriptor(arrayDescriptorPtr(test4))
print
Redim test4(0 to 9, 1 to 100)
printArrayDescriptor(arrayDescriptorPtr(test4))
Sleep
Cls
printArrayDescriptor(arrayDescriptorPtr(u.test1))
Sleep
Cls
printArrayDescriptor(arrayDescriptorPtr(u.test2))
print
Redim u.test2(0 to 9, 1 to 100)
printArrayDescriptor(arrayDescriptorPtr(u.test2))

Sleep
[edit]
Used now 'Typeof((array))' (with double parentheses) to be compatible with array inside a Namespace (see #404).
Last edited by fxm on Sep 04, 2019 18:38, edited 8 times in total.
Tourist Trap
Posts: 2958
Joined: Jun 02, 2015 16:24

Re: How accessing to the array's descriptor structure?

Post by Tourist Trap »

Do you project to upload this rather complete snapshot about array descriptors in the documentation? It would be great.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How accessing to the array's descriptor structure? (ONLY FOR FBC VERSION < 1.08)

Post by fxm »

The descriptors (for the var-len strings and for the arrays) are internal structures from compiler which allow an easily interfacing of a user with strings/arrays.
Similarly, for virtuality and polymorphism, the Vptr/Vtable/RTTinfo structures are only described in the topic "Abstract/Virtual destructor/method behaviour" (last diagram and use-cases at http://www.freebasic.net/forum/viewtopi ... 71#p206971).

But all these internal structures from compiler must not normally be described in the User Manual (they are subject to change from one revision to the next).
Their knowledge allows only better understanding of the user program implementation, and also are useful for hacking but this is not the aim of the documentation!
Last edited by fxm on Sep 04, 2019 18:39, edited 1 time in total.
Tourist Trap
Posts: 2958
Joined: Jun 02, 2015 16:24

Re: How accessing to the array's descriptor structure?

Post by Tourist Trap »

fxm wrote:Their knowledge allows only better understanding of the user program implementation, and also are useful for hacking but this is not the aim of the documentation!
The question is to know if there is today a section that fits for such details, because of course every detail can have a place in the documentation (in its extended meaning). Just my opinion from my personal usage: this remains the better place to gather valuable informations and it's comfortable when one doesn't have to extend its search over other sources.

However I understand that you are already at work on the keyword definitions which is enough huge work!
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How accessing to the array's descriptor structure? (ONLY FOR FBC VERSION < 1.08)

Post by fxm »

Descriptor use is complex!
For example the use is different between the fix-len arrays and the var-len arrays.

- For a local fix-len array, the descriptor is only used for passing the array to a procedure:

Code: Select all

Function uboundArray (array() As Integer) As Integer
  Function = Ubound(array)
End Function

Dim As Integer array(1 To 10)
Print Ubound(array)
Print uboundArray(array())
Print
Cast(Integer Ptr, @array(1))[-1] = 5 ' overwrite ubound value in descriptor
Print Ubound(array)
Print uboundArray(array())

Sleep
- For a global fix-len array, the descriptor is also only used for passing the array to a procedure (for 64-bit, compile with '-exx'):

Code: Select all

Function arrayDescriptorPtrFunction (Byval p As Any Ptr) As Any Ptr
  Return p
End function
#Define arrayDescriptorPtr(array) _
Cast(Function (() As Typeof((array))) As Any Ptr, @arrayDescriptorPtrFunction)(array())


Function uboundArray (array() As Integer) As Integer
  Function = Ubound(array)
End Function

Dim Shared As Integer array(1 To 10)
Print Ubound(array)
Print uboundArray(array())
Print
Cast(Integer Ptr, arrayDescriptorPtr(array))[7] = 5 ' overwrite ubound value in descriptor
Print Ubound(array)
Print uboundArray(array())

Sleep
- For a var-len array, the descriptor is always used (for 64-bit, compile with '-exx'):

Code: Select all

Function arrayDescriptorPtrFunction (Byval p As Any Ptr) As Any Ptr
  Return p
End function
#Define arrayDescriptorPtr(array) _
Cast(Function (() As Typeof((array))) As Any Ptr, @arrayDescriptorPtrFunction)(array())


Function uboundArray (array() As Integer) As Integer
  Function = Ubound(array)
End Function

Dim As Integer array()
Redim array(1 To 10)
Print Ubound(array)
Print uboundArray(array())
Print
Cast(Integer Ptr, arrayDescriptorPtr(array))[7] = 5 ' overwrite ubound value in descriptor
Print Ubound(array)
Print uboundArray(array())

Sleep
[edit]
Used now 'Typeof((array))' (with double parentheses) to be compatible with array inside a Namespace (see #404).
Last edited by fxm on Sep 04, 2019 18:39, edited 7 times in total.
Tourist Trap
Posts: 2958
Joined: Jun 02, 2015 16:24

Re: How accessing to the array's descriptor structure?

Post by Tourist Trap »

Thanks for those interesting showcases.
fxm wrote:- For a var-len array, the descriptor is always used
I've been sighting there and there a Bydesc keyword for function parameters. Is it that Bydesc means passing the descriptor?
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How accessing to the array's descriptor structure? (ONLY FOR FBC VERSION < 1.08)

Post by fxm »

Yes, ByDesc means "by descriptor".
Last edited by fxm on Sep 04, 2019 18:40, edited 1 time in total.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How accessing to the array's descriptor structure? (ONLY FOR FBC VERSION < 1.08)

Post by fxm »

@dkl,

When passing an array to a procedure, it seems that the behavior is different if this array is a member fix-len array of an UDT:
- For any other array kind, the same descriptor seems to be successively passed to several procedures.
- For a member len-fix array of an UDT, a different descriptor address is passed at each procedure.
(for 64-bit,compile with '-exx')

Code: Select all

Function arrayDescriptorPtrFunction (Byval p As Any Ptr) As Any Ptr
  Return p
End function
#Define arrayDescriptorPtr(array) _
Cast(Function (() As Typeof((array))) As Any Ptr, @arrayDescriptorPtrFunction)(array())


Dim As Longint test1(0 to 9, 1 to 100)
Dim Shared As Longint test2(0 to 9, 1 to 100)
Dim As Longint test3()
Dim Shared As Longint test4()
Type UDT
  Dim As Longint test1(0 to 9, 1 to 100)
  Dim As Longint test2(Any, Any)
End Type
Dim As UDT u

Print arrayDescriptorPtr(test1), arrayDescriptorPtr(test1), arrayDescriptorPtr(test1)
Print
Print arrayDescriptorPtr(test2), arrayDescriptorPtr(test2), arrayDescriptorPtr(test2)
Print
Print arrayDescriptorPtr(test3), arrayDescriptorPtr(test3), arrayDescriptorPtr(test3)
Redim test3(0 to 9, 1 to 100)
Print arrayDescriptorPtr(test3), arrayDescriptorPtr(test3), arrayDescriptorPtr(test3)
Print
Print arrayDescriptorPtr(test4), arrayDescriptorPtr(test4), arrayDescriptorPtr(test4)
Redim test4(0 to 9, 1 to 100)
Print arrayDescriptorPtr(test4), arrayDescriptorPtr(test4), arrayDescriptorPtr(test4)
Print
Print arrayDescriptorPtr(u.test1), arrayDescriptorPtr(u.test1), arrayDescriptorPtr(u.test1)
Print
Print arrayDescriptorPtr(u.test2), arrayDescriptorPtr(u.test2), arrayDescriptorPtr(u.test2)
Redim u.test2(0 to 9, 1 to 100)
Print arrayDescriptorPtr(u.test2), arrayDescriptorPtr(u.test2), arrayDescriptorPtr(u.test2)

Sleep

Code: Select all

1236872       1236872       1236872

4227244       4227244       4227244

1236756       1236756       1236756
1236756       1236756       1236756

4227288       4227288       4227288
4227288       4227288       4227288

1228588       1228540       1228492       <= ???

1236708       1236708       1236708
1236708       1236708       1236708
Does this case not hide a malfunction (for example memory leak)?


[edit]
Used now 'Typeof((array))' (with double parentheses) to be compatible with array inside a Namespace (see #404).
Last edited by fxm on Sep 04, 2019 18:41, edited 6 times in total.
dkl
Site Admin
Posts: 3235
Joined: Jul 28, 2005 14:45
Location: Germany

Re: How accessing to the array's descriptor structure?

Post by dkl »

It should be "just" a new descriptor at every code location that passes the array as argument. (that's one descriptor per argument in generated code, not per run-time execution of that code)

FB also creates temporary descriptors for local fixed-size arrays, but in that case it's very easy to re-use the same descriptor everytime. For UDT fields this re-use is currently not implemented. It's also a bit more complicated, because UDT fields can be used from multiple scopes, while local vars appear in their scope only.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How accessing to the array's descriptor structure? (ONLY FOR FBC VERSION < 1.08)

Post by fxm »

For the global fix-len arrays ('Dim Shared As Longint test2(0 to 9, 1 to 100)' in my above example), the same descriptor seems also to be passed as argument at each array pass.
Last edited by fxm on Sep 04, 2019 18:41, edited 1 time in total.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How accessing to the array's descriptor structure? (ONLY FOR FBC VERSION < 1.08)

Post by fxm »

@dkl,

By analogy with the var-len strings for which we can get the descriptor address with the syntax '@s' (and '@s[n]' to get the address of one character data):
- Could we also get the descriptor address of a var-len array with the syntax '@array' (already '@array(n)' to get the address of one element)?
Last edited by fxm on Sep 04, 2019 18:41, edited 1 time in total.
dkl
Site Admin
Posts: 3235
Joined: Jul 28, 2005 14:45
Location: Germany

Re: How accessing to the array's descriptor structure?

Post by dkl »

I'm not sure; in case of Strings there is the String type to represent the descriptor, but for arrays, what type should be used? In other words, the language doesn't currently support having pointers to arrays and then accessing the array elements through that.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How accessing to the array's descriptor structure? (ONLY FOR FBC VERSION < 1.08)

Post by fxm »

fxm wrote:An example with a Sub to print the descriptor items and several types of arrays in order to validate the different cases (for 64-bit, compile with '-exx'):

Code: Select all

Function arrayDescriptorPtrFunction (Byval p As Any Ptr) As Any Ptr
  Return p
End function
#Define arrayDescriptorPtr(array) _
Cast(Function (() As Typeof((array))) As Any Ptr, @arrayDescriptorPtrFunction)(array())

'------------------------------------------------------------------------------------

Sub printArrayDescriptor (Byval p As Any Ptr)
  Dim As Integer Ptr pu = p
  Print "[@array descriptor: "; pu; "]"
  Print "  @array(all_null_indexes)   ="; pu[0]
  Print "  @array(all_min_indexes)    ="; pu[1]
  Print "  array_total_size_in_bytes  ="; pu[2]
  Print "  array_element_size_in_bytes="; pu[3]
  Print "  number_of_array_dimensions ="; pu[4]
  For i As Integer = 1 to pu[4]
    Print "  [dimension number:"; i; "]"
    Print "    number_of_elements="; pu[5 + (i - 1) * 3]
    Print "    min_index         ="; pu[6 + (i - 1) * 3]
    Print "    max_index         ="; pu[7 + (i - 1) * 3]
  Next i
End Sub

'------------------------------------------------------------------------------------

Dim As Longint test1(0 to 9, 1 to 100)
Dim Shared As Longint test2(0 to 9, 1 to 100)
Dim As Longint test3()
Dim Shared As Longint test4()
Type UDT
  Dim As Longint test1(0 to 9, 1 to 100)
  Dim As Longint test2(Any, Any)
End Type
Dim As UDT u

Screen 0
Width , 30

printArrayDescriptor(arrayDescriptorPtr(test1))
Sleep
Cls
printArrayDescriptor(arrayDescriptorPtr(test2))
Sleep
Cls
printArrayDescriptor(arrayDescriptorPtr(test3))
Print
Redim test3(0 to 9, 1 to 100)
printArrayDescriptor(arrayDescriptorPtr(test3))
Sleep
Cls
printArrayDescriptor(arrayDescriptorPtr(test4))
print
Redim test4(0 to 9, 1 to 100)
printArrayDescriptor(arrayDescriptorPtr(test4))
Sleep
Cls
printArrayDescriptor(arrayDescriptorPtr(u.test1))
Sleep
Cls
printArrayDescriptor(arrayDescriptorPtr(u.test2))
print
Redim u.test2(0 to 9, 1 to 100)
printArrayDescriptor(arrayDescriptorPtr(u.test2))

Sleep
Another macro code, but compatible 32/64 bits and with/without -exx option:
(workaround of the bug report #854)

Code: Select all

Function arrayDescriptorPtrFunction (Byval p As Any Ptr) As Any Ptr
  Return p
End function
#macro arrayDescriptorPtr(array, p)
  Scope
    Dim As Function (() As Typeof((array))) As Any Ptr f
    f = Cast(Function (() As Typeof((array))) As Any Ptr, @arrayDescriptorPtrFunction)
    p = f(array())
  End Scope
#endmacro

'------------------------------------------------------------------------------------

Sub printArrayDescriptor (Byval p As Any Ptr)
  Dim As Integer Ptr pu = p
  Print "[@array descriptor: "; pu; "]"
  Print "  @array(all_null_indexes)   ="; pu[0]
  Print "  @array(all_min_indexes)    ="; pu[1]
  Print "  array_total_size_in_bytes  ="; pu[2]
  Print "  array_element_size_in_bytes="; pu[3]
  Print "  number_of_array_dimensions ="; pu[4]
  For i As Integer = 1 to pu[4]
    Print "  [dimension number:"; i; "]"
    Print "    number_of_elements="; pu[5 + (i - 1) * 3]
    Print "    min_index         ="; pu[6 + (i - 1) * 3]
    Print "    max_index         ="; pu[7 + (i - 1) * 3]
  Next i
End Sub

'------------------------------------------------------------------------------------

Dim As Longint test1(0 to 9, 1 to 100)
Dim Shared As Longint test2(0 to 9, 1 to 100)
Dim As Longint test3()
Dim Shared As Longint test4()
Type UDT
  Dim As Longint test1(0 to 9, 1 to 100)
  Dim As Longint test2(Any, Any)
End Type
Dim As UDT u

Dim As Any Ptr p

Screen 0
Width , 30

arrayDescriptorPtr(test1, p)
printArrayDescriptor(p)
Sleep
Cls
arrayDescriptorPtr(test2, p)
printArrayDescriptor(p)
Sleep
Cls
arrayDescriptorPtr(test3, p)
printArrayDescriptor(p)
Print
Redim test3(0 to 9, 1 to 100)
arrayDescriptorPtr(test3, p)
printArrayDescriptor(p)
Sleep
Cls
arrayDescriptorPtr(test4, p)
printArrayDescriptor(p)
print
Redim test4(0 to 9, 1 to 100)
arrayDescriptorPtr(test4, p)
printArrayDescriptor(p)
Sleep
Cls
arrayDescriptorPtr(u.test1, p)
printArrayDescriptor(p)
Sleep
Cls
arrayDescriptorPtr(u.test2, p)
printArrayDescriptor(p)
print
Redim u.test2(0 to 9, 1 to 100)
arrayDescriptorPtr(u.test2, p)
printArrayDescriptor(p)

Sleep
Last edited by fxm on Sep 04, 2019 18:42, edited 1 time in total.
Munair
Posts: 1286
Joined: Oct 19, 2017 15:00
Location: Netherlands
Contact:

Re: How accessing to the array's descriptor structure?

Post by Munair »

@fxm, thanks for sharing.
Munair
Posts: 1286
Joined: Oct 19, 2017 15:00
Location: Netherlands
Contact:

Re: How accessing to the array's descriptor structure?

Post by Munair »

Speaking of bug reports, the first ones in the list date back to October 2012. Looking at "bug" 372, it uses a suspicious pointer assignment, which doesn't look like good coding. Furthermore, there's no converting back using cvshort.

Just wondering, how many of the bug reports are actually serious cases? Who maintains it, or maintained it?
Last edited by Munair on Dec 21, 2017 8:29, edited 3 times in total.
Post Reply