How FB supports References (pass and return byref, byref variables), and How to Use them

Forum for discussion about the documentation project.
Post Reply
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

How FB supports References (pass and return byref, byref variables), and How to Use them

Post by fxm »

References and pointers are closely related.
Indeed, a variable and its different references have the same address, since they allow access to the same object.

In FB, a reference is implemented under the hood through an internal pointer which holds the address of the variable.
The reference encapsulates the manipulation of the address of the variable and is used as a dereferenced pointer.
The difference lies here in the fact that one does not have to perform the dereferencing.

References are much easier to handle than pointers, so they make code much safer.

1) Reminder on pointers
  • An address is a value. User can store this value in a variable. Pointers are precisely variables that contain the address of other objects, for example the address of another variable.
    The value of a pointer may change. This does not usually mean that the pointed variable is moved to memory, but rather that the pointer points to something else.
    In order to precise what is pointed by a pointer, the pointers have a type. This type is usually constructed from the type of the pointed object. This allows the compiler to verify that the manipulations made in memory via the pointer are valid.

    The most common use of pointers is when sent in the parameters of a procedure. They make it possible to manipulate in a simple way data which can be important (instead of providing to a procedure a very big data block, one could for example provide a pointer to this one...).
    Another use is when doing dynamic allocation in memory, with the '[C]Allocate' and 'New' keywords which return the address of the memory that has been allocated. This address must be stored somewhere and know how to use it.
    Finally, pointers can also be used to manipulate tables, but their interest is lower with FB because user has the capacity to declare static and dynamic arrays and to use the dedicated '()' (Array Index) operator to access their elements.

    It is also possible to create pointers to procedures, and to use these pointers to parameterize an algorithm, the behavior of which will depend on the procedures thus pointed out.

    It is very important to make sure that the pointers that user is manipulating are all initialized (that is, they contain the address of a valid object, not just anything). Indeed, to access a variable by means of an uninitialized pointer amounts to read or, more seriously, to write in the memory at a completely random place (according to the initial value of the pointer at the time of its creation).
    In general, pointers are initialized as soon as they are created, or if they are to be used later, they are initialized with the null value. This will allow future tests on the validity of the pointer or at least to detect errors. Indeed, using a pointer with a null value to access a variable can often generate a program protection fault, but will always generate an error message at run-time if the program was compiled with the option '-exx'.

    With the dereferencing operators '*' and '[]', it is possible to access the variables from pointers.
    The 'Any Ptr' pointers are a special type of pointer. They can point to a variable of any type. You can not use the '*' and '[]' dereferencing operators on an 'Any Ptr' pointer. It must first be converted into a pointer of a given type.
    Similarly, a typed pointer can also be converted into a pointer of any other type (if necessary converting first through an 'Any Ptr' pointer).
    For object pointers ('po'), it's more convenient to use the single '->' operator rather than combining the "*" and "." to access object members (use 'po->member' instead of '(*po).member').

    A pointer can also be declared with a type different but compatible with the address types of objects to be pointed (as a simple example, 'Zstring Ptr' and 'Ubyte Ptr' are two compatible pointer types in both directions). In a more evolved way in case of inheritance structure, derived type objects can be referenced with pointers constructed from the type of one among those of their common bases (allowing to activate the polymorphism when a virtual and overridden method is called on such pointers).

    Pointers have their own arithmetic. Increment/decrement is special for this type. This is done by multiplying the size of the type of the pointer by the value that one wants to add/remove to it. This makes it possible to properly move the pointer forward or back from the number of elements indicated.
    The only operation allowed between pointers is the subtraction, provided the pointers are of the same type. This operation only makes sense if the pointers point to the same homogeneous structure of elements (of the type of the pointers), because the result corresponds to a difference in number of elements.
2) Notion of reference
  • In higher-level languages, the use of pointers tends to be suppressed, in favor of references and dynamic arrays managed by the compiler. References fulfill some pointer functions by removing explicit user access to memory. This avoids many problems, in return some uses and optimizations are no longer possible.

    In addition to the two classic ways to access data represented in memory: use its name (if it is a variable or a constant) or dereference a pointer containing its address, there is a third method, the use of references, which is absolutely essential in some cases and provides, in addition, an alternative to the use of a pointer.

    A reference is a way to associate a new name with an already existing object.
    It is not a means to create a new object, even if the syntax to create a reference variable leads to an expression strangely resembling the definition of an initialized variable.
    This assignment of a new name does not deprive the object concerned of its original name, if it had one. The new name simply becomes a synonym for the old one, meaning that they both refer to the same object.

    Note:
    • The arrays of references are not supported yet in FB.
    • The non-static reference fields for UDT are not supported yet in FB.
    • The references to procedures are not supported in FB (only references to pointers to procedures).
    To pass/return a variable by reference instead of doing it by value avoids making a copy of the transmitted object. When the object in question is very large, the copy operation can be expensive in run-time and in memory.

    In the manner of a pointer, a reference has its own type, which must be identical to that of the referenced object, otherwise compatible at least. In this last case only (type compatible but not the same), references (like pointers) can not do all same operations than with the original object.

    The use of a reference to an object makes it possible to obtain the advantages associated with the use of a "pointer to object" (speed and memory savings) while avoiding the heaviness of writing implied by the use of an object pointer (usage of the '@' and '*/[]' operators).

    The different syntaxes used to declare a reference all use the keyword 'Byref'.
    Since a pointer is a variable, it is possible to modify its contents, and the same pointer can allow successive access to different variables. The association between a reference and the object that it designates is, however, fixed when it is declared.

    The 'byref' keyword indicates a variable that is declared by reference. It is used in three different contexts:
    • In a procedure signature, to pass an argument by reference (byref parameter).
    • In a function signature, to return a variable to the caller by reference (byref return).
    • In the body of the code, to define a reference variable (byref variable).
3) Passing parameter by reference to procedure (byref parameter)
  • Syntax of declaration:
    • Sub/Function procedure_name (Byref parameter As [Const] data_type, ...
    When used in the parameter list of a procedure, the 'byref' keyword indicates that an argument is passed by reference, not by value. The consequence is that any modification made to the argument in the called procedure is reflected in the body of the call.
    If the procedure does not need or must not to modify the transmitted object, the 'Const' qualifier can be used in the declaration (before the declaration of the 'data_type') so that the compiler checks in the body of the procedure that the passed object is not modified in any place (otherwise, a compiler error message is issued).
  • Full syntax example for passing a parameter by reference:

    Code: Select all

    Declare Sub passbyref (Byref ref As Double, Byval value As Double)  '' declaration for passing by reference
    
    Dim As Double X = 0
    Print X
    passbyref(X, 1.23)
    Print X
    
    Sleep
    
    Sub passbyref (Byref ref As Double, Byval value As Double)  '' declaration for passing by reference
      ref = value
    End Sub
    

    Code: Select all

     0
     1.23
4) Returning variable by reference from function (byref return)
  • Syntax of declaration:
    • Function function_name (...) Byref As [Const] data_type
    When used in the return type of a function, the 'byref' keyword indicates that the variable is returned by reference, not by value. The consequence is that the caller can modify the variable returned by the function and the modification is reflected in the state of the variable that the function processes.
    If the caller does not need or must not to modify the transmitted object, the 'Const' qualifier can be used in the declaration (before the declaration of the 'data_type') so that the compiler checks in the body of the caller that the returned object is not modified in any place (otherwise, a compiler error message is issued).

    Operators (member or global), when used as functions, have also the capability to return results by reference, by using the similar syntax.
  • Full syntax example for returning a variable by reference:

    Code: Select all

    Declare Function returnbyref () Byref As Double  '' declaration for returning by reference
    
    Print returnbyref()
    returnbyref() = 4.56
    Print returnbyref()
    
    Sleep
    
    Function returnbyref () Byref As Double  '' declaration for returning by reference
      Static As Double X = 0
      Return X
    End Function
    

    Code: Select all

     0
     4.56
    • As for the arguments list, it should always be surrounded with parentheses even if empty.
    Note:
    On the left-hand side of an assignment expression using the '=' symbol, the result of the function (returned by reference) must be enclosed in parentheses when the function calls one single argument, in order to solve the parsing ambiguity.
    From fbc version 0.90, '=>' can be used for assignments, in place of '=', same as for initializers, allowing to avoid parsing ambiguity (without parentheses):

    Code: Select all

    Declare Function transitbyref( ByRef _s As String ) ByRef As String
    
    Dim As String s
    
    s = "abcd"
    Print s
    
    '' the enclosing parentheses are required here.
    ( transitbyref( s ) ) = transitbyref( s ) & "efgh"
    Print s
    
    '' the enclosing parentheses are not required here.
    transitbyref( s ) => transitbyref( s ) & "ijkl"
    Print s
    
    Sleep
    
    Function transitbyref( ByRef _s As String ) ByRef As String
       '' This var-len string will transit by reference (input and output), no copy will be created.
       Return _s
    End Function
    

    Code: Select all

    abcd
    abcdefgh
    abcdefghijkl
5) Defining reference variable in code (byref variable)
  • Syntax of declaration:
    • Dim/Static [Shared] Byref As [Const] data_type ref = variable
      or
      Var [Shared] Byref ref = variable
    Unlike pointers, the reference variable must be assigned as soon as the declaration using an initializer.
    'data_type' must be the same type as that of the variable, or a compatible type (for example one from the types of its Bases in case of inheritance):
    • Only when the two types are identical (or using the second syntax with 'Var'), a reference variable can be considered as an alias of the variable. One can do the same operations through such a reference variable as one can do with the original variable.
    • Otherwise (types compatible but not identical), one can not do all same operations than with the original variable:
      • For example, a base type reference variable referring to a derived type object allows to activate polymorphism when a virtual method is called on it, similarly to a base type pointer referring to a derived type object. One can do the same operations through such a reference variable as one can do with a dereferenced pointer of same type (but for both not the same operations as using directly the derived type instance).
    If the code does not need or must not to modify the referred object, the 'Const' qualifier can be used in the declaration (before the declaration of the 'data_type' in the first syntax) so that the compiler checks in the code that the object is not modified, through the reference variable, in any place (otherwise, a compiler error message is issued).

    There is no interaction between the life of a reference and the life of the object who is referred (similarly to a pointer: destroy an object does not destroy its pointer(s)).
    Once created, each one lives his life independently.
  • Full syntax example for defining a reference variable in code:

    Code: Select all

    Dim As Double X = 0
    
    Dim Byref As Double refX = X  '' declaration for defining a reference
    Print X
    refX = 7.89
    Print X
    
    Sleep
    

    Code: Select all

     0
     7.89
6) References versus pointers by comparative examples
  • Function returning the greater variable between two integer variables:
    • Using pointers (by passing/returning pointer variables):

      Code: Select all

      Function maxPtr (Byval p1 As Integer Ptr, Byval p2 As Integer Ptr) As Integer Ptr
        If *p1 > *p2 Then
          Return p1
        Else
          Return p2
        End If
      End Function
      
      Dim As Integer i1 = 1, i2 = 2
      Print i1, i2
      *maxPtr(@i1, @i2) = 3
      Print i1, i2
      
      Sleep
      

      Code: Select all

       1             2
       1             3
    • Using references (by passing/returning reference variables):

      Code: Select all

      Function maxRef (Byref r1 As Integer, Byref r2 As Integer) Byref As Integer
        If r1 > r2 Then
          Return r1
        Else
          Return r2
        End If
      End Function
      
      Dim As Integer i1 = 1, i2 = 2
      Print i1, i2
      maxRef(i1, i2) = 3
      Print i1, i2
      
      Sleep
      

      Code: Select all

       1             2
       1             3
  • Inheritance structure with overriding subroutine and overriding function with covariant return:
    • Using pointers to objects:

      Code: Select all

      Type myBase Extends Object
        Declare Virtual Function clone () As myBase Ptr
        Declare Virtual Sub Destroy ()
      End Type
      
      Function myBase.clone () As myBase Ptr
        Dim As myBase Ptr pp = New myBase(This)
        Print "myBase.clone() As myBase Ptr", pp
        Function = pp
      End Function
      
      Sub myBase.Destroy ()
        Print "myBase.Destroy()", , @This
        Delete @This
      End Sub
      
      
      Type myDerived Extends myBase
        Declare Function clone () As myDerived Ptr override  '' overriding member function with covariant return
        Declare Sub Destroy () override                      '' overriding member subroutine
      End Type
      
      Function myDerived.clone () As myDerived Ptr      '' overriding member function with covariant return
        Dim As myDerived Ptr pc = New myDerived(This)
        Print "myDerived.clone() As myDerived Ptr", pc
        Function = pc
      End Function
      
      Sub myDerived.Destroy ()                '' overriding member subroutine
        Print "myDerived.Destroy()", , @This
        Delete @This
      End Sub
      
      
      Dim As myDerived c
      
      Dim As myBase Ptr ppc = @c                '' base type pointer to derived object c
      Dim As myDerived Ptr pcc = @c             '' derived type pointer to derived object c
      
      Dim As myBase Ptr ppc1 = ppc->clone()     '' base type pointer to clone of object c
      '                                              (through its base type pointer and polymorphism)
      Dim As myDerived Ptr pcc1 = pcc->clone()  '' derived type pointer to derived object c
      '                                              (through its derived type pointer and covariance of return value)
      Print
      ppc1->Destroy()                           '' using base type pointer and polymorphism
      pcc1->Destroy()                           '' using derived type pointer
      
      Sleep
      

      Code: Select all

      myDerived.clone() As myDerived Ptr        4663904
      myDerived.clone() As myDerived Ptr        4663952
      
      myDerived.Destroy()                       4663904
      myDerived.Destroy()                       4663952
    • Using references to objects:

      Code: Select all

      Type myBase Extends Object
        Declare Virtual Function clone () Byref As myBase
        Declare Virtual Sub Destroy ()
      End Type
      
      Function myBase.clone () Byref As myBase
        Dim As myBase Ptr pp = New myBase(This)
        Print "myBase.clone() Byref As myBase", pp
        Function = *pp
      End Function
      
      Sub myBase.Destroy ()
        Print "myBase.Destroy()", , @This
        Delete @This
      End Sub
      
      
      Type myDerived Extends myBase
        Declare Function clone () Byref As myDerived override  '' overriding member function with covariant return
        Declare Sub Destroy () override                        '' overriding member subroutine
      End Type
      
      Function myDerived.clone () Byref As myDerived      '' overriding member function with covariant return
        Dim As myDerived Ptr pc = New myDerived(This)
        Print "myDerived.clone() Byref As myDerived", pc
        Function = *pc
      End Function
      
      Sub myDerived.Destroy ()                '' overriding member subroutine
        Print "myDerived.Destroy()", , @This
        Delete @This
      End Sub
      
      
      Dim As myDerived c
      
      Dim Byref As myBase rpc = c                '' base type reference to derived object c
      Dim Byref As myDerived rcc = c             '' derived type reference to derived object c
      
      Dim Byref As myBase rpc1 = rpc.clone()     '' base type reference to clone of object c
      '                                               (through its base type reference and polymorphism)
      Dim Byref As myDerived rcc1 = rcc.clone()  '' derived type reference to derived object c
      '                                               (through its derived type reference and covariance of return value)
      Print
      rpc1.Destroy()                             '' using base typpe reference and polymorphism
      rcc1.Destroy()                             '' using derived type reference
      
      Sleep
      

      Code: Select all

      myDerived.clone() Byref As myDerived      9775712
      myDerived.clone() Byref As myDerived      9775760
      
      myDerived.Destroy()                       9775712
      myDerived.Destroy()                       9775760
7) Hacking on usage of references with the additional syntaxes allowed by FreeBASIC
  • In FB, a reference is implemented under the hood through an internal pointer which holds the address of the variable.
    The access to this internal pointer is presently allowed for user, both in read and write (unlike many other languages):
    • Therefore, the address of the referred variable (the value of the internal pointer) can be get by using the '@' operator applied on the reference variable symbol name:
      variable_address = @ref
    • And even, a reference can be reassigned (by modifying the value of the internal pointer) to refer to another variable (of compatible type) by doing:
      @ref = @other_variable
    • The address of the internal pointer can even be obtained:
      internal_pointer_address = @@ref
    Note:
    • A reference can also be re-initialized to a "null" reference:
      @ref = 0
    • A reference can even be directly declared as a null reference:
      Dim Byref As data_type ref = *Cptr(data_type Ptr, 0)
    Thus, by always using the same reference symbol name, one can mix the pure syntax on the reference with the syntax on its internal pointer.
  • Example of hacking on reference symbol name:

    Code: Select all

    Declare Function resizeZstring (Byref refZstring As Zstring, Byval length As Integer) Byref As Zstring
    Declare Sub prntZstring (Byref refZstring As Zstring)
    
    Dim Byref As Zstring refZ = *Cptr(Zstring Ptr, 0)  '' "null" reference declaration
    
    Const cz1 = "FB"
    @refZ = @(resizeZstring(refZ, Len(cz1)))           '' reference (re-)inititialization
    refZ = cz1
    prntZstring(refZ)
    
    Const cz2 = "FreeBASIC"
    @refZ = @(resizeZstring(refZ, Len(cz2)))           '' reference re-inititialization
    refZ = cz2
    prntZstring(refZ)
    
    Const cz3 = "FreeBASIC 1.06.0"
    @refZ = @(resizeZstring(refZ, Len(cz3)))           '' reference re-inititialization
    refZ = cz3
    prntZstring(refZ)
    
    Const cz4 = ""
    @refZ = @(resizeZstring(refZ, Len(cz4)))           '' reference re-inititialization to "null" reference
    refZ = cz4
    prntZstring(refZ)
    
    Sleep
    
    Function resizeZstring (Byref refZstring As Zstring, Byval length As Integer) Byref As Zstring
      If length > 0 Then
        If @refZstring = 0 Then
          Print "Zstring memory buffer allocation"
        Else
          Print "Zstring memory buffer re-allocation"
        End If
        length += 1
      Else
        Print "Zstring memory buffer de-allocation"
      End If
    '  Return *Cptr(Zstring Ptr, Reallocate(@refZstring, length * Sizeof(Zstring)))
    '  '' Using the "Return Byval ..." syntax allows to avoid casting + dereferencing as above
      Return Byval Reallocate(@refZstring, length * Sizeof(Zstring))
    End Function
    
    Sub prntZstring (Byref refZstring As Zstring)
      Print "  " & @refZstring, "'" & refZstring & "'"
      Print
    End Sub
    

    Code: Select all

    Zstring memory buffer allocation
      9513600     'FB'
    
    Zstring memory buffer re-allocation
      9513600     'FreeBASIC'
    
    Zstring memory buffer re-allocation
      9513600     'FreeBASIC 1.06.0'
    
    Zstring memory buffer de-allocation
      0           ''
badidea
Posts: 2586
Joined: May 24, 2007 22:10
Location: The Netherlands

Re: How FB supports References (pass and return byref, byref variables), and How to Use them

Post by badidea »

Maybe I should read this, long post. I'm getting confused, this code is not good:

Code: Select all

function loadData(pData as ubyte ptr) as integer
	dim as integer nBytes = 10
	pData = allocate(nBytes)
	if pData = 0 then return -1
	'<actual loading code placeholder>
	return nBytes
end function

dim as ubyte ptr pData
dim as integer nBytes

nBytes = loadData(pData)
print nBytes
if pData <> 0 then deallocate(pData) else print "Free error"
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How FB supports References (pass and return byref, byref variables), and How to Use them

Post by fxm »

In order for the caller to retrieve the value of the pointer assigned within the procedure, the corresponding parameter (the pointer) must be passed by reference and not by value (by default, a pointer is always passed by value in -lang fb):

Code: Select all

function loadData(Byref pData as ubyte ptr) as integer
Documentation wrote: - In -lang fb dialect, Byval is the default parameter passing convention for all built-in types except String and user-defined Type which are passed Byref by default. The Zstring and Wstring built-in types are also passed Byref by default, but passing Byval is forbidden. Arrays are always passed Byref and the use of the specifier Byref or Byval is forbidden.
- In -lang qb and -lang fblite dialects, Byref is the default parameter passing convention.
badidea
Posts: 2586
Joined: May 24, 2007 22:10
Location: The Netherlands

Re: How FB supports References (pass and return byref, byref variables), and How to Use them

Post by badidea »

Of course, the function needs to know the location (address) of the original pData in order to change its contents (the location of the 10 byte data). I got confused when trying a "ubyte ptr ptr" in the function.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How FB supports References (pass and return byref, byref variables), and How to Use them

Post by fxm »

It works also if we pass to the function a Ptr Ptr by value:

Code: Select all

function loadData(Byval ppData as ubyte ptr ptr) as integer
   dim as integer nBytes = 10
   *ppData = allocate(nBytes)
   if *ppData = 0 then return -1
   '<actual loading code placeholder>
   return nBytes
end function

dim as ubyte ptr pData
dim as integer nBytes

nBytes = loadData(@pData)
print nBytes
if pData <> 0 then deallocate(pData) else print "Free error"
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: How FB supports References (pass and return byref, byref variables), and How to Use them

Post by coderJeff »

grindstone wrote:
coderJeff wrote:Is it possible to discuss references without discussing pointers?
That depends. If you only want to know how to use it, it's enough to know that changing a paramter submitted BYREF will affect the value outside of the procedure, while the changes made to a BYVAL parameter are limited to the inside of the procedure. But if you want to know how it works, it's essential to understand the pointer stuff.
grindstone, true, good point.

While fbc uses pointers to implement references, it's not that is must be a pointer, it's just convenient for the compiler writer to use a pointer. The reference manager might give out references in the form of a key, or index, or hash, or any such token that uniquely identifies a "thing" to be referenced. Some languages use references exclusively such that even when assigning one variable to another and you think you are copying a value, really just copying references, and changing values only with copy-on-write. But can be sure it's pointers somewhere in there, just hidden from the programmer.

fxm, again, well written article. Even, if all the parts on pointers were deleted, would still be a nice introduction on how to use references. Good job.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How FB supports References (pass and return byref, byref variables), and How to Use them

Post by fxm »

coderJeff wrote:Even, if all the parts on pointers were deleted, ...
fxm wrote:
MrSwiss wrote:@fxm,
your introduction part about pointers, should be made a article "in itself",
because it applies to more cases, than just the "byref" stuff.
It's just a reminder.
There are already several pages on pointers in documentation (see Programmer's Guide, Community Tutorials).
Munair
Posts: 1286
Joined: Oct 19, 2017 15:00
Location: Netherlands
Contact:

Re: How FB supports References (pass and return byref, byref variables), and How to Use them

Post by Munair »

This is very useful.
Post Reply