How FB supports References (pass and return byref, byref variables), and How to Use them

[fxm]

References and pointers are closely related. Indeed, a variable and its different references have the same address, since they allow access to the same object. In FB, a reference is implemented under the hood through an internal pointer which holds the address of the variable.
To use a reference to manipulate an object is therefore the same as manipulating a pointer containing the address of this object.
The reference encapsulates the manipulation of the address of the variable and is used as a dereferenced pointer. The difference lies here in the fact that one does not have to perform the dereferencing.
References are much easier to handle than pointers, so they make code much safer.

1) Reminder on pointers

An address is a value. User can store this value in a variable. Pointers are precisely variables that contain the address of other objects, for example the address of another variable.
The value of a pointer may change. This does not usually mean that the pointed variable is moved to memory, but rather that the pointer points to something else.
In order to precise what is pointed by a pointer, the pointers have a type. This type is usually constructed from the type of the pointed object. This allows the compiler to verify that the manipulations made in memory via the pointer are valid.

The most common use of pointers is when sent in the parameters of a procedure. They make it possible to manipulate in a simple way data which can be important (instead of providing to a procedure a very big data block, one could for example provide a pointer to this one...).
Another use is when doing dynamic allocation in memory, with the '[C]Allocate' and 'New' keywords which return the address of the memory that has been allocated. This address must be stored somewhere and know how to use it.
Finally, pointers can also be used to manipulate tables, but their interest is lower with FB because user has the capacity to declare static and dynamic arrays and to use the dedicated '()' (Array Index) operator to access their elements.

It is also possible to create pointers to procedures, and to use these pointers to parameterize an algorithm, the behavior of which will depend on the procedures thus pointed out. But this is not the subject of this article, because the references can not do something equivalent (no references to procedures but only references to pointers to procedures).

It is very important to make sure that the pointers that user is manipulating are all initialized (that is, they contain the address of a valid object, not just anything). Indeed, to access a variable by means of an uninitialized pointer amounts to read or, more seriously, to write in the memory at a completely random place (according to the initial value of the pointer at the time of its creation).
In general, pointers are initialized as soon as they are created, or if they are to be used later, they are initialized with the null value. This will allow future tests on the validity of the pointer or at least to detect errors. Indeed, using a pointer with a null value to access a variable can often generate a program protection fault, but will always generate an error message at run-time if the program was compiled with the option '-exx'.

With the dereferencing operators '*' and '[]', it is possible to access the variables from pointers.
The 'Any Ptr' pointers are a special type of pointer. They can point to a variable of any type. You can not use the '*' and '[]' deferencing operators on an 'Any Ptr' pointer. It must first be converted into a pointer of a given type.
Similarly, a typed pointer can also be converted into a pointer of any other type (if necessary converting first through an 'Any Ptr' pointer).
For object pointers ('po'), it's more convenient to use the single '->' operator rather than combining the "*" and "." to access object members (use 'po->member' instead of '(*po).member').

Pointers have their own arithmetic. Increment/decrement is special for this type. This is done by multiplying the size of the type of the pointer by the value that one wants to add/remove to it. This makes it possible to properly move the pointer forward or back from the number of elements indicated.
The only operation allowed between pointers is the subtraction, provided the pointers are of the same type. This operation only makes sense if the pointers point to the same homogeneous structure of elements (of the type of the pointers), because the result corresponds to a difference in number of elements.

In higher-level languages, the use of pointers tends to be suppressed, in favor of references and dynamic arrays managed by the compiler. References fulfill some pointer functions by removing explicit user access to memory. This avoids many problems, in return some uses and optimizations are no longer possible.

2) Notion of reference

In addition to the two classic ways to access data represented in memory: use its name (if it is a variable or a constant) or dereference a pointer containing its address, there is a third method, the use of references, which is absolutely essential in some cases and provides, in addition, an alternative to the use of a pointer.

A reference is a way to associate a new name with an already existing object.
It is not a means to create a new object, even if the syntax to create a reference variable leads to an expression strangely resembling the definition of an initialized variable.
This assignment of a new name does not deprive the object concerned of its original name, if it had one. The new name simply becomes a synonym for the old one, meaning that they both refer to the same object.

The different syntaxes used to declare a reference all use the keyword 'Byref'.
Since a pointer is a variable, it is possible to modify its contents, and the same pointer can allow successive access to different variables. The association between a reference and the object that it designates is, however, fixed when it is declared.

The 'byref' keyword indicates a variable that is declared by reference. It is used in three different contexts:

• In a procedure signature, to pass an argument by reference (byref parameter).
  
• In a function signature, to return a variable to the caller by reference (byref return).
• In the body of the code, to define a reference variable (byref variable).
To pass/return a variable by reference instead of doing it by value avoids making a copy of the transmitted object. When the object in question is very large, the copy operation can be expensive in run-time and in memory.
The use of a reference to an object makes it possible to obtain the advantages associated with the use of a "pointer to object" (speed and memory savings) while avoiding the heaviness of writing implied by the use of an object pointer (usage of the '@' and '*/[]' operators).

3) Passing parameter by reference to procedure (byref parameter)

• Syntax of declaration:
  
  Sub/Function procedure_name (Byref parameter As [Const] data_type, ...
  When used in the parameter list of a procedure, the 'byref' keyword indicates that an argument is passed by reference, not by value. The consequence is that any modification made to the argument in the called procedure is reflected in the body of the call.
  If the procedure does not need or must not to modify the transmitted object, the 'Const' qualifier can be used in the declaration (before the declaration of the 'data_type') so that the compiler checks in the body of the procedure that the passed object is not modified in any place (otherwise, a compiler error message is issued).
  
  
• Full syntax example for passing a parameter by reference:
  

  Code: Select all

  Declare Sub passbyref (Byref ref As Double, Byval value As Double)  '' declaration for passing by reference

Dim As Double X = 0
Print X
passbyref(X, 1.23)
Print X

Sleep

Sub passbyref (Byref ref As Double, Byval value As Double)  '' declaration for passing by reference
  ref = value
End Sub


  Code: Select all

   0
 1.23

4) Returning variable by reference from function (byref return)

• Syntax of declaration:
  
  Function function_name (...) Byref As [Const] data_type
  When used in the return type of a function, the 'byref' keyword indicates that the variable is returned by reference, not by value. The consequence is that the caller can modify the variable returned by the function and the modification is reflected in the state of the variable that the function processes.
  If the caller does not need or must not to modify the transmitted object, the 'Const' qualifier can be used in the declaration (before the declaration of the 'data_type') so that the compiler checks in the body of the caller that the returned object is not modified in any place (otherwise, a compiler error message is issued).
  
  Operators (member or global), when used as functions, have also the capability to return results by reference, by using the similar syntax.
  
  
• Full syntax example for returning a variable by reference:
  

  Code: Select all

  Declare Function returnbyref () Byref As Double  '' declaration for returning by reference

Print returnbyref()
returnbyref() = 4.56
Print returnbyref()

Sleep

Function returnbyref () Byref As Double  '' declaration for returning by reference
  Static As Double X = 0
  Return X
End Function


  Code: Select all

   0
 4.56
  As for the arguments list, it should always be surrounded with parentheses even if empty.
  Note:
  On the left-hand side of an assignment expression using the '=' symbol, the result of the function (returned by reference) must be enclosed in parentheses when the function calls one single argument, in order to solve the parsing ambiguity.
  From fbc version 0.90, '=>' can be used for assignments, in place of '=', same as for initializers, allowing to avoid parsing ambiguity (without parentheses):
  

  Code: Select all

  Declare Function transitbyref( ByRef _s As String ) ByRef As String

Dim As String s

s = "abcd"
Print s

'' the enclosing parentheses are required here.
( transitbyref( s ) ) = transitbyref( s ) & "efgh"
Print s

'' the enclosing parentheses are not required here.
transitbyref( s ) => transitbyref( s ) & "ijkl"
Print s

Sleep

Function transitbyref( ByRef _s As String ) ByRef As String
   '' This var-len string will transit by reference (input and output), no copy will be created.
   Return _s
End Function


  Code: Select all

  abcd
abcdefgh
abcdefghijkl

5) Defining reference variable in code (byref variable)

• Syntax of declaration:
  
  Dim/Static [Shared] Byref As [Const] data_type ref = variable
  or
  Var [Shared] Byref ref = variable
  Unlike pointers, the reference variable must be assigned as soon as the declaration using an initializer.
  'data_type' must be the same type as that of the variable, or a compatible type (for example one from the types of its Bases in case of inheritance):
  
  • Only when the two types are identical (or using the second syntax with Var), a reference variable can be considered as an alias of the variable. One can do the same operations through such a reference variable as one can do with the original variable.
  • Otherwise (types compatible but not identical), one cannot do all same operations than with the original variable:
    
    For example, a base type reference variable referring to a derived type object allows to activate polymorphism when a virtual method is called on it, similarly to a base type pointer referring to a derived type object. One can do the same operations through such a reference variable as one can do with a dereferenced pointer of same type (but for both not the same operations as using directly the derived type instance).
  If the code does not need or must not to modify the referred object, the 'Const' qualifier can be used in the declaration (before the declaration of the 'data_type' in the first syntax) so that the compiler checks in the code that the object is not modified, through the reference variable, in any place (otherwise, a compiler error message is issued).
  
  There is no interaction between the life of a reference and the life of the object who is referred (similarly to a pointer: destroy an object does not destroy its pointer(s)).
  Once created, each one lives his life independently.
  
  Note:
  
  • The arrays of references are not supported yet in FB.
  • The non-static reference fields for UDT are not supported yet in FB.
• Full syntax example for defining a reference variable in code:
  

  Code: Select all

  Dim As Double X = 0

Dim Byref As Double refX = X  '' declaration for defining a reference
Print X
refX = 7.89
Print X

Sleep


  Code: Select all

   0
 7.89

6) References versus pointers by comparative examples

• Function returning the greater variable between two integer variables:
  
  • Using pointers (by passing/returning pointer variables):
    

    Code: Select all

    Function maxPtr (Byval p1 As Integer Ptr, Byval p2 As Integer Ptr) As Integer Ptr
  If *p1 > *p2 Then
    Return p1
  Else
    Return p2
  End If
End Function

Dim As Integer i1 = 1, i2 = 2
Print i1, i2
*maxPtr(@i1, @i2) = 3
Print i1, i2

Sleep


    Code: Select all

     1             2
 1             3
  • Using references (by passing/returning reference variables):
    

    Code: Select all

    Function maxRef (Byref r1 As Integer, Byref r2 As Integer) Byref As Integer
  If r1 > r2 Then
    Return r1
  Else
    Return r2
  End If
End Function

Dim As Integer i1 = 1, i2 = 2
Print i1, i2
maxRef(i1, i2) = 3
Print i1, i2

Sleep


    Code: Select all

     1             2
 1             3
• Inheritance structure with overriding subroutine and overriding function with covariant return:
  
  • Using pointers to objects:
    

    Code: Select all

    Type myBase Extends Object
  Declare Virtual Function clone () As myBase Ptr
  Declare Virtual Sub Destroy ()
End Type

Function myBase.clone () As myBase Ptr
  Dim As myBase Ptr pp = New myBase(This)
  Print "myBase.clone() As myBase Ptr", pp
  Function = pp
End Function

Sub myBase.Destroy ()
  Print "myBase.Destroy()", , @This
  Delete @This
End Sub


Type myDerived Extends myBase
  Declare Function clone () As myDerived Ptr override  '' overriding member function with covariant return
  Declare Sub Destroy () override                      '' overriding member subroutine
End Type

Function myDerived.clone () As myDerived Ptr      '' overriding member function with covariant return
  Dim As myDerived Ptr pc = New myDerived(This)
  Print "myDerived.clone() As myDerived Ptr", pc
  Function = pc
End Function

Sub myDerived.Destroy ()                '' overriding member subroutine
  Print "myDerived.Destroy()", , @This
  Delete @This
End Sub


Dim As myDerived c

Dim As myBase Ptr ppc = @c                '' base type pointer to derived object c
Dim As myDerived Ptr pcc = @c             '' derived type pointer to derived object c

Dim As myBase Ptr ppc1 = ppc->clone()     '' base type pointer to clone of object c
'                                              (through its base type pointer and polymorphism)
Dim As myDerived Ptr pcc1 = pcc->clone()  '' derived type pointer to derived object c
'                                              (through its derived type pointer and covariance of return value)
Print
ppc1->Destroy()                           '' using base type pointer and polymorphism
pcc1->Destroy()                           '' using derived type pointer

Sleep


    Code: Select all

    myDerived.clone() As myDerived Ptr        4663904
myDerived.clone() As myDerived Ptr        4663952

myDerived.Destroy()                       4663904
myDerived.Destroy()                       4663952
  • Using references to objects:
    

    Code: Select all

    Type myBase Extends Object
  Declare Virtual Function clone () Byref As myBase
  Declare Virtual Sub Destroy ()
End Type

Function myBase.clone () Byref As myBase
  Dim As myBase Ptr pp = New myBase(This)
  Print "myBase.clone() Byref As myBase", pp
  Function = *pp
End Function

Sub myBase.Destroy ()
  Print "myBase.Destroy()", , @This
  Delete @This
End Sub


Type myDerived Extends myBase
  Declare Function clone () Byref As myDerived override  '' overriding member function with covariant return
  Declare Sub Destroy () override                        '' overriding member subroutine
End Type

Function myDerived.clone () Byref As myDerived      '' overriding member function with covariant return
  Dim As myDerived Ptr pc = New myDerived(This)
  Print "myDerived.clone() Byref As myDerived", pc
  Function = *pc
End Function

Sub myDerived.Destroy ()                '' overriding member subroutine
  Print "myDerived.Destroy()", , @This
  Delete @This
End Sub


Dim As myDerived c

Dim Byref As myBase rpc = c                '' base type reference to derived object c
Dim Byref As myDerived rcc = c             '' derived type reference to derived object c

Dim Byref As myBase rpc1 = rpc.clone()     '' base type reference to clone of object c
'                                               (through its base type reference and polymorphism)
Dim Byref As myDerived rcc1 = rcc.clone()  '' derived type reference to derived object c
'                                               (through its derived type reference and covariance of return value)
Print
rpc1.Destroy()                             '' using base typpe reference and polymorphism
rcc1.Destroy()                             '' using derived type reference

Sleep


    Code: Select all

    myDerived.clone() Byref As myDerived      9775712
myDerived.clone() Byref As myDerived      9775760

myDerived.Destroy()                       9775712
myDerived.Destroy()                       9775760

7) Hacking on usage of references with the additional syntaxes allowed by FreeBASIC

In FB, a reference is implemented under the hood through an internal pointer which holds the address of the variable.
The access to this internal pointer is presently allowed for user, both in read and write (unlike many other languages):

• Therefore, the address of the referred variable (the value of the internal pointer) can be get by using the '@' operator applied on the reference variable symbol name:
  variable_address = @ref
  
• And even, a reference can be reassigned (by modifying the value of the internal pointer) to refer to another variable (of compatible type) by doing:
  @ref = @other_variable
• The address of the internal pointer can even be obtained:
  internal_pointer_address = @@ref
Note:

• A reference can also be re-initialized to a "null" reference:
  @ref = 0
• A reference can even be directly declared as a null reference:
  Dim Byref As data_type ref = *Cptr(data_type Ptr, 0)
Thus, by always using the same reference symbol name, one can mix the pure syntax on the reference with the syntax on its internal pointer.


• Example of hacking on reference symbol name:
  

  Code: Select all

  Declare Function resizeZstring (Byref refZstring As Zstring, Byval length As Integer) Byref As Zstring
Declare Sub prntZstring (Byref refZstring As Zstring)

Dim Byref As Zstring refZ = *Cptr(Zstring Ptr, 0)  '' "null" reference declaration

Const cz1 = "FB"
@refZ = @(resizeZstring(refZ, Len(cz1)))           '' reference (re-)inititialization
refZ = cz1
prntZstring(refZ)

Const cz2 = "FreeBASIC"
@refZ = @(resizeZstring(refZ, Len(cz2)))           '' reference re-inititialization
refZ = cz2
prntZstring(refZ)

Const cz3 = "FreeBASIC 1.06.0"
@refZ = @(resizeZstring(refZ, Len(cz3)))           '' reference re-inititialization
refZ = cz3
prntZstring(refZ)

Const cz4 = ""
@refZ = @(resizeZstring(refZ, Len(cz4)))           '' reference re-inititialization to "null" reference
refZ = cz4
prntZstring(refZ)

Sleep

Function resizeZstring (Byref refZstring As Zstring, Byval length As Integer) Byref As Zstring
  If length > 0 Then
    If @refZstring = 0 Then
      Print "Zstring memory buffer allocation"
    Else
      Print "Zstring memory buffer re-allocation"
    End If
    length += 1
  Else
    Print "Zstring memory buffer de-allocation"
  End If
'  Return *Cptr(Zstring Ptr, Reallocate(@refZstring, length * Sizeof(Zstring)))
'  '' Using the "Return Byval ..." syntax allows to avoid casting + dereferencing as above
  Return Byval Reallocate(@refZstring, length * Sizeof(Zstring))
End Function

Sub prntZstring (Byref refZstring As Zstring)
  Print "  " & @refZstring, "'" & refZstring & "'"
  Print
End Sub


  Code: Select all

  Zstring memory buffer allocation
  9513600     'FB'

Zstring memory buffer re-allocation
  9513600     'FreeBASIC'

Zstring memory buffer re-allocation
  9513600     'FreeBASIC 1.06.0'

Zstring memory buffer de-allocation
  0           ''

[badidea]

Maybe I should read this, long post. I'm getting confused, this code is not good:

Code: Select all

function loadData(pData as ubyte ptr) as integer
   dim as integer nBytes = 10
   pData = allocate(nBytes)
   if pData = 0 then return -1
   '<actual loading code placeholder>
   return nBytes
end function

dim as ubyte ptr pData
dim as integer nBytes

nBytes = loadData(pData)
print nBytes
if pData <> 0 then deallocate(pData) else print "Free error"


[fxm]

In order for the caller to retrieve the value of the pointer assigned within the procedure, the corresponding parameter (the pointer) must be passed by reference and not by value (by default, a pointer is always passed by value in -lang fb):

Code: Select all

function loadData(Byref pData as ubyte ptr) as integer

- In -lang fb dialect, Byval is the default parameter passing convention for all built-in types except String and user-defined Type which are passed Byref by default. The Zstring and Wstring built-in types are also passed Byref by default, but passing Byval is forbidden. Arrays are always passed Byref and the use of the specifier Byref or Byval is forbidden.
- In -lang qb and -lang fblite dialects, Byref is the default parameter passing convention.

[badidea]

Of course, the function needs to know the location (address) of the original pData in order to change its contents (the location of the 10 byte data). I got confused when trying a "ubyte ptr ptr" in the function.

[fxm]

It works also if we pass to the function a Ptr Ptr by value:

Code: Select all

function loadData(Byval ppData as ubyte ptr ptr) as integer
   dim as integer nBytes = 10
   *ppData = allocate(nBytes)
   if *ppData = 0 then return -1
   '<actual loading code placeholder>
   return nBytes
end function

dim as ubyte ptr pData
dim as integer nBytes

nBytes = loadData(@pData)
print nBytes
if pData <> 0 then deallocate(pData) else print "Free error"


[coderJeff]

Is it possible to discuss references without discussing pointers?

That depends. If you only want to know how to use it, it's enough to know that changing a paramter submitted BYREF will affect the value outside of the procedure, while the changes made to a BYVAL parameter are limited to the inside of the procedure. But if you want to know how it works, it's essential to understand the pointer stuff.

grindstone, true, good point.

While fbc uses pointers to implement references, it's not that is must be a pointer, it's just convenient for the compiler writer to use a pointer. The reference manager might give out references in the form of a key, or index, or hash, or any such token that uniquely identifies a "thing" to be referenced. Some languages use references exclusively such that even when assigning one variable to another and you think you are copying a value, really just copying references, and changing values only with copy-on-write. But can be sure it's pointers somewhere in there, just hidden from the programmer.

fxm, again, well written article. Even, if all the parts on pointers were deleted, would still be a nice introduction on how to use references. Good job.

[fxm]

Even, if all the parts on pointers were deleted, ...

@fxm,
your introduction part about pointers, should be made a article "in itself",
because it applies to more cases, than just the "byref" stuff.

It's just a reminder.
There are already several pages on pointers in documentation (see Programmer's Guide, Community Tutorials).