FB MemCopy() statement (without CRT includes)

[MrSwiss]

CRT's memcpy() statement, usually included with #Include "crt/string.bi", seems
to be often used by advanced programmers, to solve 'low level' byte copying.
The signature is:

Code: Select all

memcpy(dest as any ptr, src as any ptr, size as size_t) as any ptr

This order of parameters seems to be 'against the grain' since, a cli *copy* works
as follows:

Code: Select all

copy [path]source [path]destination [switches]

Therefore, by re-writing it to FB, I took the liberty, to rearrange the parameters &
change the return type (from any ptr, to boolean). Also, in order to prevent any
ambiguity, I've re-named it, as below:

Code: Select all

MemCopy(ByVal src As Any Ptr, ByVal dst As Any Ptr, ByVal siz As ULong=1) As Boolean

For ease of use it is in a simple .bi file, called MemCopy.bi, which only contains the
one Private Function as well as its declaration. This means, the Function is only
linked in, if it is actually called from main code.

MemCopy.bi:

Code: Select all

' MemCopy.bi -- 2018-07-25, by MrSwiss
'
' only ever include it once (the same, as include guards)
#Pragma Once
' renamed to avoid ambiguity issues with CRT's memcpy (with other param. order & return type)
Declare Function MemCopy(ByVal src As Any Ptr, ByVal dst As Any Ptr, ByVal siz As ULong=1) As Boolean
' ----- MemCopy() implementation -----
Private Function MemCopy( _             ' FB implementation of: CRT's memcpy(@dst, @src, size)
    ByVal src   As Any Ptr, _           ' source ptr (inverted params. from original)
    ByVal dst   As Any Ptr, _           ' target ptr
    ByVal siz   As ULong = 1 _          ' size (in bytes) to copy (default = 1, is min.)
    ) As Boolean                        ' FALSE = OK | TRUE = ERROR (different return-type)
    If src = 0 Or dst = 0 Or siz = 0 Then Return TRUE  ' parameters error checks
    Dim As UByte Ptr ps = src, pd = dst ' convert Any Ptr's to UByte Ptr's (for indexing)
    For i As ULong = 0 To siz - 1       ' amount of bytes to copy (0 to 0 = 1 byte = min.)
        pd[i] = ps[i]                   ' copy them
    Next
    Return FALSE                        ' everything OK (NO ERROR)
End Function
' ----- EOF -----

Test/Demo code MemCopy_test1.bas:

Code: Select all

' MemCopy_test1.bas -- 2018-07-25, by MrSwiss
'
' compile: -s console
'
#Include "MemCopy.bi"   ' instead of: #Include "crt/string.bi" or "crt.bi" (which includes a lot more)

' ===== Demo Code Start =====
Dim As ULongInt     tuli = &hF0F1F2F3F4F5F6F7       ' use hex to be able to check later
Dim As UByte        uba(0 To 7)                     ' same size as ULongInt (above)
Dim As String       msg = "Greetings from FreeBASIC!"   ' source string (to be copied)
Dim As ZString Ptr  psz = CAllocate(Len(msg) + 1)   ' allocate memory, mandatory! (incl. terminator)

Print "MemCopy(), with a numeric variable ..." : Print
Print "ULongInt   : "; Hex(tuli, 16)
If MemCopy(@tuli, @uba(0), SizeOf(ULongInt)) Then   ' SizeOf(ULongInt) can be written as literal: 8
    Print "MemCopy() failed!" : Sleep : End 1       ' in case of error: inform user, then quit
End If
Print "UByte array: ";                  ' suppress LF
For i As Long = 7 To 0 Step -1          ' MSB to LSB output
    Print Hex(uba(i), 2);               ' byte by byte, in hex (no LF)
Next : Print                            ' add a LF (when finished)

Print : Print "MemCopy(), with a string ..." : Print
Print "original: "; msg                         ' below: instead of StrPtr(), SAdd() can be used
If MemCopy(StrPtr(msg), psz, Len(msg)) Then     ' StrPtr() is needed for string-data access, _ 
    Print "MemCopy() failed!" : Sleep : End 2   ' otherwise: @msg, whould point at 'the header'
End If
Print "copy    : "; *psz                ' output: dereferenced ZString Ptr
DeAllocate(psz)                         ' clean up: free allocated memory

Locate CsrLin + 2                       ' add 2 more 'empty' lines
Print "press a key, to EXIT ... ";      ' inform user

Sleep : End 0                           ' wait for user then quit (no error)
' ===== Demo Code End ===== ' ----- EOF -----

Yet another *little code*, that could make a *big difference* ...

[StringEpsilon]

I plugged this into fbJSON and it's quite a lot slower.

My benchmark has 5700002 calls to memcpy / memcopy and 26426818 bytes are copied total (an average of 4.5 bytes per call).

With crt/memcpy: 1.22 seconds

With your function: 1.35 seconds

Not sure what the benefit of using your variant is. The FB runtime statically linked with every FB programm will also use the syscall "memcpy" - so you don't save anything by avoiding the include.

[badidea]

MrSwiss's function copies byte for byte, other implementation often use words, dwords, etc.
The difference seems quite small to me actually, this may be due to caching and/or that the CPU runs faster then memory.

[jj2007]

The CRT memcpy is very difficult to beat. We have tested many alternatives, only a few are faster, see Faster MemCopy for details.

[MrSwiss]

memcpy(dest as any ptr, src as any ptr, n as size_t) as any ptr | Copy one buffer into another .

That is the only definition, found in documentation, which for many means "close to nothing".
Even the comment #Include "crt/string.bi" is in fact, at best misleading, because:
the actual .bi that contains memcpy, is "crt/mem.bi"!

Not sure what the benefit of using your variant is.

To anybody, who knows how to use the orignal, probably nothing.
But to those needing a bit more exhaustive explanations and maybe, a look at the code,
before using a method, who knows (btw: never said a word, about speed) ...

Oohh, before I forget: it is always easier to criticise, than to create something.

[dodicat]

I did something like this a while back.
But I used zstring ptr (one byte at a time)
It was much slower than the C memcpy.
Here I have promoted the leap to long.
It is a bit faster.
But will it be valid in every case -- tall order!
It is surely bound to leak with long.
The C memcpy is obviously greatly optimised and has been tested to destruction.
The seemingly wrong order of parameters is probably justified by the fact that size is put beside the thing you want the size of.
Anyway, thank you Mr Swiss.
Here was my method, but using long instead of zstring ptr.

Code: Select all

 
#include "crt.bi"


    #define datatype long
   #macro memcopy(dest,src,size)
    for n as long=0 to (size)\sizeof(*(src))-1
      ((cast(datatype ptr,dest))[n])=((cast(datatype ptr,src))[n])
    next
   #endmacro
   
   
   dim as ubyte a(1 to 4)
   
   dim as ulong c=rgb(3,6,7)
   
   dim as long lim=10000000
   dim as double t
   
   
   for z as long=1 to 5
   t=timer
   for z as long=1 to lim
   memcopy(@a(1),@c,sizeof(ulong))
next
print timer-t;"  seconds  custom"
print c," =  rgba(";a(3);",";a(2);",";a(1);",";a(4);")"
print

 t=timer
   for z as long=1 to lim
   memcpy(@a(1),@c,sizeof(ulong))
next
print timer-t;"  seconds    crt"
print c," =  rgba(";a(3);",";a(2);",";a(1);",";a(4);")"
print
print "------------------------------------"
next z
print
dim as string s1="Hello world!"
   dim as string s2
   memcopy(@s2,@s1,len(s1)*sizeof(string))
   print "Done   "; s2
sleep

  

-gen gas always of course

[fxm]

Version with 'Long Ptr' pointers, then 'Byte Ptr' pointers:

Code: Select all

Sub memcopy (byval dest As Any Ptr, Byval src As Any Ptr, Byval size As Uinteger)
  Dim As Uinteger Nl = size Shr 2
  If Nl > 0 Then
    Dim As Long Ptr pld = dest, pls = src
    For I As Uinteger = 0 To Nl - 1
      pld[I] = pls[I]
    Next I
  End If
  Dim As Uinteger Nr = size And 3
  If Nr > 0 Then
    Dim As Uinteger Nb = Nl Shl 2
    Dim As Byte Ptr pbd = dest + Nb, pbs = src + Nb
    For I As Uinteger = 0 To Nr - 1
      pbd[I] = pbs[I]
    Next I
  End If
End Sub

[fxm]

Version with 'Integer Ptr' pointers, then 'Byte Ptr' pointers:
(compatible with 32-bit and 64-bit compilers)

Code: Select all

Sub memcopy (byval dest As Any Ptr, Byval src As Any Ptr, Byval size As Uinteger)
  Const As Uinteger K = Log(Sizeof(Integer)) / Log(2)
  Dim As Uinteger Nl = size Shr K
  If Nl > 0 Then
    Dim As Integer Ptr pld = dest, pls = src
    For I As Uinteger = 0 To Nl - 1
      pld[I] = pls[I]
    Next I
  End If
  Dim As Uinteger Nr = size And Sizeof(Integer) - 1
  If Nr > 0 Then
    Dim As Uinteger Nb = Nl Shl K
    Dim As Byte Ptr pbd = dest + Nb, pbs = src + Nb
    For I As Uinteger = 0 To Nr - 1
      pbd[I] = pbs[I]
    Next I
  End If
End Sub

[edit]
- Execution time improvement:
Replace:

Code: Select all

  Dim As Uinteger Nl = size \ Sizeof(Integer)
.....
    Dim As Uinteger Nb = Nl * Sizeof(Integer)

with:

Code: Select all

  Const As Uinteger K = Log(Sizeof(Integer)) / Log(2)
  Dim As Uinteger Nl = size Shr K
.....
    Dim As Uinteger Nb = Nl Shl K

[jj2007]

Why is FB so damn complicated? I tried to pass a pointer to the FB source string to crt memcpy, no chance with a FreeBasic string. One might be tempted to use something like this:

Code: Select all

memcpy(dest3, StrPtr(source), 100)

... because the manual says "Returns the address of a string's character data" (nota bene: "a string's", not "a ZSTRING's").

So I reverted to a ZString, and that one behaves erratically, too:

Code: Select all

#include "crt.bi"

Sub memcopy (byval dest As Any Ptr, Byval src As Any Ptr, Byval size As Uinteger)
  Dim As Uinteger Nl = size \ Sizeof(Integer)
  If Nl > 0 Then
    Dim As Integer Ptr pld = dest, pls = src
    For I As Uinteger = 0 To Nl - 1
      pld[I] = pls[I]
    Next I
  End If
  Dim As Uinteger Nr = size And Sizeof(Integer) - 1
  If Nr > 0 Then
    Dim As Uinteger Nb = Nl * Sizeof(Integer)
    Dim As Byte Ptr pbd = dest + Nb, pbs = src + Nb
    For I As Uinteger = 0 To Nr - 1
      pbd[I] = pbs[I]
    Next I
  End If
End Sub

Sub AsmCopy naked(byval dest As Any Ptr, Byval src As Any Ptr, Byval size As Uinteger)
Asm
  push esi
  push edi
  mov edi, [esp+4+8]		' +8 is correction for the two pushes
  mov esi, [esp+8+8]		' pointer to FB string descriptor
  mov esi, [esi-4]		' the actual string is 4 bytes lower
  mov ecx, [esp+12+8]
  rep movsb
  pop edi
  pop esi
  ret 12
End Asm
End Sub

Dim As ZString * 101 zsource = "Hello, this is a simple string intended for testing string algos. It has 100 characters without zero"
Dim As String source="Hello, this is a simple string intended for testing string algos. It has 100 characters without zero"
Dim As ZString Ptr dest1 = CAllocate(Len(source) + 1)
Dim As ZString Ptr dest2 = CAllocate(Len(source) + 1)
Dim As ZString Ptr dest3 = CAllocate(Len(source) + 1)

memcopy(@dest1, @source, 100)
printf("memcopy:   [%s]", dest1)
print

asmcopy(dest2, @source, 100)
printf("asmcopy:   [%s]", dest2)
print

' printf("zsource:   [%s]", zsource)
' print

memcpy(dest3, @zsource-4, 100)	' no idea why -4 is required, and why there is "characters ers without"
printf("crt memcpy: [%s]", dest3)

sleep()

Why do I have to pass @zsource-4, and why are some characters garbled towards the end of the string...? The manual doesn't mention such acrobatics, there is only one simple example:

Code: Select all

Dim As ZString * 13 str1 => "hello, world"

Of course, it's nowhere explained what is the purpose of this syntax sugar of using => instead of =. Mysteries of Free "Basic".

Anyway, the point was a different one. Check the code in Sub AsmCopy naked - this is exactly what you would see in the disassembly. In contrast, the pure FB Sub memcopy (last version by fxm above) looks like this:

Code: Select all

CPU Disasm
Address             Hex dump                 Command                       Comments
00401590            Ú$  55                   push ebp                      ; TmpFb.00401590(guessed Arg1,Arg2,Arg3)
00401591            ³.  89E5                 mov ebp, esp                  ; create stack frame
00401593            ³.  83EC 1C              sub esp, 1C
00401596            ³.  53                   push ebx
00401597            ³.  8B45 10              mov eax, [arg3]
0040159A            ³.  C1E8 02              shr eax, 2
0040159D            ³.  8945 FC              mov [local.1], eax
004015A0            ³.  837D FC 00           cmp dword ptr [local.1], 0
004015A4            ³. 76 43                jbe short 004015E9
004015A6            ³.  8B45 08              mov eax, [arg1]
004015A9            ³.  8945 F4              mov [local.3], eax
004015AC            ³.  8B45 0C              mov eax, [arg2]
004015AF            ³.  8945 F0              mov [local.4], eax
004015B2            ³.  C745 EC 00000000     mov dword ptr [local.5], 0
004015B9            ³.  8B45 FC              mov eax, [local.1]
004015BC            ³.  83C0 FF              add eax, -1
004015BF            ³.  8945 E8              mov [local.6], eax
004015C2            ³. EB 1D                jmp short 004015E1
004015C4            ³>  8B45 EC              Úmov eax, [local.5]
004015C7            ³.  C1E0 02              ³shl eax, 2
004015CA            ³.  8B5D F0              ³mov ebx, [local.4]
004015CD            ³.  01C3                 ³add ebx, eax
004015CF            ³.  8B45 EC              ³mov eax, [local.5]
004015D2            ³.  C1E0 02              ³shl eax, 2
004015D5            ³.  8B4D F4              ³mov ecx, [local.3]
004015D8            ³.  01C1                 ³add ecx, eax
004015DA            ³.  8B03                 ³mov eax, [ebx]
004015DC            ³.  8901                 ³mov [ecx], eax
004015DE            ³.  FF45 EC              ³inc dword ptr [local.5]
004015E1            ³>  8B45 E8              +mov eax, [local.6]
004015E4            ³.  3945 EC              ³cmp [local.5], eax
004015E7            ³. 76 DB                Àjbe short 004015C4
004015E9            ³>  8B45 10              mov eax, [arg3]
004015EC            ³.  83E0 03              and eax, 00000003
004015EF            ³.  8945 F8              mov [local.2], eax
004015F2            ³.  837D F8 00           cmp dword ptr [local.2], 0
004015F6            ³. 76 48                jbe short 00401640
004015F8            ³.  8B45 FC              mov eax, [local.1]
004015FB            ³.  C1E0 02              shl eax, 2
004015FE            ³.  8945 F4              mov [local.3], eax
00401601            ³.  8B45 08              mov eax, [arg1]
00401604            ³.  0345 F4              add eax, [local.3]
00401607            ³.  8945 F0              mov [local.4], eax
0040160A            ³.  8B45 0C              mov eax, [arg2]
0040160D            ³.  0345 F4              add eax, [local.3]
00401610            ³.  8945 EC              mov [local.5], eax
00401613            ³.  C745 E8 00000000     mov dword ptr [local.6], 0
0040161A            ³.  8B45 F8              mov eax, [local.2]
0040161D            ³.  83C0 FF              add eax, -1
00401620            ³.  8945 E4              mov [local.7], eax
00401623            ³. EB 13                jmp short 00401638
00401625            ³>  8B45 EC              Úmov eax, [local.5]
00401628            ³.  0345 E8              ³add eax, [local.6]
0040162B            ³.  8B4D F0              ³mov ecx, [local.4]
0040162E            ³.  034D E8              ³add ecx, [local.6]
00401631            ³.  8A18                 ³mov bl, [eax]
00401633            ³.  8819                 ³mov [ecx], bl
00401635            ³.  FF45 E8              ³inc dword ptr [local.6]
00401638            ³>  8B5D E4              +mov ebx, [local.7]
0040163B            ³.  395D E8              ³cmp [local.6], ebx
0040163E            ³. 76 E5                Àjbe short 00401625
00401640            ³>  5B                   pop ebx
00401641            ³.  89EC                 mov esp, ebp
00401643            ³.  5D                   pop ebp
00401644            À.  C2 0C00              retn 0C

64 lines of code, instead of 10. That looks efficient, doesn't it? ;-)

[caseih]

I'm glad you have your MasmBasic macros to play with so you can avoid all this unnecessary inefficiency and save a few cycles.

Efficiency is hardly the point of implementing memcpy in FB. Implementing memcpy can be a vehicle for exploring how things work at a higher abstract level (copy bytes one at a time). And for asking interesting questions like why did the C standard specify memcpy's parameters in the order that it did? (Why does AT&T assembly syntax have operands backwards to Intel's preferred syntax?)

As has been said the libc memcpy is highly optimized, nearly as much as your hand-crafted assembly, and is usually going to be your first choice. And it's portable. Assembly is fun and all, but MasmBasic is not going to work on an Arm processor. Or MIPS, or even AtMel, all processors I play with with IoT and embedded fun. Or WebAssembly, which is becoming the new hip virtual machine target of sorts.

[caseih]

Getting off topic, but doing some quick research into how memcpy is optimized I discovered that many compilers implement their own memcpy routines that they place in-line and optimize, rather than calls to libc if they can.

[jj2007]

@fxm: In the example posted above, I can't get your algo to work properly. Specifically:

Code: Select all

    For I As Uinteger = 0 To Nl - 1
      pld[I] = pls[I]		; trashes string descriptor of the string variable "source"
    Next I

Could be my fault, of course. Btw there are no private messages here, right?

[fxm]

@jj2007,

I corrected 2 erroneous lines in your main code (calling the 2 Subs):
- 'dest1' is already a pointer ('@dest1' is a Ptr Ptr),
- for 'memcopy()', the address of the character data of the var-len string 'source' is 'strptr(source)' and not '@source' which is the address of its descriptor.

Code: Select all

#include "crt.bi"

Sub memcopy (byval dest As Any Ptr, Byval src As Any Ptr, Byval size As Uinteger)
  Dim As Uinteger Nl = size \ Sizeof(Integer)
  If Nl > 0 Then
    Dim As Integer Ptr pld = dest, pls = src
    For I As Uinteger = 0 To Nl - 1
      pld[I] = pls[I]
    Next I
  End If
  Dim As Uinteger Nr = size And Sizeof(Integer) - 1
  If Nr > 0 Then
    Dim As Uinteger Nb = Nl * Sizeof(Integer)
    Dim As Byte Ptr pbd = dest + Nb, pbs = src + Nb
    For I As Uinteger = 0 To Nr - 1
      pbd[I] = pbs[I]
    Next I
  End If
End Sub

Sub AsmCopy naked(byval dest As Any Ptr, Byval src As Any Ptr, Byval size As Uinteger)
Asm
  push esi
  push edi
  mov edi, [esp+4+8]      ' +8 is correction for the two pushes
  mov esi, [esp+8+8]      ' pointer to FB string descriptor
  mov esi, [esi-4]      ' the actual string is 4 bytes lower
  mov ecx, [esp+12+8]
  rep movsb
  pop edi
  pop esi
  ret 12
End Asm
End Sub

Dim As ZString * 101 zsource = "Hello, this is a simple string intended for testing string algos. It has 100 characters without zero"
Dim As String source="Hello, this is a simple string intended for testing string algos. It has 100 characters without zero"
Dim As ZString Ptr dest1 = CAllocate(Len(source) + 1)
Dim As ZString Ptr dest2 = CAllocate(Len(source) + 1)
Dim As ZString Ptr dest3 = CAllocate(Len(source) + 1)

memcopy(dest1, strptr(source), 100)  '' memcopy(@dest1, @source, 100)
printf("memcopy:   [%s]", dest1)
print

asmcopy(dest2, @source, 100)
printf("asmcopy:   [%s]", dest2)
print

' printf("zsource:   [%s]", zsource)
' print

memcpy(dest3, @zsource, 100)  '' memcpy(dest3, @zsource-4, 100)   ' no idea why -4 is required, and why there is "characters ers without"
printf("crt memcpy: [%s]", dest3)

sleep()

Now, 'memcopy()', 'AsmCopy()' and 'memcpy()' work.

But not obvious:
- 'AsmCopy()' requests as destination address the address of the allocated character data buffer, but as source address the address of the descriptor of a var-len string.
- Otherwise, more clear by declaring ''AsmCopy()' as following: Sub AsmCopy naked(byval dest As Any Ptr, Byval src As String Ptr, Byval size As Uinteger)


[edit]
- OK now that I understood everything (see just above).

[counting_pine]

Why is FB so damn complicated? ...

Weird... A simple example works fine for me:

Code: Select all

#include "crt.bi"
dim as zstring*10 zs = "1234", zd = "ABCD"

print zs, zd
memcpy(@zd, @zs, 4)
print zs, zd

Maybe there's something elsewhere in the code that's breaking calling conventions or something.

[MrSwiss]

What I personally find somewhat distressing, in this whole discussion is:
the neglection of ERROR checking, for "speed's sake" ...

I'm not willing to do that since, it masks errors from calling code like:

• 1) a target string without allocated memory
  2) the same, with un-allocated zstring ptr's
  3) similar issues with numeric variables
  4) other programming mistakes ...

Therefore, fxm's code changed to Function again (with error checking) and,
some preprocessor code, to allow for safe use in DOS too:

Code: Select all

#Ifndef size_t          ' prevent "duplicate definition"-ERROR

#Ifdef __FB_DOS__       ' DOS has no 64 bit version! aka: _
Type size_t As ULong    ' always 32 bit (fixed size)
#Else
Type size_t As UInteger ' WIN / LIN depending on FBC's bitness: SizeOf(Any Ptr)
#EndIf  ' __FB_DOS__

#EndIf  ' size_t

Function memcopy( _
    ByVal src   As Any Ptr, _
    ByVal dst   As Any Ptr, _
    ByVal siz   As size_t = 1 _
    ) As Boolean
    ' parameters error checks first --> get out, if ERROR found
    If src = 0 OrElse dst = 0 OrElse siz = 0 Then Return TRUE
    ' depending on siz: any amount of integers to copy?
    Dim As UInteger Ni = siz \ Sizeof(Any Ptr)
    If Ni > 0 Then
        Dim As Integer Ptr pid = dst, pis = src
        For i As UInteger = 0 To Ni - 1
            pid[i] = pis[i]
        Next
    End If
    ' any amount of remaining bytes to copy?
    Dim As UInteger Nr = siz And Sizeof(Any Ptr) - 1    ' remaining bytes (mod - 1)
    If Nr > 0 Then
        Dim As Byte Ptr pbd, pbs
        If Ni = 0 Then                  ' no offset calc. required
            pbd = dst : pbs = src
        Else                            ' here: we need to calc. offset's
            Dim As UInteger No = Ni * Sizeof(Any Ptr)   ' current offset calc.
            pbd = dst + No : pbs = src + No ' adjust ptr's accordingly
        End If
        For i As UInteger = 0 To Nr - 1
            pbd[i] = pbs[i]
        Next
    End If
    ' everything OK
    Return FALSE
End Function

Also, added a minimum of comments in code.
Optimized offset calc. for Byte Ptr's (only done, when required!).