Unsupported function

Windows specific questions.
deltarho[1859]
Posts: 1014
Joined: Jan 02, 2017 0:34
Location: UK

Unsupported function

Postby deltarho[1859] » Apr 10, 2018 21:34

I have checked the forums and found several references to unsupported 'something or other' but no 'Unsupported function'. There are three types of subroutines which are not supported but I cannot see where Sub Hash is fouling fall on any of them. I am not passing a string where there is an issue.

Interestingly WinFBE is coming back with error 288 but poseidonFB is coming back with error 287; both 'Unsupported function'.

Here is a snippet taken from a project. The offending line is 'Dim stoe As Any Ptr ...'.

Code: Select all

#Include Once "windows.bi"
'#Include Once "win/bcrypt.bi"
'#Inclib "bcrypt"
#Include Once "win/wincrypt.bi"
'#Inclib "crypt32"

Sub Hash( hashalg As lpcwstr, ulData As PUCHAR, ulDataLen As Ulong, final As Long, sbinhash As ZString ptr )
Static As Byte Ptr phalg, phhash
Dim sbinary As String
Dim shex As String
Dim nlength As Long
Dim as string temp

  ' Initialize section
 
  If phhash = 0 Then ' will be the Case On, And perhaps only, first pass
    BcryptOpenAlgorithmprovider @phalg, hashalg, 0, 0 ' we want phalg
    BcryptCreatehash phalg, @phhash, null, 0, 0, 0, 0 ' we want phhash
  End If

  ' Update section
 
  BcryptHashData( phhash, Cast( PUCHAR, ulData ), ulDataLen, 0 )

  ' Finalization section
 
  If final = true Then ' finalize hash
    Dim As Ulong lhashlength
    Dim As Ulong lresult
   
    BcryptGetProperty phalg, bcrypt_hash_length, Cast( Puchar, @lhashlength ), 4, @lresult, 0
    temp = String( lhashlength, 0 )
    BcryptFinishHash phhash, Strptr( temp ), lhashlength, 0
    BcryptDestroyHash phhash
    BcryptCloseAlgorithmprovider phalg, 0
    phhash = 0
    sbinhash = Cast( ZString Ptr, @temp )
  End If

End Sub

Dim As String s = "FreeBASIC", TargetHash
Dim stoe As Any Ptr = Threadcall Hash( Wstr("SHA256"), Strptr(s), Len(s), False, TargetHash )
Threadwait stoe

Sleep
fxm
Posts: 7737
Joined: Apr 22, 2009 12:46
Location: Paris suburb, FRANCE

Re: Unsupported function

Postby fxm » Apr 10, 2018 22:09

Seems to be a bug of ThreadCall:
Trying to pass a [u]LONG type parameter produces this error message.

Code: Select all

Declare Sub test (Byval As Long)
Dim As Any Ptr p = Threadcall test (0)
    error 288: Unsupported function in 'Dim As Any Ptr p = Threadcall test (0)'
It would be a second third bug of this keyword, after the well known main bug:
Presently when Threadcall involves to pass parameters to the thread, there is no guarantee that the corresponding data are still maintained after the end of the Threadcall statement and this until the thread is launched. That can cause bad behavior.
Last edited by fxm on Apr 11, 2018 5:06, edited 1 time in total.
deltarho[1859]
Posts: 1014
Joined: Jan 02, 2017 0:34
Location: UK

Re: Unsupported function

Postby deltarho[1859] » Apr 10, 2018 22:58

Thanks fxm.

I will have a look at Thread Pools.
fxm
Posts: 7737
Joined: Apr 22, 2009 12:46
Location: Paris suburb, FRANCE

Re: Unsupported function

Postby fxm » Apr 11, 2018 4:59

deltarho[1859]
Posts: 1014
Joined: Jan 02, 2017 0:34
Location: UK

Re: Unsupported function

Postby deltarho[1859] » Apr 11, 2018 5:27

This is working:-

Code: Select all

#Include Once "windows.bi"
#Inclib "bcrypt"
#Include Once "win/wincrypt.bi"
#Inclib "crypt32"
 
Type Params
  As LPCWSTR HashAlg
  As PUCHAR ulData
  As Ulong ulDatalen
  As Long Final
  As String sHash
End Type
 
Sub Hash( ptrParams As Params Ptr )
Static As Byte Ptr phalg, phhash
Dim sBinary As String
Dim sHex As String
Dim nLength As Long
 
  ' Initialize section
 
  If phhash = 0 Then ' will be the Case On, And perhaps only, first pass
    BcryptOpenAlgorithmprovider @phalg, ptrParams->HashAlg, 0, 0 ' we want phalg
    BcryptCreatehash phalg, @phhash, null, 0, 0, 0, 0 ' we want phhash
  End If
 
  ' update section
 
  BcryptHashData( phhash, Cast( PUCHAR, ptrParams->ulData ), ptrParams->ulDataLen, 0 )
 
  ' Finalization section
 
  If ptrParams->Final = true Then ' finalize hash
    Dim As Ulong lHashLength
    Dim As Ulong lResult
    Dim sBinHash As String
 
    BcryptGetProperty phalg, BCRYPT_HASH_LENGTH, Cast( Puchar, @lHashLength ), 4, @lResult, 0
    sBinHash = String( lHashLength, 0 )
    BcryptFinishHash phhash, Strptr( sBinHash ), lHashLength, 0
    BcryptDestroyHash phhash
    BcryptCloseAlgorithmprovider phalg, 0
    phhash = 0
    nLength = Len(sBinHash)*2 + 1 ' + 1 To accomodate a null terminator
    sHex = Space( nLength )
    CryptBinaryToStringa Strptr(sBinHash), Len(sBinHash), CRYPT_STRING_HEXRAW + CRYPT_STRING_NOCRLF, _
      Strptr(sHex), @nLength ' at msdn nlength 'Out' Is Len(sBinHash) * 2, so
    ptrParams->sHash = Ucase( Left( sHex, nLength ) )
  End If
 
End Sub
 
Dim As String s = "FreeBASIC"
 
Dim As Params ArgList
ArgList.hashalg = @Wstr("SHA256")
ArgList.ulData = Strptr(s)
ArgList.ulDataLen = Len(s)
ArgList.final = True
 
Dim stoe As Any Ptr = ThreadCall Hash( @ArgList )
ThreadWait stoe
Print ArgList.sHash
 
Sleep

On execution the above gave:

C541F343E3C8BA30A990F8D869C744E70B30CE4C7752233D51AF5D76F030D999

which is the correct SHA256 of "FreeBASIC".

I think we may have a solution to ThreadCall's woes by keeping it simple, passing a pointer to a UDT and nothing else. We no longer have issues with strings, ulongs and whatever else may be found to give an issue.

It may seem odd that I am putting a hash sub in a secondary thread of execution. The reason is that if we are decrypting an encrypted buffer and hashing the very same buffer then we can use ThreadCall Hash() followed by the buffer decryption and then followed by ThreadWait. The hash takes longer than the decryption so we will have to wait for the hash before filling the buffer with the next read but we will have asynchronous computations for some of the time.

With encryption we have to encrypt first and then hash so, like it or not, we will have synchronous computations.

So, with threading we should get a performance boost with decryption/hash.
deltarho[1859]
Posts: 1014
Joined: Jan 02, 2017 0:34
Location: UK

Re: Unsupported function

Postby deltarho[1859] » Apr 11, 2018 5:44

Here is a re-write of the example in the docs using a UDT pointer.

Interestingly, the example in the docs does not work properly where the 'id' in 'Print "thread " & id;' does not get printed with the string passing bug; solved with fxm's workaround using ZString ptr. On the other hand, 'id' does get printed with the UDT pointer approach.

Admittedly, the UDT pointer approach involves more work but if it is robust then so be it.

Code: Select all

Type Params
  As String id
  As Any Ptr tlock
  As Integer count
End Type

Sub thread( ptrParams As Params Ptr )
    For i As Integer = 1 To ptrParams->count
        Mutexlock ptrParams->tlock
        Print "thread " & ptrParams->id;
        Locate , 20
        Print i & "/" & ptrParams->count
        Mutexunlock ptrParams->tlock
    Next
End Sub


Dim tlock As Any Ptr = Mutexcreate()

Dim As Params ArgList1
ArgList1.id = "A"
ArgList1.tlock = tlock
ArgList1.count = 6
Dim a As Any Ptr = Threadcall thread( @ArgList1 )

Dim As Params ArgList2
ArgList2.id = "B"
ArgList2.tlock = tlock
ArgList2.count = 4
Dim b As Any Ptr = Threadcall thread( @ArgList2 )

Threadwait a
Threadwait b
Mutexdestroy tlock
Print "All done (And without Dim Shared!)"

Sleep
fxm
Posts: 7737
Joined: Apr 22, 2009 12:46
Location: Paris suburb, FRANCE

Re: Unsupported function

Postby fxm » Apr 11, 2018 8:25

Using 'ThreadCall':
The using of a pointer (as passed parameter) is more robust but not 100% safe.
Indeed, the allocated memory may be freed before the thread launch:
- if the passed parameter is an object with destructor (like a string for example), the effect is immediate because object being destroyed before the memory deallocation, then the argument value to be passed is broken.
- if the passed parameter is not an object (like a numeric value for example), the allocated memory is not cleared before deallocation, then the argument value to be passed is broken only if the freed memory is reallocated and reassigned in the meantime for other thing.

Using 'ThreadCreate':
Now that you have modified your code to pass a pointer as only argument, there is not much work left to replace 'ThreadCall' by 'ThreadCreate' which is 100% safe:

Code: Select all

#Include Once "windows.bi"
#Inclib "bcrypt"
#Include Once "win/wincrypt.bi"
#Inclib "crypt32"
 
Type Params
  As LPCWSTR HashAlg
  As PUCHAR ulData
  As Ulong ulDatalen
  As Long Final
  As String sHash
End Type
 
Sub Hash( Byval p As Any Ptr )
Dim ptrParams As Params Ptr = p
Static As Byte Ptr phalg, phhash
Dim sBinary As String
Dim sHex As String
Dim nLength As Long
 
  ' Initialize section
 
  If phhash = 0 Then ' will be the Case On, And perhaps only, first pass
    BcryptOpenAlgorithmprovider @phalg, ptrParams->HashAlg, 0, 0 ' we want phalg
    BcryptCreatehash phalg, @phhash, null, 0, 0, 0, 0 ' we want phhash
  End If
 
  ' update section
 
  BcryptHashData( phhash, Cast( PUCHAR, ptrParams->ulData ), ptrParams->ulDataLen, 0 )
 
  ' Finalization section
 
  If ptrParams->Final = true Then ' finalize hash
    Dim As Ulong lHashLength
    Dim As Ulong lResult
    Dim sBinHash As String
 
    BcryptGetProperty phalg, BCRYPT_HASH_LENGTH, Cast( Puchar, @lHashLength ), 4, @lResult, 0
    sBinHash = String( lHashLength, 0 )
    BcryptFinishHash phhash, Strptr( sBinHash ), lHashLength, 0
    BcryptDestroyHash phhash
    BcryptCloseAlgorithmprovider phalg, 0
    phhash = 0
    nLength = Len(sBinHash)*2 + 1 ' + 1 To accomodate a null terminator
    sHex = Space( nLength )
    CryptBinaryToStringa Strptr(sBinHash), Len(sBinHash), CRYPT_STRING_HEXRAW + CRYPT_STRING_NOCRLF, _
      Strptr(sHex), @nLength ' at msdn nlength 'Out' Is Len(sBinHash) * 2, so
    ptrParams->sHash = Ucase( Left( sHex, nLength ) )
  End If
 
End Sub
 
Dim As String s = "FreeBASIC"
 
Dim As Params ArgList
ArgList.hashalg = @Wstr("SHA256")
ArgList.ulData = Strptr(s)
ArgList.ulDataLen = Len(s)
ArgList.final = True
 
Dim stoe As Any Ptr = ThreadCreate( @Hash, @ArgList )
ThreadWait stoe
Print ArgList.sHash
 
Sleep

Code: Select all

Type Params
  As String id
  As Any Ptr tlock
  As Integer count
End Type

Sub thread( Byval p As Any Ptr )
    Dim ptrParams As Params Ptr = p
    For i As Integer = 1 To ptrParams->count
        Mutexlock ptrParams->tlock
        Print "thread " & ptrParams->id;
        Locate , 20
        Print i & "/" & ptrParams->count
        Mutexunlock ptrParams->tlock
    Next
End Sub


Dim tlock As Any Ptr = Mutexcreate()

Dim As Params ArgList1
ArgList1.id = "A"
ArgList1.tlock = tlock
ArgList1.count = 6
Dim a As Any Ptr = Threadcreate( @thread, @ArgList1 )

Dim As Params ArgList2
ArgList2.id = "B"
ArgList2.tlock = tlock
ArgList2.count = 4
Dim b As Any Ptr = Threadcreate( @thread, @ArgList2 )

Threadwait a
Threadwait b
Mutexdestroy tlock
Print "All done (And without Dim Shared!)"

Sleep
fxm
Posts: 7737
Joined: Apr 22, 2009 12:46
Location: Paris suburb, FRANCE

Re: Unsupported function

Postby fxm » Apr 11, 2018 11:25

deltarho[1859]
Posts: 1014
Joined: Jan 02, 2017 0:34
Location: UK

Re: Unsupported function

Postby deltarho[1859] » Apr 11, 2018 14:13

I like the ThreadCreate( @Hash, @ArgList ) code. I am going to use that.
fxm wrote:The using of a pointer (as passed parameter) is more robust but not 100% safe.

there is not much work left to replace 'ThreadCall' by 'ThreadCreate' which is 100% safe:

It seems to me then that there is a case for removing ThreadCall from the docs or a least a reccomendation for it not to be used.
deltarho[1859]
Posts: 1014
Joined: Jan 02, 2017 0:34
Location: UK

Re: Unsupported function

Postby deltarho[1859] » Apr 11, 2018 15:10

Off topic but worth mentioning. With a one GB file we process 4096 buffers at 256KB. We have then 4096 thread creations. The hash time is greater than the decryption time but the asynchronous time is being diminished by the thread creation overhead resulting in a break even at best. If we processed a one GB buffer then we would have only one thread creation. We could, of course, increase the buffer to 512KB or 1MB but the best drive read ahead, for HDDs, is 256KB so we would gain and then lose.

We will not have multiple creations with a thread pool so I will now have a look at that approach.
fxm
Posts: 7737
Joined: Apr 22, 2009 12:46
Location: Paris suburb, FRANCE

Re: Unsupported function

Postby fxm » Apr 11, 2018 18:52

deltarho[1859] wrote:It seems to me then that there is a case for removing ThreadCall from the docs or a least a reccomendation for it not to be used.

KeyPgThreadCall → fxm [Added the advice to rather use ThreadCreate for the moment (which is it 100% safe)]
integer
Posts: 364
Joined: Feb 01, 2007 16:54
Location: usa

Re: Unsupported function

Postby integer » Apr 13, 2018 8:12

fxm wrote:
deltarho[1859] wrote:It seems to me then that there is a case for removing ThreadCall from the docs or a least a reccomendation for it not to be used.

KeyPgThreadCall → fxm [Added the advice to rather use ThreadCreate for the moment (which is it 100% safe)]

THANK YOU.
deltarho[1859]
Posts: 1014
Joined: Jan 02, 2017 0:34
Location: UK

Re: Unsupported function

Postby deltarho[1859] » Apr 14, 2018 0:55

Thread pooling is not the answer either.

It takes 1.199ms to hash a 256KB buffer. Decryption, on the other hand, only takes 0.105ms. SHA256 is far too slow. In this case any kind of threading is a waste of time. This idea is screaming to use Blake2 instead of SHA256.
deltarho[1859]
Posts: 1014
Joined: Jan 02, 2017 0:34
Location: UK

Re: Unsupported function

Postby deltarho[1859] » Apr 14, 2018 11:08

Yours truly wrote:In this case any kind of threading is a waste of time.

Perhaps not.

If the hash time is 'a' and the decryption time is 'b' then what we are interested in is a/(a+b). If a = b then we get 0.5 the maximum possible boost for threading. As 'a' increases relative to 'b' then a/(a+b) tends to 1 and threading is less effective. With the above we have 1.199/(1.199+0.105) = 0.92 so we will, in theory, get an 8% boost in throughput. The overhead for thread pooling is negligible compared with thread creation. On my machine submitting a thread, as opposed to creating one, takes only a couple of micro-seconds.

From what I have read AES CBC encryption, in software, should be slightly faster than decryption. In hardware, such as AES NI, they should be about the same.

In my case, with AES NI, I am getting decryption faster than encryption including reading the data; making sure that the filecache is always flushed. The waters are further muddied when we compare HDDs with SSDs.

At the end of the day what matters is actual times and not paper predictions.

With thread pooling a 100MB file behaves as follows; in mill-seconds.

Code: Select all

Encryption Decryption
   730        580
   726        532
   771        553

I cannot see how I would make matters worse by thread pooling so I am going to go with it.

Blake2 would make a world of difference but I do not have Blake2 in code so that I can 'stream' data ie build a hash and then finalize it. All I have is the facility to hash once.

Return to “Windows”

Who is online

Users browsing this forum: No registered users and 3 guests