wth -- Thread time .vs Subroutine time
Re: wth -- Thread time .vs Subroutine time
Indeed, it is boring !
How does the bug materialize ?
The main code hangs, or the program does anything ?
Is there a system error message ?
How many 'PoolingSubmit' do you have between two 'PoolingWait' ?
How does the bug materialize ?
The main code hangs, or the program does anything ?
Is there a system error message ?
How many 'PoolingSubmit' do you have between two 'PoolingWait' ?
-
- Posts: 4313
- Joined: Jan 02, 2017 0:34
- Location: UK
- Contact:
Re: wth -- Thread time .vs Subroutine time
CryptoRndII is robust - I will put money on it.
What we have is the latest version of ThreadPooling is failing PractRand but the version before that seems OK. ThreadInitThenMultiStart got to 1TB successfully.
I have just tried the latest version of ThreadPooling again, and it failed PractRand at 32MB; so it was OK to 16MB. I am getting a type of failure in PractRand that I have never seen before, and I don't know what it means.
This is my SwitchBuffer procedure
I will try to analyse that but, for the moment, I cannot see why that should go wrong with the latest version of ThreadPooling.
What we have is the latest version of ThreadPooling is failing PractRand but the version before that seems OK. ThreadInitThenMultiStart got to 1TB successfully.
I have just tried the latest version of ThreadPooling again, and it failed PractRand at 32MB; so it was OK to 16MB. I am getting a type of failure in PractRand that I have never seen before, and I don't know what it means.
This is my SwitchBuffer procedure
Code: Select all
Private Sub SwitchBuffer
t1.PoolingWait
t1plus.PoolingWait
Swap ptrBaseBuffer0, ptrBaseBuffer1
Swap ptrBaseBuffer0plus, ptrBaseBuffer1plus
ptrBuffer = ptrBaseBuffer0
SwitchBufferCriteria = Cast( Integer, ptrBuffer ) + BufferSize
t1.PoolingSubmit(@FillBuffer, ptrBaseBuffer1)
t1plus.PoolingSubmit(@FillBuffer, ptrBaseBuffer1plus)
End Sub
Re: wth -- Thread time .vs Subroutine time
I'm trying to find a code configuration that crashes .....
Re: wth -- Thread time .vs Subroutine time
@deltarho[1859]
In this new version of 'ThreadPooling', there is 2 changes:
1) call 'memmove' instead of use a [For..Next] loop to shift the data in the two arrays ('_pThread()' and '_p()'),
2) do not restart the internal thread at each 'PoolingWait' (using '_mutex3').
The first change seems to be safe, but it changes a little the execution time.
Could you test the below version with just the change #1 ?
Version that I call 'intermediate 1':
In this new version of 'ThreadPooling', there is 2 changes:
1) call 'memmove' instead of use a [For..Next] loop to shift the data in the two arrays ('_pThread()' and '_p()'),
2) do not restart the internal thread at each 'PoolingWait' (using '_mutex3').
The first change seems to be safe, but it changes a little the execution time.
Could you test the below version with just the change #1 ?
Version that I call 'intermediate 1':
Code: Select all
#include once "crt/string.bi"
Type ThreadPooling
Public:
Declare Constructor()
Declare Sub PoolingSubmit(Byval pThread As Function(Byval As Any Ptr) As String, Byval p As Any Ptr = 0)
Declare Sub PoolingWait()
Declare Sub PoolingWait(values() As String)
Declare Property PoolingState() As Ubyte
Declare Destructor()
Private:
Dim As Function(Byval p As Any Ptr) As String _pThread0
Dim As Any Ptr _p0
Dim As Function(Byval p As Any Ptr) As String _pThread(Any)
Dim As Any Ptr _p(Any)
Dim As Any Ptr _mutex1
Dim As Any Ptr _mutex2
Dim As Any Ptr _pt
Dim As Byte _end
Dim As String _returnF(Any)
Dim As Ubyte _state
Declare Static Sub _Thread(Byval p As Any Ptr)
End Type
Constructor ThreadPooling()
Redim This._pThread(0)
Redim This._p(0)
Redim This._returnF(0)
This._mutex1 = Mutexcreate()
This._mutex2 = Mutexcreate()
Mutexlock(This._mutex2)
This._pt= Threadcreate(@ThreadPooling._Thread, @This)
End Constructor
Sub ThreadPooling.PoolingSubmit(Byval pThread As Function(Byval As Any Ptr) As String, Byval p As Any Ptr = 0)
Mutexlock(This._mutex1)
If Ubound(This._pThread) = 0 Then
Mutexunlock(This._mutex2)
End If
Redim Preserve This._pThread(Ubound(This._pThread) + 1)
This._pThread(Ubound(This._pThread)) = pThread
Redim Preserve This._p(Ubound(This._p) + 1)
This._p(Ubound(This._p)) = p
Mutexunlock(This._mutex1)
This._state = 1
End Sub
Sub ThreadPooling.PoolingWait()
This._end = 1
Mutexunlock(This._mutex2)
.ThreadWait(This._pt)
This._end = 0
Redim This._returnF(0)
This._state = 0
This._pt= Threadcreate(@ThreadPooling._Thread, @This)
End Sub
Sub ThreadPooling.PoolingWait(values() As String)
This._end = 1
Mutexunlock(This._mutex2)
.ThreadWait(This._pt)
This._end = 0
If Ubound(This._returnF) > 0 Then
Redim values(1 To Ubound(This._returnF))
For I As Integer = 1 To Ubound(This._returnF)
values(I) = This._returnF(I)
Next I
Redim This._returnF(0)
Else
Erase values
End If
This._state = 0
This._pt= Threadcreate(@ThreadPooling._Thread, @This)
End Sub
Property ThreadPooling.PoolingState() As Ubyte
Return This._state
End Property
Sub ThreadPooling._Thread(Byval p As Any Ptr)
Dim As ThreadPooling Ptr pThis = p
Do
Mutexlock(pThis->_mutex1)
While Ubound(pThis->_pThread) = 0
Mutexunlock(pThis->_mutex1)
pThis->_state = 4
Mutexlock(pThis->_mutex2)
If pThis->_end = 1 And Ubound(pThis->_pThread) = 0 Then Exit Sub
Mutexlock(pThis->_mutex1)
Wend
pThis->_pThread0 = pThis->_pThread(1)
pThis->_p0 = pThis->_p(1)
If Ubound(pThis->_pThread) > 1 Then
memmove(@pThis->_pThread(1), @pThis->_pThread(2), (Ubound(pThis->_pThread) - 1) * Sizeof(pThis->_pThread))
memmove(@pThis->_p(1), @pThis->_p(2), (Ubound(pThis->_p) - 1) * Sizeof(pThis->_p))
End If
Redim Preserve pThis->_pThread(Ubound(pThis->_pThread) - 1)
Redim Preserve pThis->_p(Ubound(pThis->_p) - 1)
Mutexunlock(pThis->_mutex1)
Redim Preserve pThis->_ReturnF(Ubound(pThis->_returnF) + 1)
pThis->_state = 2
pThis->_returnF(Ubound(pThis->_returnF)) = pThis->_pThread0(pThis->_p0)
Loop
End Sub
Destructor ThreadPooling()
This._end = 1
Mutexunlock(This._mutex2)
.ThreadWait(This._pt)
Mutexdestroy(This._mutex1)
Mutexdestroy(This._mutex2)
End Destructor
-
- Posts: 4313
- Joined: Jan 02, 2017 0:34
- Location: UK
- Contact:
Re: wth -- Thread time .vs Subroutine time
That failed PractRand at 4KB; before we switch a buffer so the intitial buffer fills have failed.fxm wrote:Could you test the below version with just the change #1
I am double-checking the version before the latest version of ThreadPooling. I pulled out of the last test at 8Gb but thought that I'd let it run a bit longer. It has just gone past 32GB.
Last edited by deltarho[1859] on Feb 15, 2021 19:29, edited 1 time in total.
Re: wth -- Thread time .vs Subroutine time
Can you test again another version with just the change #2.
Version that I call 'Intermediate 2':
Version that I call 'Intermediate 2':
Code: Select all
Type ThreadPooling '' new version
Public:
Declare Constructor()
Declare Sub PoolingSubmit(Byval pThread As Function(Byval As Any Ptr) As String, Byval p As Any Ptr = 0)
Declare Sub PoolingWait()
Declare Sub PoolingWait(values() As String)
Declare Property PoolingState() As Ubyte
Declare Destructor()
Private:
Dim As Function(Byval p As Any Ptr) As String _pThread0
Dim As Any Ptr _p0
Dim As Function(Byval p As Any Ptr) As String _pThread(Any)
Dim As Any Ptr _p(Any)
Dim As Any Ptr _mutex1
Dim As Any Ptr _mutex2
Dim As Any Ptr _mutex3
Dim As Any Ptr _pt
Dim As Byte _end
Dim As String _returnF(Any)
Dim As Ubyte _state
Declare Static Sub _Thread(Byval p As Any Ptr)
End Type
Constructor ThreadPooling()
Redim This._pThread(0)
Redim This._p(0)
Redim This._returnF(0)
This._mutex1 = Mutexcreate()
This._mutex2 = Mutexcreate()
Mutexlock(This._mutex2)
This._mutex3 = Mutexcreate()
Mutexlock(This._mutex3)
This._pt= Threadcreate(@ThreadPooling._Thread, @This)
End Constructor
Sub ThreadPooling.PoolingSubmit(Byval pThread As Function(Byval As Any Ptr) As String, Byval p As Any Ptr = 0)
Mutexlock(This._mutex1)
If Ubound(This._pThread) = 0 Then
Mutexunlock(This._mutex2)
End If
Redim Preserve This._pThread(Ubound(This._pThread) + 1)
This._pThread(Ubound(This._pThread)) = pThread
Redim Preserve This._p(Ubound(This._p) + 1)
This._p(Ubound(This._p)) = p
Mutexunlock(This._mutex1)
This._state = 1
End Sub
Sub ThreadPooling.PoolingWait()
Mutexlock(This._mutex3)
Redim This._returnF(0)
This._state = 0
Mutexunlock(This._mutex3)
End Sub
Sub ThreadPooling.PoolingWait(values() As String)
Mutexlock(This._mutex3)
If Ubound(This._returnF) > 0 Then
Redim values(1 To Ubound(This._returnF))
For I As Integer = 1 To Ubound(This._returnF)
values(I) = This._returnF(I)
Next I
Redim This._returnF(0)
Else
Erase values
End If
This._state = 0
Mutexunlock(This._mutex3)
End Sub
Property ThreadPooling.PoolingState() As Ubyte
Return This._state
End Property
Sub ThreadPooling._Thread(Byval p As Any Ptr)
Dim As ThreadPooling Ptr pThis = p
Do
Mutexlock(pThis->_mutex1)
While Ubound(pThis->_pThread) = 0
Mutexunlock(pThis->_mutex1)
Mutexunlock(pThis->_mutex3)
pThis->_state = 4
Mutexlock(pThis->_mutex2)
If pThis->_end = 1 And Ubound(pThis->_pThread) = 0 Then Exit Sub
Mutexlock(pThis->_mutex3)
Mutexlock(pThis->_mutex1)
Wend
pThis->_pThread0 = pThis->_pThread(1)
pThis->_p0 = pThis->_p(1)
For I As Integer = 2 To Ubound(pThis->_pThread)
pThis->_pThread(I - 1) = pThis->_pThread(I)
pThis->_p(I - 1) = pThis->_p(I)
Next I
Redim Preserve pThis->_pThread(Ubound(pThis->_pThread) - 1)
Redim Preserve pThis->_p(Ubound(pThis->_p) - 1)
Mutexunlock(pThis->_mutex1)
Redim Preserve pThis->_ReturnF(Ubound(pThis->_returnF) + 1)
pThis->_state = 2
pThis->_returnF(Ubound(pThis->_returnF)) = pThis->_pThread0(pThis->_p0)
Loop
End Sub
Destructor ThreadPooling()
This._end = 1
Mutexunlock(This._mutex2)
.ThreadWait(This._pt)
Mutexdestroy(This._mutex1)
Mutexdestroy(This._mutex2)
Mutexdestroy(This._mutex3)
End Destructor
-
- Posts: 4313
- Joined: Jan 02, 2017 0:34
- Location: UK
- Contact:
Re: wth -- Thread time .vs Subroutine time
That failed at 8MB.
See my last post edit.
See my last post edit.
Re: wth -- Thread time .vs Subroutine time
I did not understand very well.
Among these 2 versions ("intermediate 1" and "intermediate 2"), is there one that works?
Among these 2 versions ("intermediate 1" and "intermediate 2"), is there one that works?
-
- Posts: 4313
- Joined: Jan 02, 2017 0:34
- Location: UK
- Contact:
Re: wth -- Thread time .vs Subroutine time
@fxm
With regard your very latest code I dumped 128MB and tested it with Fourmilab's ENT. All seemed OK but ENT really only checks distribution uniformity. I could join a file to itself and ENT would not complain; PractRand would and so it should.
With regard your very latest code I dumped 128MB and tested it with Fourmilab's ENT. All seemed OK but ENT really only checks distribution uniformity. I could join a file to itself and ENT would not complain; PractRand would and so it should.
-
- Posts: 4313
- Joined: Jan 02, 2017 0:34
- Location: UK
- Contact:
Re: wth -- Thread time .vs Subroutine time
"intermediate 1" fails at 4KBI did not understand very well.
Among these 2 versions ("intermediate 1" and "intermediate 2"), is there one that works?
"intermediate 2" fails at 8MB
The version that is working, getting to 32GB, is pre mutex3.
Re: wth -- Thread time .vs Subroutine time
Between "intermediary 1" and the one you call "pre mutex3", there is only change #1.
It's weird, unless “memmove” is not thread-safe!
It's weird, unless “memmove” is not thread-safe!
Re: wth -- Thread time .vs Subroutine time
@fxm, have you tried to force -mt in compilation?
Re: wth -- Thread time .vs Subroutine time
@deltarho[1859]MrSwiss wrote:@fxm, have you tried to force -mt in compilation?
Can you try this ?
-
- Posts: 4313
- Joined: Jan 02, 2017 0:34
- Location: UK
- Contact:
Re: wth -- Thread time .vs Subroutine time
Using -mt
"intermediate 1" froze at 1GB
"intermediate 2" fails at 4MB
I have never seen PractRand freeze before.
There is a debate at the PowerBASIC forum on whether PowerBASIC has a future or not. In one post I wrote: "It is hard work to develop a perfect design, but that is more to do with our imagination than the language we are using." One member responded with: "Over the years, I have found that a program that is easy to write is hard to use, and a program that is hard to write is easy to use."
ThreadPooling is easy to use but some code, especially _Thread, is extremely complex.
"intermediate 1" froze at 1GB
"intermediate 2" fails at 4MB
I have never seen PractRand freeze before.
There is a debate at the PowerBASIC forum on whether PowerBASIC has a future or not. In one post I wrote: "It is hard work to develop a perfect design, but that is more to do with our imagination than the language we are using." One member responded with: "Over the years, I have found that a program that is easy to write is hard to use, and a program that is hard to write is easy to use."
ThreadPooling is easy to use but some code, especially _Thread, is extremely complex.
Re: wth -- Thread time .vs Subroutine time
Do you remember this constraint:deltarho[1859] wrote:This is my SwitchBuffer procedureCode: Select all
Private Sub SwitchBuffer t1.PoolingWait t1plus.PoolingWait Swap ptrBaseBuffer0, ptrBaseBuffer1 Swap ptrBaseBuffer0plus, ptrBaseBuffer1plus ptrBuffer = ptrBaseBuffer0 SwitchBufferCriteria = Cast( Integer, ptrBuffer ) + BufferSize t1.PoolingSubmit(@FillBuffer, ptrBaseBuffer1) t1plus.PoolingSubmit(@FillBuffer, ptrBaseBuffer1plus) End Sub
I do not know what is after:Warning: The information supplied to the user thread procedure via the passed pointer (by 'PoolingSubmit') should not be changed between 'PoolingSubmit' and 'PoolingWait' due to the time uncertainty on the real call of the user thread procedure in this interval.
t1.PoolingSubmit(@FillBuffer, ptrBaseBuffer1)
t1plus.PoolingSubmit(@FillBuffer, ptrBaseBuffer1plus)
because it is the end of the 'SwitchBuffer' Sub, but I hope that the two pointers 'ptrBaseBuffer1' and 'ptrBaseBuffer1plus' (and their pointed data) are not modified by the main code (and the other secondary thread) before next:
t1.PoolingWait
t1plus.PoolingWait
Same constraint between 'ThreadStart' and 'ThreadWait'.
Last edited by fxm on Feb 15, 2021 23:09, edited 2 times in total.