Very basic process CPU load determiner

Windows specific questions.
Post Reply
deltarho[1859]
Posts: 4292
Joined: Jan 02, 2017 0:34
Location: UK
Contact:

Re: Very basic process CPU load determiner

Post by deltarho[1859] »

Yours truly wrote:I am 'maxed out' of ideas on this topic so I will call it a day. <smile>
Famous last words.<rolls eyes>

Instead of advising the average session load at the end of monitoring it now shows a running average reporting at the same interval as the other statistics. I don't know why I did not do this in the first place. <Doh!>

By the way, CPU Load could be used to examine your internet browser but that is not a good idea. Modern browsers employ multiple processes with the same name. They have different PIDs, of course, but CPU Load will only open the first one it finds.

Process Explorer and Process Hacker display all of the browser's PIDs. Interestingly, Task Manager is a bit smart in this context, at least with Windows 10. On my machine it is not showing all of the PIDs but is displaying 'FlashPeak Slimjet (11)' and there are, at the moment, 11 browser PIDs. I have done some tests and it is clear that Task Manager is totting up the load from all of the browser PIDs.
deltarho[1859]
Posts: 4292
Joined: Jan 02, 2017 0:34
Location: UK
Contact:

Re: Very basic process CPU load determiner

Post by deltarho[1859] »

For some reason the average session load was showing as an over estimate so I have brought it in line with the 'averaging of 8' that the current load is subject to.

I tried a variety of loads and it is now 'bang in line' with what Process Explorer and Process Hacker seem to be showing.

Of course, if we vary the load during a session that will be reflected in the current load, the average session load will be just that and be influenced by all the loads since the session started.

Bit of a black art this. <Ha,ha>

Added: If you have an app which is active throughout its session, such as a graphics app, then whilst the current load may be going up an down the average session load should enter a steady state. If, for some reason, your app enters a quiet period then the current load may show as zero for that time. The average session load will now start to fall. If your app then bursts into activity again the current load will reflect that and the average session load will start to rise. I mention this because we would not see this behaviour when the average session load was only being reported at the end of monitoring.
deltarho[1859]
Posts: 4292
Joined: Jan 02, 2017 0:34
Location: UK
Contact:

Re: Very basic process CPU load determiner

Post by deltarho[1859] »

I think that the 'glitch' was because I didn't test for long enough. The 'averaging of 8' hasn't made that much of a difference in fact.

Here is a 21% or so load comparing the Performance Monitor with CPU Load's latest incarnation.

Image

Any difference in approach will manifest itself in the early stages of a test but should diminish the longer the test.

I think I will leave well alone now, 21.30 compared with 21.802 isn't that bad.
deltarho[1859]
Posts: 4292
Joined: Jan 02, 2017 0:34
Location: UK
Contact:

Re: Very basic process CPU load determiner

Post by deltarho[1859] »

Just a quick one. Has anyone 'messed about' with my code - it is source code after all and you are more than welcome to 'knock seven bells out of it' if it can be improved upon. Sometimes I felt that I was in a dark room trying to find a chair to sit on. It reminds me of something in my distant youth but I cannot remember what. It wouldn't surprise me to learn that I was carrying a bottle of rum at the time. <tut, tut>
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: Very basic process CPU load determiner

Post by caseih »

Kind of funny but your screenshot really took me aback at first. In Unix, "load" has an entirely different meaning than you ascribe to it. I'm guessing you're talking about percent CPU usage, correct? In Linux a load average of 20 would indicate some serious problems with I/O hanging up.

For kicks and giggles you could implement a moving average or two. One could be over 10 seconds, another over a minute, another over 10 minutes, etc.
paul doe
Moderator
Posts: 1730
Joined: Jul 25, 2017 17:22
Location: Argentina

Re: Very basic process CPU load determiner

Post by paul doe »

caseih wrote:For kicks and giggles you could implement a moving average or two. One could be over 10 seconds, another over a minute, another over 10 minutes, etc.
A neat idea, indeed.
@deltarho: here's the code for an EMA (Exponential Moving Average):

Code: Select all

function ema( byval value as double = 0.0, byval period as double = 1.0, byval zero as boolean = false ) as double
	/'
		EMA (Exponential Moving Average) implementation
	'/
	static as double average
	
	if( zero = true ) then
		average = 0.0
	end if
	
	dim as double smoothFactor = 2.0 / ( 1.0 + period )	
	average = average * ( 1.0 - smoothFactor ) + value * smoothFactor 
	
	return( average )
end function
So you can collect individual measurements and always have an up-to-date averaged measurement. The advantage of the EMA over the SMA (Simple Moving Average, the classical average) is that not only does it not discard any value, but also gives more weight to the recent ones. Told you, I do a lot of financial coding =D

This is how you use it:

Code: Select all

'' use whatever initial value you wish (set the last flag to true to reset the EMA to 0)
ema( 100, , true )

do
	t = timer()
	/'
		do some mighty interesting stuff
	'/
	t = timer() - t
	
	'' and then add the measured time to the EMA, like this:
	? ema( t, 20.0 ) '' this will give an EMA of the last 20 measurements
loop until( inkey() <> "" )
The 'period' parameter represents over how many measurements you will like to average. The higher the slower it is to react to changes. Try it and see if you like it =D
deltarho[1859]
Posts: 4292
Joined: Jan 02, 2017 0:34
Location: UK
Contact:

Re: Very basic process CPU load determiner

Post by deltarho[1859] »

caseih wrote:I'm guessing you're talking about percent CPU usage, correct?
Yes.
you could implement a moving average or two.
I am doing that with the average session load. Polling is done at 8Hz. The oldest reading of eight is dropped, a new reading taken and the average of 8 is calculated. The result is displayed at 2Hz. My logic was to try and smooth out some of the volatility.

Task Manager, Process Explorer and Process Hacker all have the option to choose different refresh rates, up to 10s with the latter two. What none of them are clear about is the polling rate employed. A polling rate and a display rate may not be the same. Mine are not: The polling rate is 8Hz and the display rate is 2Hz.

Your suggestion of having large intervals got me to wonder what the above would look like when using large intervals. Since we are examining CPU Load, % CPU usage, then, clearly, we need at least two readings. If we 'dial in' the slowest refresh rate with any of the above there is a corresponding delay before the first readings are displayed. On the face of it then it looks like none of them are employing conditioning, but I am. The Performance Monitor also has a display delay corresponding to the refresh rate.

My next tweak was to synchronise the polling. If we poll at the system interrupt then when we subtract two readings we 'eliminate' the interrupts resolution so it doesn't matter if the resolution is 1ms or 16ms or anything in between.

When we have a tweaking regime it is sometimes worthwhile to halt the proceedings and ask if it is necessary to keep all the tweaks made to date. Some early tweaks could be dispensed with by virtue of later tweaks. Microsoft do this with consolidating updates. This happens with tax law where a consolidating tax act is introduced after several 'tweaking budgets'.

In my case I questioned the early 'average of 8' tweak, preceding the later synchronizing tweak. To my mind there is no case to not use the synchronizing tweak. The 'average of 8' tweak needs a fast polling, so I now questioned the fast polling.

We still have the issue of volatility. Contrary to my initial approach of faster sampling, I was thinking about digital sound sampling for audio CDs, the slower the sampling the lesser the volatility. However, a slow sample rate is not a smoothing mechanism. A slow sample rate rejects information. The slower the sampling rate the more important the Max Load metric becomes. We need to get a balance.

There is a statistical method worth looking at and that is confidence intervals. Here we would have not just an average but an interval encompassing that average where we can say, something like, "We are 95% confident that the CPU load is in the interval 18 - 22." With a balanced interval I would prefer to use 20 ± 2. Looking at the time graphs of the Performance Monitor it seems to me that we should use balanced intervals or, in other words, two sided confidence limits.

I do not want to bore you but for 30 or less samples we should use the t-distribution and for greater than 30 samples we should use the normal distribution for our % confidence calculations. I have opted for the normal distribution and am now sampling, and displaying, at one second. So, we need to wait for 30 seconds before accepting the confidence interval.

I have coded the above and get this with one of dodicat's master pieces. Over a session of 120 seconds we have a current load of 3.13%, a max load of 5.66% and can say, with 95% confidence, that the load is between 2.12% and 4.56%. The ± 1.22 may be regarded as an indication of the volatility. Since the max load is without the limits then it can be regarded as unusual and was not sustained for very long.

Image

Here is a snapshot of the Performance Monitor, sampling at one second intervals, for dodicat's program. The graph is not load but processor time. This is typical of a preemptive multitasking operating system - all appears calm and collected on the surface but under the bonnet we have absolute mayhem, a bit like leaving the relativistic world for a visit to the quantum world. <smile>

Image

I have tried this on a few loads and things are looking good.

I need some time to look at the 'paul doe' post and really like the idea of historical data having less credence with the passage of time.

Thanks for the feedback, guys. I had 'called it a day' earlier but it looks like I'l be at it a bit longer. I don't mind because I am enjoying myself. I should remind you that this thread is a "result of a conversation between dodicat and myself in another thread." and that thread was about graphics. <laugh>
MrSwiss
Posts: 3910
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: Very basic process CPU load determiner

Post by MrSwiss »

@paul doe,

I think, your EMA implementation, could be done simpler:

Code: Select all

Function EMA( _
    ByVal value As Double = 0.0, _
    ByVal period As ULong = 1, _
    ByVal zero As Boolean = FALSE _
    ) ByRef As Double   ' for speed use: ByRef As Double
    ' only use this, to return a internal static variable
    Static As Double    average
    Dim As Double       smoothFactor = 2.0 / (1.0 + period)
   
    If zero Then average = 0.0
    average *= (1.0 - smoothFactor) + value * smoothFactor
   
    Return average
End Function
paul doe
Moderator
Posts: 1730
Joined: Jul 25, 2017 17:22
Location: Argentina

Re: Very basic process CPU load determiner

Post by paul doe »

MrSwiss wrote:@Paul Doe,

I think your EMA implementation, could be done simpler:
Perhaps. But yours is incorrect:

Code: Select all

function ema( byval value as double = 0.0, byval period as double = 1.0, byval zero as boolean = false ) as double
   /'
      EMA (Exponential Moving Average) implementation
   '/
   static as double average
   
   if( zero = true ) then
      average = 0.0
   end if
   
   dim as double smoothFactor = 2.0 / ( 1.0 + period )   
   average = average * ( 1.0 - smoothFactor ) + value * smoothFactor 
   
   return( average )
end function

Function EMA2( _
    ByVal value As Double = 0.0, _
    ByVal period As ULong = 1, _
    ByVal zero As Boolean = FALSE _
    ) ByRef As Double   ' for speed use: ByRef As Double
    ' only use this, to return a internal static variable
    Static As Double    average
    Dim As Double       smoothFactor = 2.0 / (1.0 + period)
   
    If zero Then average = 0.0
    average *= (1.0 - smoothFactor) + value * smoothFactor
   
    Return average
End Function

'' use whatever initial value you wish (set the last flag to true to reset the EMA to 0)
dim as double t

dim as double em1 = ema( 100, , true )
dim as double em2 = ema2( 100, , true )

do
   t = timer()
   /'
      do some mighty interesting stuff
   '/
   t = timer() - t
   
   '' and then add the measured time to the EMA, like this:
   em1 = ema( t, 20.0 )
   em2 = ema2( t, 20.0 )
   
   ? em1, em2
loop until( inkey() <> "" )

sleep()
I never return byref for values that are supposed to be encapsulated, whether it's faster or not. You're essentially returning a pointer to an internal value that you weren't supposed to be accessing in the first place. So, nothing prevents you to get really clever and do something like this:

Code: Select all

dim byref foo = ema2( t, 20.0 ) '' your implementation

'' and from here on, I can directly change the value of the static variable 'average' and screw everything up =D

foo = 332 '' just set the 'average' value inside the ema2() function...
foo = ema2( t, 20.0 ) '' ...and the next call returns crap
Also, I fail to see it 'simpler', I just see it refactored. My coding conventions aren't yours, obviously ;)
Last edited by paul doe on Nov 04, 2017 23:50, edited 2 times in total.
deltarho[1859]
Posts: 4292
Joined: Jan 02, 2017 0:34
Location: UK
Contact:

Re: Very basic process CPU load determiner

Post by deltarho[1859] »

paul doe wrote:Also, I fail to see it 'simpler', I just see it refactored.
Same conclusion here.
deltarho[1859]
Posts: 4292
Joined: Jan 02, 2017 0:34
Location: UK
Contact:

Re: Very basic process CPU load determiner

Post by deltarho[1859] »

Thanks paul doe.

If we look at the Performance Monitor graph above we cannot tell when it was taken. A snapshot taken now would be similar. A snapshot taken in the future would be similar. The point being that there is no temporal influence. We cannot say the same of the FTSE 100 or the Dow Jones or whatever.

Having said that I have just found this: "Some computer performance metrics, e.g. the average process queue length, or the average CPU utilization, use a form of exponential moving average."

I can see a role for EMA with regard the system load but not for a single process. If a process suddenly 'ups the ante' do we want to see a gentle transition or a spontaneous transition. Using Cpustres.exe and the Performance Monitor with one busy thread in use if I quickly added another two busy threads the output graphing respnds immediately. I would rather CPU Load behaved in the same way.

I have put you EMA code in a folder as I can see a use for it in other projects. Thank you.
paul doe
Moderator
Posts: 1730
Joined: Jul 25, 2017 17:22
Location: Argentina

Re: Very basic process CPU load determiner

Post by paul doe »

deltarho[1859] wrote:I have put you EMA code in a folder as I can see a use for it in other projects. Thank you.
You're welcome. Use it however you like, of course. No credit is given or taken =D
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Very basic process CPU load determiner

Post by fxm »

paul doe wrote:I never return byref for values that are supposed to be encapsulated, whether it's faster or not. You're essentially returning a pointer to an internal value that you weren't supposed to be accessing in the first place. So, nothing prevents you to get really clever and do something like this .....
When the execution speed is paramount, we can also return by "Byref As Const Datatype", which protects the return reference (except from intentional hacking).
paul doe
Moderator
Posts: 1730
Joined: Jul 25, 2017 17:22
Location: Argentina

Re: Very basic process CPU load determiner

Post by paul doe »

fxm wrote:When the execution speed is paramount, we can also return by "Byref As Const Datatype", which protects the return reference (except from intentional hacking).
Of course. But why bother with a simple double value? ;)
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Very basic process CPU load determiner

Post by fxm »

Yes, I was thinking for large parameters as often the UDT instances.
Post Reply