Hi,
actually adapting a bigger datasheet for Multisort (posted in another topic).
This one has 2 mio. rows to sorts; I see significant slowdown of whole preparing process;
it takes cca 10 min. - create/load datasheet to arrays and (simple) pre-sort datasheet by 1 column for all important columns.
Cca 15 sorts, the rest is balast - columns with notes, details, explanations etc.
(Then start of multisorts as separate chapter, where no threading is thinked for now).
I would like to speed up this preparing process with threading and divide work to parts.
a/ procedure "fillArrayDS()" - loop of 1 to 2 mio creation of datasheet array divided to 4 threads, t.m. 4 loops :
1 to 500.000, 501.000 to 1.000.000, 1.000.001 to 1.500.000, 1.500.001 to 2.000.000
b/ 15 single sorts called in separate threads
c/ eventually re/creation of pivot sub-array matrix (a few columns to multisort), here I am on doubts with filling array members under threading.
As I understand threading, no matter how much CPU cores I have, right ?
Sending job for actually free core is internal matter of FB threading/mutexing functions, right ?
Howto speed-up sorts with using "Mutexes & Threads" ?
Re: Howto speed-up sorts with using "Mutexes & Threads" ?
The loading from file should be limited by the disc speed, if not then you are probably doing something wrong (inefficient) there.
For the sorting, you could give each thread a part of the data (e.g a quarter for 4 threads) to sort and when all threads are ready, one final sort on all data. This last short short be fast (merging sorted data). BUT, with a data size >> CPU cache L2, the limit will probably be memory speed and not the number of cores.
The OS will divide the load over the cores as far as I know. There is probably an optimum number of threads, but CPU dependent.
For the sorting, you could give each thread a part of the data (e.g a quarter for 4 threads) to sort and when all threads are ready, one final sort on all data. This last short short be fast (merging sorted data). BUT, with a data size >> CPU cache L2, the limit will probably be memory speed and not the number of cores.
The OS will divide the load over the cores as far as I know. There is probably an optimum number of threads, but CPU dependent.
Re: Howto speed-up sorts with using "Mutexes & Threads" ?
Indeed.Old code yet.
Combsort used, datasheet autogenerated everytime from raw data.Slow.
Quicksort, heapsort, incr.update - in betatest.Core subs change.
Combsort used, datasheet autogenerated everytime from raw data.Slow.
Quicksort, heapsort, incr.update - in betatest.Core subs change.
Re: Howto speed-up sorts with using "Mutexes & Threads" ?
Sorting 2 Million rows of a text file should take around one second, not ten minutes. Can you upload an example file somewhere for testing algos?