Blake2 hashing function

[deltarho[1859]]

I should imagine that most of you know that a few years ago the NIST held a competition to find a hash function to complement the SHA2 family of hashing functions. The winner was a subset of Keccak and it became SHA3. A lot of people felt that Blake should have won. It came in two forms; 32 bit platforms [digests from 1 to 32 bytes] and 64 bit platforms [digests from 1 to 64 bytes]. Anyway, that is water under the bridge.

Some of the authors of Blake got together and developed Blake2: Full details at blake2.net. At the website is a graph comparing Blake2b ( optimized for 64 bit platforms ) and Blake2s ( optimized for 32 bit platforms ). Both Blake2's are very fast. Blake2b is not only faster than SHA256 it is faster than SHA1 and MD5.

The reason I am looking at this now is because my current project is suffering by SHA2 being not fast enough.

Here is a link to four binaries: b2sum-bin_20130305.zip

b2sum-amd64-linux [64 bit platform]
b2sum-i686-linux [32 bit platform]
b2sum-amd64-windows [64 bit platform]
b2sum-i686-windows [32 bit platform]

There are also parallel versions for multi-core processors: Blake2bp( 4-way parallel ) and Blak2sp( 8-way parallel)

Here is an example using "b2sum-amd64-windows.exe", for 64 bit Windows. Note that I am talking about 64 bit Windows and not 64 bit mode. The two test vectors are from BLAKE (hash function) at WikiPedia.

Code: Select all

#include once "Windows.bi"
 
#define CrLf Chr(13,10)
 
Dim As String sFileName = "Empty.txt"
'Dim As String sFilename = "BrownFox.txt"
Dim As Double t
 
Chdir("F:\FreeBASIC\Blake2") ' change for your setup
t = Timer
Open Pipe "b2sum-amd64-windows.exe -a blake2b " + sFileName For Input As #1
  Dim As String HashHex
  Line Input #1, HashHex
Close #1
t = Timer - t
 
MessageBox ( Null, Ucase(Left(HashHex, 128)) + CrLf + CrLf + Str(Int(t*1000))+ "ms", "Blake2b", MB_OK )
 
'BLAKE2b-512("")
' = 786A02F742015903C6C6FD852552D272912F4740E15847618A86E217F71F5419
'   D25E1031AFEE585313896444934EB04B903A685B1448B755D56F701AFE9BE2CE
 
'BLAKE2b-512("The quick brown fox jumps over the lazy dog")
' = A8ADD4BDDDFD93E4877D2746E62817B116364A1FA7BC148D95090BC7333B3673
'   F82401CF7AA2E4CB1ECD90296E3F14CB5413F8ED77BE73045B13914CDCD6A918

I also did a test on a 100MB file coming out of the filecache on each run to maintain a level playing field. Timings in milli-seconds.

Code: Select all

SHA512   485
SHA256   520
Blake2b  226 outputs 512 bits
Blake2s  267 outputs 256 bits
Blake2bp 144 outputs 512 bits
Blake2sp 140 outputs 256 bits

On my machine I would use Blake2bp and if I wanted 256 bit I would simply halve the output.

On my machine then Blake2bp is about 3.5 times faster than SHA2. For some strange reason SHA512 is faster than SHA256 for Microsoft's CNG; noticed with a few of my apps. This was not the case with Microsoft's CryptoAPI; prior to Windows Vista.

[deltarho[1859]]

I have done further tests where the filecache is purged of the 100MB.txt file before each and every test and I am getting not much difference between SHA2 and Blake2, which is not making any sense. Oh, dear!

Obviously, a software solution would better than piping but I do not have any Blake2 code. There is some code at PureBASIC for Blake but Blake is not Blake2. RFC 7693 covers it but that is more than a five minute job, may be 20. <laugh>

[TeeEmCee]

I have done further tests where the filecache is purged of the 100MB.txt file before each and every test and I am getting not much difference between SHA2 and Blake2, which is not making any sense. Oh, dear!

Why would you expect a significant time difference? If you have a harddisk, disk latency is huge, and even the throughput of reading from the disk (a couple hundred MB/s at most, even for a cheap SSD) is far lower than the throughout of computing the hash, which is only a few cycles per byte (so close to 1GB/s).

Obviously, a software solution would better than piping but I do not have any Blake2 code.

Why not link to the BLAKE2 library?

For some strange reason SHA512 is faster than SHA256 for Microsoft's CNG; noticed with a few of my apps. This was not the case with Microsoft's CryptoAPI; prior to Windows Vista.

That is to be expected, SHA512 is faster on a 64 bit machine. It and BLAKE2 are both designed that way. See https://en.wikipedia.org/wiki/SHA-2#Comparison_of_SHA_functions. "Prior to Vista" meaning what, a 32 bit Win XP machine? On a 32 bit machine, SHA256 should be faster.

[deltarho[1859]]

If you have a harddisk, disk latency is huge, and even the throughput of reading from the disk (a couple hundred MB/s at most, even for a cheap SSD) is far lower than the throughout of computing the hash, which is only a few cycles per byte (so close to 1GB/s).

I am well aware of that and why the ratio of AES256/AES128 on paper diminishes the slower the data is read and the paper ratio is only achieved when the data is read at infinite speed. The 'real world' ratio is then greater for SSDs compared with HDDs.

However, the figures that I am getting are as if I was using two different storage media.

Thanks for the explanation re SHA512, albeit lacking in grace.

[deltarho[1859]]

Actually, I am looking at hashing AES encrypted data. Since padding is employed to the next 16 bytes it follows that the encrypted data is divisible by eight. I can then use a binary hash tree with the number of nodes determined by the number of cores available.