(Z)String UTF8 Aware

[marcov]

For example, UTF-8 is 100% ASCII compatible

Factually incorrect. Ascii can be broken into pieces on every index yet both parts will still be printable glyphs, UTF-8 cannot.

[Munair]

For example, UTF-8 is 100% ASCII compatible

Factually incorrect. Ascii can be broken into pieces on every index yet both parts will still be printable glyphs, UTF-8 cannot.

I think you know what I mean.

[marcov]

For example, UTF-8 is 100% ASCII compatible

Factually incorrect. Ascii can be broken into pieces on every index yet both parts will still be printable glyphs, UTF-8 cannot.

I think you know what I mean.

To be honest, I do not. UTF-16 also has the same base range of ASCII in each char.

[Munair]

To be honest, I do not. UTF-16 also has the same base range of ASCII in each char.

Yes, but UTF-16 allows only two code units, word or double word, whereas UTF-8 allows four units, single byte to double word. Even though &h0041 would work as a valid ASCII value, the word would have to be cast to byte, which, strictly speaking is not compatible. Even worse, with UTF-16 endianess, &h4100 would fail altogether. Simply put, one cannot guarantee that UTF-16 encoded text contains valid ASCII.

In short: "UTF-8 requires either 8, 16, 24 or 32 bits (one to four octets) to encode a Unicode character, UTF-16 requires either 16 or 32 bits to encode a character, and UTF-32 always requires 32 bits to encode a character."

[jj2007]

UTF-8 is 100% ASCII compatible

To clarify: The range of the ASCII (http://www.asciitable.com/) charset is 0...127, and they are all identical to the respective UTF-8 range. So YES, UTF-8 is 100% ASCII compatible.

[caseih]

Right. ASCII is valid utf-8 and for code points under 128, utf-8 bytes are the same as ASCII. The same cannot be said for utf-16.