Error in LEN() command

[haegartheroot]

Hello together,

I do not know if someone is already aware of an error in the LEN() command in FreeBasic?
If you use strings with international characters (like German umlauts or French accents) you get wrong results with the LEN() command.

Just to show it, I have created a small test program:

Code: Select all

'   Short test program demonstrating the wrong results of the
'	len() command in FreeBasic
'	-----------------------------------------------------------
'                                  ....+....1....+...
	dim germanstring  as string = "Märchenkönigsgrüße" 'length=18
'                                  ....+....1....+..
	dim danishstring  as string = "Rødgrød med fløde"  'length=17
'                                  ....+....1....+
	dim turkishstring as string = "Bir çay lütfen!"    'length=15
'                                  ....+....1....+...
	dim frenchstring  as string = "Académie Française" 'length=18
	dim as integer germanlen, danishlen, turkishlen, frenchlen

    print "Test of international strings with FreeBasic"
    print "--------------------------------------------"
    germanlen  = len(germanstring)
    print "Output of German string:  " + germanstring
	print "Length of German string:  "; germanlen
	danishlen  = len(danishstring)
    print "Output of Danish string:  " + danishstring
    print "Length of Danish string:  "; danishlen
	turkishlen = len(turkishstring)
	print "Output of Turkish string: " + turkishstring
    print "Length of Turkish string: "; turkishlen
	frenchlen  = len(frenchstring)
    print "Output of French string:  " + frenchstring
    print "Length of French string:  "; frenchlen

The output is:

• Test of international strings with FreeBasic
  --------------------------------------------
  Output of German string: Märchenkönigsgrüße
  Length of German string: 22
  Output of Danish string: Rødgrød med fløde
  Length of Danish string: 20
  Output of Turkish string: Bir çay lütfen!
  Length of Turkish string: 17
  Output of French string: Académie Française
  Length of French string: 20

As you can see from this, all returned string lengths do not match the number of displayed characters.

If I do the same in other languages, e.g. Mono Basic, the results are correct.
To prove it, here's another small test program written in Mono Basic:

Code: Select all

Public Class Application
	Public Shared Sub Main()
'                                  ....+....1....+...
	dim germanstring  as string = "Märchenkönigsgrüße" 'length=18
'                                  ....+....1....+..
	dim danishstring  as string = "Rødgrød med fløde"  'length=17
'                                  ....+....1....+
	dim turkishstring as string = "Bir çay lütfen!"    'length=15
'                                  ....+....1....+...
	dim frenchstring  as string = "Académie Française" 'length=18
	dim germanlen, danishlen, turkishlen, frenchlen as integer

	System.Console.WriteLine("Test of international strings with Mono Basic")
	System.Console.WriteLine("---------------------------------------------")

	germanlen  = germanstring.length()
	System.Console.WriteLine("Output of German string:  " + germanstring)
	System.Console.WriteLine("Length of German string:  " + germanlen.ToString())
	danishlen  = danishstring.length()
	System.Console.WriteLine("Output of Danish string:  " + danishstring)
	System.Console.WriteLine("Length of Danish string:  " + danishlen.ToString())
	turkishlen = turkishstring.length()
	System.Console.WriteLine("Output of Turkish string: " + turkishstring)
	System.Console.WriteLine("Length of Turkish string: " + turkishlen.ToString())
	frenchlen  = frenchstring.length()
	System.Console.WriteLine("Output of French string:  " + frenchstring)
	System.Console.WriteLine("Length of French string:  " + frenchlen.ToString())
	End Sub
End Class

And here is the output:

• Test of international strings with Mono Basic
  ---------------------------------------------
  Output of German string: Märchenkönigsgrüße
  Length of German string: 18
  Output of Danish string: Rødgrød med fløde
  Length of Danish string: 17
  Output of Turkish string: Bir çay lütfen!
  Length of Turkish string: 15
  Output of French string: Académie Française
  Length of French string: 18

In this case the string lengths are correct (equal to the displayed characters).

Both programs (FreeBasic and Mono Basic) were executed on the same box running Ubuntu 16.04LTS.
I also tried the FreeBasic test program under Window which gives me the same (wrong) results.

Is there a way to get the correct string lengths (=number of displayed characters) in FreeBasic?

Thanks!

//haegar

[FreeFox]

I've just tried your FB code myself and the lengths are displayed correctly even if the non-English characters themselves are not:

Code: Select all

Test of international strings with FreeBasic
--------------------------------------------
Output of German string:  Mõrchenk÷nigsgr³▀e
Length of German string:   18
Output of Danish string:  R°dgr°d med fl°de
Length of Danish string:   17
Output of Turkish string: Bir þay l³tfen!
Length of Turkish string:  15
Output of French string:  AcadÚmie Franþaise
Length of French string:   18

I'm using FB version 1.05.0 for Windows 64 bit. The console is set to code page 850.

What are you using?

[caseih]

What operating system are you using? What unicode encoding is your source code file using?

Outside of the WString type, FreeBASIC has no in-built unicode support. What you are seeing is simply the length in bytes of the unicode string as encoded in the source code file. On Linux this would likely be UTF-8. In your example code, FB has no idea that the string has unicode characters in it, and it's just because the console knows how to interpret those encoded bytes that it prints the string out correctly to the console. I suspect you're using Linux, which is normally set to use UTF-8 in the terminal.

If you want to use unicode in FB, either you'll have to encode everything to UTF-8 and store it in normal strings (as your example does), and create your own length function to measure it (requiring processing the string one byte at a time using UTF-8 parsing), or use the WString type. FB ships with some examples of working with unicode and ships with a helper file called utf_conv.bi that you can use to help with encoding and decoding unicode into bytes.

Apparently finding the length of a unicode string is kind of hard anyway because length doesn't always make sense. You could divide the string up by code points, but that doesn't necessarily correspond to characters. And the definition of a character is a bit muddy too as modifiers can affect the appearance of a single character.

[MrSwiss]

I'm using FB version 1.05.0 for Windows 64 bit. The console is set to code page 437.
Same result as FreeFox. wrong String, correct Len(String).
BTW: String in source is already mangled after copying.

[FreeFox]

Just realized I had saved the file in ASCII rather than UTF-8.

Having changed it to the latter, everything's now displaying fine:

Code: Select all

Test of international strings with FreeBasic
--------------------------------------------
Output of German string:  Märchenkönigsgrüße
Length of German string:   18
Output of Danish string:  Rødgrød med fløde
Length of Danish string:   17
Output of Turkish string: Bir çay lütfen!
Length of Turkish string:  15
Output of French string:  Académie Française
Length of French string:   18

The reason it's now working without the use of WString is that all the non-English characters used can be accommodated by code page 850 for code points within the range 128-255 (i.e. single byte) and the Windows console can display them.

It's a pity we don't have a variable length unicode string type as this would eliminate the code page problem even at the expense of using more memory.

[caseih]

Yes if you add in strings from languages like chinese that are all multi-byte, you'll definitely see inconsistent lengths with FB's byte-based len() if you just use normal strings. And it's only luck of the draw whether print statements work correctly under Windows as it depends on the encoding of the source code file vs the encoding expected by the console.

I don't think the graphics library works at all with unicode of any kind.

There are several implementations of dynamic wide strings that you can find on the forums. Some are rather complete. One came up just a few months back, but there wasn't a lot of interest, for whatever reason.

[FreeFox]

Thanks @caseih.

I'll look into that :)

I do know that VB.Net (and probably therefore Mono Basic) has always used unicode strings from the start and doesn't support ASCII strings at all (you have to use their Byte type instead).

Whilst I think this is generally a bad idea (ideally you need to support both), it probably explains why the OP was getting the correct results when using Mono Basic with the Ubuntu terminal.

[haegartheroot]

After having tried it under different Operating systems (Linux, Windows XP/7/10) and FBC compiler versions 1.05 and 0.23 on different PC's I can confirm that I always get the wrong lengths:

• Length German string: 22 (wrong) instead of 18 (right)
  Length of Danish string: 20 (wrong) instead of 17 (right)
  Length of Turkish string: 17 (wrong) instead of 15 (right)
  Length of French string: 20 (wrong) instead of 18 (right)

How did you achieve the right lengths with your FreeBasic compiler????

I made also another test with Gambas3 under Ubuntu 16.04 - here's my test code:

Code: Select all

' Gambas module file

Public Sub Main()
'   Test Program to determine String length in Gambas3
'	-----------------------------------------------------------
'                                  ....+....1....+...
	Dim germanstring As String = "Märchenkönigsgrüße" 'length=18
'                                  ....+....1....+..
	Dim danishstring As String = "Rødgrød med fløde"  'length=17
'                                  ....+....1....+
	Dim turkishstring As String = "Bir çay lütfen!"    'length=15
'                                  ....+....1....+...
	Dim frenchstring As String = "Académie Française" 'length=18
	Dim germanlen, danishlen, turkishlen, frenchlen As Integer 

  Print "Test of international strings with Gambas3"
  Print "------------------------------------------"
  germanlen = String.Len(germanstring)
  Print "Output of German string:  " & germanstring
	Print "Length of German string:  "; germanlen
	danishlen = String.Len(danishstring)
  Print "Output of Danish string:  " & danishstring
  Print "Length of Danish string:  "; danishlen
	turkishlen = String.Len(turkishstring)
	Print "Output of Turkish string: " & turkishstring
  Print "Length of Turkish string: "; turkishlen
	frenchlen = String.Len(frenchstring)
  Print "Output of French string:  " & frenchstring
  Print "Length of French string:  "; frenchlen

End

And also Gambas gives me the correct results:

• Test of international strings with Gambas3
  ------------------------------------------
  Output of German string: Märchenkönigsgrüße
  Length of German string: 18
  Output of Danish string: Rødgrød med fløde
  Length of Danish string: 17
  Output of Turkish string: Bir çay lütfen!
  Length of Turkish string: 15
  Output of French string: Académie Française
  Length of French string: 18

What's wrong with FreeBasic?

[St_W]

Nothing is wrong with FreeBasic, its Unicode support (out of the box) is just quite limited (unfortunately ...).

As other people have already explained above FreeBasic's Len() function does not understand UTF-8 and thus counts single bytes. While this is no problem for plain-english texts without any special characters it results in possibly strange results for other languages. For example characters like '"ä" need multiple bytes in UTF-8, thus count as several letters in FreeBasic's Len() function.

To fix this you've got several choices:
- use a single-byte encoding instead of UTF-8
- use WString instead of String (which uses UCS-4 on Linux AFAIR)
- use some helper methods / custom library which supports UTF-8

As you do not seem to understand the significance of different character encodings yet, as we've seen, I'd suggest the following Wikipedia article that explains it quite well:
https://en.wikipedia.org/wiki/Character_encoding

[haegartheroot]

However, what I do not understand is: why are the forum users MrSwiss and FreeFox getting the correct results?
As they stated, they saved the source code as UTF-8 (like me) and they are using FBC 1.05 (like me).

And: I just tested with the old QB45 DOS compiler and it gives me the correct string lengths for all international strings shown in the example as well.

So for me the FreeBasic LEN() command is not working correctly.
LEN() is working correct in QB45, Gambas or Mono Basic.

Nevertheless I tried to create a workaround by programming "quick & dirty" intLEN() and intMID() functions.
These functions work correctly with all given international strings. Here is the modified source of my initial example:

Code: Select all

'   Short test program demonstrating the new intLEN() and 
'	intMID functions returning the correct length and the
'   correct part of international (non-ASCII) strings
'	-----------------------------------------------------------
' 
#include once "gtk/gtk.bi" 'includes g_convert
#define NULL 0
Declare Function UTF8toISO( Byval src As String ) As String
Declare Function ISOtoUTF8( Byval src As String ) As String
Declare Function intLEN (byval src as string) as integer
Declare Function intMID ( Byval src As String, Byval strt as integer, Byval leng as integer ) As String

'                                 ....+....1....+...
	dim germanstring  as string = "Märchenkönigsgrüße" 'length=18
'                                  ....+....1....+..
	dim danishstring  as string = "Rødgrød med fløde"  'length=17
'                                  ....+....1....+
	dim turkishstring as string = "Bir çay lütfen!"    'length=15
'                                  ....+....1....+...
	dim frenchstring  as string = "Académie Française" 'length=18
	dim as integer germanlen, danishlen, turkishlen, frenchlen


    print "Test of international strings with FreeBasic"
    print "--------------------------------------------"
    germanlen  = intLEN(germanstring)
    print "Output of German string  : " + germanstring
	print "Length of German string  :"; germanlen
	print "2nd char in German string: " + intMID(germanstring,2,1)
	danishlen  = intLEN(danishstring)
    print "Output of Danish string  : " + danishstring
    print "Length of Danish string  :"; danishlen
	print "Char 1-3 in Danish string: " + intMID(danishstring,1,3)
	turkishlen = intLEN(turkishstring)
	print "Output of Turkish string : " + turkishstring
    print "Length of Turkish string :"; turkishlen
	print "Char 10 in Turkish string: " + intMID(turkishstring,10,1)
	frenchlen  = intLEN(frenchstring)
    print "Output of French string  : " + frenchstring
    print "Length of French string  :"; frenchlen
	print "Char 5 in French string  : " + intMID(frenchstring,5,1)


Function intLEN ( Byval src As String ) As Integer
  intlen = len(UTF8toISO(src))
End Function

Function intMID ( Byval src As String, Byval strt as integer, Byval leng as integer ) As String
  dim tmpstr as string
  dim midstr as string
  
  tmpstr = (UTF8toISO(src))
  midstr = mid(tmpstr,strt,leng)
  intMID = (ISOtoUTF8(midstr))
  
End Function

Function UTF8toISO( Byval src As String ) As String
  Dim As Zstring Ptr buff
  Dim As GError Ptr ptr subrc
  Dim savestring as string
  Function = ""
  savestring = src
' Convert UTF-8 in ISO-8859-1 (Latin Alphabet No. 1)
  buff = g_convert( src, Len( src ), "ISO-8859-1", "UTF-8", NULL, NULL, NULL )
  if len (*buff) = 0 then
     Function = savestring
  else
     Function = *buff
  end if
  g_free( buff )
End Function

Function ISOtoUTF8( Byval src As String ) As String
  Dim As Zstring Ptr buff
  Dim As GError Ptr ptr subrc
  Dim savestring as string
  Function = ""
  savestring = src
' Convert ISO-8859-1 in UTF-8 
  buff = g_convert( src, Len( src ), "UTF-8","ISO-8859-1", NULL, NULL, NULL )
  if len (*buff) = 0 then
     Function = savestring
  else
     Function = *buff
  end if
  g_free( buff )
End Function

Now the resulting output is this:

• Test of international strings with FreeBasic
  --------------------------------------------
  Output of German string : Märchenkönigsgrüße
  Length of German string : 18
  2nd char in German string: ä
  Output of Danish string : Rødgrød med fløde
  Length of Danish string : 17
  Char 1-3 in Danish string: Rød
  Output of Turkish string : Bir çay lütfen!
  Length of Turkish string : 15
  Char 10 in Turkish string: ü
  Output of French string : Académie Française
  Length of French string : 18
  Char 5 in French string : é

It's not nice, but it works.

And my point remains: LEN() in FreeBasic is not working correctly with all code pages (especially UTF-8).

//haegar

[caseih]

I do know that VB.Net (and probably therefore Mono Basic) has always used unicode strings from the start and doesn't support ASCII strings at all (you have to use their Byte type instead).

Whilst I think this is generally a bad idea (ideally you need to support both), it probably explains why the OP was getting the correct results when using Mono Basic with the Ubuntu terminal.

Yup you're right about Mono Basic and also Dot Net in general. They all use unicode for all strings, stored internally as UTF-16. I disagree about this being a bad idea. UTF-16 can represent any ASCII string. I think, though, you are referring to situations where you need to read data from a text file, and ASCII is still quite prevalent. That's a different issue, though. Unicode is something that's abstract. It doesn't exist in the real, byte-based world. So anytime you have to place a unicode string into bytes somewhere, whether that's memory (a string) or a file, the unicode must be encoded to bytes. Encoding schemes are wide and varied, and, depending on what characters you need to represent, you can use ASCII, traditional proprietary code pages, UTF-8, UTF-16, UTF-32, or something variation thereof. The rule thumb is anytime you're reading a file, decode the bytes into unicode, and anytime you're writing a file, you encode unicode to bytes. This does add complications, especially while reading files because you have to know what encoding the file might be in, and it's hard to guess correctly, especially with files that are encoded using older Windows code pages.

Languages with in-build unicode support have routines for decoding and encoding data as you move back and forth between abstract unicode and the real world.

After having tried it under different Operating systems (Linux, Windows XP/7/10) and FBC compiler versions 1.05 and 0.23 on different PC's I can confirm that I always get the wrong lengths:

how did you achieve the right lengths with your FreeBasic compiler????

Did you read what everyone wrote? The reason the lengths were right was because the user who did this had his source code file encoded in a character set that could represent all of characters in those strings with one byte. Then his console was set to the same character set. So it worked, but only for him. If he had given me his executable and I ran it on any of my windows machines I'd have just gotten garbage out.

I made also another test with Gambas3 under Ubuntu 16.04 - here's my test code:
What's wrong with FreeBasic?

Did you read anything of what any of us wrote? There's nothing wrong with FreeBASIC. The problem is your understanding of unicode combined with FB's lack of support for unicode. Your example file has no unicode in it, at least from FB's point of view. On Linux, your source code is UTF-8, and those unicode characters encode to multiple bytes. When you run the program it blindly outputs those UTF-8 bytes to the terminal, which recognizes them and prints out unicode for you. But FB had no idea what it was doing and LEN() does not decode UTF-8, so it can only count bytes not characters.

GamBas3 works because it has a basic understanding of unicode. GamBas uses UTF-8 for encoding unicode internally, so it's LEN() is UTF-8 aware and can count characters in a string, not just bytes.

Sounds like you need to familiarize yourself with what unicode is and how to deal with it. You can work with unicode in FB if you understand how encoding and decoding works. You can even use normal, dynamic strings to hold the encoded data. If you do, you can't use MID(), LEN(), etc. There are other third-party libraries you could use. And as we said if you use WSTRING for your strings (unfortunately they aren't dynamically-sized), then you can use MID(), LEN(), WCHR(), etc.

Here's a link to introduce you to the concept of unicode and the difference between encoding and decoding, and other related issues:
http://www.joelonsoftware.com/articles/Unicode.html

[caseih]

However, what I do not understand is: why are the forum users MrSwiss and FreeFox getting the correct results?
As they stated, they saved the source code as UTF-8 (like me) and they are using FBC 1.05 (like me).

It's a confluence of lucky choices of source code file encoding and console character set. The one's that "worked" were not using UTF-8 in their source code files. Their source code files were some other single-byte code page (they essentially weren't encoded in any encoding). In short it worked only by accident, and if the EXE that worked were to be used on a system with a different character set in the console, it would just print out garbage. As well, consider this example:

Code: Select all

dim a as string
dim b as wstring * 100

a = "你好，世界！Märchenkönigsgrüße Rødgrød med fløde"
b = "你好，世界！"
print a
print b

print LEN(a)
print LEN(b)

There are no codepages on the planet that can support ASCII, the german and dutch characters *and* these chinese characters in one byte. Such an example source code file would have to be saved to UTF-8 or UTF-16, and I guarantee the output from LEN(a) will always be "wrong."

And: I just tested with the old QB45 DOS compiler and it gives me the correct string lengths for all international strings shown in the example as well.

On your system your ancient MS-DOS is using a codepage that happens to support all of the characters you need in one byte. Again this is essentially a lucky coincidence that it worked.

Nevertheless I tried to create a workaround by programming "quick & dirty" intLEN() and intMID() functions.
These functions work correctly with all given international strings. Here is the modified source of my initial example:

Yes you have to draw on external libraries to work with UTF-8 in FB. However your idea to convert them back and forth to ISO-8859-1 is not a very good one. ISO-8859-1 (like your MS-DOS code page used in QB45) can only support a very small number of non-english characters. You're better off on Linux using UTF-8 everywhere (especially for output). Windows is a potential problem; I'm not sure if FB is using the unicode-aware version of the win32 console API. If it is, then UTF-16 is what you want to be outputting when printing to the screen. Alternatively use the built-in WSTRING type!

And my point remains: LEN() in FreeBasic is not working correctly with all code pages (especially UTF-8).

We already mentioned this a long time back! LEN() is only unicode aware when using WSTRING. Otherwise it counts bytes. Whether or not LEN() gives the right length depends entirely on the encoding of the string. This is unlikely to change in the near future.

[haegartheroot]

Did you read anything of what any of us wrote? There's nothing wrong with FreeBASIC. The problem is your understanding of unicode combined with FB's lack of support for unicode. Your example file has no unicode in it, at least from FB's point of view. On Linux, your source code is UTF-8, and those unicode characters encode to multiple bytes. When you run the program it blindly outputs those UTF-8 bytes to the terminal, which recognizes them and prints out unicode for you. But FB had no idea what it was doing and LEN() does not decode UTF-8, so it can only count bytes not characters.

I guess with "everyone" you mean user "FreeFox".

He wrote:

(1st post)
I've just tried your FB code myself and the lengths are displayed correctly even if the non-English characters themselves are not
....

(2nd post)
Just realized I had saved the file in ASCII rather than UTF-8.
Having changed it to the latter, everything's now displaying fine.

What I don't understand: at least user FreeFox stated that he has saved his source in UTF-8 and got the correct string lengths.
That's the strange thing for me.

FreeFox tells us that the LEN() commands works correctly with UTF-8 (like I would have expected it), but here is does not work!

[haegartheroot]

Did you try your example?

Code: Select all

dim a as string
dim b as wstring * 100

a = "你好，世界！Märchenkönigsgrüße Rødgrød med fløde"
b = "你好，世界！"
print a
print b

print LEN(a)
print LEN(b)

LEN() for wstring does not work!

The result of your example is:

• 你好，世界！Märchenkönigsgrüße Rødgrød med fløde
  
  61
  0

I.e. no output of the string given in b and len of string b is zero.

I have no problems with your critics on my quick & dirty solution, but it would be nicer and more constructive to criticize and to deliver a better solution!

[MrSwiss]

However, what I do not understand is: why are the forum users MrSwiss and FreeFox getting the correct results?
As they stated, they saved the source code as UTF-8 (like me) and they are using FBC 1.05 (like me).

Not correct, I've just "copy & paste" the source and then save it (FBEdit).
ANSI, I expect, because I'm on WIN, and not any other format.

Switzerland is using CP437 - aka: USA (it has to support German Umlaut & French Accents).
But, unlike Germany, it doesn't use the "sharp s" [ß], the swiss people write it as "ss".

Your example looks as follows (before being saved):

Code: Select all

dim a as string
dim b as wstring * 100

a = "??,??!Märchenkönigsgrüße Rødgrød med fløde"
b = "??,??!"
print a
print b

print LEN(a)
print LEN(b)
sleep

??,??!MΣrchenk÷nigsgrⁿ▀e R°dgr°d med fl°de
??,??!
42
6