curses/ncurses

[skystrick]

Any progress on this?

[sir_mud]

Any progress on this?

I'm investigating.

[skystrick]

Bump.

Using debug-monitor mode on my terminal, this is what I figured out:

FB is not properly using Curses or Ncurses. Evidently, FB has some hard-coded VT100 escape sequences in it, because it's sending those ALONG WITH actual Wyse 60 escape sequences.

My terminal is a Wyse 60, and Linux recognizes it as such. All other applications work fine. But FB seems to only want to "speak" VT100 commands, regardless of what type of terminal you're using.

For the record - I am not using any PRINT, LOCATE, COLOR, CLS, or other commands which depend on FB's built-in screen-management capabilities. Instead, I open a raw line to the terminal ("open "CONS:" for output as #1", then print to the screen using "print #1"), with escape sequences for everything I want to do (cursor positioning, bold/underline/reverse video, etc.)

When I launch my FB-compiled program, the very first thing FB does (and it should not) is send across "Esc [6n", which is the VT100 escape sequence for "request cursor position". This is not something I've coded into my program; FB is doing this on its own. It should not be sending VT100 commands to a non-VT100 terminal.

If I manually respond with the VT100 cursor-location response ("Esc [ line;column R"), FB sends out a whole 'nother slew of escape sequences, some of which are actual Wyse 60 commands (which shows FB is using curses/ncurses to some degree), but some of them are also VT100 sequences - again showing FB has been hard-coded in some places.

So what it sends across at that point is:

Esc ( - this is the proper Wyse 60 command for "write-protect mode off"
Esc H - this is the first of two Wyse 60 commands for "regular (non-line-drawing) character set"
Crtl-C - this is the second of two Wyse 60 commands for "regular (non-line-drawing) character set"
Esc G0 - this is the proper Wyse 60 command for "regular video attributes" (no reverse video, no underline, no blinking, etc.)
Esc cD - this is the proper Wyse 60 command for "select U.S. national character set"
Esc `1 - this is the proper Wyse 60 command for "make cursor visible"
Esc [6n - but now we're back to a VT100 command for "what is the cursor's current location (row, column)?"


And every time the FB program sends that "Esc [6n" - which, again, it should not be doing - my system hangs waiting for a response that a non-VT100 terminal doesn't know how to send it.

So can that hard-coded VT100 escape sequence be removed from FB, please? It's a bug, and I need an Orkin man! :-)

[dkl]

Hello again,

That's a lot of useful information there, thanks. What you're saying makes a lot of sense. FB does have some hard-coded escape sequences (while others are retrieved via tgetstr()).

That includes the cursor position query and a bunch of others:

Code: Select all

"\e(U"               INIT_CHARSET  inits PC 437 characters set
"\e(B"               EXIT_CHARSET  exits PC 437 characters set
"\e[6n"              QUERY_CURSOR  query cursor position (not in termcap)
"\e[18t"             QUERY_WINDOW  query terminal window size (not in termcap)
"\e[?1000h\e[?1003h" INIT_XMOUSE   enable X11 mouse
"\e[?1003l\e[?1000l" EXIT_XMOUSE   disable X11 mouse
"\e[H\e[J\e[0m"      EXIT_GFX_MODE cleanup after console gfx mode

It's a bit ugly that the FB runtime initialization already uses QUERY_WINDOW and QUERY_CURSOR. Maybe it could be changed to lazy initialization, then it would only be used when actually needed.

However, the real issue for now are the hard-coded sequences. Does the termcap/terminfo data base contain these sequences? I could not find anything related to querying cursor position on termcap/terminfo manpages. (notice also the "(not in termcap)" comments from FB source code)

If we can't use tgetstr() to query the right sequences, then we have to hard-code them or completely omit them.

With regards to hard-coding, does Wyse 60 have an escape sequence for cursor position query? How could FB determine whether the terminal only supports Wyse 60?

And about omitting them completely... I'm not sure that's a good idea. It seems like we need the cursor position query to have things work properly. Perhaps we can assume to always start at position 1,1; but what if the program uses a printf() or something else which changes cursor position without going through the FB runtime? The FB runtime's internal position would get messed up, but it needs to be accurate for LOCATE and PRINT alignment and who knows what else (a lot of the runtime code depends on the cursor position... I don't fully understand it).

Of course, doing lazy initialization would help, in the sense that you can use FB with Wyse 60 terminal as long as you don't use any of FB's console commands (so you can use the CRT functions instead for example).

[skystrick]

However, the real issue for now are the hard-coded sequences. Does the termcap/terminfo data base contain these sequences? I could not find anything related to querying cursor position on termcap/terminfo manpages. (notice also the "(not in termcap)" comments from FB source code)

Wow, I never even considered that terminfo would lack such a simple command!

On this site - "terminfo source code for NCURSES" - http://invisible-island.net/ncurses/terminfo.src.html - they had this to say:

"# The System V Release 4 and XPG4 terminfo format defines ten string
# capabilities for use by applications, <u0>...<u9>. In this file, we use
# certain of these capabilities to describe functions which are not covered
# by terminfo. The mapping is as follows:
#
# u9 terminal enquire string (equiv. to ANSI/ECMA-48 DA)
# u8 terminal answerback description
# u7 cursor position request (equiv. to VT100/ANSI/ECMA-48 DSR 6)
# u6 cursor position report (equiv. to ANSI/ECMA-48 CPR)"

Looking through the terminfo entries there, I do see several which contain "\e[6n" as the u7 value.

Unfortunately, there isn't "one version" of the terminfo database. I don't know what percentage of Linux FB users have a version which incorporates that command. I hesistate, therefore, to declare it a reliable enough thing to be a solution to this problem.

Of course, leaving FB "as-is" would make any use of curses/ncurses completely useless. There's no point in having FB get command strings from terminfo, if it's going to mix in VT100 command strings regardless of what kind of terminal is actually being used. Effectively, FB only supports one kind of terminal.

With regards to hard-coding, does Wyse 60 have an escape sequence for cursor position query?

I don't know whether the Wyse 60 native command set has a command for requesting the cursor location. I can't find one in the Wyse specs I've got. I've never had a need for one; I just tell the cursor where I want it to go (and thus I know where it is, becuase it's wherever I sent it).

Regardless, this app is planned for deployment in a legacy office which contains various models of terminals (Wyse 60, VT100, ADDS A2, and a few others). Adding support for the Wyse 60 wouldn't solve the overall problem of portability.

How could FB determine whether the terminal only supports Wyse 60?

To answer your question - in Linux/Unix, typing "echo $TERM" at the command line will tell you what kind of terminal you're using. My FB program uses that at the beginning to determine what kind of terminal it's running on:

Code: Select all

SHELL "echo $TERM > foo.tmp"
OPEN "foo.tmp" FOR INPUT AS #1
LINE INPUT #1, TERMTYPE$
CLOSE #1
KILL "foo.tmp"

IF TERMTYPE$ = "vt100" then ...
IF TERMTYPE$ = "wy60" then ...

...you get the picture.

Of course, doing lazy initialization would help, in the sense that you can use FB with Wyse 60 terminal as long as you don't use any of FB's console commands (so you can use the CRT functions instead for example).

I think this is the best solution. Since terminfo cannot be relied upon to supply information vis-a-vis the cursor-location command, the next best option is "lazy initialization" where - as long as the program doesn't use any of FB's screen-handling commands (LOCATE, PRINT, etc.) - it doesn't send out any commands (i.e. the Esc [6n).

What that would do is preserve FB as-is for the majority of VT100 users out there, with no changes to their code, but also enable users of other terminals (like me) to be able to use the program effectively.

What it would do would be to place the burden on programmers of alternative terminals - like me - to manage the terminal entirely with our own commands within our programs. I feel that's totally fair.

Alternatively, your other idea is good too - if FB automatically cleared the screen upon initializing, and automatically "homed" the cursor (sent it to position 1,1), then FB could internally "remember" the cursor's location from then on, because every subsequent LOCATE and PRINT command would move the cursor by a known amount (or to a known location). And, luckily, the cursor-homing command IS part of terminfo.

This does beg the question, though - why does FB need to know where the cursor is? Every PRINT command either comes after a LOCATE command (so we don't need FB to ask the terminal where the cursor is, because we know where we just sent it), or (if no LOCATE command is used) it simply outputs text wherever the cursor happens to be. Only if we're using something like POS(0), do we need FB to "know" where the cursor is...but again, that goes back to the idea that if the program initialized by clearing the screen and homing to 1,1 then FB could keep track internally instead of having to query the terminal. Certainly that's how QB did it?

[marcov]

Code: Select all

"\e(U"               INIT_CHARSET  inits PC 437 characters set
"\e(B"               EXIT_CHARSET  exits PC 437 characters set

Those are afaik not vt100, but only for emulations of it by PCs. Thus not standarized

Code: Select all

"\e[6n"              QUERY_CURSOR  query cursor position (not in termcap)
"\e[18t"             QUERY_WINDOW  query terminal window size (not in termcap)
"\e[?1000h\e[?1003h" INIT_XMOUSE   enable X11 mouse
"\e[?1003l\e[?1000l" EXIT_XMOUSE   disable X11 mouse
"\e[H\e[J\e[0m"      EXIT_GFX_MODE cleanup after console gfx mode

IIRC in the past you only did these when terminal was xterm. But with emergence of other more prominent Xterminals like gnome terminal etc that rule got a bit muddled. Maybe test $DISPLAY ?

If we can't use tgetstr() to query the right sequences, then we have to hard-code them or completely omit them.

Personally I would simply define a boolean in the RTLIB somewhere, to allow users to forcedly disable them.

In FPC we also hardcode them, but they are hardcoded in a more application specific app layer (of the textmode IDE), which has a commandline switch to disable them.

IIRC this was also done because the console (not X terminal) of BSDs gave problems too.

[skystrick]

Code: Select all

"\e(U"               INIT_CHARSET  inits PC 437 characters set
"\e(B"               EXIT_CHARSET  exits PC 437 characters set

Those are afaik not vt100, but only for emulations of it by PCs. Thus not standarized

Correct.

Let's note, though, that the *ACTUAL* VT100 standard as published in 1978 doesn't include several commands which are commonly accpted as part of the VT100 command set today (i.e. the color commands). Many emulators based off the VT100 command set - such as the Linux console, and ANSI.SYS under MS-DOS - have extended the set of commands commonly ascribed to the VT100 set. Every modern VT100 emulator I've come across accepts the extended commands.

For the record, codepage 437 was a Microsoft rip-off of an old Wang word-processing character set.

It gets kind of complicated to describe VT100 character sets - national vs. international, user-loaded "soft" sets vs. factory standards, etc. But in simple terms, there are two useful character sets in an average VT100: the "standard" character set (for normal text), and the "special graphics and line-drawing" character set, which enables you to draw boxes on the screen without having to use cheap-looking hyphens and pipe characters. You can go back and forth between the two by "shifting in" and "shifting out" of the line-drawing set.

There are at least two different ways to do that shift in/shift out procedure, but the simplest is simply to send CHR$(14) and CHR$(15) to the terminal. 14 puts you into the line-drawing set, 15 brings you back out into the normal text set.

Personally I would simply define a boolean in the RTLIB somewhere, to allow users to forcedly disable them.

In FPC we also hardcode them, but they are hardcoded in a more application specific app layer (of the textmode IDE), which has a commandline switch to disable them.

Either way - "lazy initialization", or a command-line switch at compile time - I'd be thrilled. I just want to be able to make portable software! :-)

[dkl]

As for why FB "needs" to know the cursor position:

what if the program uses a printf() or something else which changes cursor position without going through the FB runtime? The FB runtime's internal position would get messed up, but it needs to be accurate for LOCATE and PRINT alignment and who knows what else

and besides, the current behaviour is to start at whatever position the cursor happens to be in the terminal, not to clear the screen and start at 1,1. Add to that that we want to support the Pos/Csrlin/Screen functions...

On the other hand, for *BSD systems, FB does assume a start position of 1,1 already (and does not query the cursor position in any way).

So we can say that FB does not really need to know the cursor position; knowing it just increases the accuracy of Pos/Crslin which would otherwise return incorrect results until Locate was used. That's not even that bad - programs wanting to use Pos/Crslin are likely based on Locate and Cls anyways.

[skystrick]

On the other hand, for *BSD systems, FB does assume a start position of 1,1 already (and does not query the cursor position in any way).

Okay, so is that a functionality you can add - either via a command-line switch or "lazy initialization" - to the Linux form of FB?

[dkl]

Ok, I've adjusted the FB runtime to do the lazy initialization thing for this (so the VT100-specific escape sequences won't be used until the first use of an FB console I/O command), and I've added a global variable which can be used to disable the VT100-specific escape sequences completely (perhaps then it's possible to use the FB console I/O commands with your Wyse terminal, albeit potentially with limited functionality).

Unix rtlib: Delay querying terminal size & cursor position until needed
Unix rtlib: Provide __fb_enable_vt100_escapes global variable

[skystrick]

Okay, thank you for making this work for me! :-)

How do I use that take that hinit.c code and apply it to the copy of FreeBASIC on my computer? Or will it be included in the next release?

[dkl]

Yes, it should be in the next release which hopefully can be made in ~3 months. Besides waiting for that you can also compile FB from sources (it's fairly easy on Linux) or use one of the daily builds provided by St_W.

[skystrick]

... you can also ... use one of the daily builds provided by St_W.

Okay, excellent. Thank you so much, dkl! This is true community support!

[skystrick]

Okay, so I downloaded and installed the new build that incorporates the changes you made.

I added the line "__fb_enable_vt100_escapes = 0" at the beginning of my program, and compiled it using the regular "fbc -lang qb foo.bas" command line.

The program is still sending across the VT100 escape sequence Esc [6n (the one for cursor location request). In fact, it's sending it twice in a row now!

As previously discussed, it used to send this when I launched a program:

Esc [6n - this is the VT100 command for cursor-location request

If I manually answered it by typing in Esc [ row;column R, the program would then continue with:

Esc ( - this is the proper Wyse 60 command for "write-protect mode off"
Esc H - this is the first of two Wyse 60 commands for "regular (non-line-drawing) character set"
Crtl-C - this is the second of two Wyse 60 commands for "regular (non-line-drawing) character set"
Esc G0 - this is the proper Wyse 60 command for "regular video attributes" (no reverse video, no underline, no blinking, etc.)
Esc cD - this is the proper Wyse 60 command for "select U.S. national character set"
Esc `1 - this is the proper Wyse 60 command for "make cursor visible"
Esc [6n - but now we're back to a VT100 command for "what is the cursor's current location (row, column)?"


Now, when I launch a program, it sends this:

Esc [6n - this is the VT100 command for cursor-location request

If I manually answer it by typing in Esc [ row;column R, the program then continues by sending the exact same thing again:

Esc [6n - this is the VT100 command for cursor-location request

If I manually answer it a second time, it continues with:

Esc ( - this is the proper Wyse 60 command for "write-protect mode off"
Esc H - this is the first of two Wyse 60 commands for "regular (non-line-drawing) character set"
Crtl-C - this is the second of two Wyse 60 commands for "regular (non-line-drawing) character set"
Esc G0 - this is the proper Wyse 60 command for "regular video attributes" (no reverse video, no underline, no blinking, etc.)
Esc cD - this is the proper Wyse 60 command for "select U.S. national character set"
Esc `1 - this is the proper Wyse 60 command for "make cursor visible"


Notice that it's still asking the Esc[6n thing twice, but instead of doing it once at the beginning and once at the end, it's just doing it twice in a row at the beginning.

The behavior is exactly the same regardless of whether I start my program with "__fb_enable_vt100_escapes = 0", or with "__fb_enable_vt100_escapes = 1", or with nothing at all.

Just to be sure there wasn't a problem with my code, I tried the following:

Code: Select all

__fb_enable_vt100_escapes = 0

OPEN "CONS:" FOR OUTPUT AS #1
PRINT #1, "Hello, world!"
SYSTEM

But unfortunately, same problem. So evidently "lazy initialization" is not working, since I didn't use any of FB's console commands?

I'm sorry, I don't mean to keep bothering you, but I'm sure this is definitely not what you intended for FBC to do! :-)

Also, I have one other question, because I'm a little confused - if we have lazy initialization now, what's the point of having the "__fb_enable_vt100_escapes = 0" boolean? Do I even need to be including it in my program?

Oh by the way: just for the heck of it, I tried compiling with -lang fb instead of -lang qb. The compiler returned the following error message when I did that:

error 41: Variable not declared, __fb_enable_vt100_escapes in '__fb_enable_vt100_escapes = 1'

[dkl]

Here's an example of how __fb_enable_vt100_escapes has to be used:
https://github.com/freebasic/fbc/blob/m ... scapes.bas

Adapting it for #lang "qb", I think it would have to be:

Code: Select all

#lang "qb"

__extern __fb_enable_vt100_escapes alias "__fb_enable_vt100_escapes" as long
dim shared __fb_enable_vt100_escapes as long

__fb_enable_vt100_escapes = 0

The point of __fb_enable_vt100_escapes is that it might just allow the console I/O commands to be used on such older terminals, even if under certain assumptions such as starting out at cursor position 1,1 (same as what the FB runtime already did on *BSD systems so it can't be that bad).

With your code example, the Print# command is what's causing the terminal size to be queried. Apparently Print# is also setting up the screen for plain Print, which seems to me like another bug. And there is indeed a redundant terminal size query in the code there, another bug. I guess it wasn't visible before because it would hang up at initialization already, thus never reach the Print# statement. In other words, the lazy initialization is revealing more issues. We're getting there...