Pure FB Runtime Library (in progress)

User projects written in or related to FreeBASIC.
Post Reply
adeyblue
Posts: 299
Joined: Nov 07, 2019 20:08

Re: Pure FB Runtime Library (in progress)

Post by adeyblue »

I'm sure it's been on the todo list for 10+ years
Well, now's definitely the time to break with tradition. So elbow the list of temp descriptors and:
return all FBStrings descriptors by value - or
each take another pointer to a descriptor to fill in - or
Allocate() all FBStrings descriptors?

For instance, what does this look like in the new regime?

Code: Select all

function fb_StrMid FBCALL ( src as FBSTRING ptr, start as ssize_t, _len as ssize_t ) as FBSTRING ptr
   dim as FBSTRING ptr dst
	dim as ssize_t src_len

	FB_STRLOCK()

	if ( (src <> NULL) and (src->data <> NULL) andalso (FB_STRSIZE( src ) > 0) ) then
		src_len = FB_STRSIZE( src )

		if ( (start > 0) and (start <= src_len) and (_len <> 0) ) then
			start -= 1

			if ( _len < 0 ) then
				_len = src_len
			end if

			if ( start + _len > src_len ) then
				_len = src_len - start
			end if

			/' alloc temp string '/
			dst = fb_hStrAllocTemp_NoLock( NULL, _len ) '' 
			if ( dst <> NULL ) then
				FB_MEMCPY( dst->data, src->data + start, _len )
				/' null term '/
				dst->data[_len] = 0
			else
				dst = @__fb_ctx.null_desc
			end if
		else
			dst = @__fb_ctx.null_desc
		end if
	else
		dst = @__fb_ctx.null_desc
	end if

	/' del if temp '/
	fb_hStrDelTemp_NoLock( src )

	FB_STRUNLOCK()

	return dst
end function
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: Pure FB Runtime Library (in progress)

Post by coderJeff »

adeyblue wrote:return all FBStrings descriptors by value - or
each take another pointer to a descriptor to fill in - or
Allocate() all FBStrings descriptors?
I've been down this road a few times now and I have a few ideas. I started implementation for var-len wstrings, but this should be dealt with first.
- on windows, ensure that allocate (aka malloc) is thread safe and uses the same process heap in DLL's - maybe some old versions of windows don't?
- caller is in control of descriptors - which could be allocated on the stack or on the heap (if that makes sense for the situation)
- avoid allocating descriptors on the heap (or more accurately, separately) if it isn't needed; which would allow arrays and structs containing string descriptors to be allocated fully in one allocation - except for actual string data.
- passing in an extra pointer to receive the result seems like the most straight forward approach.
- However, currently in fbc for procedures that return structs byval: if the struct is larger than 2 registers - which for string descriptors it is, then the caller will allocate a hidden variable and pass in a hidden pointer to the procedure to receive the result. Currently there's no way to expose the hidden pointer in fbc code. But if there was, we could act on the byval return result directly.
- Ideally, if the byval result is passed returned through a hidden pointer, then the pointer can also be passed as an argument to another string function and the temporary result is automatically cleaned up when the hidden variable goes out of scope. The compiler will know that it is temporary, so in the case of assignment or as another returned string, should be able to optimize by copying the descriptor only.

Maybe:

Code: Select all

function fb_StrMid FBCALL _
	( _
		byval src as const FBSTRING ptr, _
		start as ssize_t, _
		_len as ssize_t _
	) as FBSTRING '' byval return through hidden pointer

	'' not legal in current fbc - for concept only
	#define dst *function

	'' expect that dst descriptor is already initialized to zero
	assert( dst <> NULL )
	assert( dst.data = NULL )
	assert( dst.len = 0 )
	assert( dst.size = 0 )

	dim as ssize_t src_len

	if ( (src <> NULL) and (src->data <> NULL) andalso (FB_STRSIZE( src ) > 0) ) then
		src_len = FB_STRSIZE( src )

	if ( (start > 0) and (start <= src_len) and (_len <> 0) ) then
		start -= 1

	if ( _len < 0 ) then
		_len = src_len
	end if

	if ( start + _len > src_len ) then
		_len = src_len - start
	end if

	if( fb_hStrAlloc( dst, _len ) <> NULL ) then
		FB_MEMCPY( dst->data, src->data + start, _len )
		/' null term '/
		dst->data[_len] = 0
	else
		'' memory error
	end if

	'' solve out to no-operation
	return *function
end function
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: Pure FB Runtime Library (in progress)

Post by coderJeff »

The other direction is to completely implement string as a class. Which is a direction I also worked on when adding Extending Wstring and Zstring with UDTs, and fixing or changing other subtleties with string/UDT function overloading, casting, and usage. Still missing is ability within fbc syntax for declaring and using quirk words like mid statement and string functions like instr and ltrim which are expected to accept the 'any' clause for some arguments.

Along these lines:

Josep Roca posted DWSTRING.bi - Dynamic null terminated unicode string data type. And has broader string class implementations (like CWStr) in his frame work https://github.com/JoseRoca/WinFBX

Juergen Kuehlwein did a lot of work with his 'ustring' class type for a var-len wstring implementation, but the code still needed a lot of work to include with fbc.
marcov
Posts: 3455
Joined: Jun 16, 2005 9:45
Location: Netherlands
Contact:

Re: Pure FB Runtime Library (in progress)

Post by marcov »

coderJeff wrote: - on windows, ensure that allocate (aka malloc) is thread safe and uses the same process heap in DLL's - maybe some old versions of windows don't?
Don't worry too much, such old version (win9x probably), won't do unicode anyway. People will keep on wining about MSLU, but in over a decade I never saw somebody actually using it for non testing/demonstration purposes.
- However, currently in fbc for procedures that return structs byval: if the struct is larger than 2 registers
That's highly ABI dependent, and somewhat a risk. It might be smart to make it a per target decision with a fall back to using a hidden pointer for structured results. (first or last parameter, opinions vary, as usual)
Last edited by marcov on Nov 14, 2021 12:23, edited 1 time in total.
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: Pure FB Runtime Library (in progress)

Post by coderJeff »

marcov wrote:That's highly ABI dependent, and somewhat a risk.
Ah, good point. I'm getting side tracked by some details too early.

The goal is to get rid of the string locks and temporary string management out of the run time library. It makes sense to me to solve this for var-len STRINGS before creating new code for var-len WSTRINGS.

So, I think key steps would be:
- functions returning strings return results by value. That is, the string returned must be newly allocated / constructed.
- let the compiler manage the scope and lifetime of the string in a hidden variable.
- since fbc functions can return structs by value now, so use that capability as-is for now
- once the API is all working, optimize to the ABI to minimize operations (typically memory copy in case of strings).
marcov
Posts: 3455
Joined: Jun 16, 2005 9:45
Location: Netherlands
Contact:

Re: Pure FB Runtime Library (in progress)

Post by marcov »

coderJeff wrote: - functions returning strings return results by value. That is, the string returned must be newly allocated / constructed.
And copied. This leads to sequences of copies down e.g.. a OOP class hierarchy. This consideration is why we have refcounted strings :-)
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: Pure FB Runtime Library (in progress)

Post by coderJeff »

marcov wrote:And copied.
Indeed, as I was not really expecting to reference count strings.

So, in fb code, return value needs to be a copy of the string data argument:

Code: Select all

function f( byref s as string ) as string
	return s
end function

var s = "hi"
var r = f(s)
In above, 'r' must be a separate copy of 's'.
adeyblue
Posts: 299
Joined: Nov 07, 2019 20:08

Re: Pure FB Runtime Library (in progress)

Post by adeyblue »

OK, so I've been through all of Paul Squires comments and now it builds on Windows without ignoring any files, except for file_datetime.bas due to this:
/'
!!! FIXME !!! - update the crt headers
- have issues here with duplicate definitions in the crt headers
- need to update crt headers to allow these includes to work together
'/
It happens when both of crt/wchar.bi and crt/sys/stat.bi are included.
adeyblue
Posts: 299
Joined: Nov 07, 2019 20:08

Re: Pure FB Runtime Library (in progress)

Post by adeyblue »

coderJeff wrote: Along these lines:

Josep Roca posted DWSTRING.bi - Dynamic null terminated unicode string data type. And has broader string class implementations (like CWStr) in his frame work https://github.com/JoseRoca/WinFBX
So apart from it being Windows dependent, what's wrong with it, design wise? Or rather, what stopped it being adopted? Just nobody to do it?
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: Pure FB Runtime Library (in progress)

Post by coderJeff »

adeyblue wrote:So apart from it being Windows dependent, what's wrong with it, design wise? Or rather, what stopped it being adopted? Just nobody to do it?
Only myself to blame I guess.

The design is quite good and a reasonable solution, in that it works within the capabilities of fb. But it also highlights the short comings of fb. Using the class is kind of like STRING but not quite. Some syntax is not expressible in user code. But it relies on (zero terminated) WSTRING support already in the rtlib, so as long that works OK, don't need to re-implement everything.

Usually whenever I put some effort towards an implementation, I get side tracked on other things that I think should be improved first, like wstring bugs, or allowing built-in fbc stuff to be also expressible in user code. For me, I tend to bounce between trying to add features that will help with writing a really good var-len-wstring class in user code, or features that will help with implementing it directly in the compiler and rtlib only.
adeyblue
Posts: 299
Joined: Nov 07, 2019 20:08

Re: Pure FB Runtime Library (in progress)

Post by adeyblue »

coderJeff wrote: - avoid allocating descriptors on the heap (or more accurately, separately) if it isn't needed; which would allow arrays and structs containing string descriptors to be allocated fully in one allocation - except for actual string data.
- passing in an extra pointer to receive the result seems like the most straight forward approach.
This is what I'm doing now, and there are quite a few functions which return fbstring ptr. Instead of making them all subs, I'm making them return the passed in pointer.

A little bit of gfxlib will have to be rewritten to be compatible with this system though. The InKey hook it implements returns a fbstring ptr, and what it returns comes from an rtlib function.

I've also added in a little wrapper for fbstring that'll delete it in a destructor. A few functions use fbstrings themselves, and as all the 'delete temp descriptor' calls are being removed, I will definitely forget to add in the calls to delete them in at least one function, so now I don't have to!

I'm going to do a few and then wait for an OK to do to the rest. I don't really want to get to the end before somebody looks at it and tells me that's not what we wanted
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: Pure FB Runtime Library (in progress)

Post by coderJeff »

adeyblue wrote:I'm going to do a few and then wait for an OK to do to the rest. I don't really want to get to the end before somebody looks at it and tells me that's not what we wanted
This sounds like a good approach. It's been on my mind a little what the transition will look like to go from a C rtlib to an FB rtlib. Because some things that are painful in C become less so in FB. And there will be improvements / changes in the API between compiler and rtlib. Fork the compiler? Support both rtlibs through a command line switch? Those sorts of questions roll around in my head.
adeyblue
Posts: 299
Joined: Nov 07, 2019 20:08

Re: Pure FB Runtime Library (in progress)

Post by adeyblue »

The commit is here
https://github.com/adeyblue/fbrtLib/com ... e220d06c88

The tree
https://github.com/adeyblue/fbrtlib/tree/new_strings

They mostly follow a pattern.
Add a FBSTRING ptr as the last parameter (except for Chr since that's variadic, so it's added first)
Change the FBSTRING ptr local variable to a destructable_string
Then at the end, swap the descriptors between the passed in string and the destructable_string.

A few functions have been simplified since the null string descriptor is the default for destructable_string, so all the elses assigning that aren't required.
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: Pure FB Runtime Library (in progress)

Post by coderJeff »

@adeyblue, cool. I'll take a look as soon as I can. #945 Scoping rules have changed inside methods is giving me grief at the moment.

I had a quick glance at your rtlib changes. I think on the compiler side under the hood I will need to translate:

Code: Select all

declare function func( args* ) as string
=>
declare function func( args*, byval result as FBSTRING ptr ) as FBSTRING ptr
Plus some supporting bits. And except for varargs which will put the result parameter first.
And if all that is done, then I kind of think that if the rtlib function doesn't actually operate on FBSTRING structure internals, then could potentially be declared as a "normal" function returning STRING.

When I get to it, I am thinking I will try supporting both C rtlib and FB rtlib through the '-z fbrt' command line argument. So each version of the library will need to have different names in the lib/<target> directory.
adeyblue
Posts: 299
Joined: Nov 07, 2019 20:08

Re: Pure FB Runtime Library (in progress)

Post by adeyblue »

That diff is about half of them, it's missing all the core functions in the str_ files. The library won't compile as is as I've been compiling the files as I change them, so some declares have changed but not the actual function.

I am keeping a list of all the changed functions, so you won't have to crawl through the files to find what functions needs changing or how.
then could potentially be declared as a "normal" function returning STRING.
Yeah, I've fought several urges to find replace all ssize_t to Integer and ubyte ptr to zstring, even if it isn't that simple
Post Reply