Dynamic link libraries in DOS

DOS specific questions.
monochromator
Posts: 42
Joined: Mar 05, 2013 5:37

Dynamic link libraries in DOS

Post by monochromator »

FreeBASIC doesn't support DLL for DOS.
But as is known, DJGPP, used as a FreeBASIC backend, supports dynamic libraries in many different
variations. It is DXE1 in DJGPP 2.03p2 and DXE3 in DJGPP 2.04. There is even such version, as
DJELF (GCC/G++ 4.0.0).
Is it possible to use these features for the organization of work with DLL in FreeBASIC?
monochromator
Posts: 42
Joined: Mar 05, 2013 5:37

Re: Dynamic link libraries in DOS

Post by monochromator »

Once again I would like to return to this subject.

I added support for dynamic link libraries in FreeBASIC for DOS.
At first glance, everything works fine, but, of course, requires further testing.
All required additional or changed by me source files of the compiler
and run-time library are attached to this post.
There are few words of the changes.
Support for dynamic libraries uses the DXE3 format and the corresponding DXE3
tools and features included with DJGPP 2.04.
First of all, a set of binary tools used by the compiler must be supplemented
by one additional program - DXE3GEN, borrowed from the DJGPP 2.04.
This program is put together with all other binary utilities in the directory BIN\DOS.

Some changes was made in the file fbc.bas from the source code of FreeBASIC compiler.
These changes provide receiving of DXE dynamic link library from compiled object files,
if there is a corresponding parameter in the command line (-dll or-dylib).
The changes consist in inserting a small block of code to the procedure hLinkFiles( )
in lines from 481 to 513.

Code: Select all

        if (fbGetOption( FB_COMPOPT_TARGET ) = FB_COMPTARGET_DOS) and _
           (fbGetOption( FB_COMPOPT_OUTTYPE ) = FB_OUTTYPE_DYNAMICLIB) then
                dim as string argsfile = hStripFilename(fbc.outname) + "temp.res"
                ldcline += " -I """ + hStripExt( fbc.outname ) + "_il.a"""
                ldcline += " -U"
                scope
                 dim as string ptr objfile = listGetHead( @fbc.objlist )
                 while( objfile )
                        ldcline += " """ + *objfile + """"
                        objfile = listGetNext( objfile )
                 wend
                end scope
                scope
                 dim as string ptr libfile = listGetHead(@fbc.libfiles)
                 if (libfile) then
                        ldcline +=  " -lc"
                 end if
                 while (libfile)
                        ldcline += " """ + *libfile + """"
                        libfile = listGetNext(libfile)
                 wend
                end scope
                if (createArgsFile(argsfile, ldcline) = FALSE) then
                        exit function
                end if
                fbcAddTemp(argsfile)
                if (fbc.verbose) then
                        print "dxe3gen options in '" & argsfile & "': ", ldcline
                end if
                ldcline = "@" + argsfile
                function = fbcRunBin( "making DXE", FBCTOOL_DXEGEN, ldcline )
                exit function
        end if
Also there is an additional "sys_dylib.c" file in the run-time library source code.
This file is located in the directory RTLIB\DOS and contains the code that implements
the functions of the standard routines DyLibLoad, DyLibFree and DyLibSymbol.

Besides, in the RTLIB\DOS catalog one more file "symb_reg.txt" is added.
This file is included in a body of "sys_dylib.c" by the "#include" directive
and contains a specially formatted list of all routines of a library libfb.a.

This is required to register procedures addresses, which will be imported
into loaded dynamic libraries and provide standard operations of the memory
control, string manipulations and provide other support during runtime.

"symb_reg.txt" isn't required to make manually, it is generated by the
special auxiliary utility MAKSYMBR on the basis of the simple list of all names
of the symbols which are available in library libfb.a.

That is, for its receiving it is necessary to do the following operations:

dxe3gen -o libfb.dxe --whole-archive -U libfb.a
dxe3gen --show-exp libfb.dxe > libfblst.txt
maksymbr -o symb_reg.txt libfblst.txt
del libfb.dxe
del libfblst.txt

Then the demanded symb_reg.txt file will be created.
In this file it is possible to add procedures from other libraries, for example,
from the graphic BASIC libraries gfxlib2.

In the project one more file is added. It is dxe.bi
It contains descriptions of several additional auxiliary procedures for work with
DXE (GetDyLibError, DyLibRegSymTab and DyLibUnregSymTab).

Besides, it is required to replace library LIBC in LIB\DOS with newer version, borrowed from the DJGPP 2.04.

Also I have to note that the FreeBASIC is compiled now only by DJGPP 2.04, but not DJGPP 2.03.

Then after full recompilation of FreeBASIC we will receive the compiler with support
dynamic libraries in DOS.

Here the simplest example of its work.

Source text of first dynamic library:

Code: Select all

'' mydll.dxe

'' in the DXE the function must be declared as CDECL and export

Function AddNumbers cdecl Alias "AddNumbers" _
  ( _
    ByVal operand1 As Integer, ByVal operand2 As Integer _
  ) As Integer Export

   AddNumbers = operand1 + operand2

End Function

Function AddStrings cdecl Alias "AddStrings" ( byval operand1 as string, byval operand2 as string ) as string export

        AddStrings = operand1 + operand2

End Function
Source text of second dynamic library:

Code: Select all

''mydll2.dxe

Function AddStrNumber cdecl Alias "AddStrNumber" _
  ( _
    ByVal operand1 As string, ByVal operand2 As Integer _
  ) As Integer Export

   AddStrNumber = VAL(operand1) + operand2

End Function

Source text of the main program:

Code: Select all

#include "dxe.bi"

'' create a function pointer, arguments must be the same
'' as in the original function

Dim AddNumbers As Function cdecl ( ByVal operand1 As Integer, ByVal operand2 As Integer ) As Integer

DIM AddStrings As Function cdecl ( byval operand1 as string, byval operand2 as string ) as string

DIM AddStrNumber As Function cdecl ( ByVal operand1 As string, ByVal operand2 As Integer ) As Integer

Dim hndl As Any Ptr, hndl2 As Any Ptr

hndl = DyLibLoad("noexit.dxe")

IF hndl = 0 THEN PRINT "Ошибка при загрузке NOEXIT "; *GetDyLibError

hndl = DyLibLoad("mydll.dxe")

IF hndl = 0 THEN PRINT "Ошибка при загрузке DXE"; *GetDyLibError: END

'' find the procedure address (case matters!)
AddNumbers = DyLibSymbol( hndl, "_AddNumbers" )

IF AddNumbers = 0 THEN PRINT "Ошибка при поиске функции AddNumbers"; *GetDyLibError: END

AddStrings = DyLibSymbol( hndl, "_AddStrings" )

IF AddStrings = 0 THEN PRINT "Ошибка при поиске функции AddStrings"; *GetDyLibError: END

hndl2 = DyLibLoad("mydll2.dxe")

IF hndl2 = 0 THEN PRINT "Ошибка при загрузке DXE 2"; *GetDyLibError: END

AddStrNumber = DyLibSymbol( hndl2, "_AddStrNumber" )

IF AddStrNumber = 0 THEN PRINT "Ошибка при поиске функции AddStrNumber "; *GetDyLibError: END

'' then call it..
Print "1 + 2 ="; AddNumbers( 1, 2 )

Print "ququ + mumu = "; AddStrings("ququ", "mumu")

Print "67 + 58 = "; AddStrNumber("67", 58)

DyLibFree hndl

DyLibFree hndl2

Sleep

Operations for compile project:

fbc -g -dll MYDLL.BAS
fbc -g -dll MYDLL2.BAS
fbc -g USEDLL.BAS

Result of its execution:

C:\FREBASIC\examples\export>USEDLL.EXE
Ошибка при загрузке NOEXIT No such file or directory (ENOENT)
1 + 2 = 3
ququ + mumu = ququmumu
67 + 58 = 125

Archive with all files mentioned above can be downloaded here: http://tempfile.ru/file/2822493

May I count on adding these changes into the main branch of FreeBASIC?
dkl
Site Admin
Posts: 3235
Joined: Jul 28, 2005 14:45
Location: Germany

Re: Dynamic link libraries in DOS

Post by dkl »

It seems interesting, I think it could be added to FB. Including dxe3gen should be no problem as it's part of DJGPP.

Do I understand correctly that symb_reg.txt is needed to allow a .dxe to use the fb_* functions from the libfb compiled into the .exe that is trying to use the .dxe? What about other functions such as those from libc, will they be available automatically already? Or will .exe and .dxe contain separate copies of libfb, like win32 DLLs? Are the exports needed in rtlib source code, or would it be enough to use libfb.dxe instead of libfb.a?
monochromator
Posts: 42
Joined: Mar 05, 2013 5:37

Re: Dynamic link libraries in DOS

Post by monochromator »

I apologize for the delay with the response.
Last week I was there, where for 200 kilometer around completely there is no access to the Internet.
Do I understand correctly that symb_reg.txt is needed to allow a .dxe to use the fb_* functions from the libfb compiled into the .exe that is trying to use the .dxe?
Yes, here you are absolutely right. The symb_reg.txt file actually is
a some sort of an "export table", which contains names of functions of those libraries,
which are statically linked to the executed module,
and will be imported by DXE loader into loaded dynamic libraries if these functions are necessary for them
(contained in their "unresolved tables").
Or will .exe and .dxe contain separate copies of libfb, like win32 DLLs?
No. All static libraries are available in the single copy which are joined
to the executed module (*.exe) during compilation by a linker. Main module, and also all
dynamically loaded libraries will use the same copies of static libraries, which are contained in a body of the executed module.
What about other functions such as those from libc, will they be available automatically already?
All procedures of the loaded dynamic libraries automatically become public.
For statically linked libraries (libc, libfb, gfxlib2 and others),
only those functions become public which are enumerated in the file symb_reg.txt
or are exported explicitly by dlregsym function.
Are the exports needed in rtlib source code...
Since addresses for all statically linked libraries are defined at compile-time,
the building of the symbol table should explicitly present in the source code.
In the source code of rtlib (ie in file symb_reg.txt) is resonable to place only export table of
those libraries, which are always linked to the FreeBASIC program.
There are two such libraries - libc and libfb.

Moreover, by default LIBC is linked in FreeBASIC program only partially but not full,
so the need to include it in symb_reg.txt is questionable.
But, nevertheless, I have prepared a file symb_reg.txt,
which contains the names of the functions of both these libraries.
You can download it here: http://tempfile.ru/file/2834573

For all other libraries that are linked to the program as needed (for example, gfxlib2), registration of the export tables better include in their initialization procedure or directly in the code of the
main program.
...or would it be enough to use libfb.dxe instead of libfb.a?
The files *. DXE and *. A - it's a completely different files with completely different format, just replace
them against each other is quite impossible.

Nevertheless, such an operation is feasible, but not directly, but through a special mechanism
of "import libraries" (dxe3gen knows how to build them). Such libraries are formally conventional static libraries
but in the body instead of the functions they contain only simple wrappers, that pass control to the appropriate entry point
in the *. DXE file.

For example, the command

dxe3gen -o LIBFB.DXE -I LIBFB_IL.A -U --whole-archive LIBFB.A

for static library LIBFB.A build a dynamic library LIBFB.DXE and import library LIBFB_IL.A.
This "import library" LIBFB_IL.A really can be used instead LIBFB.A.
During first access to any function from LIBFB_IL.A corresponding dynamic library LIBFB.DXE will be
automatically loaded and used.
This eliminates the need to pre-register the export table.

But there is also a shortcoming. If earlier the program contained all necessary code
in a body of one executed file (except the DPMI server, and it too could be included there), now except for the executable file the library file LIBFB.DXE will be needed also.
Without it the program will be unusable.

Besides, even if the program isn't going to use the mechanism of dynamic libraries at all
(and such programs are absolute majority), all the same it will be compelled to do it to load main runtime library LIBFB. It reduces reliability of the program and VERY STRONGLY complicates debugging.

It is also necessary to note that for LIBC it is impossible to make such thing as the DXE loader contains in her body,
therefore its functions all the same should be registered explicitly.
In my opinion, advantages of this approach in any way don't outweigh its shortcomings, therefore
I decided not to use it.
marcov
Posts: 3454
Joined: Jun 16, 2005 9:45
Location: Netherlands
Contact:

Re: Dynamic link libraries in DOS

Post by marcov »

monochromator wrote:
No. All static libraries are available in the single copy which are joined
to the executed module (*.exe) during compilation by a linker. Main module, and also all
dynamically loaded libraries will use the same copies of static libraries, which are contained in a body of the executed module.
So that probably precludes any dead code removal options (--gc-sections and friends on gcc) too. Dead code removal is hard if the code that might be using the symbols is compiled after the main program, so the main program links everything in.

With increasing libraries, this causes such programs to swell enormously, which is why one usually tries to also have the bulk of the libraries in a DLL/shared lib.
angros47
Posts: 2321
Joined: Jun 21, 2005 19:04

Re: Dynamic link libraries in DOS

Post by angros47 »

Looks interesting...

Why haven't you included a compiled FBC.EXE?
monochromator
Posts: 42
Joined: Mar 05, 2013 5:37

Re: Dynamic link libraries in DOS

Post by monochromator »

For angros47 and others, who want to participate in testing DLL mechanism.
Here is short instruction about installation process.
1. Put "dxe3gen.exe" in BIN\DOS.
2. Put "dxe.bi" into INC directory.
2. Replace "libc.a" in LIB\DOS, "libfb.a" in LIB\DOS and "fbc.exe" by files from archive.
3. Try to compile and execute test project (MYDLL.BAS, MYDLL2.BAS, USEDLL.BAS). Compilation is made by the following operations.
fbc -dll mydll.bas
fbc -dll mydll2.bas
fbc usedll.bas

Download link: http://tempfile.ru/file/2835379
angros47
Posts: 2321
Joined: Jun 21, 2005 19:04

Re: Dynamic link libraries in DOS

Post by angros47 »

I can't get it to work:

1) It told me to set DXE_LD_LIBRARY_PATH variable: I set it to lib
2) It complained it cannot find LD, I put bin\dos in the path
3) It complained about the file dxe.ld, I added this one:
http://www.filewatcher.com/p/djgpp-crx- ... xe.ld.html
4) Now, it tell me "linker output file has more than one section"

I'm using dosbox, under linux. Hints?
monochromator
Posts: 42
Joined: Mar 05, 2013 5:37

Re: Dynamic link libraries in DOS

Post by monochromator »

I altered a few DXE3GEN specifically for FreeBASIC.
Try this new version.

Instruction on installation:

1 . Download archive at the link: http://tempfile.ru/file/2865664
2 . Replace the compiler FBC.EXE with the file from archive.
3 . Add DXE3GEN.EXE from archive to the BIN\DOS catalog.
4 . Add DXE.LD from archive to the LIB\DOS catalog.
5 . Replace LIBFB.A and LIBC.A from LIB\DOS with corresponding files from archive.
6 . Try to compile and start the test project as it is described above.
angros47
Posts: 2321
Joined: Jun 21, 2005 19:04

Re: Dynamic link libraries in DOS

Post by angros47 »

Thank you, now it works (the first time I ran USEDLL.EXE, it returned an error on the exit, the second time it didn't)

Oh, in your new file, there was no DXE.BI (required), I used the one in your previous upload.

Nice work!
monochromator
Posts: 42
Joined: Mar 05, 2013 5:37

Re: Dynamic link libraries in DOS

Post by monochromator »

In the archive attached to this message, there are all materials, connected
with support of dynamic libraries in FreeBASIC for DOS.
There some errors noticed at testing are corrected, and also is added
the additional test example showing some expanded possibilities of work
with dynamic libraries.

Content of the archive:
1 . DXE3BAS - source code for DXE3GEN adapted for use in FreeBASIC.
2 . SOURCES - source texts of some files of the compiler which were added
or changed for the organization of support of dynamic libraries.
2a - COMPILER - the changed files of source codes of the compiler.
2b - RTLIB - the changed files of source codes of runtime library.
3 . MAKSYMBR - a source text of the utility of maksymbr for obtaining the symb_reg.txt file.
4 . TEST1 - the first test example of use of dynamic libraries in DOS.
5 . TEST2 - the second test example for demonstration of some expanded opportunities.
6 . The root catalog contains binary files of the compiler (fbc.exe), auxiliary
utilities (dxe3gen.exe and maksymbr.exe), runtime libraries for the FreeBASIC (libfb.a)
and C (libc.a), the header file for use of expanded opportunities (dxe.bi),
and also the script of configuration used by the dxe3gen program (dxe.ld).

Instruction on installation:

1 . Download archive at the link: http://tempfile.ru/file/2866247
2 . Replace the compiler FBC.EXE with the file from archive.
3 . Add DXE3GEN.EXE from archive in the BIN\DOS catalog.
4 . Add DXE.LD from archive in the LIB\DOS catalog.
5 . Add the DXE.BI file in the INC catalog
6 . Replace LIBFB.A and LIBC.A from LIB\DOS corresponding файлпми from archive.
7 . Try to compile and start test projects with the following commands:
TEST1 program:
fbc - dll MYDLL.BAS
fbc - dll MYDLL2.BAS
fbc USEDLL.BAS
TEST2 program:
fbc - dll restsdll.bas
fbc restest.bas

I want to appeal also to dkl, counting_pine and other responsible persons with a request
nevertheless to include described function in the main branch of the FreeBASIC.
rugxulo
Posts: 219
Joined: Jun 30, 2006 5:31
Location: Usono (aka, USA)
Contact:

Re: Dynamic link libraries in DOS

Post by rugxulo »

marcov wrote:So that probably precludes any dead code removal options (--gc-sections and friends on gcc) too.
Last I checked, this wasn't supported by BinUtils on any COFF targets, so that won't work here. MS COFF is probably the only COFF target that will get any work done in the future, due to Windows popularity (as Cygwin and MinGW are tier two). But even that isn't "top tier" for GCC / BinUtils developers. They mostly only focus on ELF and consider COFF (and others?) as heavily deprecated (and even COFF "-gcoff" debugging since GCC 4.5.x seems broken, at least for DJGPP, in lieu of newer DWARF, e.g. v4, which is heavily favored these days). Well, DJGPP isn't even on GNU's list of supported platforms, so it's a miracle it lasted this long! (Seems Juan Guerrero ported / patched most things like GDB 7.6 and BinUtils 2.23.2 for DJGPP in recent times.)
rugxulo
Posts: 219
Joined: Jun 30, 2006 5:31
Location: Usono (aka, USA)
Contact:

Re: Dynamic link libraries in DOS

Post by rugxulo »

angros47 wrote:I can't get it to work:

1) It told me to set DXE_LD_LIBRARY_PATH variable: I set it to lib
2) It complained it cannot find LD, I put bin\dos in the path
3) It complained about the file dxe.ld, I added this one:
http://www.filewatcher.com/p/djgpp-crx- ... xe.ld.html
4) Now, it tell me "linker output file has more than one section"

I'm using dosbox, under linux. Hints?
Don't use DOSBox for development. For one thing, it's dirt slow, e.g. one thing I compiled (with ancient DJGPP GCC) took 7 mins. when native would've been under 1 minute. Also, it's buggy, "only for games", hence it doesn't support a lot of stuff that a "real" PC or "real" DOS would support, e.g. truncating file names didn't work (and crashed the VM) last I tried, e.g. try opening a file named "GNUmakefile" in DOS will open "gnumakef" instead, if found, but DOSBox doesn't emulate that correctly (probably because no game needed it, and that is their mantra). Also, DOSBox only supports max. 64 MB of memory (but default 16 MB).

If you're on Linux, I know that some distros don't make it obvious (check "multiverse"), and I know it has some compatibility issues and bugs (stupid SELinux!), but DOSEMU is probably your best bet. (But I didn't try this particular topic's patches / tools, just saying advice in general.)
angros47
Posts: 2321
Joined: Jun 21, 2005 19:04

Re: Dynamic link libraries in DOS

Post by angros47 »

Sorry to bring back this old thread, but I wonder if it is a feature that should be added to the compiler. This one, and http://freebasic.net/forum/viewtopic.php?f=4&t=21274
St_W
Posts: 1618
Joined: Feb 11, 2009 14:24
Location: Austria
Contact:

Re: Dynamic link libraries in DOS

Post by St_W »

angros47 wrote:Sorry to bring back this old thread, but I wonder if it is a feature that should be added to the compiler. This one, and http://freebasic.net/forum/viewtopic.php?f=4&t=21274
All links to both of them are down and thus there's probably nothing to add. Or do you have a copy?
Post Reply