Using dylibsymbol instead of import libraries (.dll.a)

[AGS]

Every now and then I find an interesting dll and build an import library/convert the C header files in order to use the dll. The import library tends to break with newer releases of a dll.

Converting the header files is something that must be done. But you can actually do without the import library (no need for pexports/dlltool). An extension of the header file is all that is needed.

You can use dylibload/dylibsymbol to get access to the dll. To be able to access all the functions inside a dll you need to dimension a lot of variables (needed for dylibsymbol).

As an example the code for the header file of the zlib library (Zlib can be downloaded from this page, if you have FreeBASIC you should have zlib.bi). I've put the dimensioning of variables inside a namespace.

The header file for zlib now becomes:

Code: Select all

Namespace zlib

#ifndef __zlib_bi__
#define __zlib_bi__


'' zconf.bi
#define MAX_MEM_LEVEL 9
#define MAX_WBITS 15

type uInt as uinteger
type Bytef as Byte
type charf as byte
type intf as integer
type uIntf as uInt
type uLongf as uLong
type voidpc as any ptr
type voidpf as any ptr
type voidp as any ptr

#ifndef SEEK_SET
#define SEEK_SET 0
#define SEEK_CUR 1
#define SEEK_END 2
#endif


#define ZLIB_VERSION "1.2.2"
#define ZLIB_VERNUM &h1220

type alloc_func as function cdecl(byval as voidpf, byval as uInt, byval as uInt) as voidpf
type free_func as sub cdecl(byval as voidpf, byval as voidpf)

type internal_state
	dummy as integer
end type

type z_stream_s
	next_in as Bytef ptr
	avail_in as uInt
	total_in as uLong
	next_out as Bytef ptr
	avail_out as uInt
	total_out as uLong
	msg as byte ptr
	state as internal_state ptr
	zalloc as alloc_func
	zfree as free_func
	opaque_ as voidpf
	data_type as integer
	adler as uLong
	reserved as uLong
end type

type z_stream as z_stream_s
type z_streamp as z_stream ptr

#define Z_NO_FLUSH 0
#define Z_PARTIAL_FLUSH 1
#define Z_SYNC_FLUSH 2
#define Z_FULL_FLUSH 3
#define Z_FINISH 4
#define Z_BLOCK 5
#define Z_OK 0
#define Z_STREAM_END 1
#define Z_NEED_DICT 2
#define Z_ERRNO (-1)
#define Z_STREAM_ERROR (-2)
#define Z_DATA_ERROR (-3)
#define Z_MEM_ERROR (-4)
#define Z_BUF_ERROR (-5)
#define Z_VERSION_ERROR (-6)
#define Z_NO_COMPRESSION 0
#define Z_BEST_SPEED 1
#define Z_BEST_COMPRESSION 9
#define Z_DEFAULT_COMPRESSION (-1)
#define Z_FILTERED 1
#define Z_HUFFMAN_ONLY 2
#define Z_RLE 3
#define Z_DEFAULT_STRATEGY 0
#define Z_BINARY 0
#define Z_ASCII 1
#define Z_UNKNOWN 2
#define Z_DEFLATED 8
#define Z_NULL 0

type in_func as function cdecl(byval as any ptr, byval as ubyte ptr ptr) as uinteger
type out_func as function cdecl(byval as any ptr, byval as ubyte ptr, byval as uinteger) as integer

type gzFile as voidp

Dim zlibVersion As Function cdecl () as zstring ptr
Dim deflate As Function cdecl (byval strm as z_streamp, byval flush as integer) as integer
Dim deflateEnd As Function cdecl (byval strm as z_streamp) as integer
Dim inflate As Function cdecl (byval strm as z_streamp, byval flush as integer) as integer
Dim inflateEnd As Function cdecl (byval strm as z_streamp) as integer
Dim deflateSetDictionary As Function cdecl (byval strm as z_streamp, byval dictionary as Bytef ptr, byval dictLength as uInt) as integer
Dim deflateCopy As Function cdecl (byval dest as z_streamp, byval source as z_streamp) as integer
Dim deflateReset As Function cdecl (byval strm as z_streamp) as integer
Dim deflateParams As Function cdecl (byval strm as z_streamp, byval level as integer, byval strategy as integer) as integer
Dim deflateBound As Function cdecl (byval strm as z_streamp, byval sourceLen as uLong) as uLong
Dim deflatePrime As Function cdecl (byval strm as z_streamp, byval bits as integer, byval value as integer) as integer
Dim inflateSetDictionary As Function cdecl (byval strm as z_streamp, byval dictionary as Bytef ptr, byval dictLength as uInt) as integer
Dim inflateSync As Function cdecl (byval strm as z_streamp) as integer
Dim inflateCopy As Function cdecl (byval dest as z_streamp, byval source as z_streamp) as integer
Dim inflateReset As Function cdecl (byval strm as z_streamp) as integer
Dim inflateBack As Function cdecl (byval strm as z_stream ptr, byval in as in_func, byval in_desc as any ptr, byval out as out_func, byval out_desc as any ptr) as integer
Dim inflateBackEnd As Function cdecl (byval strm as z_stream ptr) as integer
Dim zlibCompileFlags As Function cdecl () as uLong
Dim compress As Function cdecl (byval dest as Bytef ptr, byval destLen as uLongf ptr, byval source as Bytef ptr, byval sourceLen as uLong) as integer
Dim compress2 As Function cdecl (byval dest as Bytef ptr, byval destLen as uLongf ptr, byval source as Bytef ptr, byval sourceLen as uLong, byval level as integer) as integer
Dim compressBound As Function cdecl (byval sourceLen as uLong) as uLong
Dim uncompress As Function cdecl (byval dest as Bytef ptr, byval destLen as uLongf ptr, byval source as Bytef ptr, byval sourceLen as uLong) as integer
Dim gzopen As Function cdecl (byval path as zstring ptr, byval mode as zstring ptr) as gzFile
Dim gzdopen As Function cdecl (byval fd as integer, byval mode as zstring ptr) as gzFile
Dim gzsetparams As Function cdecl (byval file as gzFile, byval level as integer, byval strategy as integer) as integer
Dim gzread As Function cdecl (byval file as gzFile, byval buf as voidp, byval len as uinteger) as integer
Dim gzwrite As Function cdecl (byval file as gzFile, byval buf as voidpc, byval len as uinteger) as integer
Dim gzprintf As Function cdecl (byval file as gzFile, byval format as zstring ptr, ...) as integer
Dim gzputs As Function cdecl (byval file as gzFile, byval s as zstring ptr) as integer
Dim gzgets As Function cdecl (byval file as gzFile, byval buf as zstring ptr, byval len as integer) as zstring ptr
Dim gzputc As Function cdecl (byval file as gzFile, byval c as integer) as integer
Dim gzgetc As Function cdecl (byval file as gzFile) as integer
Dim gzungetc As Function cdecl (byval c as integer, byval file as gzFile) as integer
Dim gzflush As Function cdecl (byval file as gzFile, byval flush as integer) as integer
Dim gzseek As Function cdecl (byval file as gzFile, byval offset as integer, byval whence as integer) as integer
Dim gzrewind As Function cdecl (byval file as gzFile) as integer
Dim gztell As Function cdecl (byval file as gzFile) as integer
Dim gzeof As Function cdecl (byval file as gzFile) as integer
Dim gzclose As Function cdecl (byval file as gzFile) as integer
Dim gzerror As Function cdecl (byval file as gzFile, byval errnum as integer ptr) as zstring ptr
Dim gzclearerr As Sub cdecl (byval file as gzFile)
Dim adler32 As Function cdecl (byval adler as uLong, byval buf as Bytef ptr, byval len as uInt) as uLong
Dim crc32 As Function cdecl (byval crc as uLong, byval buf as Bytef ptr, byval len as uInt) as uLong
Dim deflateInit_ As Function cdecl (byval strm as z_streamp, byval level as integer, byval version as zstring ptr, byval stream_size as integer) as integer
Dim inflateInit_ As Function cdecl (byval strm as z_streamp, byval version as zstring ptr, byval stream_size as integer) as integer
Dim deflateInit2_ As Function cdecl (byval strm as z_streamp, byval level as integer, byval method as integer, byval windowBits as integer, byval memLevel as integer, byval strategy as integer, byval version as zstring ptr, byval stream_size as integer) as integer
Dim inflateInit2_ As Function cdecl (byval strm as z_streamp, byval windowBits as integer, byval version as zstring ptr, byval stream_size as integer) as integer
Dim inflateBackInit_ As Function cdecl (byval strm as z_stream ptr, byval windowBits as integer, byval window as ubyte ptr, byval version as zstring ptr, byval stream_size as integer) as integer
Dim zError As Function cdecl (byval as integer) as zstring ptr
Dim inflateSyncPoint As Function cdecl (byval z as z_streamp) as integer
Dim get_crc_table As Function cdecl () as uLongf ptr

Dim zlib_handle As Any Ptr

End Namespace

The zlib_handle is going to be used in the next part of the header file. Which looks like this (the previous code and the following code should be put in one file, code below is outside the namespace):

Code: Select all

zlib.zlib_handle = DyLibLoad("zlib.dll")
zlib.zlibVersion = DyLibSymbol(zlib.zlib_handle,"zlibVersion")
zlib.deflate = DyLibSymbol(zlib.zlib_handle,"deflate")
zlib.deflateEnd = DyLibSymbol(zlib.zlib_handle,"deflateEnd")
zlib.inflate = DyLibSymbol(zlib.zlib_handle,"inflate")
zlib.inflateEnd = DyLibSymbol(zlib.zlib_handle,"inflateEnd")
zlib.deflateSetDictionary = DyLibSymbol(zlib.zlib_handle,"deflateSetDictionary")
zlib.deflateCopy = DyLibSymbol(zlib.zlib_handle,"deflateCopy")
zlib.deflateReset = DyLibSymbol(zlib.zlib_handle,"deflateReset")
zlib.deflateParams = DyLibSymbol(zlib.zlib_handle,"deflateParams")
zlib.deflateBound = DyLibSymbol(zlib.zlib_handle,"deflateBound")
zlib.deflatePrime = DyLibSymbol(zlib.zlib_handle,"deflatePrime")
zlib.inflateSetDictionary = DyLibSymbol(zlib.zlib_handle,"inflateSetDictionary")
zlib.inflateSync = DyLibSymbol(zlib.zlib_handle,"inflateSync")
zlib.inflateCopy = DyLibSymbol(zlib.zlib_handle,"inflateCopy")
zlib.inflateReset = DyLibSymbol(zlib.zlib_handle,"inflateReset")
zlib.inflateBack = DyLibSymbol(zlib.zlib_handle,"inflateBack")
zlib.inflateBackEnd = DyLibSymbol(zlib.zlib_handle,"inflateBackEnd")
zlib.zlibCompileFlags = DyLibSymbol(zlib.zlib_handle,"zlibCompileFlags")
zlib.compress = DyLibSymbol(zlib.zlib_handle,"compress")
zlib.compress2 = DyLibSymbol(zlib.zlib_handle,"compress2")
zlib.compressBound = DyLibSymbol(zlib.zlib_handle,"compressBound")
zlib.uncompress = DyLibSymbol(zlib.zlib_handle,"uncompress")
zlib.gzopen = DyLibSymbol(zlib.zlib_handle,"gzopen")
zlib.gzdopen = DyLibSymbol(zlib.zlib_handle,"gzdopen")
zlib.gzsetparams = DyLibSymbol(zlib.zlib_handle,"gzsetparams")
zlib.gzread = DyLibSymbol(zlib.zlib_handle,"gzread")
zlib.gzwrite = DyLibSymbol(zlib.zlib_handle,"gzwrite")
zlib.gzprintf = DyLibSymbol(zlib.zlib_handle,"gzprintf")
zlib.gzputs = DyLibSymbol(zlib.zlib_handle,"gzputs")
zlib.gzgets = DyLibSymbol(zlib.zlib_handle,"gzgets")
zlib.gzputc = DyLibSymbol(zlib.zlib_handle,"gzputc")
zlib.gzgetc = DyLibSymbol(zlib.zlib_handle,"gzgetc")
zlib.gzungetc = DyLibSymbol(zlib.zlib_handle,"gzungetc")
zlib.gzflush = DyLibSymbol(zlib.zlib_handle,"gzflush")
zlib.gzseek = DyLibSymbol(zlib.zlib_handle,"gzseek")
zlib.gzrewind = DyLibSymbol(zlib.zlib_handle,"gzrewind")
zlib.gztell = DyLibSymbol(zlib.zlib_handle,"gztell")
zlib.gzeof = DyLibSymbol(zlib.zlib_handle,"gzeof")
zlib.gzclose = DyLibSymbol(zlib.zlib_handle,"gzclose")
zlib.gzerror = DyLibSymbol(zlib.zlib_handle,"gzerror")
zlib.gzclearerr = DyLibSymbol(zlib.zlib_handle,"gzclearerr")
zlib.adler32 = DyLibSymbol(zlib.zlib_handle,"adler32")
zlib.crc32 = DyLibSymbol(zlib.zlib_handle,"crc32")
zlib.deflateInit_ = DyLibSymbol(zlib.zlib_handle,"deflateInit_")
zlib.inflateInit_ = DyLibSymbol(zlib.zlib_handle,"inflateInit_")
zlib.deflateInit2_ = DyLibSymbol(zlib.zlib_handle,"deflateInit2_")
zlib.inflateInit2_ = DyLibSymbol(zlib.zlib_handle,"inflateInit2_")
zlib.inflateBackInit_ = DyLibSymbol(zlib.zlib_handle,"inflateBackInit_")
zlib.zError = DyLibSymbol(zlib.zlib_handle,"zError")
zlib.inflateSyncPoint = DyLibSymbol(zlib.zlib_handle,"inflateSyncPoint")
zlib.get_crc_table = DyLibSymbol(zlib.zlib_handle,"get_crc_table")


#define deflateInit(strm, level) zlib.deflateInit_((strm), (level), ZLIB_VERSION, sizeof(z_stream))
#define inflateInit(strm) zlib.inflateInit_((strm), ZLIB_VERSION, sizeof(z_stream))
#define deflateInit2(strm, level, method, windowBits, memLevel, strategy) zlib.deflateInit2_((strm),(level),(method),(windowBits),(memLevel),(strategy), ZLIB_VERSION, sizeof(z_stream))
#define inflateInit2(strm, windowBits) zlib.inflateInit2_((strm), (windowBits), ZLIB_VERSION, sizeof(z_stream))
#define inflateBackInit(strm, windowBits, window) zlib.inflateBackInit_((strm), (windowBits), (window), ZLIB_VERSION, sizeof(z_stream))

#endif

Put the above file in the /freebasic/inc/ directory and name it zlib.bi2 (you don't want to overwrite the original zlib header file) and you will not be needing an import library when using zlib.dll any more.

If there is a change in the interface of the zlib library you just install the new dll, change the header file and you're done. No pexports/dlltool needed (no import library needed).

In case you are wondering why the assignments are not inside the namespace: assignments inside a namespace at the given level of scope yield a syntax error.

Below an example usage of the above header file.

Code: Select all

#include "zlib.bi2"

Function main() As Integer
	
	Dim As zlib.gzfile test_file

	test_file = zlib.gzopen("test_file","wb9")
	If (0 = test_file) Then
		Print "Could not create file, exiting"
		Exit Function
	End If
	For i As Integer = 0 To 100
		zlib.gzputc(test_file,i)
	Next i	
	Var error_ = zlib.gzclose(test_file)
	If (error_ <> Z_OK) Then
		Print "An error occured, aborting program"
		Exit Function
	End If
	test_file = zlib.gzopen("test_file","rb9")
	If (0 = test_file) Then
		Print "Could not open file for reading, aborting program"
		Exit Function
	End If	
	For k As Integer = 0 To 100
		error_ = zlib.gzgetc(test_file)
		If (error_ = -1) Then
			Print "An error occured, aborting program"
			zlib.gzclose(test_file)
			Exit Function
		End If
		Print error_
	Next k
       'have to close library yourself
	DyLibFree(zlib.zlib_handle)
	Return 0

End Function

main()

[jcfuller]

Also with this method you don't NEED to translate ALL the header file; Just the items you need.

James

[MichaelW]

One problem that I see is that DyLibLoad and DyLibSymbol can fail, and there is no code to handle this. The major advantage of run-time dynamic linking over load-time dynamic linking is the ability to present the user with a nice, easy to understand error message in the event that the DLL or one or more of the functions are not present.

And for what it’s worth, an import library is frequently included with the DLL. For zlib, I can simply rename the included import library and subsititute it for the one included with FreeBASIC, and assuming the DLL is present, the functions can be called no problem. But I should point out that I have encountered import libraries that would not work, and I can't recall having any Microsoft import library work.

Also, even when using an import library, only the functions that are referenced in the source need to be declared.

[jcfuller]

One problem that I see is that DyLibLoad and DyLibSymbol can fail, and there is no code to handle this. The major advantage of run-time dynamic linking over load-time dynamic linking is the ability to present the user with a nice, easy to understand error message in the event that the DLL or one or more of the functions are not present.

Example code?

James

[1000101]

Code: Select all

myLib = DyLibLoad( "myLib" )
If( myLib = 0 )Then
   Print "Sorry, could not locate myLib in the path."
   End
End If

etc

[Landeel]

In my WIP game I use SDL and SDL_mixer for sound. I'm currently using dyloadlib instead of dynamic linking.

I think this approach is better, specially for Linux.

1) because I can provide a helpful error message if the lib can't be loaded, or if it can't be found.
2) because I can load libs from any directory (by default Linux programs will only look up for libs in the system directories).

X86_64 installations usually don't have the 32-bit version of libSDL_mixer.so, wich made my game fail on most x86_64 machines. With this method I can simply ship the lib with the game.

[Loe]

3. we can treat dll as addin/plugin (like fbedit addin), I mean we can use dylibload and dylibfree, to change dll at runtime. that what I did in axsuite code generator.

[AGS]

A tiny adjustment of the sourcecode: the namespace should be inside the #ifndef as in

Code: Select all

#ifndef __zlib_bi__
#define __zlib_bi__

Namespace zlib

otherwise the whole purpose of #ifndef... #define... #endif kinda gets defeated.

I do not agree with MichaelW that an import library (that is, a .dll.a MinGW kind of import library) is frequently included with the dll.

Most dlls I download have been compiled using VisualC and these do not come with usable import libraries.

[voodooattack]

Here's a trick I added to FB long time ago, but my patch got dropped for some vague reason that I don't remember any longer, my aim at the time was to make FB behave more like VB when it comes to automatic DLL detection (although done at compile time), and perhaps add DLL delay-loading later on (which does it at run-time, but needs some sort of exception mechanism to do it right).

This is all based on a usually overlooked feature of LD (the gnu linker used by FBC to link objects) , and that feature is DLL hot-linking.. LD does NOT require an import library to statically link against a DLL to begin with, if you've used Linux, you probably know that there are no "import libraries" used, when ported to windows, for the sake of compatibility, that feature was added to LD to keep things consistent.

In short, you can link to a DLL directly by passing the DLL file as a library parameter on the command-line, like this:

LD <myparameters> -( TARGETLIB.DLL -)

The -( and -) parameters respectively indicate the start and the end of a library group, to the last of my knowledge, you can't use -l with a DLL directly, and you must include it in a library group.

To avoid manual linking (which is a real pain), you can pass parameters to the linker through FBC itself, by invoking it with the -Wl parameter like this:

fbc myfile.bas <myotherargs> -Wl -(,"P:\ATH\TO\DLL\TARGETLIB.DLL",-)

I hope you find this easier than rewriting headers and rebuilding import libraries.

[srvaldez]

that's a good tip voodooattack, I was a bit puzled at the fact that you did not need an import lib in OS X.

[jcfuller]

voodooattack,
Don't you still have the issue of not being able to show a informative messge when the library is not available on the target machine?

James

[Landeel]

This is new to me. After all the work I had writing subs/functions to use SDL: http://www.freebasic.net/forum/viewtopi ... highlight=
Well, this is useful.

[voodooattack]

voodooattack,
Don't you still have the issue of not being able to show a informative messge when the library is not available on the target machine?

James

Yes, this method is more or less a transparent way to handle DLL static-linking.. but that's about it.

Although that would be really easy to do if DLL delay-loading gets implemented into the compiler/linker, from that point onwards, it would all be a matter of exceptions triggered by the linker/compiler-generated thunks when something goes wrong, those are small automatically-generated routines that the import-table points to initially, which load the DLL and patch the IAT entries to point at the target's appropriate functions when first called, but this would still require a good exception handling mechanism, which is something we lack in FB.

[jcfuller]

Is it possible to do this with dll functions that take optional parameters like the printf function from crt?

James

[voodooattack]

Is it possible to do this with dll functions that take optional parameters like the printf function from crt?

James

Yes, those are still functions as far as the compiler is concerned, vararg functions use the cdecl convention.

It's all about how the compiler sees it, when you give it a function prototype pointer then call it, it emits the correct assembly instructions for calling that function prototype, now no matter what address you assign your variable to, it still gets called the same.