Portability carrying shared libraries (and perhaps AppImage)

External libraries (GTK, GSL, SDL, Allegro, OpenGL, etc) questions.
Post Reply
xlucas
Posts: 334
Joined: May 09, 2014 21:19
Location: Argentina

Portability carrying shared libraries (and perhaps AppImage)

Post by xlucas »

Hi! Whenever I can, I always like writing everything myself. If not, then at least link everything statically, so no dependencies. If this would not be possible (as I see the many libraries make it really hard), then I would like to make sure that whoever has a copy of my program can run it right away without having to download 3rd party libraries. So I have these questions:

- How can I have my program read the shared object files from the same directory, so that I can distribute it with whatever libraries I use? I see there's DyLibLoad and I've tried it turning a module into a .so file some time ago, but can I use this with, say, SDL or ogg/vorbis? In other words, dynamically loading shared objects and dynamically linking my program against a library are clearly different things and I don't know if I can do one whenever I can do the other. I get really confused about this. If you guys can give me examples or hints, I'll appreciate it
- Secondly and more specifically, I'm very interested in AppImage, but when I've tried to make an AppImage out of my programs, I get really lost. It seems to be super complicated and I'm sure it must be simpler than I think. Again, I reckon if I could see one example or some hints, I could get to do that. If any of you guys have done this, can you point me in the right direction? And if you think AppImage isn't a good idea for some reason, I would like to know that too.

Thanks!!!
D.J.Peters
Posts: 8586
Joined: May 28, 2005 3:28
Contact:

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by D.J.Peters »

With DyLibLoad () you can load a shared "*.so" or "*.dll" library from any folder !

Joshy

%HOME/libMyLib.so
%HOME/TestMyLib.bas"

TestMyLib.bas

Code: Select all

 chdir exepath() ' be sure you used the right path !
var hLib = DyLibLoad ("MyLib") 
if hLib=0 then
  print "error: can't load 'MyLib' !"
  beep: sleep : end 1
end if
' ... MyStuff1 = DyLibSymbol ( hLib,"the_exported_name" )  
' ... MyStuff2 = DyLibSymbol ( hLib,"the_exported_name2" )  
xlucas
Posts: 334
Joined: May 09, 2014 21:19
Location: Argentina

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by xlucas »

D.J.Peters wrote:With DyLibLoad () you can load a shared "*.so" or "*.dll" library from any folder !
Thank you for your answer! Actually, I already know this. Sorry, I wrote a long message and maybe my question wasn't clear because of that. What I would like to know is if loading a ".so" library with DyLibLoad can replace dynamically linking to.

In other words, say I want to use the SDL library, which programs usually link to dynamically. Can I instead load the library using DyLibLoad and not link it at all, so that I can put the ".so" files wherever I want and avoid the natural mechanism the OS uses to locate the library in the system when trying to run a program that dynamically links to it? Or are these two completely different things and not interchangeable?

And then, can I construct an AppImage from a program I make that using DyLibLoad or does this only apply to programs that dynamically link to libraries?

Cheers!
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by caseih »

No you can't replace dynamic linking with DyLibLoad calls, at least not without code changes. The reason is the DyLibLoad returns pointers to the functions which you can then call by pointer. Whereas symbols that are provided by dynamically-linked libraries are called directly in your code. So to use DyLibLoad you'd have to replace every call to your SDL functions with calling a dereferenced function pointer. Hope that makes sense. You'd have to change every call to an SDL function to calling via function pointer.

In Linux, most applications that include their own copies of shared libraries place them in a directory somewhere and then use a starter wrapper script to set up LD_LIBRARY_PATH and then exec the real binary. For example, firefox when installed from the official binary tarball places all its files in, say, /opt/firefox, and puts a script in /usr/bin/firefox that sets up the library paths before execing /opt/firefox/firefox-bin. This method is somewhat similar to how you'd distribute an application on Windows, except that Windows by default will load dlls from the same directory as the binary, which isn't the default on Linux.

I'm not familiar with making AppImages, but I imagine it involves making a directory tree somewhere with its own bin lib etc share directories, and then the AppImage mechanism makes sure the paths are set up properly before execing the actual binary. To answer your question, you can DyLibLoad inside an AppImage bundle. Or you can just directly link to an included shared library. The latter is easier since the linker takes care of the actual loading.

The problem of software distribution on Linux remains open. Clearly things work best if you build a package for a distro's built-in package manager that depends on system-shipped libraries, but this isn't always feasible. Hence AppImage, Flatpak, and Snap systems. I think AppImage is probably your easiest path to a self-contained FB app on Linux that includes its own shared libraries.
D.J.Peters
Posts: 8586
Joined: May 28, 2005 3:28
Contact:

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by D.J.Peters »

The ELF loader of the OS looks in the import section for open references of the application.
If the shared lib not loaded in process space before it used the dynamic linker "ld_linux.so"
to load the needed libraries in the process space before the application is running.

The environment var "LD_LIBRARY_PATH" tells the loader where to look for libraries.

The command ldconfig can be used to add new paths permanently to LD_LIBRARY_PATH.

Some lib's comes with two sets of include files one for dynamic loading and for "normal" dynamic linking.

If you don't have for example the right header files for dynamic loading of SDL you have to write it self :-(

if you used DyLibLoad () in your app to load SDL in the process space it would be to late
be course the ELF loader comes before your app are started :-)

With other words you can use dynamic linking or dynamic loading of a lib but not both !

Joshy
xlucas
Posts: 334
Joined: May 09, 2014 21:19
Location: Argentina

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by xlucas »

Thank you, guys!

I understand... although I still have to research a lot to be able to do anything. So then, DyLibLoad is more for my own libraries, most of the time. And this LD_LIBRARY_PATH... I could make a bash script to change it, then call the compiler and then exit so that it returns to normal, but then, I figure that would change the path to every library. So, say I want SDL to be found in a directory under my program files, but I still want other libraries that I know that will be present in the system, such as XLib, to be looked for in the same place as usual. Would this mess up with that?

Then, if I use AppImage, do I not need to set up the LD_LIBRARY_PATH variable? I suppose I have to tell the AppImage maker which libraries I have and where they are. From what I've read, I too think that AppImage is a better option than FlatPak and Snap. It seems a lot more independent.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by caseih »

xlucas wrote:So, say I want SDL to be found in a directory under my program files, but I still want other libraries that I know that will be present in the system, such as XLib, to be looked for in the same place as usual. Would this mess up with that?
No. The LD_LIBRARY_PATH in effect just prepends your custom paths to the system path list. So if the library is not in your custom directory, the linker will look for it in the normal system places.
Then, if I use AppImage, do I not need to set up the LD_LIBRARY_PATH variable? I suppose I have to tell the AppImage maker which libraries I have and where they are. From what I've read, I too think that AppImage is a better option than FlatPak and Snap. It seems a lot more independent.
If your libraries are stored in standard places inside your AppImage tree, then no, you don't need to mess with LD_LIBRARY_PATH in the AppImage. I have never made an AppImage so I don't know what's involved with setting it up. But I imagine you make a standard file tree with bin, lib, etc, share, etc that bets bundled up somehow. AppImage itself must set the LD_LIBRARY_PATH automatically.
D.J.Peters
Posts: 8586
Joined: May 28, 2005 3:28
Contact:

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by D.J.Peters »

xlucas wrote:Would this mess up with that?
No you don't clear or overwrite the var LD_LIBRARY_PATH you add your path to it !
let say LD_LIBRARY_PATH is currently "/lib;usr/lib;/usr/local/lib;" etc ...
now you add your path to it $HOME/myapp/lib/
($HOME is your path I don't know XXX)

LD_LIBRARY_PATH looks than "/lib;usr/lib;/usr/local/lib;XXX/myapp/lib;"
or you add only the current folder to it

LD_LIBRARY_PATH looks than "/lib;usr/lib;/usr/local/lib;./;"


export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/your/path/lib
or
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./

You know what i mean ?

Joshy
Last edited by D.J.Peters on Jun 16, 2021 3:13, edited 1 time in total.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by caseih »

Typically LD_LIBRARY_PATH is not preset on any Linux system I'm familiar with. The system uses the default list of pathes in /etc/ld.so.conf. So it's totally fine to just define it (overwrite it). That's what it's there for. If your system is using this variable and requires to preserve it, something is broken.

And you don't need to always use the export keyword either. For example:

LD_LIBRARY_PATH=/opt/mypath /opt/mypath/mybinary

works as well. The binary will see the LD_LIBRARY_PATH variable as you set it, but your current shell will not. But in a script you should use export so that all child processes started by the script will inherit that variable.
D.J.Peters
Posts: 8586
Joined: May 28, 2005 3:28
Contact:

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by D.J.Peters »

@caseih while the system is in the init process with different runlevels (after the kernel booted)
some permanent paths are "hard coded with root rights" with ldconfig in "/etc/ld.so.conf" !

But if you run for example different desktop managers before a startx command are manually or by the system scripts executed
LD_LIBRARY_PATH are manually extended or created by the desktop manager and some other system tools.

Or if you plugin some hardware like professional high resolution scanners the kernel module loader manipulates LD_LIBRARY_PATH also.

In general some USB rules hard or self coded used LD_LIBRARY_PATH also.

If you used girly Linux like Ubuntu or Kbuntu I'm sure there is only one desktop manager and no user used startx or USB rules manually.
In this case it can be save overwriting the system var LD_LIBRARY_PATH but don't tell any BASIC user it's save to do it on his Linux distro also!

Joshy
xlucas
Posts: 334
Joined: May 09, 2014 21:19
Location: Argentina

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by xlucas »

Thank you, guys. Yes, I don't think it'd be too damaging if I just overwrite the variable in this case because I wouldn't be exporting it and it would only be to compile my program, but still, I would prefer to take the precaution to see if it contains something and append instead. That sounds safer. Still, from this, it seems to me that it'd be even more elegant if I could just DyLibLoad the .so file, but if there's not declaration file that allows it, then well, I'll have to resort to this environment variable.

So... this variable change, I would also need to do to compile the program and dynamically link to the library, right? Not to run it later. Am I correct? This makes me wonder... say I append to the variable the directory "./", which is relative... is this going to expand into an absolute path at link time or is it going to be stored as "./" in the binary? In other words, will my binary then always look for the library at the directory it is being run from or will it try to find the original directory? I think the latter would be too silly, so surely is the former, but please confirm.

Also, I assume relative paths refer to the current directory at the time of linking and not to the path where the binary is being written, which doesn't have to coincide. I have a serious confusion regarding current directories, but I'll ask that in another thread.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by caseih »

D.J.Peters wrote:In this case it can be save overwriting the system var LD_LIBRARY_PATH but don't tell any BASIC user it's save to do it on his Linux distro also!

Joshy
Sorry Joshy, but you're incorrect. Despite everything you said, and regardless of where LD_LIBRARY_PATH might be used in a system, it is safe to overwrite LD_LIBRARY_PATH in any script, or in any terminal shell. It really is. Changing this variable has no effect on any USB rules or anything else. Nor will it affect the system in any way. This is because of the way the environment variables work and how they are inherited. It's actually safe to overwrite *any* environment variable, even PATH if you desired. LD_LIBRARY_PATH only has effect within the local environment. In other words if you set it in a script, it only effects the child processes of that script, and no effect anywhere else, regardless of whether LD_LIBRARY_PATH is used elsewhere. This environment variable is for the express purpose of a shell or script to influence the linker. It won't hurt the USB rules or anything else.

I do maintain that it is highly unusual for any Linux distro to permanently set LD_LIBRARY_PATH to something. I've never seen any distro that sets it in /etc/profile, or /etc/bashrc. What distro are you using that when you open a shell and run "env | grep LD_LIBRARY_PATH" do you have something already set? I can think of no reason to ever do that when it's trivial for a package to drop a file in /etc/ld.so.conf.d.
Last edited by caseih on Jun 21, 2021 13:52, edited 1 time in total.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by caseih »

xlucas wrote:So... this variable change, I would also need to do to compile the program and dynamically link to the library, right? Not to run it later. Am I correct?
No, you can pass search paths directly to the compiler with the -p option. LD_LIBRARY_PATH is only needed at run time.
Also, I assume relative paths refer to the current directory at the time of linking and not to the path where the binary is being written, which doesn't have to coincide. I have a serious confusion regarding current directories, but I'll ask that in another thread.
LD_LIBRARY_PATH must use full paths, not relative. EDIT: possibly the linker would handle relative paths to the current working directory. But I always use full paths.
xlucas
Posts: 334
Joined: May 09, 2014 21:19
Location: Argentina

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by xlucas »

caseih wrote:No, you can pass search paths directly to the compiler with the -p option. LD_LIBRARY_PATH is only needed at run time.
Oh, but then, wouldn't it be a better idea for me to use the -p option to compile my programs so that later on, they don't have to be run from within a script? And can I use relative paths with the -p option?
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: Portability carrying shared libraries (and perhaps AppImage)

Post by caseih »

I think -p only tells the compiler where the dlls are at compile time. AT run time the linker uses the normal search path mechanisms: first look at LD_LIBRARY_PATH, second look at paths provided by /etc/ld.so.conf.

You can always find out where the linker will look at runtime by using the following command:

ldd /path/to/exe

For example:

Code: Select all

$ ldd /usr/bin/pluma                                  
	linux-vdso.so.1 (0x00007fff4379a000)
	libICE.so.6 => /lib64/libICE.so.6 (0x00007fb4260e6000)
	libgtksourceview-3.0.so.1 => /lib64/libgtksourceview-3.0.so.1 (0x00007fb426037000)
	libpeas-gtk-1.0.so.0 => /lib64/libpeas-gtk-1.0.so.0 (0x00007fb426026000)
	libgtk-3.so.0 => /lib64/libgtk-3.so.0 (0x00007fb42585a000)
	libgdk-3.so.0 => /lib64/libgdk-3.so.0 (0x00007fb42574f000)
	libpango-1.0.so.0 => /lib64/libpango-1.0.so.0 (0x00007fb425700000)
	libatk-1.0.so.0 => /lib64/libatk-1.0.so.0 (0x00007fb4256d4000)
	libcairo.so.2 => /lib64/libcairo.so.2 (0x00007fb4255ac000)
	libgdk_pixbuf-2.0.so.0 => /lib64/libgdk_pixbuf-2.0.so.0 (0x00007fb425581000)
	libpeas-1.0.so.0 => /lib64/libpeas-1.0.so.0 (0x00007fb425567000)
	libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x00007fb425383000)
	libX11.so.6 => /lib64/libX11.so.6 (0x00007fb42523c000)
	libSM.so.6 => /lib64/libSM.so.6 (0x00007fb42522f000)
	libgirepository-1.0.so.1 => /lib64/libgirepository-1.0.so.1 (0x00007fb4251f5000)
	libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007fb42519c000)
	libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007fb425071000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb42504f000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fb424e85000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fb424d3e000)
	libpangocairo-1.0.so.0 => /lib64/libpangocairo-1.0.so.0 (0x00007fb424d2c000)
	libharfbuzz.so.0 => /lib64/libharfbuzz.so.0 (0x00007fb424c4a000)
	libcairo-gobject.so.2 => /lib64/libcairo-gobject.so.2 (0x00007fb424c3e000)
	libxml2.so.2 => /lib64/libxml2.so.2 (0x00007fb424acd000)
	libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007fb424ac7000)
	libXi.so.6 => /lib64/libXi.so.6 (0x00007fb424ab3000)
	libXfixes.so.3 => /lib64/libXfixes.so.3 (0x00007fb424aaa000)
	libatk-bridge-2.0.so.0 => /lib64/libatk-bridge-2.0.so.0 (0x00007fb424a71000)
	libwayland-client.so.0 => /lib64/libwayland-client.so.0 (0x00007fb424a60000)
	libepoxy.so.0 => /lib64/libepoxy.so.0 (0x00007fb42492d000)
	libfribidi.so.0 => /lib64/libfribidi.so.0 (0x00007fb42490e000)
	libpangoft2-1.0.so.0 => /lib64/libpangoft2-1.0.so.0 (0x00007fb4248f3000)
	libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007fb4248a8000)
	libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007fb4247e3000)
	libXinerama.so.1 => /lib64/libXinerama.so.1 (0x00007fb4247de000)
	libXrandr.so.2 => /lib64/libXrandr.so.2 (0x00007fb4247d1000)
	libXcursor.so.1 => /lib64/libXcursor.so.1 (0x00007fb4247c4000)
	libXcomposite.so.1 => /lib64/libXcomposite.so.1 (0x00007fb4247bd000)
	libXdamage.so.1 => /lib64/libXdamage.so.1 (0x00007fb4247b8000)
	libxkbcommon.so.0 => /lib64/libxkbcommon.so.0 (0x00007fb424775000)
	libwayland-cursor.so.0 => /lib64/libwayland-cursor.so.0 (0x00007fb42476a000)
	libwayland-egl.so.1 => /lib64/libwayland-egl.so.1 (0x00007fb424765000)
	libXext.so.6 => /lib64/libXext.so.6 (0x00007fb424750000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fb424743000)
	libthai.so.0 => /lib64/libthai.so.0 (0x00007fb424737000)
	libpixman-1.so.0 => /lib64/libpixman-1.so.0 (0x00007fb42468b000)
	libpng16.so.16 => /lib64/libpng16.so.16 (0x00007fb424654000)
	libxcb-shm.so.0 => /lib64/libxcb-shm.so.0 (0x00007fb42464f000)
	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007fb424621000)
	libxcb-render.so.0 => /lib64/libxcb-render.so.0 (0x00007fb424611000)
	libXrender.so.1 => /lib64/libXrender.so.1 (0x00007fb424604000)
	libz.so.1 => /lib64/libz.so.1 (0x00007fb4245ea000)
	libmount.so.1 => /lib64/libmount.so.1 (0x00007fb42458a000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fb42455d000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fb424541000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fb42453a000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007fb424531000)
	libffi.so.6 => /lib64/libffi.so.6 (0x00007fb424526000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fb4244ad000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fb4261fa000)
	libgraphite2.so.3 => /lib64/libgraphite2.so.3 (0x00007fb424487000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fb42445d000)
	libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x00007fb424406000)
	libatspi.so.0 => /lib64/libatspi.so.0 (0x00007fb4243cd000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007fb42439e000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007fb424389000)
	libdatrie.so.1 => /lib64/libdatrie.so.1 (0x00007fb42437f000)
	libXau.so.6 => /lib64/libXau.so.6 (0x00007fb424379000)
	libblkid.so.1 => /lib64/libblkid.so.1 (0x00007fb424326000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007fb42428d000)
	libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007fb4241d4000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007fb4241b3000)
	libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007fb424092000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb424077000)
	libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007fb424053000)
Post Reply