Creating Static lib with function level granularity

For other topics related to the FreeBASIC project or its community.
marpon
Posts: 340
Joined: Dec 28, 2012 13:31
Location: Paris - France

Creating Static lib with function level granularity

Postby marpon » Mar 08, 2015 8:52

Hi all
I want to make a static lib for windows platform including different general purpose functions , some working toogether , some are independant,
and in order to only embbed the needed functions/objects in the target programs at link time
i would like to create the static lib with each function in 1 object module.

Is it a tool to do it in batch mode : from a .bas file with all the needed stuff
able to split that source file into multiple source modules , managing the function/sub declarations , the type declarations ...

I think, the brilliant people creating freebasic probably have something in this area ...
Please help / thanks in advance
TJF
Posts: 3481
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Re: Creating Static lib with function level granularity

Postby TJF » Mar 08, 2015 11:37

marpon wrote:I want to make a static lib for windows platform including different general purpose functions , some working toogether , some are independant,
and in order to only embbed the needed functions/objects in the target programs at link time
i would like to create the static lib with each function in 1 object module.

I don't understand your target. The linker selects the necessary object code from a static library, and embeds that only. You need not care about it. And that's totally independand from the source code.
fxm
Posts: 9126
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Creating Static lib with function level granularity

Postby fxm » Mar 08, 2015 13:51

With QB, I seem to remember that each linked library was entirely embedded in the ".exe" file.
marpon
Posts: 340
Joined: Dec 28, 2012 13:31
Location: Paris - France

Re: Creating Static lib with function level granularity

Postby marpon » Mar 08, 2015 15:04

@TJF
The linker selects the necessary object code from a static library


its true , but as you said only at object level ( .o) not at function level

that means , if you put several functions on a source code and compile to object all the functions on that object code will be linked (even if you use only 1 on the object ).
exactly what i don't want ( the exe will include not-used code , waste of mem , size on disk...)

My intention is to create as many object modules as functions and after put them on a lib file ,manually is easy when few functions , so I look for a batch solution for numerous functions
taking every function from a source file to a distinct .bas file ( only with that specific function) compile to object ( -o) taking care of the needed refs ( declaration of functions, type, shared...)
and after putting all that collection of objects into the lib.a file

as I said function level granularity...
TJF
Posts: 3481
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Re: Creating Static lib with function level granularity

Postby TJF » Mar 08, 2015 15:31

It gets more and more unclear. All that object magic and several source files in not necessary when you proceed as you started

marpon wrote:I want to make a static lib ...

Just compile all your SUBs and FUNCTION in a library and link your program against that library. The linker does not include the complete static library, it picks out the necessary stuff.

You can test this with the static fb library. It's about 900 kB in size. All executables get linked against that library, but the executables are much smaller. The more functions you use, the more stuff comes from libfb to your executable.
caseih
Posts: 1369
Joined: Feb 26, 2007 5:32

Re: Creating Static lib with function level granularity

Postby caseih » Mar 08, 2015 15:36

If you compile the static library with the appropriate GCC flags, you can then at link stage tell the linker to discard unused symbols. To do this on fbc, you'll have to figure out how to have fbc pass the appropriate flags to the gcc compiler, and the appropriate flag to the linker:

http://stackoverflow.com/questions/5307 ... g-with-gcc
dkl
Site Admin
Posts: 3209
Joined: Jul 28, 2005 14:45
Location: Germany

Re: Creating Static lib with function level granularity

Postby dkl » Mar 08, 2015 15:54

Yea, the GNU linker works at .o file level (or perhaps at section level?) when including/excluding code.

I don't know any tool to move functions into a separate .bas file each, at the source code level, but maybe it's not even hard to write that...

Although, there are gcc -ffunction-sections and ld --gc-sections options. This can be used with fbc -gen gcc:
1.bas:
sub f1()
end sub
sub f2()
end sub

2.bas:
declare sub f1()
f1()

$ fbc 1.bas -c -gen gcc -Wc -ffunction-sections

$ fbc 2.bas 1.o -Wl --gc-sections,--print-gc-sections
[...]
ld: Removing unused section '.text.F2' in file '1.o'
[...]


Looks like this only works on Linux at the moment though:
https://sourceware.org/bugzilla/show_bug.cgi?id=11539
marcov
Posts: 2762
Joined: Jun 16, 2005 9:45
Location: Eindhoven, NL
Contact:

Re: Creating Static lib with function level granularity

Postby marcov » Mar 08, 2015 17:15

dkl wrote:Yea, the GNU linker works at .o file level (or perhaps at section level?) when including/excluding code.

I don't know any tool to move functions into a separate .bas file each, at the source code level, but maybe it's not even hard to write that...


The compiler should simply generate a .s file for each symbol (whatever would have an own section). But that is horribly slow with binutils. (think 10-20 times assembling+linking time and more, worse on platforms that handle many small files inefficiently, like Windows and Dos)
Worse, you might hit certain internal limitations in binutils that you don't hit otherwise.

I think compiling the import library for the windows headers this way took 15 minutes on 500MHz machines or so, and still 2-3 minutes on Athlon XP's. And that FPC Windows header only had a subset of headers. (I think 20000 symbols or so)

Still for some very legacy targets/binary formats for which section based smart linking is not implement or unreliable, this can be a solution. But it is really a last resort. And then only in a general case and for not too big binaries and libraries.

For an one off effort, it is easier to take the RTL and make a cut down version of it though. The binaries will be smaller and it is less work. If it is done smartly, and the RTL has some general ifdef knobs to turn on and off optional functionality, such cutting down can be kept as a patch, so that the work hasn't be redone mostly for newer versions.

Note that smart linking is not just turning on section based compilation and linking. The code-to-link must be properly setup, and unused code must not be reachable.

An RTL that registers or initializes certain parts too early might add references that pull in quite some code. The same with the compiler that generates tables (RTTI etc) that are not weakly linked. (e.g. assume you can enumerate classes in a namespace. Unused classes VMTs will be linked in if the entry in the "classes per namespace" table is not weak)
marpon
Posts: 340
Joined: Dec 28, 2012 13:31
Location: Paris - France

Re: Creating Static lib with function level granularity

Postby marpon » Mar 09, 2015 10:14

@marcov
@dkl

Thanks for your replies.

Now I'm confused , dkl says it exists some options to realize the function level granularity but only for linux, I'am interrested for win

And sorry marcov , my level of understanding is not enougth to understand your answer.

I understood at least : no automatic tool.


What i was figuring , such kind of tool could be interresting to realize libraries with fine granularity.

The basic idea was to produce a "normal" lib , for the users no need of extra options to link to that lib,
but as slim result as possible.

I will continue my manual tests to evaluate the interrest and if some other coder have already experiment someting
please advise.
TJF
Posts: 3481
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Re: Creating Static lib with function level granularity

Postby TJF » Mar 09, 2015 14:00

marpon wrote:I understood at least : no automatic tool.

fb-doc is able to parse the code. You could extend it by an external module, in order to generate a common header file and separate function modules.

As far as I understand the discussion, you won't compile a static library. Instead you want to compile a minimal object file from your granular sources and bind this object file to the executable.

marcow wrote:..., such cutting down can be kept as a patch, so that the work hasn't be redone mostly for newer versions.

I'd use CMake build system in this case, and create an OBJECT library

[code=CMake file=CMakeLists.txt]# list the files to include
SET(SNIPPETS_BAS
"snippet1.bas"
"snippet2.bas"
"snippet3.bas"
"snippet4.bas"
)

# declare the object library
ADD_LIBRARY(fbsnippets OBJECT ${SNIPPETS_BAS})
# set compiler flags
SET_TARGET_PROPERTIES(fbsnippets PROPERTIES
COMPILE_FLAGS "-Wc -fPIC"
)
# add auto dependencies check
ADD_Fbc_SRC_DEPS(fbsnippets)
[/code]
Then bind this object library to the final executable

[code=CMake file=CMakeLists.txt]# declare the final executable
ADD_CUSTOM_TARGET(MyExe $<TARGET_OBJECTS:fbsnippets>)[/code]
marcov
Posts: 2762
Joined: Jun 16, 2005 9:45
Location: Eindhoven, NL
Contact:

Re: Creating Static lib with function level granularity

Postby marcov » Mar 10, 2015 12:22

TJF wrote:
marcov wrote:..., such cutting down can be kept as a patch, so that the work hasn't be redone mostly for newer versions.


I'd use CMake build system in this case, and create an OBJECT library


By editing source, I meant not compiling parts that you don't need, with some larger granularity. (things like local time support, locale dependent formatting, I don't know what FB's libraries have)

If you mean use AR to archive objects together? Please so instead of talking about a totally different tool, that only confuses the subject.

Archiving with AR is the same problem as multiple AS files. AR is slow with many small files, and only the file seek overhead is eliminated, not LD's systematic inefficiency with large numbers of object files (in an archive or not).

And even if your code the challenge of generating an object file per symbol must be overcome first.
marpon
Posts: 340
Joined: Dec 28, 2012 13:31
Location: Paris - France

Re: Creating Static lib with function level granularity

Postby marpon » Mar 10, 2015 13:22

@ TJF you probably are thinking in different topic ...

Short summary of what i've done & found.

I have a source .bi file with 80 function/subs for generic usage with activex stuff (> 45 kb)
very few of these functions/subs are used at the same time in the program
because it depends on the user usage , the only possible optimization is by conditionnal compilation with #define.
But in most of time the link will include at least 70 % of not used functions/subs.

That why my intention was to make a static lib with all of the code from that .bi file ( thinking link was able to know what function is needed and take only that : false )
no problem to make it but much more difficult to optimize with #define
as a result the size of the outpout exe file compiled with the lib,have increased around 5% , because all the code is on 1 .o file and the link seems to operate at .o level (confirmed by @dkl )
and the lib file size 40 kb

After cutting to make group of function working toogether and creation 6 .o files the output exe have been reduced about 20%, with lot of time analysing the code to group coherent code.
Better solution but not so easy and time consuming.(the lib file size 60 kb)

In my first post , I was asking for batch process to make it more efficiently and at function/sub level.

I made it manually and got 78 .o files , done a lib from them with ar (the lib inflated a lot 95ko)
but when compiling the exe file reduced 45% , it works , it takes less than 30 s to compile the .o and produce lib,
so Ok
Its now time for a tool able to split the code into mono procedure parts, isolate the declarations(func/sub) , isolate the types, isolate the shared var ( and transform them to common shared ) putting that cummon parts into .bi file to compile to .o, create a lib file with ar
Then Create a specific .bi file with the cummon stuff , loading the lib file and compile for exe.

Thats all ... after you also have to manage the different versions of freebasic /lib/inc
from my tests using Obj stuff i need at least 2 libs : one for : 0.24 up to .90 and 1 for more recent .

I hope is clear enougth for people interrested on making static libs, tanks for your attention /help
TJF
Posts: 3481
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Re: Creating Static lib with function level granularity

Postby TJF » Mar 10, 2015 17:51

It's getting more confusing!

marpon wrote:Then Create a specific .bi file with the cummon stuff , loading the lib file and compile for exe.

You should know that fbc doesn't compile bi files. Headers (bi files) are designed to contain declarations only, but no executable source code (as SUBs or FUNCTIONs).

marpon wrote:Its now time for a tool able to split the code into mono procedure parts, isolate the declarations(func/sub) , isolate the types, isolate the shared var ( and transform them to common shared )

I already told you that fb-doc is such a tool. You just need to extend it by an external module to generate the files you desire.

Sorry, there's no progress in this thread. I'm out.
caseih
Posts: 1369
Joined: Feb 26, 2007 5:32

Re: Creating Static lib with function level granularity

Postby caseih » Mar 10, 2015 18:54

Of course fbc compiles .bi files! #include merely brings in code from another file, which is parsed and compiled like any other code.

Now of course using a .bi file the way the OP is using is unusual, and perhaps not the best practice. And in fact there are times when code in the .bi file cannot work, as it would cause duplicated symbols, especially when the declarations and types defined in the .bi file are needed by more than one .bas file in your project.

My advise is to make a library and not worry so much about a few KB of unused code.
marpon
Posts: 340
Joined: Dec 28, 2012 13:31
Location: Paris - France

Re: Creating Static lib with function level granularity

Postby marpon » Mar 10, 2015 18:59

@TJF something wrong with you

what about your remark
You should know that fbc doesn't compile bi files. Headers (bi files) are designed to contain declarations only, but no executable source code (as SUBs or FUNCTIONs).

who is saying compiling bi file,
I've said : "to make a static lib with all of the code from that .bi file"

And you should know : the bi files can hold any type of code , not only declarations , the .bi file is just a convention of extension for include file, you can even use other extension, no problem.

I will check your Fb-doc proposal , but in the mean time I have done my own tool, and my lib is done !

Return to “Community Discussion”

Who is online

Users browsing this forum: No registered users and 41 guests