fbfrog header translator

User projects written in or related to FreeBASIC.
dkl
Site Admin
Posts: 3156
Joined: Jul 28, 2005 14:45
Location: Germany

fbfrog header translator

Postby dkl » Oct 22, 2011 1:53

https://github.com/dkl/fbfrog

fbfrog is a command line tool that reads *.h files (C API declarations) and generates a corresponding *.bi file (FreeBASIC API declarations). It's intended to automate most of the work needed to create and maintain FB bindings for C libraries.

----------------------------------------
Original message about the first experimental version:

This is a new C-to-FB header translation helper. It's based on rearranging and translating tokens, and at the moment it gets through the libcurl headers pretty well.

I started this project a while back, wanting to somehow automate header translation. Then about two weeks ago I talked about it on IRC and that made me finish it. Now it can translate the important things, like structures, fields, function declarations, function pointers. And it can do some tricks like merging in #includes. But you can read more about that in the readme.

Well, by design it is very limited, but at the same time it sort of preserves the look and feel of the original headers, not perfectly, but as close as I have ever seen. I don't know whether this is the right way to go about making FB bindings, since it's more of a 1:1 translation than a binding, but so far it was a cool experiment. No idea where this could go from here. By the way, I have never used h_2_bi or FBSWIG, so I don't know what the differences are.

fbfrog itself is a command line tool, but there's a separate GUI program for it, take a look:
Linux (GTK+) | Win32

Aaand phew, thank god it's out now, so I can go back to other stuff... Have fun with it, and take a look at the source!
TJF
Posts: 3324
Joined: Dec 06, 2009 22:27
Location: N47°, E15°

Postby TJF » Oct 22, 2011 12:32

Interesting! It's very similar to h_2_bi.

I compiled the source without problems (on Ubuntu11.10 / fbc 0.23 -- good readme.txt). But I've trouble using it, ie:
  • the GUI sends lots of warnings like
    `menu_proxy_module_load': ./fbfrog-gui: undefined symbol: menu_proxy_module_load

    (fbfrog-gui:3087): Gtk-WARNING **: Failed to load type module: (null)

    or
    (fbfrog-gui:3087): Gtk-CRITICAL **: gtk_text_layout_real_invalidate: assertion `layout->wrap_loop_count == 0' failed
  • Translating GLib headers I got
    translating: ../glib-2.28.0/glib/gdebug.h
    bug: failure at translate_decl(1187): tk_get(x) =TK_SEMI

    (I would like to see something like a character position or line# where the bug occurs.)

Some hints:
  • I don't like the TODO messages included in the source. And untranslated C-source results in compile errors. In h_2_bi I append messages at the end of the line and mark C-source as comment.
  • Some basics don't get translated (ie int a = 5;)
  • It's pretty easy to translate ( ? : ) -> IIF( , , ), its more complicated to identify CASTs.
  • Translating MACROs is important for cross-platform libraries

    Code: Select all

    #define MY_EXTERN /'__declspec(dllexport)'/
    #define MY_CALL __attribute__((__stdcall__))/'__stdcall'/

    /' some #defines in front, as is pretty common '/
    '' TODO: unknown construct
    MY_EXTERN MY_CALL TT1 f7(void);
    h_2_bi can expand these macros before translation and generates the desired results (that's why the control file is mandatory).
  • Named nested unions/structs can be translated as (original h_2_bi translation of unsupported.h file fragment)

    Code: Select all

    ' struct T {
    '    union {
    '       int a;
    '    } foo;
    ' };
    UNION T_foo
       AS INTEGER a
    END UNION

    TYPE T
       AS T_foo foo
    END TYPE


Can you please explain what is done in the preparsing phase? (I'm just curious and I'm too lazy to study the source.)
JohnK
Posts: 256
Joined: Sep 01, 2005 5:20
Location: Earth, usually
Contact:

Postby JohnK » Oct 22, 2011 17:24

I am very interested in these C -> FB translators. However, now we have 3 (?). It is a little confusing. There was the original by Victor1, then h_2_bi, now this.
Are they all improving?
Does one have a strength the other does not?
Converting C++ to bi is still a "holy grail"?

Thanks for these really cool tools
-JohnK
maddogg6
Posts: 824
Joined: Dec 07, 2005 22:58
Contact:

Re: fbfrog header translator

Postby maddogg6 » Oct 22, 2011 19:03

dkl wrote:Well, by design it is very limited, but at the same time it sort of preserves the look and feel of the original headers, not perfectly, but as close as I have ever seen. ......

.... By the way, I have never used h_2_bi or FBSWIG, so I don't know what the differences are.


This rather confuses me.

I mean, I realize you are comparing to hand translated headers (right?), but how do you know if it was not a combination (ie. first swig, then fix/hand translate.) that translated something you may assume to be hand translated??

I admit my ignorance, so please don't take this as snark or anything, but it does seem like a claim made in admitted ignorance.
Sorta like saying I am the best ________ in my family... only then later explaining when pressed that I'm also an only child and my parents are dead, but.. ;)
dkl
Site Admin
Posts: 3156
Joined: Jul 28, 2005 14:45
Location: Germany

Postby dkl » Oct 22, 2011 20:42

There I tried to describe a happy moment while testing fbfrog, but well, don't take it all too serious. It was just a good feeling to have made this program work.

And I really don't know what the differences are to the other tools, in terms of how they work or how they are used or what exactly they can and cannot do. Sure enough the results must be similar somehow. Although, I am assuming them to work differently, in a way that does not allow them to preserve commentary and whitespace (but instead to translate more C constructs to FB), because I haven't seen any FB headers like that, except for some nice manually translated ones.

I'd say the strengths of fbfrog are the close-to-the-original translation and the TODOs. I think it's cool to have it do 90% of the work, then go through the TODOs to fix up the rest manually and be done that much faster compared to working completely manually.


TJF, during preparsing it collects some information on #includes and #defines. It figures out the dependencies between multiple header files, so it can decide whether to merge them in, or to concatenate them, or leave them alone (and all that merging etc. is optional too), and for #defines, it checks whether they represent a calling convention or similar, and currently this is what it does:

Code: Select all

#define ITS_EMPTY
#define ITS_EXTERN __declspec(dllexport)
#define ITS_STDCALL __stdcall
#define ITS_CDECL __attribute__((cdecl))

int f();
ITS_EMPTY int f();
ITS_EXTERN int f();
ITS_STDCALL int f();
ITS_CDECL int f();
ITS_EXTERN ITS_CDECL int f();


Code: Select all

#define ITS_EMPTY
#define ITS_EXTERN __declspec(dllexport)
#define ITS_STDCALL __stdcall
#define ITS_CDECL __attribute__((cdecl))

declare function f() as integer
declare function f() as integer
declare function f() as integer
declare function f ITS_STDCALL () as integer
declare function f ITS_CDECL () as integer
'' TODO: unknown construct
ITS_EXTERN ITS_CDECL int f();


You can see it's very basic. It accepts a single define in front of function declarations, but not two, and that would need to be added to the parser. But yea, it doesn't do macro expansion at all. I mean, it could also do a preprocessing pass or something, but that was too much work for me, and I think it's better to preserve #defines without expanding them where they're used (to stay close to the original). And yea, since there are endless possibilities of how to hide C constructs behind macros, it's very easy to come across something that fbfrog can't translate. That's definitely a limitation, but it doesn't really bother me, as long as fbfrog can handle the set of common constructs found in most headers.
maddogg6
Posts: 824
Joined: Dec 07, 2005 22:58
Contact:

Postby maddogg6 » Oct 23, 2011 2:53

dkl wrote:There I tried to describe a happy moment while testing fbfrog, but well, don't take it all too serious. It was just a good feeling to have made this program work.

lol - I see... oh and kudos. Thanks for explanation.

I wish my C was stronger but it just messes with me something fierce.
TJF
Posts: 3324
Joined: Dec 06, 2009 22:27
Location: N47°, E15°

Postby TJF » Oct 23, 2011 7:53

JohnK wrote:I am very interested in these C -> FB translators. However, now we have 3 (?). It is a little confusing. There was the original by Victor1, then h_2_bi, now this.
Are they all improving?
Does one have a strength the other does not?
Converting C++ to bi is still a "holy grail"?

Thanks for these really cool tools
-JohnK

You're wellcome. I hope you aren't not only interested in translator tools but use them as well and publish your results.

Here's my rating (in chronologic order)
  • FBSWIG (the original by v1ctor) is the dinosaur. It's the first attempt to this topic. It's a real compiler (not lightweight) and it's windows only. I couldn't get it running so I don't realy know about it's features.
  • h_2_bi is like a Rolls Royce. It has many features (allthough it's a lightweight solution -- no compiler). It can expand macros (recursiv, VARARG parameters) and resolve naming conflicts. It has more than 100 switches to control file input, translation and file output. It protocols info on the translation process. It can be integrated into Geany as a custom command. But it needs some experience to handle all these features.
  • fbfrog is the solution inbetween (leightweight -- no compiler). It has a good set of basic features and it's easy to use. It's best choise for beginners to get started with header translations (when it gets more stabile).
TJF wrote:I don't like the TODO messages included in the source. And untranslated C-source results in compile errors.

The next day I think this is a good solution for beginners. They get precise information about the pitfalls in the translation process and where they occur.

dkl wrote:Although, I am assuming them to work differently, in a way that does not allow them to preserve commentary and whitespace ...

In the early versions (<0.2) h_2_bi preserved whitespace and it still can preserve commentary (on demand). The commentary (and/or if required the original C-source) is placed in front of the translation.

dkl wrote:TJF, during preparsing it collects some information on #includes and #defines. It figures out the dependencies between multiple header files, so it can decide whether to merge them in, or to concatenate them, or leave them alone (and all that merging etc. is optional too), and for #defines, it checks whether they represent a calling convention or similar, and currently this is what it does:

Thanks for explaining that. h_2_bi collects information to support these decisions in each run. It's up to the user to make them before the next run. This is less convenient but more flexible.
dkl
Site Admin
Posts: 3156
Joined: Jul 28, 2005 14:45
Location: Germany

Re: fbfrog header translator

Postby dkl » Jul 20, 2014 18:59

Hello!

A new version of fbfrog has long been in the making, and now it's pretty much finished. It has a much more extensive C parser than before, and even a full C preprocessor for macro expansion and #if evaluation. It supports 64bit and automatically adds Extern blocks and fixes symbol identifier conflicts, and lots of other details beyond parsing C.

Doing full C preprocessing is probably the detail with the biggest impact: It means that it absolutely does not matter what deeply nested #ifs or ugly macros are used in the input headers, it's all stripped out. All that remains is the pure C API (and #defines, which are given special treatment). This really helps when translating C headers like libpng's png.h where the function declarations are hidden behind multiple layers of conditionally defined macros...

Another side-effect of the full CPP is that fbfrog has to read the input headers at least 5 times: once for each target system (DOS/DJGPP, Linux/x86, Linux/x86_64, Win32, Win64). And in the end, that produces 5 APIs that will be merged into one binding. It will automatically insert #if __FB_WIN32__ and such where needed. This feature could easily be extended to support further targets (e.g. Linux/ARM or FreeBSD).

And this feature can also be used to combine the APIs of multiple versions of a library into one binding, for example the v1.2.* and v1.6.* series of libpng. The binding's user could then select the desired API version by doing something like #define __LIBPNG_VERSION 12 before #include "png.bi". I think this will be useful for compiling an FB program against a specific version of libpng installed on the system, in case it's different than what png.bi defaults to. libpng is a good example here: Most Linux distros still have libpng 1.2, but the png.bi from FB 0.90.1 only supports libpng 1.5, and the two are incompatible (I think they're ABI-incompatible, and libpng will literally abort the program if it detects a version mismatch like that).

Internally, fbfrog now works a lot like a C compiler, that happens to generate FB code. It's based on an AST now, and even though it still allows preserving commentary to some degree, it's essentially the opposite of the original idea of "just rearranging tokens". But it's also what has allowed for all the new features.
Amundo
Posts: 55
Joined: Feb 26, 2007 0:25

Re: fbfrog header translator

Postby Amundo » Jul 24, 2014 2:09

Haven't been around much, so just saw this. Amazing work, dude, thanks for sharing.

I really must test it by feeding it some headers from the Lotus Notes C API, but because FB isn't case-sensitive, it always trips up on some variable somewhere :-(


Awesome, just awesome...
frisian
Posts: 231
Joined: Oct 08, 2009 17:25

Re: fbfrog header translator

Postby frisian » Jan 16, 2015 22:46

Converted my first .h file into a .bi file with your FBfrog (1.5), worked very well.

The only thing that wend wrong was it missed a "#endif" at the end of the .bi file.
The .h file had at the end "#endif /* __GMP_H__ */" perhaps the comment messed things up, I removed the comment and tried again but it was still missing.

Working with FBfrog made me realize it would be nice if it includes some things.
It would be nice to start a converted file with "Converted by FBfrog <version nr.>" so everybody knows with what program and version number it was converted.

Most .bi files start with something like

#ifndef __gmp_bi__
#define __gmp_bi__

#inclib "gmp"

the #inclib part is what it's all about, the header file belongs to a certain library file.

Regards
dkl
Site Admin
Posts: 3156
Joined: Jul 28, 2005 14:45
Location: Germany

Re: fbfrog header translator

Postby dkl » Jan 17, 2015 16:34

Yea, fbfrog will currently emit #pragma once, instead of the #ifndef/#define/#endif include guards. But it can still emit #if blocks sometimes (to handle target-specific sections of code), so if there was an #endif missing there, that's probably a bug.

It could emit it's name/version number into the .bi file of course, but I've been going with the idea that fbfrog is just the tool to do the initial bulk conversion - but not the full conversion. Because there's almost always some further fixes needed... so it's more like "initially converted by fbfrog, and then hand-edited by ...".

About #inclib, I'm thinking that it should just be added manually. fbfrog had an -inclib option for a while to allowing adding them this way, but it was more complicated than doing it manually.
caseih
Posts: 1011
Joined: Feb 26, 2007 5:32

Re: fbfrog header translator

Postby caseih » Apr 26, 2015 3:07

Found a problem trying to convert a simple header file on Linux that includes limits.h. Inside of the default limits.h it uses a "#include_next" declaration, which fbfrog can't deal with. I'm not totally sure what include_next does, and I'm not sure which of the limits.h files I have (there's about 8 on my hard drive) get pulled in by this include_next statement. How does fbfrog search for include files? just /usr/include, or does it pull in some from the gcc system itself (say /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed/)?
dkl
Site Admin
Posts: 3156
Joined: Jul 28, 2005 14:45
Location: Germany

Re: fbfrog header translator

Postby dkl » Apr 26, 2015 12:40

It only searches #includes relative to the parent file, and doesn't use the gcc include directories. Although, you could specify those via -incdir <path> options, and #include_next support could be added based on that...

Is limits.h really needed though? Maybe it's enough to copy the header file you want to translate out of /usr/include, then fbfrog won't resolve the #include <limits.h> to /usr/include/limits.h anymore (unless you specified -incdir /usr/include). Often headers can be translated even though not all #includes were found.

If the header can't be preprocessed without limits.h then perhaps -fbfroginclude limits.h will do the trick - I added a stripped down, generic version of limits.h (at include/fbfrog/limits.h) which was enough to solve this kind of problem for me in the past. (where headers triggered #error directives if the limits.h #defines were missing)

As a last resort (while #include_next support is missing), you can copy the involved headers out of /usr/include, and expand #include_next's manually... I think the #include_next <limits.h> refers to gcc's private one (/usr/lib/gcc/i686-linux-gnu/4.9/include-fixed/limits.h on my system).

Edit: just implemented #include_next based on -incdir options
TeeEmCee
Posts: 191
Joined: Jul 22, 2006 0:54
Location: Auckland

Re: fbfrog header translator

Postby TeeEmCee » Apr 16, 2017 5:59

(Forgive me if reviving this old thread isn't appropriate, but it's been done several times before)

Is there any way to get fbfrog to preserve size_t instead of converting it to uinteger? (I'm converting system headers on FreeBSD)

Also, in a separate project I successfully converted a simple C++ header to FB using it. Aside from dropping inline function bodies, the only thing I had to do was manually convert by-reference arguments like "int &foo" to "byref foo as integer". fbfrog can't parse the &. It would be quite useful if it could do that, although I'm sure you have no desire to write a C++ parser.
dkl
Site Admin
Posts: 3156
Joined: Jul 28, 2005 14:45
Location: Germany

Re: fbfrog header translator

Postby dkl » Apr 16, 2017 10:10

The size_t handling is currently hard-coded, so the only way to disable it would be to remove it from that internal table. It might not be a 100% precise translation, but I assumed that size_t will always be the same as UInteger, so might aswell use that.

Indeed, adding support for & => byref is quite easy, also the struct parser could be extended to support simple classes. I think C++ default parameters already had to be supported aswell, due to some headers using them.

Return to “Projects”

Who is online

Users browsing this forum: No registered users and 1 guest