h_2_bi.bas, a tool for translating .h files into .bi

User projects written in or related to FreeBASIC.
Post Reply
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Post by TJF »

AGS wrote:...What if you could change that into
regular_expression>regular_expression_replacement
I think it's a good idea. Using regular expressions for replacing words in the FB code after translation may be need. But, is this a task for h_2_bi? Or can it be done by a code beautyfier, running after h_2_bi (ie started by h2bigui)?

I have good reasons to keep h_2_bi as small and handy as possible.

Lets discuss about this issue when I finished version 0.2.
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Version 0.2

Post by TJF »

Release 0.2 is out now.

Bugfixes/changes:
  • * more comments in source
    * review of the C code parser
    * result file gets writen during translation
    * extended protocol section, output can be controlled now
    * updated readme.txt file (~430 lines, including tutorial)
    * options: some moved, unused removed (details in h_2_bi.bi)
    * source code spread over several files, cleaned-up, some OOP now
    * some speed-up: ie GTK-2.22 from 85 s (vers 0.1.9.2) to < 30 s (vers 0.2)
    * all #else/#endif statements can be marked now (before, only #ifdef/#ifndef were marked)
New features
  • * uncommenting of unsave #define lines
    * input file buffering (especially for AGS :-)
    * new translations: EXTERN, STATIC, CONST
    * new options: (details in h_2_bi.bi)
    • - new group _P: protocol output (ie -P_pLF = no list of done files)
      - new group _T: translation output (ie -P_tBT = no type blocks)
      - some options moved from _O group to others.
    * new option -g (Geany mode)
Details on option -g:

The -g option (Geany mode) is for use in Geany editor (an FB-IDE). It's like magic: select in your document a C header source block, press a key, and your selection will be replaced by the matching FB source.

How to:
First, you have to compile h_2_bi and install the binary as a 'send text custom command' in Geany: go to the 'Set Custom Commands' dialog in the 'Format' sub menu of the 'Edit' and 'Popup' menu. Then click on 'Add' to get a new text entry. Fill the entry with command for calling the h_2_bi binary, followed by the option ' -g'. From now on, the current selection can be sent to h_2_bi. Geany will replace the selection by the output of h_2_bi.

Technical background:
In this modus, h_2_bi does neither read the input from a .h file nor writes the output to a .bi file. Instead, it reads from the stdin stream and writes to stdout. This means no file name will be specified -- h_2_bi cannot read any config file. Instead, h_2_bi uses the config file 'Geany.h2bi' in the EXEPATH of the h_2_bi binary to get specifications (MACROS, HEADERS, etc.). All sections can be used, apart from the file name section (file names will be ignored).


Download:

http://www.freebasic-portal.de/download ... s-134.html

Enjoy, test, test, test -- and report!
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Post by TJF »

@AGS:

The h_2_bi source changed a bit. You need to adapt your h2bigui code. Here is a new example on how to read the config file:

Code: Select all

' File AGS_Example.bas
'
' Shows how to use the new h_2_bi modules of release 0.2 to read a
' config file and handle the input.
'
' Copyright 2010 (C) by Thomas[ dot ]Freiherr[ at ]gmx[ dot ]net
' Licence: LGPL
'

#DEFINE PROG_NAME "" '        Workaround. Sorry, I forgot to fix it.

#INCLUDE ONCE "vbcompat.bi"
#INCLUDE ONCE "h_2_bi.bi" '             parameter/option definitions
#INCLUDE ONCE "h_2_bi_Helpers.bas" '   div. routines for small tasks
#INCLUDE ONCE "h_2_bi_Options.bas" ' evaluates command line pars/options
#INCLUDE ONCE "h_2_bi_CReader.bas" '         reades C files .h/.h2bi
#INCLUDE ONCE "h_2_bi_Config.bas" '      evaluates config file .h2bi



' callback for output: HEADERS, TYPES, POSTs
FUNCTION RepOut(BYREF Se AS ZSTRING PTR, _
                BYREF Re AS ZSTRING PTR, _
                BYREF Az AS INTEGER, _
                BYVAL Po AS ANY PTR) AS INTEGER
  VAR no = CAST(INTEGER PTR, Po) : IF *no < 0 THEN *no = ABS(*no) : RETURN 0
  *no -= 1 : IF *no < 0 THEN RETURN 1
  VAR t = ">", p = 0
  WHILE (*Se)[p] = (*Re)[p] '                        compare strings
    p += 1
    IF (*Se)[p] = 0 THEN
      t = "&" : Re += p + IIF(*(Re + p) = TYKENN, LEN(TYKENN), 0) : EXIT WHILE
    END IF : IF (*Re)[p] = 0 THEN EXIT WHILE
  WEND

' ******************************************************************
' instead of printing, handle the output here as needed
  ? RTRIM(*Se) & t & *Re
' ******************************************************************

  RETURN 0
END FUNCTION



' read the config file and handle the input
SUB readH2Bi(BYREF F AS STRING)
  DIM AS Options PTR Op = NEW Options("")
  DIM AS CFileReader PTR In = NEW CFileReader(F)
  VAR inmsg = In->Erro()

  IF inmsg THEN '                                 error reading file
    ? *inmsg & F
  ELSE
    DIM AS Config PTR Co = NEW Config(In, Op)
    ? Co->Msg ' output messages from reading process (can get removed)

' ******************************************************************
' place your stuff here

    ?"The first section (file names):"
    VAR aa = 0, ee = 5
    DO
      aa = ee + 1 : ee = INSTR(aa, Co->FNams, CHR(1))
      IF ee <= aa THEN EXIT DO
      ?MID(Co->FNams, aa, ee - aa) '            output one file name
    LOOP
    ?"__________________________________________________________________"

    ?"The " & KEY_FOLDERS & " section:"
    aa = 0 : ee = 5
    DO
      aa = ee + 1 : ee = INSTR(aa, Co->Fold, CHR(1))
      IF ee <= aa THEN EXIT DO
      ?MID(Co->Fold, aa, ee - aa) '           output one folder name
    LOOP
    ?"__________________________________________________________________"

    ?"The " & KEY_START & " section:"
    ? Co->Start
    ?"__________________________________________________________________"

    ?"The " & KEY_END & " section:"
    ? Co->Ende
    ?"__________________________________________________________________"

    ?"The " & KEY_MACROS & " section:"
    aa = 0 : ee = 5
    DO
      aa = ee + 1 : ee = INSTR(aa, Co->Macros, CHR(1))
      IF ee <= aa THEN EXIT DO
      ?MID(Co->Macros, aa, ee - aa) '          output one macro line
    LOOP
    ?"__________________________________________________________________"

    ?"The " & KEY_PARAMETERS & " section:"
    ?"the complete group C: "; BIN(Op->FLC, 32)
    ?"ie the option -P_C" & TCMA & ": "; Op->C.MA
    ?"the complete group T: "; BIN(Op->FLT, 32)
    ?"ie the option -P_T" & TTBE & ": "; Op->T.BE
    ?"__________________________________________________________________"

    VAR no = 0
    ?"The " & KEY_POST_REPS & " section:"
    no = &h7FFFFFFF '                high integer to get all entries
    Co->Post.list(@RepOut, @no) '             output all POSTs lines
    ?"__________________________________________________________________"

    ?"The " & KEY_HEADERS & " section:"
    no = Co->AzHe '              only the user defined (no defaults)
    Co->Head.list(@RepOut, @no) '           output all HEADERS lines
    ?"__________________________________________________________________"

    ?"The " & KEY_TYPES & " section:"
    no = -Co->AzTy + 1 '   !!! negative to skip the first 'void>ANY'
    Co->Typ_.list(@RepOut, @no) '             output all TYPES lines

' end of your stuff
' ******************************************************************

    DELETE Co
  END IF

  DELETE In
  DELETE Op
END SUB



' ******************************************************************
' main

readH2Bi("Geany" & H2BI_FILE_SUFFIX)

END 0
In my next version, the 'h_2_bi_Config.bas' file will get changed for better error messages. I can send you this file in an email, if required.
AGS
Posts: 1284
Joined: Sep 25, 2007 0:26
Location: the Netherlands

Post by AGS »

I downloaded version 0.2 today. Don't now why I did not look at it before (I somehow did not see it until today???).

Looked at your code and it's.... changed. A lot. Funny thing is I experimented a bit with file buffering myself. And I've come to the conclusion that #get is just a slow function.

I tried replacing get# with my own version of get# (get_).

get_ uses a small buffer (1024bytes seems to be a nice size). When you call get_ it returns one byte from the buffer. If a read will lead to a read beyond the buffer limits new data gets read by calling get#,,buffer,SIZE_OF_BUFFER. If there is no more data then eof is set (get_ uses it's own data structure that contains the buffer and some other stuff).

The whole trick is to call get# as little as possible. When reading a small file calling get# on every byte does a bit worse than using the buffer (factor 2?). As the size of the file increases the advantage of using a buffer grows. Reading a file-at-once is the fastest. I tested a bit and I think the numbers were something like (reading large file eg filesize > 600k):

- read whole file at once ==> speed 240 (it's just a number)
- read whole file using buffer ==> speed 12 (relative to the 240)
- read whole file byte-for-byte ==> speed 1 (relative to the 240)

But file size matters.
Big file ==> big advantage using buffering.
Small file ==> little advantage using buffering.

For a single C header file buffering gets you some (factor 2?) or no speed up. It's all in the number of calls to get#. For some reason that routine is slow. Reading file at once is best when file size is small. Buffering is a reasonable choice as file size increases (reading a huge file at once would involve allocating a LOT of memory for the buffer).

Anyway, I'm going to be looking at your code and test it a bit.

I have decided to drop my iup gui as I am not willing to write one gui for windows and another one for linux. I tried to translate what I had to gtk+ but in the end I decided a bit of glade will get me better results faster.

I'm going to be using GTK+ only (glade) from now on. Great stuff, iup, but writing a nice GUI is difficult enough as it is without having to worry whether or not it will work on Linux.

I'm trying to write some sort of c2fb. I'm not writing it in FB yet as I'm using two tools (GPP and dparser) to get C parsing going. And both of these tools have been written in C.

GPP is a Generic PreProcessor that I use as a C preprocessor preprocessing (its about 2500 lines of code).

dparser is a compiler compiler that you can use to create a parser+scanner (the results is a slow parser but still, a parser). It creates a parse tree. As dparser preserves whitespace (including comments) it's a great tool for source-to-source translation. dparser is a lot bigger than GPP (thousands of lines).

And now I'm wondering if you, me or you and me could put something on paper regarding the translation of C to FB. I know you must have some ideas as to what the problems are when translating C to FreeBASIC. You implemented translation of casting/operators/other things and I think it would be good to have the kind of knowledge needed to do those kind of things on paper.

Not only for my benefit but for others that want to try and translate C to FreeBASIC as well. I'm thinking along the lines of some sort of checklist, a table with C operators versus FB operators, some do's and don't s, something about code that is not directly translatable to freebasic etc....

And finally: would you mind if I would do a spell check on the h2bi README file (and the comments inside the different parts of h2bi)? I'll post a link to my spell checking results (if any) in this thread.
AGS
Posts: 1284
Joined: Sep 25, 2007 0:26
Location: the Netherlands

Post by AGS »

I checked out the new lookup function (keyword lookup). Interesting solution. I benchmarked it a bit and it basically performs at a speed comparable to that of a binary search. And no slowdown when feeding it words that are not keywords. Nice.
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Post by TJF »

AGS wrote:I checked out the new lookup function (keyword lookup). Interesting solution. I benchmarked it a bit and it basically performs at a speed comparable to that of a binary search. And no slowdown when feeding it words that are not keywords. Nice.
The code is inspired from FB2HTML.bas by ThePuppetMaster. He has a lot of high quality code at the portal.

I used FBeauty to test it. I also tested the Geany mode in this little tool.

The code performs a little slower on words starting with characters like 'c' or 's' (due to a large number of FB keywords starting with this characters) and is fast at words starting with charaters like 'k', 'x' or 'y'. After all, it is a fast solution that can be easy maintained further on.
AGS wrote:... And finally: would you mind if I would do a spell check on the h2bi README file (and the comments inside the different parts of h2bi)? I'll post a link to my spell checking results (if any) in this thread.
Any help on improving h_2_bi is wellcome. I concentrate on the source code, so that the compiler understands me. If you can improve the comments and the docu to help new users understanding the tool, it will be a good improvement. You may post the results here or send it as a private mail, so that I can implement it in the next version. Let me know if (and how) I can support you.
AGS wrote:I have decided to drop my iup gui as I am not willing to write one gui for windows and another one for linux. I tried to translate what I had to gtk+ but in the end I decided a bit of glade will get me better results faster.

I'm going to be using GTK+ only (glade) from now on. Great stuff, iup, but writing a nice GUI is difficult enough as it is without having to worry whether or not it will work on Linux.
This meets my experience too: concentrate on one problem to solve.

You'll have to make a decision first: LibGlade or GtkBuilder. LibGlade is easy to code, but needs one more .DLL to install on windows. I switched to GtkBuilder which is more universal, but needs a little more code in FB. The main advantage: all the windows runtime libs can get installed by one prebuild package.
AGS wrote:For a single C header file buffering gets you some (factor 2?) or no speed up. ...
I think your speed test results are typical for windows. On Linux the difference isn't that much.

In h_2_bi version 0.1.9.2 the input is byte-by-byte and in version 0.2 the complete C header file(s) is(/are) buffered. Regarding speed up I get a factor of ~3 for translating GTK-2.22.0 headers (vers. 0.1.9.2: 85 sec, vers. 0.2: 30 sec).

Most of the speed up is due to the parser optimization (OOP, less variables, less function calls, ...). The file buffering is on second place in the speed up ranking. (The speed up is less than factor 1.5.)
AGS wrote:I'm trying to write some sort of c2fb. I'm not writing it in FB yet as I'm using two tools (GPP and dparser) to get C parsing going. And both of these tools have been written in C.
This would be nice to have. Don't mind to code in C. When realized, the tool can translate itself to FB ;-)
AGS wrote:And now I'm wondering if you, me or you and me could put something on paper regarding the translation of C to FB. I know you must have some ideas as to what the problems are when translating C to FreeBASIC. You implemented translation of casting/operators/other things and I think it would be good to have the kind of knowledge needed to do those kind of things on paper.
I'm not sure if I can help here. Before I started h_2_bi, I had no clue of C coding. I learned by trial and error. Up to now, I understand a bit of the header source.

To give advices on translating C source files, I'll have to learn new structures like 'while', 'for' or 'if' blocks (ins't impossible). But the main problem I see at translating the little expressions like '++i', 'j--' or 'a == b == c'. There are no direct expressions in FB and the code has to get translated into more than one FB code line. This needs clever solution.
AGS
Posts: 1284
Joined: Sep 25, 2007 0:26
Location: the Netherlands

Post by AGS »

TJF wrote:
AGS wrote:I checked out the new lookup function (keyword lookup). Interesting solution. I benchmarked it a bit and it basically performs at a speed comparable to that of a binary search. And no slowdown when feeding it words that are not keywords. Nice.
The code is inspired from FB2HTML.bas by ThePuppetMaster. He has a lot of high quality code at the portal.
I'll have to check out his code then.

TJF wrote: You'll have to make a decision first: LibGlade or GtkBuilder. LibGlade is easy to code, but needs one more .DLL to install on windows. I switched to GtkBuilder which is more universal, but needs a little more code in FB. The main advantage: all the windows runtime libs can get installed by one prebuild package.
I'm going with GtkBuilder too.
TJF wrote:
AGS wrote:For a single C header file buffering gets you some (factor 2?) or no speed up. ...
I think your speed test results are typical for windows. On Linux the difference isn't that much.
That's very likely, yes.
TJF wrote: In h_2_bi version 0.1.9.2 the input is byte-by-byte and in version 0.2 the complete C header file(s) is(/are) buffered. Regarding speed up I get a factor of ~3 for translating GTK-2.22.0 headers (vers. 0.1.9.2: 85 sec, vers. 0.2: 30 sec).

Most of the speed up is due to the parser optimization (OOP, less variables, less function calls, ...). The file buffering is on second place in the speed up ranking. (The speed up is less than factor 1.5.)
For small files the whole file buffering thing is not all that important. When reading a big file (several hundreds of kb) there might be a bigger difference in runtime performance. Might (as Linux is not Windows).
TJF wrote:
AGS wrote:And now I'm wondering if you, me or you and me could put something on paper regarding the translation of C to FB. I know you must have some ideas as to what the problems are when translating C to FreeBASIC. You implemented translation of casting/operators/other things and I think it would be good to have the kind of knowledge needed to do those kind of things on paper.
I'm not sure if I can help here. Before I started h_2_bi, I had no clue of C coding. I learned by trial and error. Up to now, I understand a bit of the header source.

To give advices on translating C source files, I'll have to learn new structures like 'while', 'for' or 'if' blocks (ins't impossible). But the main problem I see at translating the little expressions like '++i', 'j--' or 'a == b == c'. There are no direct expressions in FB and the code has to get translated into more than one FB code line. This needs clever solution.
I basically started writing right away and have split the writing about translation into translation of declarations (that's what h_2_bi mostly does) and translation of definitions. I'll post the PDF when it's in readable shape.

There are a bunch of pitfalls when translating C to fb and you need a symbol table to make it work. I'm aiming for the translation of reasonably simple C examples that come with external libraries. Anything more would be a bonus.

The new code for h2bi looks very different from the old code. I recognize some code from the previous version but it's organised differently. More use of macros (STACK), constructors etc....

Is there anything in particular that should be tested more thoroughly? I usually just start running a new version of h2bi on some basic examples and then continue with ever more complicated header files until I try to make h2bi translate something like this
http://svn.freebsd.org/viewvc/base/rele ... iew=markup
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Post by TJF »

AGS wrote:The new code for h2bi looks very different from the old code. I recognize some code from the previous version but it's organised differently. More use of macros (STACK), constructors etc....
Yes, I cleaned up and re-structured nearly everything. I didn't see the code running for more than 4 weeks, before I could start to find and fix bugs and do some opimization.
AGS wrote:Is there anything in particular that should be tested more thoroughly? I usually just start running a new version of h2bi on some basic examples and then continue with ever more complicated header files until I try to make h2bi translate something like this ...
Since I changed a lot, there is no particular feature to test. Feel free to test whatever you want in the way you like. Every report is welcome (from AGS and from other users as well).
AGS
Posts: 1284
Joined: Sep 25, 2007 0:26
Location: the Netherlands

Post by AGS »

First bug found. And it's a minor one.

When using a commandline option that starts out as a valid command line option (for example -P_TBT) but then ends with a _ (-P_TBT_) the program crashes.

Example

When you run h2bi using the following commandline option
h2bi -P_TBT_
this is what happens
Aborting due to runtime error 7 (null pointer access) at line 87 of d:\Projects\
h2bi\version0.2\FB_h_2_bi\src\h_2_bi_Options.bas::CHECKOPTP()
When running the above example I was using a valid h2bi file . And it does not matter where you specify the -P_TBT_ (either on the command line or in the .h2bi file) the result is always the same (runtime error 7).

Compiled using fbc version 0.21.1 (options -g -exx).

I tested the program on some small examples and, apart from the minor bug I found, so far so good. I'm having some trouble getting the output to look the way I want though. I want to have

--> no C source code in the output
and I also want
--> all comments preserved

I've tried several command line options.

I thought -P_ISC_ISB would do the trick but all I am getting is C source code (with the comments inside the C code preserved).
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Post by TJF »

@AGS:

Thanks for the report. I did a short test and found the bugs you described. After looking at the source, I found out that I cannot fix it fast. Since I'm currently moving, I have no time for a deep look into the problem.

I'll fix this ASAP when I'm in the new house (and internet connection is running again).

Best regards!
Dinosaur
Posts: 1478
Joined: Jul 24, 2005 1:13
Location: Hervey Bay (.au)

Post by Dinosaur »

Hi all

Have a need for a .h to .bi converter, but when I download the zip files
I get one single line of what appears to be only declares, and I can't see any
basic code in any of the .bas files ???


I am using FbEdit

Regards
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Post by TJF »

Thanks for your interest on h_2_bi!

Altough you don't see the code, you can compile and run it.

Background:

h_2_bi is coded on Linux with CHR(10) line ends. Most win apps and progs (like FreeBasic) can handle this. Unfortunately FBEdit cannot. To see the code you need to convert line ends from CHR(10) to CHR(13, 10).

I recomment to use a more powerful IDE like Geany. This also helps to view and handle the .h files for your project. And you get cross-platform capability - importent for testing your app on other OSs.
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

h_2_bi-0.2.0.4 is available (new release)

Post by TJF »

Should be fixed now:
AGS wrote:First bug found. And it's a minor one.

...

I want to have

--> no C source code in the output
and I also want
--> all comments preserved

Download:

http://www.freebasic-portal.de/download ... s-134.html

Have fun!
rdc
Posts: 1741
Joined: May 27, 2005 17:22
Location: Texas, USA
Contact:

Post by rdc »

Very nice work here. Thanks for keeping up with this.
AGS
Posts: 1284
Joined: Sep 25, 2007 0:26
Location: the Netherlands

Post by AGS »

@TJF
Is there anything that can be done to make h2bi even better than it already is or do you think you've done all you can? In other words: is the current version the 'ultimate' version of h2bi?

And I really liked the following sentence:
* input file buffering (especially for AGS :-)
:-D That's a good one :-D I think it did not help that much, did it? (buffering).

I just started using the Geany option (-g) trick. And it is GOOD! Very good.

Occasionaly h_2_bi -g fails. When I try to convert too much at a time it crashes.

Example

Code: Select all


 hid_t H5Dopen2(hid_t file_id, const char *name, hid_t dapl_id);
 herr_t H5Dclose(hid_t dset_id);
 hid_t H5Dget_space(hid_t dset_id);
 herr_t H5Dget_space_status(hid_t dset_id, H5D_space_status_t *allocation);
 hid_t H5Dget_type(hid_t dset_id);
 hid_t H5Dget_create_plist(hid_t dset_id);
 hid_t H5Dget_access_plist(hid_t dset_id);
 hsize_t H5Dget_storage_size(hid_t dset_id);
 haddr_t H5Dget_offset(hid_t dset_id);
 herr_t H5Dread(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id,
			hid_t file_space_id, hid_t plist_id, void *buf);
 herr_t H5Dwrite(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id,
			 hid_t file_space_id, hid_t plist_id, const void *buf);
 herr_t H5Diterate(void *buf, hid_t type_id, hid_t space_id,
            H5D_operator_t op, void *operator_data);
 herr_t H5Dvlen_reclaim(hid_t type_id, hid_t space_id, hid_t plist_id, void *buf);
 herr_t H5Dvlen_get_buf_size(hid_t dataset_id, hid_t type_id, hid_t space_id, hsize_t *size);
 herr_t H5Dfill(const void *fill, hid_t fill_type, void *buf,
        hid_t buf_type, hid_t space);
 herr_t H5Dset_extent(hid_t dset_id, const hsize_t size[]);
 herr_t H5Ddebug(hid_t dset_id);
When I select all but the final three declarations all is well. When I select one more declaration h_2_bi crashes.

For me the Geany thing is THE best solution. First I unleash a regular expression or two on the C source (getting rid of some C nastiness) and then it's select -> control one -> done.

I have tried setting a couple of options in geany.h2bi but I could not get the options to work (yet). But I have not really tried all that hard as I'm too pleased with the Geany thing :). Option -g = option gut (option sehr gut).
Post Reply