Parsing 101

Post your FreeBASIC source, examples, tips and tricks here. Please don’t post code without including an explanation.
Post Reply
notthecheatr
Posts: 1759
Joined: May 23, 2007 21:52
Location: Cut Bank, MT
Contact:

Parsing 101

Post by notthecheatr »

I wrote a simple object called words_list which takes a string and splits it into words. Its main use would probably be parsing, since for example it'll take something like this:

Code: Select all

Dim a As Integer
and give you the words

Code: Select all

Dim

Code: Select all

a

Code: Select all

As

Code: Select all

Integer
It leaves line comments (since it works line-by-line, multi-line comments can't be supported) and whitespace out of the words it produces, and strings inside quotation marks are all one word.

May be useful to someone anyways, here's the ZIP with the code, example, and readme: http://notthecheatr.phatcode.net/downlo ... s_list.zip
1000101
Posts: 2556
Joined: Jun 13, 2005 23:14
Location: SK, Canada

Post by 1000101 »

I hate to be a buzz-kill, but...strtok()
duke4e
Posts: 717
Joined: Dec 04, 2005 0:16
Location: Varazdin, Croatia, Europe
Contact:

Post by duke4e »

strtok()
what the hell is that?
Peter
Posts: 66
Joined: May 29, 2006 22:16

Post by Peter »

strtok = String Tokenize, basically it breaks down a string into the component pieces i.e. words. It pretty much does what your trying to do with your code. That said, I think your code is great, it gives you the basis to make much more complex stuff that is (and I think this is important sometimes) completely independent of outside calls.

Back to lurkerdom for me now.
MichaelW
Posts: 3500
Joined: May 16, 2006 22:34
Location: USA

Post by MichaelW »

For general-purpose use your whitespace should probably include tab characters. Strtok is fast and easy to use, but a byte scanner is probably a better choice where you need the length of each token and/or where you need to detect errors such as unmatched quotes.

Finding Tokens in a String
notthecheatr
Posts: 1759
Joined: May 23, 2007 21:52
Location: Cut Bank, MT
Contact:

Post by notthecheatr »

Now I know why people complain.

Actually, it's mostly a learning thing. Like I said, the code's pretty much a hack, but it might be useful and that's why I wrote it. I don't really care if there's something else that already does that, as using that other tool won't really teach me anything. Sure, it's good to use other tools, but there's nothing wrong with writing your own thing, is there?


P.S. This is an object. I could write an object based on strtok() but in any case it doesn't matter... my point is that using an object makes it a much more natural interface.
Peter
Posts: 66
Joined: May 29, 2006 22:16

Post by Peter »

Sure, it's good to use other tools, but there's nothing wrong with writing your own thing, is there?
No, there's nothing wrong with it. Especially when your learning to program I feel it is important to do as much coding as possible. When you have to code to a deadline it is right nice to know that someone else did some of the work for you already though. That all said.... when I was a wee lad all excited because my aunt showed me a few commands on my parents BBC Micro I spent several days writing a way to perform a a set number of times. I was so proud of it and wanted to show it off. When I did I was shocked when told about this "FOR...NEXT" thing did the same blasted thing!
yetifoot
Posts: 1710
Joined: Sep 11, 2005 7:08
Location: England
Contact:

Post by yetifoot »

Peter wrote:
Sure, it's good to use other tools, but there's nothing wrong with writing your own thing, is there?
No, there's nothing wrong with it. Especially when your learning to program I feel it is important to do as much coding as possible.
I also think it's good to learn how it's done, at least the first time. I prefer to hand write the lexer/scanner part, rather than use strtok, as strtok is not flexible enough sometimes.
Post Reply