Parsing 101

Source-code only - please, don't post questions here.
notthecheatr
Posts: 1759
Joined: May 23, 2007 21:52
Location: Cut Bank, MT
Contact:

Parsing 101

Postby notthecheatr » Dec 14, 2007 20:18

I wrote a simple object called words_list which takes a string and splits it into words. Its main use would probably be parsing, since for example it'll take something like this:

Code: Select all

Dim a As Integer


and give you the words

Code: Select all

Dim

Code: Select all

a

Code: Select all

As

Code: Select all

Integer


It leaves line comments (since it works line-by-line, multi-line comments can't be supported) and whitespace out of the words it produces, and strings inside quotation marks are all one word.

May be useful to someone anyways, here's the ZIP with the code, example, and readme: http://notthecheatr.phatcode.net/downlo ... s_list.zip
1000101
Posts: 2556
Joined: Jun 13, 2005 23:14
Location: SK, Canada

Postby 1000101 » Dec 15, 2007 1:56

I hate to be a buzz-kill, but...strtok()
duke4e
Posts: 717
Joined: Dec 04, 2005 0:16
Location: Varazdin, Croatia, Europe
Contact:

Postby duke4e » Dec 15, 2007 3:04

strtok()
what the hell is that?
Peter
Posts: 66
Joined: May 29, 2006 22:16

Postby Peter » Dec 15, 2007 3:21

strtok = String Tokenize, basically it breaks down a string into the component pieces i.e. words. It pretty much does what your trying to do with your code. That said, I think your code is great, it gives you the basis to make much more complex stuff that is (and I think this is important sometimes) completely independent of outside calls.

Back to lurkerdom for me now.
MichaelW
Posts: 3500
Joined: May 16, 2006 22:34
Location: USA

Postby MichaelW » Dec 15, 2007 12:22

For general-purpose use your whitespace should probably include tab characters. Strtok is fast and easy to use, but a byte scanner is probably a better choice where you need the length of each token and/or where you need to detect errors such as unmatched quotes.

Finding Tokens in a String
notthecheatr
Posts: 1759
Joined: May 23, 2007 21:52
Location: Cut Bank, MT
Contact:

Postby notthecheatr » Dec 16, 2007 1:16

Now I know why people complain.

Actually, it's mostly a learning thing. Like I said, the code's pretty much a hack, but it might be useful and that's why I wrote it. I don't really care if there's something else that already does that, as using that other tool won't really teach me anything. Sure, it's good to use other tools, but there's nothing wrong with writing your own thing, is there?


P.S. This is an object. I could write an object based on strtok() but in any case it doesn't matter... my point is that using an object makes it a much more natural interface.
Peter
Posts: 66
Joined: May 29, 2006 22:16

Postby Peter » Dec 16, 2007 2:43

Sure, it's good to use other tools, but there's nothing wrong with writing your own thing, is there?

No, there's nothing wrong with it. Especially when your learning to program I feel it is important to do as much coding as possible. When you have to code to a deadline it is right nice to know that someone else did some of the work for you already though. That all said.... when I was a wee lad all excited because my aunt showed me a few commands on my parents BBC Micro I spent several days writing a way to perform a a set number of times. I was so proud of it and wanted to show it off. When I did I was shocked when told about this "FOR...NEXT" thing did the same blasted thing!
yetifoot
Posts: 1710
Joined: Sep 11, 2005 7:08
Location: England
Contact:

Postby yetifoot » Dec 17, 2007 15:42

Peter wrote:
Sure, it's good to use other tools, but there's nothing wrong with writing your own thing, is there?

No, there's nothing wrong with it. Especially when your learning to program I feel it is important to do as much coding as possible.


I also think it's good to learn how it's done, at least the first time. I prefer to hand write the lexer/scanner part, rather than use strtok, as strtok is not flexible enough sometimes.

Return to “Tips and Tricks”

Who is online

Users browsing this forum: Baidu [Spider] and 1 guest