Fun with Formal Language

Tokenization

It’s been a while since I’ve updated you all on my progress regarding my final year project, in which I’m building an integrated development environment for the PHP scripting language.

At this stage I’m researching different methodologies and algorithms for what I want to achieve in my first prototype, as well as different methodologies for the actual software engineering experience in, and of, itself.

Whilst doing this I’ve found that writing code for the algorithms I’ve been researching has been super useful in determining the ease of implementation, and actually understanding the ideas involved themselves. In the above screenshot you can see an example of one algorithm I’ve implemented to tokenize my code.

Tokenization is the process of identifying the name and value of each token in a piece of code. A token is just a set of symbols, such as a $variableName or a keyword such as true — or even a simple code delimiter such as ; or { or }.

The above screenshot shows the code as highlighted by Visual Studio and as highlighted by my program, some of the formatting is a tad different than you would expect, but this is a very early version of the program and won’t be used as part of my final product — it’s just to help me visualize how an algorithm works.

Danny

Advertisements

Tags: , , , , , , , ,

2 responses to “Fun with Formal Language”

  1. cameronwilby says :

    You’re going to love Languages and Compilers if you’re taking that in your third year, you use tools like YACC and Bison to do this sort of thing.

    Another term for Tokenization is “Lexical Analysis”. I think the latter sounds cooler ๐Ÿ™‚

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s