Chapter: Principles of Compiler Design : Lexical Analysis

Recognition of Tokens

For this language fragment the lexical analyzer will recognize the keywords if, then, else, as well as the lexemes denoted by relop, id, and num.

RECOGNITION OF TOKENS

Consider the following grammar fragment: stmt → if expr then stmt

| if expr then stmt else stmt | ε

expr → term relop term

| term

term → id | num

where the terminals if , then, else, relop, id and num generate sets of strings given by the following regular definitions:

if → if

then → then

else → else

relop → <|<=|=|<>|>|>=

id → letter(letter|digit)^*

num → digit⁺ (.digit⁺)?(E(+|-)?digit⁺)?

For this language fragment the lexical analyzer will recognize the keywords if, then, else, as well as the lexemes denoted by relop, id, and num. To simplify matters, we assume keywords are reserved; that is, they cannot be used as identifiers.

Transition diagrams

It is a diagrammatic representation to depict the action that will take place when a lexical analyzer is called by the parser to get the next token. It is used to keep track of information about the characters that are seen as the forward pointer scans the input.

Study Material, Lecturing Notes, Assignment, Reference, Wiki description explanation, brief detail

Principles of Compiler Design : Lexical Analysis : Recognition of Tokens |