### Implementing the Lexer Logic in Rust The lexer is a critical component of any interpreter or compiler. Its primary responsibility is to take the input source code as a string and convert it into a sequence of tokens. These tokens represent the basic elements of the language such as keywords, identifiers, literals, operators, and symbols. In this task, we will implement the lexer logic that will tokenize the input source code based on the regular expressions we defined in the previous task. We will create a `Lexer` struct that will handle the tokenization process and produce a stream of tokens. #### Step-by-Step Implementation 1. **Lexer Struct Definition** We start by defining a `Lexer` struct that will hold the input string, the current position in the input, and the current line and column numbers. This struct will be used to maintain the state of the lexer as it processes the input. ```rust /// A struct that represents the lexer state pub struct Lexer { input: String, position: usize, line: usize, column: usize, } impl Lexer { /// Creates a new Lexer instance with the given input string pub fn new(input: String) -> Self { Self { input, position: 0, line: 1, column: 1, } } /// The main method that tokenizes the input and returns tokens pub fn lex(&mut self) -> Token { // Implementation of the lex method } } ``` 2. **Token Definition** Before we implement the lexer logic, we need to define the `Token` enum that will represent the different types of tokens. Each token will have a `TokenKind` and a `String` value. ```rust /// Represents the different types of tokens #[derive(Debug, PartialEq)] pub enum TokenKind { Identifier, Keyword, Integer, Float, Operator, Symbol, Comment, Whitespace, Error, } /// Represents a token produced by the lexer #[derive(Debug)] pub struct Token { pub kind: TokenKind, pub value: String, pub line: usize, pub column: usize, } ``` 3. **Lexer Logic Implementation** The `lex` method is where the core tokenization logic resides. This method will process each character in the input string and determine the type of token it represents. ```rust pub fn lex(&mut self) -> Token { // Skip whitespace and comments self.skip_whitespace_and_comments(); // Check if we've reached the end of the input if self.position >= self.input.len() { return Token { kind: TokenKind::Error, value: "Unexpected end of input".to_string(), line: self.line, column: self.column, }; } // Get the current character let current_char = self.input[self.position..].chars().next().unwrap(); // Check for identifier or keyword if current_char.is_alphabetic() { let mut identifier = String::new(); while self.position < self.input.len() && self.input[self.position..].chars().next().unwrap().is_alphanumeric() { identifier.push(current_char); self.position += 1; current_char = self.input[self.position..].chars().next().unwrap(); } // Determine if the identifier is a keyword let kind = match identifier.as_str() { "fn" => TokenKind::Keyword, "let" => TokenKind::Keyword, _ => TokenKind::Identifier, }; self.column += identifier.len(); return Token { kind, value: identifier, line: self.line, column: self.column - identifier.len(), }; } // Check for number (integer or float) if current_char.is_digit(10) || current_char == '.' { let mut number = String::new(); let mut has_decimal = false; let mut has_exponent = false; while self.position < self.input.len() && (current_char.is_digit(10) || (current_char == '.' && !has_decimal) || (current_char == 'e' && !has_exponent) || (current_char == 'E' && !has_exponent)) { number.push(current_char); self.position += 1; current_char = self.input[self.position..].chars().next().unwrap(); if current_char == '.' { has_decimal = true; } else if current_char == 'e' || current_char == 'E' { has_exponent = true; } } return Token { kind: TokenKind::Float, value: number, line: self.line, column: self.column, }; } // Check for operator or symbol for op in &["+", "-", "*", "/", "%", "=", "==", "!=", ">", "<", ">=", "<="] { if self.input[self.position..].starts_with(op) { let value = op.to_string(); let length = op.len(); self.position += length; self.column += length; return Token { kind: TokenKind::Operator, value, line: self.line, column: self.column - length, }; } } // Handle remaining cases match current_char { '(' => { self.position += 1; self.column += 1; return Token { kind: TokenKind::Symbol, value: "(".to_string(), line: self.line, column: self.column, }; } ')' => { self.position += 1; self.column += 1; return Token { kind: TokenKind::Symbol, value: ")".to_string(), line: self.line, column: self.column, }; } ',' => { self.position += 1; self.column += 1; return Token { kind: TokenKind::Symbol, value: ",".to_string(), line: self.line, column: self.column, }; } ';' => { self.position += 1; self.column += 1; return Token { kind: TokenKind::Symbol, value: ";".to_string(), line: self.line, column: self.column, }; } _ => { // If none of the above, it's an error self.position += 1; self.column += 1; return Token { kind: TokenKind::Error, value: format!("Unexpected character: {}", current_char), line: self.line, column: self.column, }; } } } /// Skips whitespace and comments in the input fn skip_whitespace_and_comments(&mut self) { while self.position < self.input.len() { let current_char = self.input[self.position..].chars().next().unwrap(); // Skip whitespace if current_char.is_whitespace() { self.position += 1; if current_char == '\n' { self.line += 1; self.column = 1; } else { self.column += 1; } continue; } // Skip line comments starting with // if current_char == '/' && self.position + 1 < self.input.len() && self.input[self.position + 1..].chars().next().unwrap() == '/' { // Move to the end of the line let mut i = self.position; while i < self.input.len() && self.input[i..].chars().next().unwrap() != '\n' { i += 1; } self.position = i; self.line += 1; self.column = 1; continue; } // Skip block comments enclosed in /* */ if current_char == '/' && self.position + 1 < self.input.len() && self.input[self.position + 1..].chars().next().unwrap() == '*' { let mut i = self.position + 2; while i < self.input.len() && !(self.input[i..].chars().next().unwrap() == '*' && i + 1 < self.input.len() && self.input[i + 1..].chars().next().unwrap() == '/') { i += 1; } if i < self.input.len() && self.input[i..].chars().next().unwrap() == '*' && i + 1 < self.input.len() && self.input[i + 1..].chars().next().unwrap() == '/' { i += 2; } self.position = i; // Update line and column counts let mut lines = self.input[self.position..].chars().collect::>(); let line_count = lines.iter().filter(|c| **c == '\n').count(); self.line += line_count; self.column = self.position - self.position.lastIndexOf('\n').unwrap_or(self.position); continue; } // If none of the above, break the loop break; } } } ``` 4. **Testing the Lexer** To ensure that the lexer works correctly, we can write some test cases: ```rust #[cfg(test)] mod tests { use super::*; #[test] fn test_lexer() { let input = " fn main() { let x = 10; let y = 20.5; println!(\"Sum: {}\", x + y); } "; let mut lexer = Lexer::new(input.to_string()); let tokens: Vec = loop { let token = lexer.lex(); if token.kind == TokenKind::Error { break vec![token]; } // Add more conditions to collect tokens }; // Assert tokens // ... } } ``` 5. **Handling Line and Column Tracking** The lexer keeps track of the current line and column as it processes the input. This is essential for providing accurate error messages. The `skip_whitespace_and_comments` method updates the line and column counts as it processes whitespace and comments. 6. **Error Handling** The lexer includes basic error handling for unexpected characters. If the lexer encounters a character that doesn't match any token pattern, it returns an error token with the unexpected character and its position. #### Next Steps Now that we have implemented the lexer, the next step is to implement the parser. The parser will take the tokens produced by the lexer and construct an abstract syntax tree (AST) based on the grammar rules we defined earlier. #### Further Reading - [Lexer Implementation in Rust](https://doc.rust-lang.org/book/ch08-00-parsing-command-line-arguments.html) - [Tokenization in Programming Languages](https://en.wikipedia.org/wiki/Lexical_analysis) - [Rust String Manipulation](https://doc.rust-lang.org/std/string/) This implementation provides a solid foundation for the lexer component of our interpreter. It handles various token types, skips whitespace and comments, and includes basic error handling.

Rust Programming Language Interpreter

Learning Outcomes