class

Xerp::Tokenize::Tokenizer

Inherits Reference < Object

Main tokenizer class.

Constants

BLOCK_COMMENT_END = /\*\/|-\}|=end|"""/
BLOCK_COMMENT_START = /\/\*|\{-|=begin|"""/
IDENT_PATTERN = /[a-zA-Z_][a-zA-Z0-9_]*/

Regex patterns for token extraction

LINE_COMMENT_PATTERNS = [/(?:#|\/\/)(.*)$/]

Comment patterns

NUMBER_PATTERN = /\b\d+(?:\.\d+)?(?:[eE][+-]?\d+)?\b/
STRING_DQ_PATTERN = /"(?:[^"\\]|\\.)*"/
STRING_SQ_PATTERN = /'(?:[^'\\]|\\.)*'/

Constructors

new(max_token_len : Int32 = MAX_TOKEN_LEN)
Source

Instance methods

tokenize(lines : Array(String)) : TokenizeResult

Tokenizes an array of lines. Lines are 0-indexed in the array, but TokenOcc.line is 1-indexed.

Source