class

Mint::SemanticTokenizer

Inherits Reference < Object

Constants

TOKEN_MAP = {Ast::TypeVariable => TokenType::TypeParameter, Ast::Comment => TokenType::Comment, Ast::RegexpLiteral => TokenType::Regexp, Ast::NumberLiteral => TokenType::Number, Ast::Id => TokenType::Type}

This represents which token types are used for which node.

TOKEN_TYPES = TokenType.names.map!(&.camelcase(lower: true))

This is used by the language server.

Class methods

highlight(path : String, *, html : Bool = false) : String

Highlights the file.

Source
tokenize(ast : Ast) : Array(Item)

Returns the tokenized version of the AST.

Source
tokenize_with_lines(ast : Ast) : Array(Array(Item))

Returns the tokenized version of the AST where the lines contain their respective tokens.

Source

Instance methods

add(from : Parser::Location, to : Parser::Location, type : TokenType)
Source
add(node : Ast::Node, type : TokenType)
Source
cache

We keep a cache of all tokenized nodes to avoid duplications

Source
tokenize(nodes : Array(Ast::Node))
Source
tokenize(node : Ast::CssDefinition)
Source
tokenize(node : Ast::HtmlComponent)
Source
tokenize(node : Ast::HtmlElement)
Source
tokenize(node : Ast::Variable)
Source
tokenize(node : Ast::StringLiteral | Ast::HereDocument)
Source
tokenize(node : Ast::Node | Nil)
Source
tokenize(ast : Ast) : Nil
Source
tokens

This is where the resulting tokens are stored.

Source

Nested types