public class Tokenizer
- Object
- Tokenizer
ImplementsSyntaxHighlighter
A small, allocation light, stateful lexer for the pure code editor. It tokenizes a single line at a time given the lexer state carried over from the previous line, and returns the state to carry into the next line. This makes rehighlighting incremental: after an edit only the changed line and any following lines whose entry state changed need to be retokenized.
Only the spans that carry color (keyword, string, comment, number) are emitted; the gaps between them are drawn in the default text color.
Fields
public static final int KEYWORD = 1 | Token kind: a language keyword. |
public static final int STRING = 2 | Token kind: a string or character literal. |
public static final int COMMENT = 3 | Token kind: a comment. |
public static final int NUMBER = 4 | Token kind: a numeric literal. |
public static final int TYPE = 5 | Token kind: an XML/HTML tag or a declared type. |
public static final int PROPERTY = 6 | Token kind: an attribute name or object property. |
public static final int STATE_NORMAL = 0 | Lexer state: normal code. |
public static final int STATE_BLOCK_COMMENT = 1 | Lexer state: inside a block comment that opened on a previous line. |
public static final int STATE_TEMPLATE = 2 | Lexer state: inside a backtick template string that opened on a previous line. |
public static final int STATE_XML_COMMENT = 3 | Lexer state: inside an XML/HTML comment. |
public static final int STATE_TRIPLE_SINGLE = 4 | Lexer state: inside a Python triple-single-quoted string. |
public static final int STATE_TRIPLE_DOUBLE = 5 | Lexer state: inside a Python triple-double-quoted string. |
public static final int STATE_CSS_DECLARATION = 6 | Lexer state: inside a CSS declaration block. |
public static final int STATE_CSS_COMMENT_DECLARATION = 7 | Lexer state: inside a CSS comment opened from a declaration block. |
Constructors
public Tokenizer(LanguageDef def) | Creates a tokenizer for the given language. |
Methods
public SyntaxHighlightResult tokenize(String line, int startState) | Tokenizes one line. |
Inherited methods
Field details
KEYWORD
public static final int KEYWORD = 1Token kind: a language keyword.
STRING
public static final int STRING = 2Token kind: a string or character literal.
COMMENT
public static final int COMMENT = 3Token kind: a comment.
NUMBER
public static final int NUMBER = 4Token kind: a numeric literal.
TYPE
public static final int TYPE = 5Token kind: an XML/HTML tag or a declared type.
PROPERTY
public static final int PROPERTY = 6Token kind: an attribute name or object property.
STATE_NORMAL
public static final int STATE_NORMAL = 0Lexer state: normal code.
STATE_BLOCK_COMMENT
public static final int STATE_BLOCK_COMMENT = 1Lexer state: inside a block comment that opened on a previous line.
STATE_TEMPLATE
public static final int STATE_TEMPLATE = 2Lexer state: inside a backtick template string that opened on a previous line.
STATE_XML_COMMENT
public static final int STATE_XML_COMMENT = 3Lexer state: inside an XML/HTML comment.
STATE_TRIPLE_SINGLE
public static final int STATE_TRIPLE_SINGLE = 4Lexer state: inside a Python triple-single-quoted string.
STATE_TRIPLE_DOUBLE
public static final int STATE_TRIPLE_DOUBLE = 5Lexer state: inside a Python triple-double-quoted string.
STATE_CSS_DECLARATION
public static final int STATE_CSS_DECLARATION = 6Lexer state: inside a CSS declaration block.
STATE_CSS_COMMENT_DECLARATION
public static final int STATE_CSS_COMMENT_DECLARATION = 7Lexer state: inside a CSS comment opened from a declaration block.
Constructor details
Tokenizer
public Tokenizer(LanguageDef def)Creates a tokenizer for the given language.
Method details
tokenize
public SyntaxHighlightResult tokenize(String line, int startState)Tokenizes one line.
Parameters
lineString- the line text (without its trailing newline)
startStateint- the lexer state carried from the previous line
Returns
the colored spans and exit state