Whitespace and comment tokens attached to this token.
All trivia tokens must have the text property set to the text with
which they identify with. This means you can map all trivia tokens to
their .text property and join them together to get the source code back
without any loss of information.
Trivia is only included when calling getTokensForParser. When iterating
over DLexer all tokens will be in their raw form and none will be
converted to trivia.
Note: in the future you might need to explicitly pass
WhitespaceBehavior.include (or keep the default) as getTokensForParser
currently overrides it to include.
Whitespace and comment tokens attached to this token.
All trivia tokens must have the text property set to the text with which they identify with. This means you can map all trivia tokens to their .text property and join them together to get the source code back without any loss of information.
Trivia is only included when calling getTokensForParser. When iterating over DLexer all tokens will be in their raw form and none will be converted to trivia.
Note: in the future you might need to explicitly pass WhitespaceBehavior.include (or keep the default) as getTokensForParser currently overrides it to include.
Contains: comment, whitespace, specialTokenSequence