Lexer
in package
Table of Contents
Methods
- tokenize() : array<string|int, Token>
- Tokenize the provided source code.
- postprocessTokens() : void
- handleInvalidCharacter() : void
- isUnterminatedComment() : bool
Methods
tokenize()
Tokenize the provided source code.
public
tokenize(string $code[, ErrorHandler|null $errorHandler = null ]) : array<string|int, Token>
The token array is in the same format as provided by the PhpToken::tokenize() method in PHP 8.0. The tokens are instances of PhpParser\Token, to abstract over a polyfill implementation in earlier PHP version.
The token array is terminated by a sentinel token with token ID 0. The token array does not discard any tokens (i.e. whitespace and comments are included). The token position attributes are against this token array.
Parameters
- $code : string
-
The source code to tokenize.
- $errorHandler : ErrorHandler|null = null
-
Error handler to use for lexing errors. Defaults to ErrorHandler\Throwing.
Return values
array<string|int, Token> —Tokens
postprocessTokens()
protected
postprocessTokens(array<int, Token> &$tokens, ErrorHandler $errorHandler) : void
Parameters
- $tokens : array<int, Token>
- $errorHandler : ErrorHandler
handleInvalidCharacter()
private
handleInvalidCharacter(Token $token, ErrorHandler $errorHandler) : void
Parameters
- $token : Token
- $errorHandler : ErrorHandler
isUnterminatedComment()
private
isUnterminatedComment(Token $token) : bool
Parameters
- $token : Token