Documentation

Tokens extends SplFixedArray
in package

Collection of code tokens.

Its role is to provide the ability to manage collection and navigate through it.

As a token prototype you should understand a single element generated by token_get_all.

Tags
author

Dariusz Rumiński dariusz.ruminski@gmail.com

extends

\SplFixedArray<Token>

final

Table of Contents

Constants

BLOCK_TYPE_ARRAY_INDEX_CURLY_BRACE  = 7
BLOCK_TYPE_ARRAY_SQUARE_BRACE  = 4
BLOCK_TYPE_ATTRIBUTE  = 11
BLOCK_TYPE_BRACE_CLASS_INSTANTIATION  = 10
BLOCK_TYPE_COMPLEX_STRING_VARIABLE  = 14
BLOCK_TYPE_CURLY_BRACE  = 2
BLOCK_TYPE_DESTRUCTURING_SQUARE_BRACE  = 9
BLOCK_TYPE_DISJUNCTIVE_NORMAL_FORM_TYPE_PARENTHESIS  = 12
BLOCK_TYPE_DYNAMIC_CLASS_CONSTANT_FETCH_CURLY_BRACE  = 13
BLOCK_TYPE_DYNAMIC_PROP_BRACE  = 5
BLOCK_TYPE_DYNAMIC_VAR_BRACE  = 6
BLOCK_TYPE_GROUP_IMPORT_BRACE  = 8
BLOCK_TYPE_INDEX_SQUARE_BRACE  = 3
BLOCK_TYPE_PARENTHESIS_BRACE  = 1

Properties

$blockEndCache  : array<int, int>
Cache of block ends. Any change in collection will invalidate it.
$blockStartCache  : array<int, int>
Cache of block starts. Any change in collection will invalidate it.
$cache  : array<non-empty-string, self>
Static class cache.
$changed  : bool
Flag is collection was changed.
$codeHash  : non-empty-string|null
A MD5 hash of the code string.
$foundTokenKinds  : array<int|non-empty-string, int>
Set of found token kinds.
$namespaceDeclarations  : null|array<int, NamespaceAnalysis>

Methods

__clone()  : mixed
Clone tokens collection.
clearAt()  : void
clearCache()  : void
Clear cache - one position or all of them.
clearChanged()  : void
Clear internal flag if collection was changed and flag for all collection's items.
clearEmptyTokens()  : void
Clear empty tokens.
clearRange()  : void
Clear tokens in the given range.
clearTokenAndMergeSurroundingWhitespace()  : void
countTokenKind()  : int
detectBlockType()  : null|array{type: self::BLOCK_TYPE_*, isStart: bool}
Detect type of block.
ensureWhitespaceAtIndex()  : bool
Ensure that on given index is a whitespace with given kind.
findBlockEnd()  : int<0, max>
findBlockStart()  : int<0, max>
findGivenKind()  : mixed
findSequence()  : array<string|int, mixed>|null
Find a sequence of meaningful tokens and returns the array of their locations.
fromArray()  : self
Create token collection from array.
fromCode()  : self
Create token collection directly from code.
generateCode()  : string
generatePartialCode()  : string
Generate code from tokens between given indices.
getBlockEdgeDefinitions()  : array<string|int, mixed>
getCodeHash()  : string
Get hash of code.
getMeaningfulTokenSibling()  : int|null
Get index for closest sibling token that is not a whitespace, comment or attribute.
getNextMeaningfulToken()  : int|null
Get index for closest next token that is not a whitespace or comment.
getNextNonWhitespace()  : int|null
Get index for closest next token which is non whitespace.
getNextTokenOfKind()  : int|null
Get index for closest next token of given kind.
getNonEmptySibling()  : int|null
Get index for closest sibling token which is not empty.
getNonWhitespaceSibling()  : int|null
Get index for closest sibling token which is non whitespace.
getPrevMeaningfulToken()  : int|null
Get index for closest previous token that is not a whitespace or comment.
getPrevNonWhitespace()  : int|null
Get index for closest previous token which is non whitespace.
getPrevTokenOfKind()  : int|null
Get index for closest previous token of given kind.
getTokenNotOfKindSibling()  : int|null
Get index for closest sibling token not of given kind.
getTokenNotOfKindsSibling()  : int|null
Get index for closest sibling token not of given kind.
getTokenOfKindSibling()  : int|null
Get index for closest sibling token of given kind.
hasAlternativeSyntax()  : bool
insertAt()  : void
Insert instances of Token inside collection.
isAllTokenKindsFound()  : bool
Check if all token kinds given as argument are found.
isAnyTokenKindsFound()  : bool
Check if any token kind given as argument is found.
isChanged()  : bool
Check if collection was change: collection itself (like insert new tokens) or any of collection's elements.
isEmptyAt()  : bool
isMonolithicPhp()  : bool
Checks for monolithic PHP code.
isPartialCodeMultiline()  : bool
isTokenKindFound()  : bool
Check if token kind given as argument is found.
offsetGet()  : Token
offsetSet()  : void
Set collection item.
offsetUnset()  : void
Unset collection item.
overrideRange()  : void
Override tokens at given range.
removeLeadingWhitespace()  : void
removeTrailingWhitespace()  : void
setCode()  : void
Set code. Clear all current content and replace it by new Token items generated from code directly.
setSize()  : bool
Set new size of collection.
toJson()  : string
calculateCodeHash()  : non-empty-string
Calculate hash for code.
changeCodeHash()  : void
Change code hash.
extractTokenKind()  : int|non-empty-string
findOppositeBlockEdge()  : int<0, max>
getCache()  : self
Get cache value for given key.
getTokenNotOfKind()  : int|null
hasCache()  : bool
Check if given key exists in cache.
isKeyCaseSensitive()  : bool
A helper method used to find out whether a certain input token has to be case-sensitively matched.
registerFoundToken()  : void
Register token as found.
removeWhitespaceSafely()  : void
setCache()  : void
unregisterFoundToken()  : void
Unregister token as not found.

Constants

BLOCK_TYPE_ARRAY_INDEX_CURLY_BRACE

public mixed BLOCK_TYPE_ARRAY_INDEX_CURLY_BRACE = 7

BLOCK_TYPE_ARRAY_SQUARE_BRACE

public mixed BLOCK_TYPE_ARRAY_SQUARE_BRACE = 4

BLOCK_TYPE_ATTRIBUTE

public mixed BLOCK_TYPE_ATTRIBUTE = 11

BLOCK_TYPE_BRACE_CLASS_INSTANTIATION

public mixed BLOCK_TYPE_BRACE_CLASS_INSTANTIATION = 10

BLOCK_TYPE_COMPLEX_STRING_VARIABLE

public mixed BLOCK_TYPE_COMPLEX_STRING_VARIABLE = 14

BLOCK_TYPE_CURLY_BRACE

public mixed BLOCK_TYPE_CURLY_BRACE = 2

BLOCK_TYPE_DESTRUCTURING_SQUARE_BRACE

public mixed BLOCK_TYPE_DESTRUCTURING_SQUARE_BRACE = 9

BLOCK_TYPE_DISJUNCTIVE_NORMAL_FORM_TYPE_PARENTHESIS

public mixed BLOCK_TYPE_DISJUNCTIVE_NORMAL_FORM_TYPE_PARENTHESIS = 12

BLOCK_TYPE_DYNAMIC_CLASS_CONSTANT_FETCH_CURLY_BRACE

public mixed BLOCK_TYPE_DYNAMIC_CLASS_CONSTANT_FETCH_CURLY_BRACE = 13

BLOCK_TYPE_DYNAMIC_PROP_BRACE

public mixed BLOCK_TYPE_DYNAMIC_PROP_BRACE = 5

BLOCK_TYPE_DYNAMIC_VAR_BRACE

public mixed BLOCK_TYPE_DYNAMIC_VAR_BRACE = 6

BLOCK_TYPE_GROUP_IMPORT_BRACE

public mixed BLOCK_TYPE_GROUP_IMPORT_BRACE = 8

BLOCK_TYPE_INDEX_SQUARE_BRACE

public mixed BLOCK_TYPE_INDEX_SQUARE_BRACE = 3

BLOCK_TYPE_PARENTHESIS_BRACE

public mixed BLOCK_TYPE_PARENTHESIS_BRACE = 1

Properties

$blockEndCache

Cache of block ends. Any change in collection will invalidate it.

private array<int, int> $blockEndCache = []

$blockStartCache

Cache of block starts. Any change in collection will invalidate it.

private array<int, int> $blockStartCache = []

$cache

Static class cache.

private static array<non-empty-string, self> $cache = []

$changed

Flag is collection was changed.

private bool $changed = false

It doesn't know about change of collection's items. To check it run isChanged method.

$codeHash

A MD5 hash of the code string.

private non-empty-string|null $codeHash = null

$foundTokenKinds

Set of found token kinds.

private array<int|non-empty-string, int> $foundTokenKinds = []

When the token kind is present in this set it means that given token kind was ever seen inside the collection (but may not be part of it any longer). The key is token kind and the value is always true.

$namespaceDeclarations

private null|array<int, NamespaceAnalysis> $namespaceDeclarations = null

Methods

__clone()

Clone tokens collection.

public __clone() : mixed

clearAt()

public clearAt(int $index) : void
Parameters
$index : int

clearCache()

Clear cache - one position or all of them.

public static clearCache([null|non-empty-string $key = null ]) : void
Parameters
$key : null|non-empty-string = null

position to clear, when null clear all

clearChanged()

Clear internal flag if collection was changed and flag for all collection's items.

public clearChanged() : void

clearEmptyTokens()

Clear empty tokens.

public clearEmptyTokens() : void

Empty tokens can occur e.g. after calling clear on item of collection.

clearRange()

Clear tokens in the given range.

public clearRange(int $indexStart, int $indexEnd) : void
Parameters
$indexStart : int
$indexEnd : int

clearTokenAndMergeSurroundingWhitespace()

public clearTokenAndMergeSurroundingWhitespace(int $index) : void
Parameters
$index : int

countTokenKind()

public countTokenKind(int|string $tokenKind) : int
Parameters
$tokenKind : int|string
Return values
int

detectBlockType()

Detect type of block.

public static detectBlockType(Token $token) : null|array{type: self::BLOCK_TYPE_*, isStart: bool}
Parameters
$token : Token
Return values
null|array{type: self::BLOCK_TYPE_*, isStart: bool}

ensureWhitespaceAtIndex()

Ensure that on given index is a whitespace with given kind.

public ensureWhitespaceAtIndex(int $index, int $indexOffset, string $whitespace) : bool

If there is a whitespace then it's content will be modified. If not - the new Token will be added.

Parameters
$index : int

index

$indexOffset : int

index offset for Token insertion

$whitespace : string

whitespace to set

Return values
bool

if new Token was added

findBlockEnd()

public findBlockEnd(self::BLOCK_TYPE_* $type, int $searchIndex) : int<0, max>
Parameters
$type : self::BLOCK_TYPE_*

type of block

$searchIndex : int

index of opening brace

Return values
int<0, max>

index of closing brace

findBlockStart()

public findBlockStart(self::BLOCK_TYPE_* $type, int $searchIndex) : int<0, max>
Parameters
$type : self::BLOCK_TYPE_*

type of block

$searchIndex : int

index of closing brace

Return values
int<0, max>

index of opening brace

findGivenKind()

public findGivenKind(int|array<int, int> $possibleKind[, int $start = 0 ][, null|int $end = null ]) : mixed
Parameters
$possibleKind : int|array<int, int>

kind or array of kinds

$start : int = 0

optional offset

$end : null|int = null

optional limit

findSequence()

Find a sequence of meaningful tokens and returns the array of their locations.

public findSequence(array<int, array{0: int, 1?: string}|string|Token$sequence[, int $start = 0 ][, null|int $end = null ][, array<int, bool>|bool $caseSensitive = true ]) : array<string|int, mixed>|null
Parameters
$sequence : array<int, array{0: int, 1?: string}|string|Token>

an array of token (kinds)

$start : int = 0

start index, defaulting to the start of the file

$end : null|int = null

end index, defaulting to the end of the file

$caseSensitive : array<int, bool>|bool = true

global case sensitiveness or a list of booleans, whose keys should match the ones used in $sequence. If any is missing, the default case-sensitive comparison is used

Return values
array<string|int, mixed>|null

fromArray()

Create token collection from array.

public static fromArray(array<int, Token$array[, bool|null $saveIndices = null ]) : self
Parameters
$array : array<int, Token>

the array to import

$saveIndices : bool|null = null

save the numeric indices used in the original array, default is yes

Return values
self

fromCode()

Create token collection directly from code.

public static fromCode(string $code) : self
Parameters
$code : string

PHP code

Return values
self

generateCode()

public generateCode() : string
Return values
string

generatePartialCode()

Generate code from tokens between given indices.

public generatePartialCode(int $start, int $end) : string
Parameters
$start : int

start index

$end : int

end index

Return values
string

getBlockEdgeDefinitions()

public static getBlockEdgeDefinitions() : array<string|int, mixed>
Return values
array<string|int, mixed>

getCodeHash()

Get hash of code.

public getCodeHash() : string
Return values
string

getMeaningfulTokenSibling()

Get index for closest sibling token that is not a whitespace, comment or attribute.

public getMeaningfulTokenSibling(int $index, -1|1 $direction) : int|null
Parameters
$index : int

token index

$direction : -1|1
Return values
int|null

getNextMeaningfulToken()

Get index for closest next token that is not a whitespace or comment.

public getNextMeaningfulToken(int $index) : int|null
Parameters
$index : int

token index

Return values
int|null

getNextNonWhitespace()

Get index for closest next token which is non whitespace.

public getNextNonWhitespace(int $index[, null|string $whitespaces = null ]) : int|null

This method is shorthand for getNonWhitespaceSibling method.

Parameters
$index : int

token index

$whitespaces : null|string = null

whitespaces characters for Token::isWhitespace

Return values
int|null

getNextTokenOfKind()

Get index for closest next token of given kind.

public getNextTokenOfKind(int $index[, array<int, array{: int}|string|Token$tokens = [] ][, bool $caseSensitive = true ]) : int|null

This method is shorthand for getTokenOfKindSibling method.

Parameters
$index : int

token index

$tokens : array<int, array{: int}|string|Token> = []

possible tokens

$caseSensitive : bool = true

perform a case sensitive comparison

Return values
int|null

getNonEmptySibling()

Get index for closest sibling token which is not empty.

public getNonEmptySibling(int $index, -1|1 $direction) : int|null
Parameters
$index : int

token index

$direction : -1|1
Return values
int|null

getNonWhitespaceSibling()

Get index for closest sibling token which is non whitespace.

public getNonWhitespaceSibling(int $index, -1|1 $direction[, null|string $whitespaces = null ]) : int|null
Parameters
$index : int

token index

$direction : -1|1
$whitespaces : null|string = null

whitespaces characters for Token::isWhitespace

Return values
int|null

getPrevMeaningfulToken()

Get index for closest previous token that is not a whitespace or comment.

public getPrevMeaningfulToken(int $index) : int|null
Parameters
$index : int

token index

Return values
int|null

getPrevNonWhitespace()

Get index for closest previous token which is non whitespace.

public getPrevNonWhitespace(int $index[, null|string $whitespaces = null ]) : int|null

This method is shorthand for getNonWhitespaceSibling method.

Parameters
$index : int

token index

$whitespaces : null|string = null

whitespaces characters for Token::isWhitespace

Return values
int|null

getPrevTokenOfKind()

Get index for closest previous token of given kind.

public getPrevTokenOfKind(int $index[, array<int, array{: int}|string|Token$tokens = [] ][, bool $caseSensitive = true ]) : int|null

This method is shorthand for getTokenOfKindSibling method.

Parameters
$index : int

token index

$tokens : array<int, array{: int}|string|Token> = []

possible tokens

$caseSensitive : bool = true

perform a case sensitive comparison

Return values
int|null

getTokenNotOfKindSibling()

Get index for closest sibling token not of given kind.

public getTokenNotOfKindSibling(int $index, -1|1 $direction[, array<int, array{: int}|string|Token$tokens = [] ]) : int|null
Parameters
$index : int

token index

$direction : -1|1
$tokens : array<int, array{: int}|string|Token> = []

possible tokens

Return values
int|null

getTokenNotOfKindsSibling()

Get index for closest sibling token not of given kind.

public getTokenNotOfKindsSibling(int $index, -1|1 $direction[, array<int, int> $kinds = [] ]) : int|null
Parameters
$index : int

token index

$direction : -1|1
$kinds : array<int, int> = []

possible tokens kinds

Return values
int|null

getTokenOfKindSibling()

Get index for closest sibling token of given kind.

public getTokenOfKindSibling(int $index, -1|1 $direction[, array<int, array{: int}|string|Token$tokens = [] ][, bool $caseSensitive = true ]) : int|null
Parameters
$index : int

token index

$direction : -1|1
$tokens : array<int, array{: int}|string|Token> = []

possible tokens

$caseSensitive : bool = true

perform a case sensitive comparison

Return values
int|null

hasAlternativeSyntax()

public hasAlternativeSyntax() : bool
Return values
bool

insertAt()

Insert instances of Token inside collection.

public insertAt(int $index, array<int, Token>|Token|Tokens $items) : void
Parameters
$index : int

start inserting index

$items : array<int, Token>|Token|Tokens

instances of Token to insert

isAllTokenKindsFound()

Check if all token kinds given as argument are found.

public isAllTokenKindsFound(array<int, int|string> $tokenKinds) : bool
Parameters
$tokenKinds : array<int, int|string>
Return values
bool

isAnyTokenKindsFound()

Check if any token kind given as argument is found.

public isAnyTokenKindsFound(array<int, int|string> $tokenKinds) : bool
Parameters
$tokenKinds : array<int, int|string>
Return values
bool

isChanged()

Check if collection was change: collection itself (like insert new tokens) or any of collection's elements.

public isChanged() : bool
Return values
bool

isEmptyAt()

public isEmptyAt(int $index) : bool
Parameters
$index : int
Return values
bool

isMonolithicPhp()

Checks for monolithic PHP code.

public isMonolithicPhp() : bool

Checks that the code is pure PHP code, in a single code block, starting with an open tag.

Return values
bool

isPartialCodeMultiline()

public isPartialCodeMultiline(int $start, int $end) : bool
Parameters
$start : int

start index

$end : int

end index

Return values
bool

isTokenKindFound()

Check if token kind given as argument is found.

public isTokenKindFound(int|string $tokenKind) : bool
Parameters
$tokenKind : int|string
Return values
bool

offsetGet()

public offsetGet([mixed $offset = ]) : Token
Parameters
$offset : mixed =
Return values
Token

offsetSet()

Set collection item.

public offsetSet(int $index, Token $newval) : void

Warning! $newval must not be typehinted to be compatible with ArrayAccess::offsetSet method.

Parameters
$index : int
$newval : Token

offsetUnset()

Unset collection item.

public offsetUnset(int $index) : void
Parameters
$index : int

overrideRange()

Override tokens at given range.

public overrideRange(int $indexStart, int $indexEnd, array<int, Token>|Tokens $items) : void
Parameters
$indexStart : int

start overriding index

$indexEnd : int

end overriding index

$items : array<int, Token>|Tokens

tokens to insert

removeLeadingWhitespace()

public removeLeadingWhitespace(int $index[, null|string $whitespaces = null ]) : void
Parameters
$index : int
$whitespaces : null|string = null

optional whitespaces characters for Token::isWhitespace

removeTrailingWhitespace()

public removeTrailingWhitespace(int $index[, null|string $whitespaces = null ]) : void
Parameters
$index : int
$whitespaces : null|string = null

optional whitespaces characters for Token::isWhitespace

setCode()

Set code. Clear all current content and replace it by new Token items generated from code directly.

public setCode(string $code) : void
Parameters
$code : string

PHP code

setSize()

Set new size of collection.

public setSize(int $size) : bool
Parameters
$size : int
Return values
bool

toJson()

public toJson() : string
Return values
string

calculateCodeHash()

Calculate hash for code.

private static calculateCodeHash(string $code) : non-empty-string
Parameters
$code : string
Return values
non-empty-string

changeCodeHash()

Change code hash.

private changeCodeHash(non-empty-string $codeHash) : void

Remove old cache and set new one.

Parameters
$codeHash : non-empty-string

new code hash

extractTokenKind()

private extractTokenKind(array{: int}|string|Token $token) : int|non-empty-string
Parameters
$token : array{: int}|string|Token

token prototype

Return values
int|non-empty-string

findOppositeBlockEdge()

private findOppositeBlockEdge(self::BLOCK_TYPE_* $type, int $searchIndex, bool $findEnd) : int<0, max>
Parameters
$type : self::BLOCK_TYPE_*

type of block

$searchIndex : int

index of starting brace

$findEnd : bool

if method should find block's end or start

Return values
int<0, max>

index of opposite brace

getCache()

Get cache value for given key.

private static getCache(non-empty-string $key) : self
Parameters
$key : non-empty-string

item key

Return values
self

getTokenNotOfKind()

private getTokenNotOfKind(int $index, -1|1 $direction, callable(int): bool $filter) : int|null
Parameters
$index : int

token index

$direction : -1|1
$filter : callable(int): bool
Return values
int|null

hasCache()

Check if given key exists in cache.

private static hasCache(non-empty-string $key) : bool
Parameters
$key : non-empty-string

item key

Return values
bool

isKeyCaseSensitive()

A helper method used to find out whether a certain input token has to be case-sensitively matched.

private static isKeyCaseSensitive(array<int, bool>|bool $caseSensitive, int $key) : bool
Parameters
$caseSensitive : array<int, bool>|bool

global case sensitiveness or an array of booleans, whose keys should match the ones used in $sequence. If any is missing, the default case-sensitive comparison is used

$key : int

the key of the token that has to be looked up

Return values
bool

registerFoundToken()

Register token as found.

private registerFoundToken(array{: int}|string|Token $token) : void
Parameters
$token : array{: int}|string|Token

token prototype

removeWhitespaceSafely()

private removeWhitespaceSafely(int $index, -1|1 $direction[, string|null $whitespaces = null ]) : void
Parameters
$index : int
$direction : -1|1
$whitespaces : string|null = null

setCache()

private static setCache(non-empty-string $key, Tokens $value) : void
Parameters
$key : non-empty-string

item key

$value : Tokens

item value

unregisterFoundToken()

Unregister token as not found.

private unregisterFoundToken(array{: int}|string|Token $token) : void
Parameters
$token : array{: int}|string|Token

token prototype


        
On this page

Search results