If token_get_all() is used, we still need to correctly distinguish
LNUMBER vs DNUMBER here for backwards compatibility.

<?php followed by EOF is valid since PHP 7.4.

Around a quarter of all strings in array tokens would have a string that's one
character long (e.g. ` `, `\`, `1`)

For parsing a large number of php files,
The memory increase dropped from 378374248 to 369535688 (2.5%)

Closes GH-4753.

To clean up the mess here a bit, check for invalid octal digits
with an explicit loop instead of mixing this into the string to
number conversion.

Also clean up some type usage.

As these do not operate on expressions, precedence is meaningless
for them.

This is an annoying edge-case for canonicalization.

Avoid having holes in the token stream which are annoying and
inefficient to reconstruct on the consumer side.

If <?php occurs without required trailing whitespace, we should keep
it as part of a single T_INLINE_HTML region.