lexer API

Attached is a patch that adds a public API for the lowest layers of
lexing.  This is meant to provide a solid foundation for the parsing that
Devel::Declare and similar modules do, and it complements the pluggable
keyword mechanism.  The API consists of some existing variables combined
with some new functions, all marked as experimental (which making them
public certainly is).

1 files changed, 14 insertions, 0 deletions

diff --git a/pod/perldiag.pod b/pod/perldiag.pod
index d38244e4ef..c46927c18b 100644
--- a/pod/perldiag.pod
+++ b/pod/perldiag.pod
@@ -2260,6 +2260,20 @@ effective uids or gids failed.
 length/code combination tried to obtain more data. This results in
 an undefined value for the length. See L<perlfunc/pack>.
 
+=item Lexing code attempted to stuff non-Latin-1 character into Latin-1 input
+
+(F) An extension is attempting to insert text into the current parse
+(using L<lex_stuff_pvn_flags|perlapi/lex_stuff_pvn_flags> or similar), but
+tried to insert a character that couldn't be part of the current input.
+This is an inherent pitfall of the stuffing mechanism, and one of the
+reasons to avoid it.  Where it is necessary to stuff, stuffing only
+plain ASCII is recommended.
+
+=item Lexing code internal error (%s)
+
+(F) Lexing code supplied by an extension violated the lexer's API in a
+detectable way.
+
 =item listen() on closed socket %s
 
 (W closed) You tried to do a listen on a closed socket.  Did you forget