Make identifiers str (not str8) objects throughout.

This affects the parser, various object implementations,
and all places that put identifiers into C string literals.

In testing, a number of crashes occurred as code would
fail when the recursion limit was reached (such as the
Unicode interning dictionary having key/value pairs where
key is not value). To solve these, I added an overflowed
flag, which allows for 50 more recursions after the
limit was reached and the exception was raised, and
a recursion_critical flag, which indicates that recursion
absolutely must be allowed, i.e. that a certain call
must not cause a stack overflow exception.

There are still some places where both str and str8 are
accepted as identifiers; these should eventually be
removed.

1 files changed, 12 insertions, 1 deletions

diff --git a/Include/ceval.h b/Include/ceval.h
index 15b29c6615..c9c59eb17c 100644
--- a/Include/ceval.h
+++ b/Include/ceval.h
@@ -50,7 +50,10 @@ PyAPI_FUNC(int) Py_GetRecursionLimit(void);
 	    (_Py_MakeRecCheck(PyThreadState_GET()->recursion_depth) &&  \
 	     _Py_CheckRecursiveCall(where))
 #define Py_LeaveRecursiveCall()				\
-	    (--PyThreadState_GET()->recursion_depth)
+    do{ if((--PyThreadState_GET()->recursion_depth) <   \
+	   _Py_CheckRecursionLimit - 50);               \
+	  PyThreadState_GET()->overflowed = 0;          \
+    } while(0)
 PyAPI_FUNC(int) _Py_CheckRecursiveCall(char *where);
 PyAPI_DATA(int) _Py_CheckRecursionLimit;
 #ifdef USE_STACKCHECK
@@ -59,6 +62,14 @@ PyAPI_DATA(int) _Py_CheckRecursionLimit;
 #  define _Py_MakeRecCheck(x)  (++(x) > _Py_CheckRecursionLimit)
 #endif
 
+#define Py_ALLOW_RECURSION \
+  do { unsigned char _old = PyThreadState_GET()->recursion_critical;\
+    PyThreadState_GET()->recursion_critical = 1;
+
+#define Py_END_ALLOW_RECURSION \
+    PyThreadState_GET()->recursion_critical = _old; \
+  } while(0);
+
 PyAPI_FUNC(const char *) PyEval_GetFuncName(PyObject *);
 PyAPI_FUNC(const char *) PyEval_GetFuncDesc(PyObject *);