- Added back the missing PY3PORT.rst file, with updates.

- Disallow appending unicode objects with 'y' (bytes) signatures.  This now
  requires either a bytes object or an integer.  Update the tests to reflect
- this change.
- Fix broken __all__ in Python 3.

6 files changed, 249 insertions, 29 deletions

diff --git a/PY3PORT.rst b/PY3PORT.rst
new file mode 100644
index 0000000..c873c2c
--- /dev/null
+++ b/PY3PORT.rst
@@ -0,0 +1,227 @@
+===============================
+Porting python-dbus to Python 3
+===============================
+
+This is an experimental port to Python 3.x where x >= 2.  There are lots of
+great sources for porting C extensions to Python 3, including:
+
+ * http://python3porting.com/toc.html
+ * http://docs.python.org/howto/cporting.html
+ * http://docs.python.org/py3k/c-api/index.html
+
+I also consulted an early take on this port by John Palmieri and David Malcolm
+in the context of Fedora:
+
+ * https://bugs.freedesktop.org/show_bug.cgi?id=26420
+
+although I have made some different choices.  The patches in that tracker
+issue also don't cover porting the Python bits (e.g. the test suite), nor the
+pygtk -> pygi porting, both which I've also attempted to do in this branch.
+
+This document outlines my notes and strategies for doing this port.  Please
+feel free to contact me with any bugs, issues, disagreements, suggestions,
+kudos, and curses.
+
+Barry Warsaw
+barry@python.org
+2011-11-11
+
+
+User visible changes
+====================
+
+You've got some dbus-python code that works great in Python 2.  This branch
+should generally allow your existing Python 2 code to continue to work
+unchanged.  There are a few changes you'll notice in Python 2 though::
+
+ - The minimum supported Python 2 version is 2.6.
+ - All object reprs are unicodes.  This change was made because it greatly
+   simplifies the implementation and cross-compatibility with Python 3.
+ - Some values which were ints are now longs.  Primarily, this affects the
+   type of the `variant_level` attributes.
+ - `dbus.Byte` can be constructed from a 1-character str or unicode object.
+ - Some exception strings have changed.
+ - `MethodCallMessage` and `SignalMessage` objects have better reprs now.
+
+What do you need to do to port that to Python 3?  Here are the user visible
+changes you should be aware of.  Python 3.2 is the minimal required version::
+
+ - `ByteArray`s must be initialized with bytes objects, not unicodes.  Use
+   `b''` literals in the constructor.  This also works in Python 2, where
+   bytes objects are aliases for 8-bit strings.
+ - byte signatures (i.e. `y` type codes) must be passed either a length-1
+   bytes object or an integer.  unicodes are not allowed.
+ - `ByteArray` is now a subclass of `bytes`, where in Python 2 it is a
+   subclass of `str`.
+ - `dbus.Byte` can be constructed from a 1-character byte or str object, or an
+   integer.
+ - `dbus.UTF8String` is gone, use `dbus.String`.  Also `utf8_string` arguments
+   are no longer allowed.
+ - All longs are now ints, since Python 3 has only a single int type.  This
+   also means that the class hierarchy for the dbus numeric types has changed
+   (all derive from int in Python 3).
+ - Some exception strings have changed.
+ - `MethodCallMessage` and `SignalMessage` objects have better reprs now.
+
+
+Bytes vs. Strings
+=================
+
+All strings in dbus are defined as UTF-8:
+
+http://dbus.freedesktop.org/doc/dbus-specification.html#message-protocol-signatures
+
+However, the dbus C API accepts `char*` which must be UTF-8 strings NUL
+terminated and no other NUL bytes.
+
+This page describes the mapping between Python types and dbus types:
+
+    http://dbus.freedesktop.org/doc/dbus-python/doc/tutorial.html#basic-types
+
+Notice that it maps dbus `string` (`'s'`) to `dbus.String` (unicode) or
+`dbus.UTF8String` (str).  Also notice that there is no direct dbus equivalent
+of Python's bytes type (although dbus does have byte arrays), so I am mapping
+dbus strings to unicodes in all cases, and getting rid of `dbus.UTF8String` in
+Python 3.  I've also added a `dbus._BytesBase` type which is unused in Python
+2, but which forms the base class for `dbus.ByteArray` in Python 3.  This is
+an implementation detail and not part of the public API.
+
+In Python 3, object paths (`'o'` or `dbus.ObjectPath`), signatures (`'g'` or
+`dbus.Signature`), bus names, interfaces, and methods are all strings.  A
+previous aborted effort was made to use bytes for these, which at first blush
+may makes some sense, but on deeper consideration does not.  This approach
+also tended to impose too many changes on user code, and caused lots of
+difficult to track down problems.
+
+In Python 3, all such objects are subclasses of `str` (i.e. `unicode`).
+
+(As an example, dbus-python's callback dispatching pretty much assumes all
+these things are strings.  When they are bytes, the fact that `'foo' != b'foo'`
+causes dispatch matching to fail in difficult to debug ways.  Even bus names
+are not immune, since they do things like `bus_name[:1] == ':'` which fails in
+multiple ways when `bus_name` is a bytes.  For sanity purposes, these are all
+unicode strings now, and we just eat the complexity at the C level.)
+
+I am using `#include <bytesobject.h>`, which exposes the PyBytes API to Python
+2.6 and 2.7, and I have converted all internal PyString calls to PyBytes
+calls.  Where this is inappropriate, we'll use PyUnicode calls explicitly.
+E.g. all repr() implementations now return unicodes.  Most of these changes
+shouldn't be noticed, even in existing Python 2 code.
+
+Generally, I've left the descriptions and docstrings saying "str" instead of
+"unicode" since there's no distinction in Python 3.
+
+APIs which previously returned PyStrings will usually return PyUnicodes, not
+PyBytes.
+
+
+Ints vs. Longs
+==============
+
+Python 3 only has PyLong types; PyInts are gone.  For that reason, I've
+switched all PyInt calls to use PyLong in both Python 2 and Python 3.  Python
+3.0 had a nice `<intobject.h>` header that aliased PyInt to PyLong, but that's
+gone as of Python 3.1, and the minimal required Python 3 version is 3.2.
+
+In the above page mapping basic types, you'll notice that the Python int type
+is mapped to 32-bit signed integers ('i') and the Python long type is mapped
+to 64-bit signed integers ('x').  Python 3 doesn't have this distinction, so
+ints map to 'i' even though ints can be larger in Python 3.  Use the
+dbus-specific integer types if you must have more exact mappings.
+
+APIs which accepted ints in Python 2 will still do so, but they'll also now
+accept longs.  These APIs obviously only accept longs in Python 3.
+
+Long literals in Python code are an interesting thing to have to port.  Don't
+use them if you want your code to work in both Python versions.
+
+`dbus._IntBase` is removed in Python 3, you only have `dbus._LongBase`, which
+inherits from a Python 3 int (i.e. a PyLong).  Again, this is an
+implementation detail that users should never care about.
+
+
+Macros
+======
+
+In types-internal.h, I define `PY3K` when `PY_MAJOR_VERSION` >= 3, so you'll
+see ifdefs on the former symbol within the C code.
+
+Python 3 really could use a PY_REFCNT() wrapper for ob_refcnt access.
+
+
+PyCapsule vs. PyCObject
+=======================
+
+`_dbus_bindings._C_API` is an attribute exposed to Python in the module.  In
+Python 2, this is a PyCObject, but these do not exist in Python >= 3.2, so it
+is replaced with a PyCapsules for Python 3.  However, since PyCapsules were
+only introduced in Python 2.7, and I want to support Python 2.6, PyCObjects
+are still used when this module is compiled for Python 2.
+
+
+Python level compatibility
+==========================
+
+`from dbus import _is_py3` gives you a flag to check if you must do something
+different in Python 3.  In general I use this flag to support both versions in
+one set of sources, which seems better than trying to use 2to3.  It's not part
+of the dbus-python public API, so you may not need it.
+
+
+Miscellaneous
+=============
+
+The PyDoc_STRVAR() documentation is probably out of date.  Once the API
+choices have been green-lighted upstream, I'll make a pass through the code to
+update them.  It might be tricky based on any differences between Python 2 and
+Python 3.
+
+There were a few places where I noticed what might be considered bugs,
+unchecked exception conditions, or possible reference count leaks.  In these
+cases, I've just fixed what I can and hopefully haven't made the situation
+worse.
+
+`dbus_py_variant_level_get()` did not check possible error conditions, nor did
+their callers.  When `dbus_py_variant_level_get()` encounters an error, it now
+returns -1, and callers check this.
+
+As much as possible, I've refrained from general code cleanups (e.g. 80
+columns), unless it just bugged me too much or I touched the code for reasons
+related to the port.  I've also tried to stick to existing C code style,
+e.g. through the use of pervasive `Py_CLEAR()` calls, comparison against NULL
+usually with `!foo`, and such.  As Bart Simpson might write on his classroom
+blackboard::
+
+    This is not a rewrite
+    This is not a rewrite
+    This is not a rewrite
+    This is not a rewrite
+    ...
+
+and so on.  Well, mostly ;).
+
+I think I fixed a reference leak in `DBusPyServer_set_auth_mechanisms()`.
+`PySequence_Fast()` returns a new reference, which wasn't getting decref'd in
+any return path.
+
+ - Instantiation of metaclasses uses different, incompatible syntax in Python
+   2 and 3.  You have to use direct calling of the metaclass to work across
+   versions, i.e. `Interface = InterfaceType('Interface', (object,), {})`
+ - `iteritems()` and friends are gone.  I dropped the "iter" prefixes.
+ - `xrange() is gone.  I changed them to use `range()`.
+ - `isSequenceType()` is gone in Python 3, so I use a different idiom there.
+ - `__next__()` vs. `next()`
+ - `PyUnicode_FromFormat()` `%V` flag is a clever hack!
+ - `sys.version_info` is a tuple in Python 2.6, not a namedtuple.  i.e. there
+   is no `sys.version_info.major`
+ - `PyArg_Parse()`: No 'y' code in Python 2; in Python 3, no equivalent of 'z'
+   for bytes objects.
+
+
+Open issues
+===========
+
+Here are a few things that still need to be done, or for which there may be
+open questions::
+
+ - Update all C extension docstrings for accuracy.
diff --git a/_dbus_bindings/message-append.c b/_dbus_bindings/message-append.c
index c08f498..bbf1286 100644
--- a/_dbus_bindings/message-append.c
+++ b/_dbus_bindings/message-append.c
@@ -587,35 +587,21 @@ _message_iter_append_byte(DBusMessageIter *appender, PyObject *obj)
 
     if (PyBytes_Check(obj)) {
         if (PyBytes_GET_SIZE(obj) != 1) {
-            PyErr_Format(PyExc_ValueError, "Expected a string of "
-                         "length 1 byte, but found %d bytes",
-                         (int) PyBytes_GET_SIZE(obj));
+            PyErr_Format(PyExc_ValueError,
+                         "Expected a length-1 bytes but found %d bytes",
+                         (int)PyBytes_GET_SIZE(obj));
             return -1;
         }
         y = *(unsigned char *)PyBytes_AS_STRING(obj);
     }
-    else if (PyUnicode_Check(obj)) {
-        PyObject *obj_as_bytes = PyUnicode_AsUTF8String(obj);
-
-        if (!obj_as_bytes)
-            return -1;
-        if (PyBytes_GET_SIZE(obj_as_bytes) != 1) {
-            PyErr_Format(PyExc_ValueError, "Expected a string of "
-                         "length 1 byte, but found %d bytes",
-                         (int)PyBytes_GET_SIZE(obj_as_bytes));
-            Py_CLEAR(obj_as_bytes);
-            return -1;
-        }
-        y = *(unsigned char *)PyBytes_AS_STRING(obj_as_bytes);
-        Py_CLEAR(obj_as_bytes);
-    }
     else {
         long i = PyLong_AsLong(obj);
 
         if (i == -1 && PyErr_Occurred()) return -1;
         if (i < 0 || i > 0xff) {
-            PyErr_Format(PyExc_ValueError, "%d outside range for a "
-                         "byte value", (int)i);
+            PyErr_Format(PyExc_ValueError,
+                         "%d outside range for a byte value",
+                         (int)i);
             return -1;
         }
         y = i;
diff --git a/dbus/types.py b/dbus/types.py
index a134495..c33acff 100644
--- a/dbus/types.py
+++ b/dbus/types.py
@@ -1,7 +1,7 @@
-__all__ = ('ObjectPath', 'ByteArray', 'Signature', 'Byte', 'Boolean',
+__all__ = ['ObjectPath', 'ByteArray', 'Signature', 'Byte', 'Boolean',
            'Int16', 'UInt16', 'Int32', 'UInt32', 'Int64', 'UInt64',
            'Double', 'String', 'Array', 'Struct', 'Dictionary',
-           'UTF8String', 'UnixFd')
+           'UnixFd']
 
 from _dbus_bindings import (
     Array, Boolean, Byte, ByteArray, Dictionary, Double, Int16, Int32, Int64,
@@ -11,3 +11,4 @@ from _dbus_bindings import (
 from dbus._compat import is_py2
 if is_py2:
     from _dbus_bindings import UTF8String
+    __all__.append('UTF8String')
diff --git a/test/cross-test-client.py b/test/cross-test-client.py
index ecc6f8a..74d957c 100644
--- a/test/cross-test-client.py
+++ b/test/cross-test-client.py
@@ -234,7 +234,8 @@ class Client(SignalTestsImpl):
         # "Single tests"
         if have_signatures:
             self.assert_method_eq(INTERFACE_SINGLE_TESTS, 6, 'Sum', [1, 2, 3])
-            self.assert_method_eq(INTERFACE_SINGLE_TESTS, 6, 'Sum', ['\x01', '\x02', '\x03'])
+            self.assert_method_eq(INTERFACE_SINGLE_TESTS, 6, 'Sum', 
+                                  [b'\x01', b'\x02', b'\x03'])
         self.assert_method_eq(INTERFACE_SINGLE_TESTS, 6, 'Sum', [Byte(1), Byte(2), Byte(3)])
         self.assert_method_eq(INTERFACE_SINGLE_TESTS, 6, 'Sum', ByteArray(b'\x01\x02\x03'))
 
@@ -282,7 +283,7 @@ class Client(SignalTestsImpl):
             self.assert_method_eq(INTERFACE_TESTS, '\xa9', 'IdentityString', i)
 
         if have_signatures:
-            self.assert_method_eq(INTERFACE_TESTS, Byte(0x42), 'IdentityByte', '\x42')
+            #self.assert_method_eq(INTERFACE_TESTS, Byte(0x42), 'IdentityByte', '\x42')
             self.assert_method_eq(INTERFACE_TESTS, True, 'IdentityBool', 42)
             self.assert_method_eq(INTERFACE_TESTS, 42, 'IdentityInt16', 42)
             self.assert_method_eq(INTERFACE_TESTS, 42, 'IdentityUInt16', 42)
@@ -342,7 +343,9 @@ class Client(SignalTestsImpl):
                               'IdentityByteArray',
                               ByteArray(b'\x01\x02\x03'))
         if have_signatures:
-            self.assert_method_eq(INTERFACE_TESTS, [1,2,3], 'IdentityByteArray', ['\x01', '\x02', '\x03'])
+            self.assert_method_eq(INTERFACE_TESTS, [1,2,3], 
+                                  'IdentityByteArray', 
+                                  [b'\x01', b'\x02', b'\x03'])
         self.assert_method_eq(INTERFACE_TESTS, [False,True], 'IdentityBoolArray', [False,True])
         if have_signatures:
             self.assert_method_eq(INTERFACE_TESTS, [False,True,True], 'IdentityBoolArray', [0,1,2])
diff --git a/test/run-test.sh b/test/run-test.sh
index 516e876..2530b3f 100755
--- a/test/run-test.sh
+++ b/test/run-test.sh
@@ -63,6 +63,7 @@ fi
 dbus-monitor > "$DBUS_TOP_BUILDDIR"/test/monitor.log &
 
 echo "PYTHONPATH=$PYTHONPATH"
+echo "PYTHON=$PYTHON"
 
 echo "running test-standalone.py"
 $PYTHON "$DBUS_TOP_SRCDIR"/test/test-standalone.py || die "test-standalone.py failed"
diff --git a/test/test-standalone.py b/test/test-standalone.py
index a76ad25..48cbe09 100755
--- a/test/test-standalone.py
+++ b/test/test-standalone.py
@@ -265,10 +265,12 @@ class TestMessageMarshalling(unittest.TestCase):
     def test_get_args_options(self):
         aeq = self.assertEqual
         s = _dbus_bindings.SignalMessage('/', 'foo.bar', 'baz')
-        s.append('b', 'bytes', -1, 1, 'str', 'var', signature='yayiusv')
-        aeq(s.get_args_list(), [ord('b'),
-                                [ord('b'),ord('y'),ord('t'),ord('e'), ord('s')],
-                                -1, 1, 'str', 'var'])
+        s.append(b'b', b'bytes', -1, 1, 'str', 'var', signature='yayiusv')
+        aeq(s.get_args_list(), [
+            ord('b'), 
+            [ord('b'),ord('y'),ord('t'),ord('e'), ord('s')], 
+            -1, 1, 'str', 'var'
+            ])
         byte, bytes, int32, uint32, string, variant = s.get_args_list()
         aeq(byte.__class__, types.Byte)
         aeq(bytes.__class__, types.Array)