Revision 19 as of 2010-02-17 20:20:28

Clear message

Porting Python Code to 3.0

There are three ways to support Python 3: * drop support for Python older than 2.6 * maintain separate releases for Python 2 and Python 3 * support Python 2 and Python 3 simultaneously from one code base

Each approach has its strengths and weaknesses.

Dropping Support for Python Older than 2.6

Make sure the code runs in Python 2.6 and use 2to3

2to3 is a Python program that reads Python 2.x source code and applies a series of fixers to transform it into valid Python 3.x code. The standard library contains a rich set of fixers that will handle almost all code. 2to3 supporting library lib2to3 is, however, a flexible and generic library, so it is possible to write your own fixers for 2to3. lib2to3 could also be adapted to custom applications in which Python code needs to be edited automatically. For more information about 2to3, see: http://doc.python.org/library/2to3.html

Python 2.6 introduces forward-compatibility for many new Python 3 features. The Python 3 builtins are accessible by doing an import from future_builtins. To use the new print function, use from __future__ import print_function, and to make string literals be interpreted as unicode, use from __future__ import unicode_literals. These forward-compatability features, and many others, are described in greater detail in What's New in Python 2.6:

Manual changes (not done by 2to3): - os.path.walk => os.walk: see issue4601. - rfc822 => email

Maintain Separate Releases for Python 2 and Python 3

Convert a Python 2 tree to Python 3 with the 2to3 tool: http://doc.python.org/library/2to3.html

Support Python 2 and Python 3 Simultaneously

Supporting code that runs in both Python 2.6 and Python 3 is not much more difficult than porting to Python 3 (see above for more details). However, supporting older versions--like 2.3, 2.4, and 2.5--is much more difficult. This requires avoiding any syntax that only works in one version or the other, so it often involves resorting to hacks. These hacks make the code more awkward than it would be in Python 3 or in Python 2, but many people find that this is better than supporting only one version or the other. When support for Python 2 is eventually dropped, these hacks can be removed.

Strings and Unicode

As of Python 2.6 the 2.x line includes a "bytes = str" alias in the builtins. Along with the bytes literal syntax, this allows binary data stored in a str instance to be clearly flagged so that it will use the correct type when the code is run in 3.x (either directly or via the 2to3 conversion tool). The reason it is done this way rather than backporting the 3.x bytes type is that most 2.x APIs that expect immutable binary data expect it as an 8-bit str instance - trying to pass in a backported 3.x bytes type wouldn't have the desired effect. To support versions earlier than 2.6, it is possible to define the alias at the top of the module:

try:
  bytes # Forward compatibility with Py3k
except NameError:
  bytes = str

When using "from __future__ import unicode_literals" in a module, it may also be useful to insert "str = unicode" near the top of the module. This will ensure that "isinstance('', str)" remains true in that module:

try:
  str = unicode
except NameError:
  pass # Forward compatibility with Py3k

Martin's notes from psycopg2

  • bytes vs. strings 1. Design decisions needs to be taken what exactly must be represented as bytes, and what as strings. In many cases, that was easy for psycopg, except for the question how SQL queries are represented. It appears clear that they are plain text, however, the Postgres API requires them to be transmitted in the connection encoding. I still decided to represent them internally in Unicode, but converting them to the connection encoding as early as possible probably would have worked as well.
  • the buffer object is gone; I use memoryview in 3.x.
  • various tests where in the code of the form if version_major == and version_minor > 4 (say, or 5) This will break for 3.x; you have to write if (version_major == 2 and version_minor > 4) or version_major > 2
  • Python code 1: I used the 2to3 support in distutils
  • Python code 2: setup.py needs to run in both versions. I had to replace popen2 with subprocess if available. Also, map() now returns an iterator, which I explicitly convert into list, and so on.
  • Python code 3: the test suite doesn't get installed, and hence not auto-converted with 2to3 support. I explicitly added a 2to3 conversion into the test runner, which copies the py3 version of the test into a separate directory.

Unable to edit the page? See the FrontPage for instructions.