Differences between revisions 37 and 38
Revision 37 as of 2010-03-12 22:59:53
Size: 8423
Comment: Move 2.6 features paragraph into right section.
Revision 38 as of 2010-03-12 23:14:26
Size: 9237
Comment: Explain how to use distutils
Deletions are marked like this. Additions are marked like this.
Line 22: Line 22:

Usage of 2to3 can be integrated into the installation process: 2to3 can be run as a build step in distutils. With distutils, the following fragments are needed in a setup.py::

  try:
      from distutils.command.build_py import build_py_2to3 as build_py
  except ImportError:
      # 2.x
      from distutils.command.build_py import build_py
  ...
  setup(...
    cmdclass = {'build_py':build_py}
  )

This will leave intact all source files, and convert them to Python 3 in the build area, if setup.py is run on Python 3. 'setup.py install' will then copy the 3.x version of the code into the target directory. If run on 2.x, nothing will change at all (as 2.x doesn't provide the build_py_2to3 class).

For distribute, this approach can be expressed as::

  setup(...
    use_2to3=True
  )

Porting Python Code to 3.0

There are three ways to support Python 3:

  1. maintain a Python 2 base and use 2to3 to generate Python 3 code
  2. make code run unmodified in both Python 2 and Python 3
  3. maintain separate releases for Python 2 and Python 3

Each approach has its strengths and weaknesses.

Approach 1: Maintain a Python 2 base and use 2to3 to generate Python 3 code

Make sure the code runs in Python 2.6 and use 2to3

2to3 is a Python program that reads Python 2.x source code and applies a series of fixers to transform it into valid Python 3.x code. The standard library contains a rich set of fixers that will handle almost all code. 2to3 supporting library lib2to3 is, however, a flexible and generic library, so it is possible to write your own fixers for 2to3. lib2to3 could also be adapted to custom applications in which Python code needs to be edited automatically. For more information about 2to3, see: http://doc.python.org/library/2to3.html

Usage of 2to3 can be integrated into the installation process: 2to3 can be run as a build step in distutils. With distutils, the following fragments are needed in a setup.py:

try:
    from distutils.command.build_py import build_py_2to3 as build_py
except ImportError:
    # 2.x
    from distutils.command.build_py import build_py
...
setup(...
  cmdclass = {'build_py':build_py}
)

This will leave intact all source files, and convert them to Python 3 in the build area, if setup.py is run on Python 3. 'setup.py install' will then copy the 3.x version of the code into the target directory. If run on 2.x, nothing will change at all (as 2.x doesn't provide the build_py_2to3 class).

For distribute, this approach can be expressed as:

setup(...
  use_2to3=True
)

Manual changes (not done by 2to3):

  • os.path.walk => os.walk: see issue4601.
  • rfc822 => email

Strings and Bytes

Design decisions needs to be taken what exactly must be represented as bytes, and what as strings (Unicode data). In many cases, this is easy. However, data sent or received over pipes or sockets are usually in binary mode, so explicit conversions may be necessary. For example, the Postgres API requires SQL queries to be transmitted in the connection encoding. It is probably easiest to convert the queries to the connection encoding as early as possible.

The standard IO streams (sys.stdin, sys.stdout, sys.stderr), are in text mode by default in Python 3. Calling sys.stdout.write(some_binary_data) will fail. To write binary data to standard out, you must do the following:

sys.stdout.flush()
sys.stdout.buffer.write(some_binary_data)

The flush() clears out any text (unicode) data that is stored in the TextIOWrapper, and the buffer attribute provides access to the lower-level binary stream. If you are only writing binary data, you can remove the text layer with sys.stdout = sys.stdout.detach() in Python 3.1; in Python 3.0, do sys.stdout = sys.stdout.buffer.

Approach 2: Make code run unmodified in both Python 2 and Python 3

Supporting code that runs in both Python 2.6 and Python 3 is not much more difficult than porting to Python 3 (see above for more details). However, supporting older versions--like 2.3, 2.4, and 2.5--is much more difficult. This requires avoiding any syntax that only works in one version or the other, so it often involves resorting to hacks. These hacks make the code more awkward than it would be in Python 3 or in Python 2, but many people find that this is better than supporting only one version or the other. When support for Python 2 is eventually dropped, these hacks can be removed.

Python 2.6 introduces forward-compatibility for many new Python 3 features. The Python 3 builtins are accessible by doing an import from future_builtins. To use the new print function, use from __future__ import print_function, and to make string literals be interpreted as unicode, use from __future__ import unicode_literals. These forward-compatability features, and many others, are described in greater detail in What's New in Python 2.6:

Additional resources:

Strings and Unicode

As of Python 2.6 the 2.x line includes a "bytes = str" alias in the builtins. Along with the bytes literal syntax, this allows binary data stored in a str instance to be clearly flagged so that it will use the correct type when the code is run in 3.x (either directly or via the 2to3 conversion tool). The reason it is done this way rather than backporting the 3.x bytes type is that most 2.x APIs that expect immutable binary data expect it as an 8-bit str instance - trying to pass in a backported 3.x bytes type wouldn't have the desired effect. To support versions earlier than 2.6, it is possible to define the alias at the top of the module:

try:
  bytes # Forward compatibility with Py3k
except NameError:
  bytes = str

When using "from __future__ import unicode_literals" in a module, it may also be useful to insert "str = unicode" near the top of the module. This will ensure that "isinstance('', str)" remains true in that module:

try:
  str = unicode
except NameError:
  pass # Forward compatibility with Py3k

Exceptions

Python before 2.6 does not have the as keyword, so Python 2 and Python 3 have incompatible syntax for accessing the value of an exception. Compatible code should use the following idiom to save the value of an exception:

import sys
try:
    open('/path/to/some/file')
except IOError:
    _, e, _ = sys.exc_info()

Relative Imports

Python 3 makes a distinction between relative and absolute imports. In Python 2.5, use from __future__ import absolute_import to get the same behavior as Python 3. To support older versions as well, only use absolute imports. Replace a relative import:

from xyz import abc

with an absolute import:

from mypackage.xyz import abc

Integer Division

Make sure to use from __future__ import division (introduced in Python 2.2) to get the non-truncating behavior, which is default in Python 3.

Approach 3: Maintain Separate Releases for Python 2 and Python 3

Convert a Python 2 tree to Python 3 with the 2to3 tool: http://doc.python.org/library/2to3.html

Martin's notes from psycopg2

  • the buffer object is gone; I use memoryview in 3.x.
  • various tests where in the code of the form if version_major == and version_minor > 4 (say, or 5) This will break for 3.x; you have to write if (version_major == 2 and version_minor > 4) or version_major > 2
  • Python code 2: setup.py needs to run in both versions. I had to replace popen2 with subprocess if available. Also, map() now returns an iterator, which I explicitly convert into list, and so on.
  • Python code 3: the test suite doesn't get installed, and hence not auto-converted with 2to3 support. I explicitly added a 2to3 conversion into the test runner, which copies the py3 version of the test into a separate directory.

PortingPythonToPy3k (last edited 2019-10-26 22:24:27 by FrancesHocutt)

Unable to edit the page? See the FrontPage for instructions.