Differences between revisions 31 and 32
Revision 31 as of 2010-03-12 18:53:00
Size: 8268
Editor: drclaw
Comment: added a link to the python 2 and 3 metaclasses page
Revision 32 as of 2010-03-12 20:03:43
Size: 9145
Comment: Comment on proposed strategies.
Deletions are marked like this. Additions are marked like this.
Line 18: Line 18:

.. note::
   I think this section gives bad advise. It is not necessary *at all* to drop support for python older than 2.6
   when trying to port to 3.x. It is also not strictly necessary to test that the code runs on 2.6 (although it
   can't hurt to test that). If versions before 2.6 need to be supported simultaneously with 3.x, features
   new in 2.6 obviously can't be used for porting; in most cases, they aren't necessary, anyway, if 2to3 is used.
   -- Martin v. Löwis
Line 55: Line 62:

.. note::
   If the objective is to support both 2.x and 3.x from a single source, the strategies in this
   section should not be followed. Instead, 2to3 should be used - it is not necessarily just for
   separate releases, but allows well to support all versions from a single source tree. It also
   easily supports Python back to 2.3 (and probably earlier). -- Martin v. Löwis

Porting Python Code to 3.0

There are three ways to support Python 3:

  • drop support for Python older than 2.6
  • maintain separate releases for Python 2 and Python 3
  • support Python 2 and Python 3 simultaneously from one code base

Each approach has its strengths and weaknesses.

Dropping Support for Python Older than 2.6

Note

I think this section gives bad advise. It is not necessary at all to drop support for python older than 2.6 when trying to port to 3.x. It is also not strictly necessary to test that the code runs on 2.6 (although it can't hurt to test that). If versions before 2.6 need to be supported simultaneously with 3.x, features new in 2.6 obviously can't be used for porting; in most cases, they aren't necessary, anyway, if 2to3 is used. -- Martin v. Löwis

Make sure the code runs in Python 2.6 and use 2to3

2to3 is a Python program that reads Python 2.x source code and applies a series of fixers to transform it into valid Python 3.x code. The standard library contains a rich set of fixers that will handle almost all code. 2to3 supporting library lib2to3 is, however, a flexible and generic library, so it is possible to write your own fixers for 2to3. lib2to3 could also be adapted to custom applications in which Python code needs to be edited automatically. For more information about 2to3, see: http://doc.python.org/library/2to3.html

Python 2.6 introduces forward-compatibility for many new Python 3 features. The Python 3 builtins are accessible by doing an import from future_builtins. To use the new print function, use from __future__ import print_function, and to make string literals be interpreted as unicode, use from __future__ import unicode_literals. These forward-compatability features, and many others, are described in greater detail in What's New in Python 2.6:

Manual changes (not done by 2to3):

  • os.path.walk => os.walk: see issue4601.
  • rfc822 => email

Strings and Bytes

Design decisions needs to be taken what exactly must be represented as bytes, and what as strings (Unicode data). In many cases, this is easy. However, data sent or received over pipes or sockets are usually in binary mode, so explicit conversions may be necessary. For example, the Postgres API requires SQL queries to be transmitted in the connection encoding. It is probably easiest to convert the queries to the connection encoding as early as possible.

The standard IO streams (sys.stdin, sys.stdout, sys.stderr), are in text mode by default in Python 3. Calling sys.stdout.write(some_binary_data) will fail. To write binary data to standard out, you must do the following:

sys.stdout.flush()
sys.stdout.buffer.write(some_binary_data)

The flush() clears out any text (unicode) data that is stored in the TextIOWrapper, and the buffer attribute provides access to the lower-level binary stream. If you are only writing binary data, you can remove the text layer with sys.stdout = sys.stdout.detach() in Python 3.1; in Python 3.0, do sys.stdout = sys.stdout.buffer.

Maintain Separate Releases for Python 2 and Python 3

Convert a Python 2 tree to Python 3 with the 2to3 tool: http://doc.python.org/library/2to3.html

Support Python 2 and Python 3 Simultaneously

Note

If the objective is to support both 2.x and 3.x from a single source, the strategies in this section should not be followed. Instead, 2to3 should be used - it is not necessarily just for separate releases, but allows well to support all versions from a single source tree. It also easily supports Python back to 2.3 (and probably earlier). -- Martin v. Löwis

Supporting code that runs in both Python 2.6 and Python 3 is not much more difficult than porting to Python 3 (see above for more details). However, supporting older versions--like 2.3, 2.4, and 2.5--is much more difficult. This requires avoiding any syntax that only works in one version or the other, so it often involves resorting to hacks. These hacks make the code more awkward than it would be in Python 3 or in Python 2, but many people find that this is better than supporting only one version or the other. When support for Python 2 is eventually dropped, these hacks can be removed.

Additional resources:

Strings and Unicode

As of Python 2.6 the 2.x line includes a "bytes = str" alias in the builtins. Along with the bytes literal syntax, this allows binary data stored in a str instance to be clearly flagged so that it will use the correct type when the code is run in 3.x (either directly or via the 2to3 conversion tool). The reason it is done this way rather than backporting the 3.x bytes type is that most 2.x APIs that expect immutable binary data expect it as an 8-bit str instance - trying to pass in a backported 3.x bytes type wouldn't have the desired effect. To support versions earlier than 2.6, it is possible to define the alias at the top of the module:

try:
  bytes # Forward compatibility with Py3k
except NameError:
  bytes = str

When using "from __future__ import unicode_literals" in a module, it may also be useful to insert "str = unicode" near the top of the module. This will ensure that "isinstance('', str)" remains true in that module:

try:
  str = unicode
except NameError:
  pass # Forward compatibility with Py3k

Exceptions

Python before 2.6 does not have the as keyword, so Python 2 and Python 3 have incompatible syntax for accessing the value of an exception. Compatible code should use the following idiom to save the value of an exception:

import sys
try:
    open('/path/to/some/file')
except IOError:
    _, e, _ = sys.exc_info()

Relative Imports

Python 3 makes a distinction between relative and absolute imports. In Python 2.5, use from __future__ import absolute_import to get the same behavior as Python 3. To support older versions as well, only use absolute imports. Replace a relative import:

from xyz import abc

with an absolute import:

from mypackage.xyz import abc

Integer Division

Make sure to use from __future__ import division (introduced in Python 2.2) to get the non-truncating behavior, which is default in Python 3.

Martin's notes from psycopg2

  • the buffer object is gone; I use memoryview in 3.x.
  • various tests where in the code of the form if version_major == and version_minor > 4 (say, or 5) This will break for 3.x; you have to write if (version_major == 2 and version_minor > 4) or version_major > 2
  • Python code 2: setup.py needs to run in both versions. I had to replace popen2 with subprocess if available. Also, map() now returns an iterator, which I explicitly convert into list, and so on.
  • Python code 3: the test suite doesn't get installed, and hence not auto-converted with 2to3 support. I explicitly added a 2to3 conversion into the test runner, which copies the py3 version of the test into a separate directory.

PortingPythonToPy3k (last edited 2019-10-26 22:24:27 by FrancesHocutt)

Unable to edit the page? See the FrontPage for instructions.