Abstract

This PEP proposes new string representation form for Python 3000. In Python prior to Python 3000, the repr() built-in function converts arbitrary objects to printable ASCII strings for debugging and logging. For Python 3000, a wider range of characters, based on the Unicode standard, should be considered 'printable'.

Motivation

The current repr() converts 8-bit strings to ASCII using following algorithm.

• Convert CR, LF, TAB and '\' to '\r', '\n', '\t', '\\'.
• Convert other non-printable characters(0x00-0x1f, 0x7f) and non-ASCII characters(>=0x80) to '\xXX'.
• Backslash-escape quote characters(apostrophe, ') and add the quote character at the beginning and the end.

For Unicode strings, the following additional conversions are done.

• Convert leading surrogate pair characters without trailing character(0xd800-0xdbff, but not followed by 0xdc00-0xdfff) to '\uXXXX'.
• Convert 16-bit characters(>=0x100) to '\uXXXX'.
• Convert 21-bit characters(>=0x10000) and surrogate pair characters to '\U00xxxxxx'.

This algorithm converts any string to printable ASCII, and repr() is used as handy and safe way to print strings for debugging or for logging. Although all non-ASCII characters are escaped, this does not matter when most of the string's characters are ASCII. But for other languages, such as Japanese where most characters in a string are not ASCII, this is very inconvenient. Python 3000 has a lot of nice features for non-Latin users such as non-ASCII identifiers, so it would be helpful if Python could also progress in a similar way for printable output.

Some users might be concerned that such output will mess up their console if they print binary data like images. But this is unlikely to happen in practice because bytes and strings are different types in Python 3000, so printing an image to the console won't mess it up.

This issue was once discussed by Hye-Shik Chang [1] , but was rejected.

Specification

• Add Python API int PY_UNICODE_ISPRINTABLE(Py_UNICODE ch). PY_UNICODE_ISPRINTABLE() return 0 if repr() should escape the Unicode character ch, 1 otherwise. Characters should be escaped are
  • Characters defined in the Unicode character database as "Other"(Cc, Cf, Cs, Co, Cn).
  • Characters defined in the Unicode character database as "Separator"(Zl, Zp, Zs) other than ASCII space(0x20).
• The algorithm to build repr() strings should be changed to:
  • Convert CR, LF, TAB and '\' to '\r', '\n', '\t', '\\'.
  • Convert non-printable ASCII characters(0x00-0x1f, 0x7f) to '\xXX'.
  • Convert leading surrogate pair characters without trailing character(0xd800-0xdbff, but not followed by 0xdc00-0xdfff) to '\uXXXX'.
  • Convert non-printable characters(PY_UNICODE_ISPRINTABLE() returns 0) to 'xXX', '\uXXXX' or '\U00xxxxxx'.
  • Backslash-escape quote characters(apostrophe, ') and add quote character at the beginning and the end.
• Set the Unicode error-handler for sys.stderr to 'backslashreplace' by default.
• Set the Unicode error-handler for sys.stdout in the Python interactive session to 'backslashreplace' by default.
• Add '%a' string format operator. '%a' converts any python object to string using repr() and then hex-escape all non-ASCII characters. '%a' operator generates same string as '%r' in Python 2.
• Add ascii() builtin function. ascii() converts any python object to string using repr() and then hex-escape all non-ASCII characters. ascii() generates same string as repr() in Python 2.
• Add isprintable() method to the string type. str.isprintable() returns False if repr() should escape the characters in the string, True otherwise. isprintable() method calls PY_UNICODE_ISPRINTABLE() internally.

Rationale

The repr() in Python 3000 should be Unicode not ASCII based, just like Python 3000 strings. Also, conversion should not be affected by the locale setting, because the locale is not necessarily the same as the output device's locale. For example, it is common for a daemon process to be invoked in an ASCII setting, but writes UTF-8 to its log files. Also, web applications might want to report the error information in more readable form based on the HTML page's encoding.

Characters not supported by user's console are hex-escaped on printing, by the Unicode encoder's error-handler. If the error-handler of the output file is 'backslashreplace', such characters are hex-escaped without raising UnicodeEncodeError. For example, if your default encoding is ASCII, print('Hello ¢') will prints 'Hello \xa2'. If your encoding is ISO-8859-1, 'Hello ¢' will be printed.

For non-interactive session, default error-handler of sys.stdout should be default to 'strict'. Other applications reading the output might not understand hex-escaped characters, so un-supported characters should be trapped when writing.

Backwards Compatibility

Changing repr() may break some existing codes, especially testing code. Five of Python's regression test fail with this modification. If you need repr() strings without non-ASCII character as Python 2, you can use following function.

def repr_ascii(obj):
    return str(repr(obj).encode("ASCII", "backslashreplace"), "ASCII")

For logging or for debugging, following code can raise UnicodeEncodeError.

log = open("logfile", "w")
log.write(repr(data))     # UnicodeEncodeError will be raised
                          # if data contains unsupported characters.

To avoid exceptions raised, you can specify error-handler explicitly.

log = open("logfile", "w", errors="backslashreplace")
log.write(repr(data))  # Unsupported characters will be escaped.

For the console with Unicode-based encoding, for example, en_US.utf8 and de_DE.utf8, the backslashescape trick doesn't work and all printable characters are not escaped. This will cause a problem of similarly drawing characters in Western,Greek and Cyrillic languages. These languages use similar (but different) alphabets (descended from the common ancestor) and contain letters that look similar but has different character codes. For example, it is hard to distinguish Latin 'a', 'e' and 'o' from Cyrillic 'а', 'е' and 'о'. (The visual representation, of course, very much depends on the fonts used but usually these letters are almost indistinguishable.) To avoid the problem, user can adjust terminal encoding to get desired result suitable for their environment.

Open Issues

• Is ascii() function necessary, or documentation is just fine? If necessary, should ascii() belong to builtin namespace?

Rejected Proposals

• Add encoding and errors arguments to the builtin print() function, with defaults of sys.getfilesystemencoding() and 'backslashreplace'.

  Complicated to implement, and in general, this is not seem to good idea. [2]

• Use character names to escape characters, instead of hex character codes. For example, repr('\u03b1') can be converted to "\N{GREEK SMALL LETTER ALPHA}".

  Using character names get verbose compared to hex-escape. e.g., repr("\ufbf9") is converted to "\N{ARABIC LIGATURE UIGHUR KIRGHIZ YEH WITH HAMZA ABOVE WITH ALEF MAKSURA ISOLATED FORM}".

Reference Implementation

http://bugs.python.org/issue2630

References

Copyright

This document has been placed in the public domain.