Differences between revisions 12 and 13
Revision 12 as of 2007-07-13 03:50:41
Size: 2043
Editor: cscfpc15
Comment:
Revision 13 as of 2007-07-13 03:55:29
Size: 2074
Editor: cscfpc15
Comment:
Deletions are marked like this. Additions are marked like this.
Line 2: Line 2:

Paradoxically, a {{{UnicodeDecodeError}}} may happen when _encoding_. The cause of it seems to be the coding-specific {{{encode()}}} functions that normally expect a parameter of type {{{unicode}}}. It appears that on seeing an {{{str}}} parameter, the {{{encode()}}} functions "up-convert" it into {{{unicode}}} before converting to their own coding. It also appears that such "up-conversion" makes no assumption of {{{str}}} parameter's coding, choosing a default {{{ascii}}} decoder. Hence a decoding failure inside an encoder.

Unlike a similar case with UnicodeEncodeError, such a failure cannot be always avoided. This is because the {{{str}}} result of {{{encode()}}} must be a legal coding-specific sequence. However, a more flexible treatment of an unexpected {{{str}}} argument might first validate the {{{str}}} argument by attempting to decode it, then return it unmodified if the validation was successful. As of Python2.5, this is not implemented.
Line 19: Line 15:
"""
}}}
Line 20: Line 18:
Paradoxically, a {{{UnicodeDecodeError}}} may happen when _encoding_. The cause of it seems to be the coding-specific {{{encode()}}} functions that normally expect a parameter of type {{{unicode}}}. It appears that on seeing an {{{str}}} parameter, the {{{encode()}}} functions "up-convert" it into {{{unicode}}} before converting to their own coding. It also appears that such "up-conversion" makes no assumption of {{{str}}} parameter's coding, choosing a default {{{ascii}}} decoder. Hence a decoding failure inside an encoder.

Unlike a similar case with UnicodeEncodeError, such a failure cannot be always avoided. This is because the {{{str}}} result of {{{encode()}}} must be a legal coding-specific sequence. However, a more flexible treatment of an unexpected {{{str}}} argument might first validate the {{{str}}} argument by attempting to decode it, then return it unmodified if the validation was successful. As of Python2.5, this is not implemented.

{{{
#!python
r"""

The UnicodeDecodeError normally happens when decoding an str string from a certain coding. Since codings map only a limited number of str strings to unicode characters, an illegal sequence of str characters will cause the coding-specific decode() to fail.

   1 r"""
   2 Decoding from str to unicode.
   3 
   4 >>> "a".decode("utf-8")
   5 u'a'
   6 >>> "\x81".decode("utf-8")
   7 Traceback (most recent call last):
   8   File "<stdin>", line 1, in <module>
   9   File "encodings/utf_8.py", line 16, in decode
  10 UnicodeDecodeError: 'utf8' codec can't decode byte 0x81 in position 0: unexpected code byte
  11 """

Paradoxically, a UnicodeDecodeError may happen when _encoding_. The cause of it seems to be the coding-specific encode() functions that normally expect a parameter of type unicode. It appears that on seeing an str parameter, the encode() functions "up-convert" it into unicode before converting to their own coding. It also appears that such "up-conversion" makes no assumption of str parameter's coding, choosing a default ascii decoder. Hence a decoding failure inside an encoder.

Unlike a similar case with UnicodeEncodeError, such a failure cannot be always avoided. This is because the str result of encode() must be a legal coding-specific sequence. However, a more flexible treatment of an unexpected str argument might first validate the str argument by attempting to decode it, then return it unmodified if the validation was successful. As of Python2.5, this is not implemented.

   1 r"""
   2 Encoding from unicode to str.
   3 
   4 >>> u"a".encode("utf-8")
   5 'a'
   6 >>> u"\u0411".encode("utf-8")
   7 '\xd0\x91'
   8 >>> "a".encode("utf-8")         # Unexpected argument type.
   9 'a'
  10 >>> "\xd0\x91".encode("utf-8")  # Unexpected argument type.
  11 Traceback (most recent call last):
  12   File "<stdin>", line 1, in <module>
  13 UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)
  14 """


CategoryUnicode

UnicodeDecodeError (last edited 2008-11-15 13:59:56 by localhost)

Unable to edit the page? See the FrontPage for instructions.