Python Unicode Encode Error

unicodeencodeerror python 3
python unicode encode error charmap
python unicode encode error character maps to <undefined
python encode/decode
how to solve unicode error in python
python print unicode variable
python 2 unicode
decode utf-8 python

I'm reading and parsing an Amazon XML file and while the XML file shows a ' , when I try to print it I get the following error:

'ascii' codec can't encode character u'\u2019' in position 16: ordinal not in range(128) 

From what I've read online thus far, the error is coming from the fact that the XML file is in UTF-8, but Python wants to handle it as an ASCII encoded character. Is there a simple way to make the error go away and have my program print the XML as it reads?

Likely, your problem is that you parsed it okay, and now you're trying to print the contents of the XML and you can't because theres some foreign Unicode characters. Try to encode your unicode string as ascii first:

unicodeData.encode('ascii', 'ignore')

the 'ignore' part will tell it to just skip those characters. From the python docs:

>>> u = unichr(40960) + u'abcd' + unichr(1972)
>>> u.encode('utf-8')
'\xea\x80\x80abcd\xde\xb4'
>>> u.encode('ascii')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character '\ua000' in position 0: ordinal not in range(128)
>>> u.encode('ascii', 'ignore')
'abcd'
>>> u.encode('ascii', 'replace')
'?abcd?'
>>> u.encode('ascii', 'xmlcharrefreplace')
'&#40960;abcd&#1972;'

You might want to read this article: http://www.joelonsoftware.com/articles/Unicode.html, which I found very useful as a basic tutorial on what's going on. After the read, you'll stop feeling like you're just guessing what commands to use (or at least that happened to me).

Unicode HOWTO, Likely, your problem is that you parsed it okay, and now you're trying to print the contents of the XML and you can't because theres some  When you print, you need bytes - unicode is an abstract concept. You need a mapping from the abstract unicode string into bytes - in python terms, you must convert your unicode object to str. You can do this be calling encode with an encoding that tells it how to translate from the abstract string into concrete bytes. Generally you want to use the utf-8 encoding.

A better solution:

if type(value) == str:
    # Ignore errors even if the string is not proper UTF-8 or has
    # broken marker bytes.
    # Python built-in function unicode() can do this.
    value = unicode(value, "utf-8", errors="ignore")
else:
    # Assume the value object has proper __unicode__() method
    value = unicode(value)

If you would like to read more about why:

http://docs.plone.org/manage/troubleshooting/unicode.html#id1

Python Unicode Encode Error, So, if I manually convert everything to either byte str or unicode strings, will I be in <module> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9'  When python print string, it may report UnicodeEncodeError: 'gbk' codec can't encode character and the python will be terminated. In this tutorial, we will introduce you how to fix this error.

Don't hardcode the character encoding of your environment inside your script; print Unicode text directly instead:

assert isinstance(text, unicode) # or str on Python 3
print(text)

If your output is redirected to a file (or a pipe); you could use PYTHONIOENCODING envvar, to specify the character encoding:

$ PYTHONIOENCODING=utf-8 python your_script.py >output.utf8

Otherwise, python your_script.py should work as is -- your locale settings are used to encode the text (on POSIX check: LC_ALL, LC_CTYPE, LANG envvars -- set LANG to a utf-8 locale if necessary).

To print Unicode on Windows, see this answer that shows how to print Unicode to Windows console, to a file, or using IDLE.

Understanding Python unicode, str, UnicodeEncodeError and , This is why trying to convert Unicode characters beyond 128 produces the error. The good news is that you can encode Python bytestrings in  To fix the error, encode the bytes explicitly with.encode and tell python what codec to use: Voil\u00E0! The issue is that when you call str(), python uses the default character encoding to try and encode the bytes you gave it, which in your case are sometimes representations of unicode characters.

Excellent post : http://www.carlosble.com/2010/12/understanding-python-and-unicode/

# -*- coding: utf-8 -*-

def __if_number_get_string(number):
    converted_str = number
    if isinstance(number, int) or \
            isinstance(number, float):
        converted_str = str(number)
    return converted_str


def get_unicode(strOrUnicode, encoding='utf-8'):
    strOrUnicode = __if_number_get_string(strOrUnicode)
    if isinstance(strOrUnicode, unicode):
        return strOrUnicode
    return unicode(strOrUnicode, encoding, errors='ignore')


def get_string(strOrUnicode, encoding='utf-8'):
    strOrUnicode = __if_number_get_string(strOrUnicode)
    if isinstance(strOrUnicode, unicode):
        return strOrUnicode.encode(encoding)
    return strOrUnicode

Overcoming frustration: Correctly using unicode in , How do I get around the Python error "UnicodeEncodeError: 'ascii' codec can't encode character" when using a Python script on the command line? Python byte strings (str type) have an encoding, Unicode does not. You can convert a Unicode string to a Python byte string using uni.encode(encoding) , and you can convert a byte string to a Unicode string using s.decode(encoding) (or equivalently, unicode(s, encoding) ).

You can use something of the form

s.decode('utf-8')

which will convert a UTF-8 encoded bytestring into a Python Unicode string. But the exact procedure to use depends on exactly how you load and parse the XML file, e.g. if you don't ever access the XML string directly, you might have to use a decoder object from the codecs module.

Python UnicodeEncodeError: 'ascii' codec can't encode character , If you've just run into the Python 2 Unicode brick wall, here are three line 1, in <​module> UnicodeEncodeError: 'ascii' codec can't encode  Typical error on Windows because the default user directory is C:\user\<your_user>, so when you want to use this path as an string parameter into a Python function, you get a Unicode error, just because the \u is a Unicode escape. Any character not numeric after this produces an error.

How to get around the Python error 'UnicodeEncodeError: 'ascii , str(u'café') Traceback (most recent call last): File "<input>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 3:  When ‘.decode()’ is called, Python default thinks that the string was encoded using ‘ascii’. So it tries to find the Unicode codepoint which corresponds to this encoded representation. In ascii, no Unicode codepoint corrresponds to ‘\xc3\xa4’ and so an error is raised.

Solving Unicode Problems in Python 2.7, test.py", line 3, in <module> print out UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 0: ordinal not in range(128). Since Python 3.0, the language’s str type contains Unicode characters, meaning any string created using "unicode rocks!", 'unicode rocks!', or the triple-quoted string syntax is stored as Unicode. The default encoding for Python source code is UTF-8, so you can simply include a Unicode character in a string literal:

Python 2.7. Unicode Errors Simply Explained · GitHub, If it is unicode, you must also give the encoding (and optionally, errors) parameters; bytearray() then converts the unicode to bytes using unicode.encode(). If it is an integer, the array will have that size and will be initialized with null bytes.

Comments
  • I was just coming to SO to post this question. Is there an easy way to sanitize a string for unicode()?
  • Please check also this answer to a related question: "Python UnicodeDecodeError - Am I misunderstanding encode?"
  • I'm trying to make the following string safe: ' foo "bar bar" df'(note the curly quotes), but the above still fails for me.
  • @Rosarch: Fails how? same error? And which error-handling rule did you use?
  • @Rosarch, your problem is probably earlier. Try this code: # -- coding: latin-1 -- u = u' foo "bar bar" df' print u.encode('ascii', 'ignore') For you, it was probably converting your string INTO unicode given the encoding you specified for the python scrip that threw the error.
  • I went ahead and made my issue into its own question: stackoverflow.com/questions/3224427/…
  • .encode('ascii', 'ignore') loses data unnecessarily even if OP's environment may support non-ascii characters (most cases)
  • It does not help with OP's issue: "can't encode character u'\u2019'". u'\u2019 is already Unicode.
  • It's already encoded in UTF-8 The error is specifically: myStrings = deque([u'Dorf and Svoboda\u2019s text builds on the str... and Computer Engineering\u2019s subdisciplines.']) The string is in UTF-8 as you can see, but it gets mad about the internal '\u2019'
  • Oh, OK, I thought you were having a different problem.
  • @Alex B: No, the string is Unicode, not Utf-8. To encode it as Utf-8 use '...'.encode('utf-8')