Skip to content

Instantly share code, notes, and snippets.

@lukmdo
Created November 14, 2012 15:42
Show Gist options
  • Select an option

  • Save lukmdo/4072854 to your computer and use it in GitHub Desktop.

Select an option

Save lukmdo/4072854 to your computer and use it in GitHub Desktop.
Convert unicode to ASCII
#!/usr/bin/python3
"""
>>> some_unicode = u'èṽèṙẏbody ♥ utf8'
>>>
>>> some_unicode.encode('ascii', errors='replace')
b'?????body ? utf8'
>>>
>>> import unicodedata
>>> unicodedata.normalize('NFKD', some_unicode).encode(
... 'ascii', errors='replace')
b'e?v?e?r?y?body ? utf8'
>>>
>>> # what you want is errors='ignore' - ... scary
>>> asciified = unicodedata.normalize('NFKD', some_unicode).encode(
... 'ascii', errors='ignore')
>>> asciified
b'everybody utf8'
>>> # now lets just fix whitespace
>>> b' '.join(asciified.split())
b'everybody utf8'
"""
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment