Skip to content

Instantly share code, notes, and snippets.

@bcho
Created February 28, 2015 14:21
Show Gist options
  • Select an option

  • Save bcho/dd08494765c505b9e9f1 to your computer and use it in GitHub Desktop.

Select an option

Save bcho/dd08494765c505b9e9f1 to your computer and use it in GitHub Desktop.
trick
import unicodedata
def must_be_ascii(raw):
'''Convert all non-ascii characters to ascii substitutes.
:param raw: raw input.
'''
if not isinstance(raw, (str, unicode)):
raise TypeError('Cannot convert to ascii, expected str or unicode.')
if not isinstance(raw, unicode):
raw = raw.decode('u8')
return unicodedata.normalize('NFKD', raw) \
.encode('ascii', 'ignore') \
.decode('u8')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment