Skip to content

Instantly share code, notes, and snippets.

@limhoff-r7
Last active August 29, 2015 14:01
Show Gist options
  • Save limhoff-r7/329aa446944a63a10dd1 to your computer and use it in GitHub Desktop.
Save limhoff-r7/329aa446944a63a10dd1 to your computer and use it in GitHub Desktop.
Encoding::UndefinedConversionError when msf/sanity is required and sets encodings to binary and something explicitly sets the external encoding to utif-8
Encoding.default_external: UTF-8
Encoding.default_internal:
Encoding.default_external: ASCII-8BIT
Encoding.default_internal: ASCII-8BIT
Setting no encoding on file
f.external_encoding: ASCII-8BIT
f.internal_encoding:
Contents: امثلة
Only setting external encoding on file
f.external_encoding: UTF-8
f.internal_encoding: ASCII-8BIT
U+0627 from UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError)
Setting external and internal encoding on file
i18n-fail.rb:21: warning: Ignoring internal encoding utf-8: it is identical to external encoding utf-8
f.external_encoding: UTF-8
f.internal_encoding:
Contents: امثلة
Encoding.default_external: UTF-8
Encoding.default_internal: UTF-8
Only setting external encoding on file
f.external_encoding: UTF-8
f.internal_encoding:
Contents: امثلة
# -*- encoding: utf-8 -*-
#
# CONSTANTS
#
PATH = 'arabic.txt'
#
# Methods
#
def read_arabic(encoding=nil)
mode = "r"
if encoding
mode = "#{mode}:#{encoding}"
end
begin
contents = File.open(PATH, mode) do |f|
puts "f.external_encoding: #{f.external_encoding}"
puts "f.internal_encoding: #{f.internal_encoding}"
f.read
end
rescue Encoding::UndefinedConversionError => error
puts "#{error} (#{error.class})"
else
puts "Contents: #{contents}"
end
end
#
# Script
#
puts "Encoding.default_external: #{Encoding.default_external}"
puts "Encoding.default_internal: #{Encoding.default_internal}"
File.open(PATH, 'w') do |f|
# 'examples' in Arabic
# @see https://github.com/cucumber/gherkin/blob/ae88a979d0f29c25f6c0c25198d770a45d96d2ba/lib/gherkin/i18n.json#L37
f.puts('امثلة')
end
# @see https://github.com/rapid7/metasploit-framework/blob/3a3d038904feb9729281af43e5ef97d2530d59aa/lib/msf/sanity.rb#L44
Encoding.default_external = Encoding.default_internal = "binary"
puts
puts "Encoding.default_external: #{Encoding.default_external}"
puts "Encoding.default_internal: #{Encoding.default_internal}"
puts
puts 'Setting no encoding on file'
read_arabic
puts
puts 'Only setting external encoding on file'
read_arabic('utf-8')
puts
puts 'Setting external and internal encoding on file'
read_arabic('utf-8:utf-8')
Encoding.default_external = Encoding.default_internal = 'utf-8'
puts
puts "Encoding.default_external: #{Encoding.default_external}"
puts "Encoding.default_internal: #{Encoding.default_internal}"
puts
puts 'Only setting external encoding on file'
read_arabic('utf-8')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment