<U+FEFF> character showing up in files. How to remove them?

gistfile1.txt

1) In your terminal, open the file using vim:

vim file_name
2) Remove all BOM characters:

:set nobomb
3) Save the file:

:wq


// taken from:

http://stackoverflow.com/questions/7297888/ufeff-character-showing-up-in-files-how-to-remove-them

Another recipe using awk 
http://stackoverflow.com/questions/1068650/using-awk-to-remove-the-byte-order-mark

awk '{ if (NR==1) sub(/^\xef\xbb\xbf/,""); print }' INFILE > OUTFILE

This helped me.

The vim method did not work. The awk recipe only removes <U+FEFF> (or as it is shown in vim) from the first column. I modified it to remove ALL <U+FEFF>

awk '{ gsub(/\xef\xbb\xbf/,""); print }' INFILE > OUTFILE

See this StackOverflow Answer for a better place for this information: https://stackoverflow.com/a/15137601/293280

Worked perfectly, thank you!

Thank you!

thanks @Sebastian-D

thank you for this.

A quick method if you don;t want to go into code. Just copy the file content and paste it in gedit (or notepad) editor. Copy the same content again from editor and replace in original file. You will get the clean file.

Thanks a lot!

you are a Fucking legend!

awesome, many thanks for this gist! @Sebastian-D also thanks to you. i used your awk command to solve my problem :)

thamks

In a script:

#!/bin/bash

awk '{ gsub(/\xef\xbb\xbf/,""); print }' $1 > /tmp/feff && mv /tmp/feff $1