Unicode on Mac is insane. Mac OS X uses NFD while everything else uses NFC. This fixes that.

mac_utf8_insanity.md

convmv manpage

Install convmv if you don't have it

sudo apt-get install convmv

Convert all files in a directory from NFD to NFC:

convmv -r -f utf8 -t utf8 --nfc --notest .

Convert all files in a directory from NFC to NFD:

convmv -r -f utf8 -t utf8 --nfd --notest .

I've created a repository with a pair of bijective scripts that do the conversion to and from NFD and does not rely on iconv:
https://github.com/hwdbk/synology-scripts/tree/master/mac-nfd-conversion
The scripts run on Mac OS X and other unixes (it uses bash and sed only). I use them on a Synology NAS, hence the names syn2mac and mac2syn, but what's in a name?
Also contains the script generating these scripts, if you want to play with it.

Very simple and nice

... another day saved - thanks so much!

Hello everyone.

I am on mac and I can't make the script work.

After a cd to the directory i want to change UTF, I copy the script path and press enter. But nothing happen.
Can you help me to make it work?
Thank you very very much by advance.

Best

Hi fguern, if you're referring to mac2syn or syn2mac, these scripts read from file or stdin, and output on stdout.
So, suppose you have file with NFD UTF text, called my_nfd_utf.txt (for instance), you type
mac2syn my_nfd_utf.txt
or
mac2syn < my_nfd_utf.txt
or
some-other-program-producing-the-text | mac2syn
make sure the script is executable (chmod 750 mac2syn)

If you have a string that needs translating, the syntax is
echo $(mac2syn <<< "string_or_variable_with_nfd_utf_text")

Cheers, Henk

Hello Henk,

Thanks for your help. It's still not clear for me.

Using a mac, does this mean to type that in the terminal ?
However, my goal is to change an entire folder with sub folders and sub files to the syno compatible UTF.

I tried that in the terminal :
Francoiss-MacBook-Air:~ francois$ mac2syn /Users/francois/Documents/01.\ Documents/2008_03_26\ -\ A\ voir\ à\ paris.rtf
-bash: mac2syn: command not found

I guess it didn't work :D.

And the script mac2syn is read and write for everyone.

Cheers, François

Hi François, It seems you're almost there. 1) yes, you need the Terminal application 2) the script mac2syn needs to be executable, you can do this by a) chmod 755 mac2syn followed by b) ./mac2syn .... (what you typed) note that typically the current directory is not in PATH (i.e. typing mac2syn doesn't work, but ./mac2syn does) 3) your command looks good. It will convert the 'sightseeing paris.rtf' to normal UTF. You might want to capture the output by adding 'sightseeing paris utf.rtf' at the end of the command (this is unix redirection to file) Let me know if this works. Then, it is easy to do this for an entire folder: find folder-with-your-rtfs -type f -name '*.rtf' -print | while read f ; do ./mac2syn "$f" > "$f.utf.rtf" ; done (try this with a small copy of the directory in case something unexpected happens) Henk

…

On 23 Dec 2020, at 18:17, fguern ***@***.***> wrote: @fguern commented on this gist. Hello Henk, Thanks for your help. It's still not clear for me. Using a mac, does this mean to type that in the terminal ? However, my goal is to change an entire folder with sub folders and sub files to the syno compatible UTF. I tried that in the terminal : Francoiss-MacBook-Air:~ francois$ mac2syn /Users/francois/Documents/01.\ Documents/2008_03_26\ -\ A\ voir\ à\ paris.rtf -bash: mac2syn: command not found I guess it didn't work :D. Cheers, François — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://gist.github.com/8448512#gistcomment-3571275>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH35YYBV24HF7CPFS2OOPVDSWIQZFANCNFSM4HRUSQ5Q>.

Hey Henk,

I tried your command, after installing the homebrew package + convmv (see : http://macappstore.org/convmv/). Because I understood that the script is based on these two package, right?

When launching the script with your command, even by adding 'sightseeing paris utf.rtf' it doesn't do anything : the file is still not synchronized with my syno.

francois@Francoiss-MacBook-Air ~ % cd /Users/francois/Downloads/synology-scripts-master/mac-nfd-conversion
francois@Francoiss-MacBook-Air mac-nfd-conversion % ./mac2syn /Users/francois/Documents/01.\ Documents/2008_03_26\ -\ A\ voir\ à\ paris.rtf 'sightseeing paris utf.rtf'
{\rtf1\ansi\ansicpg1252\cocoartf1265
\cocoascreenfonts1{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
\paperw11900\paperh16840\margl1440\margr1440\vieww9000\viewh8400\viewkind0
\pard\tx566\tx1133\tx1700\tx2267\tx2834\tx3401\tx3968\tx4535\tx5102\tx5669\tx6236\tx6803\pardirnatural

\f0\fs24 \cf0 * Caf'e9 Branly + mus'e9e
m'e9tro alma marceau
rer, pont de l'alma
\

• Caf'e9 des deux moulin _ Am'e9lie Poulain
  15 rue Lepic _ Montmartre
  -> fait
  \
• Mus'e9e Gr'e9vin, 10 boulevard Montmartre
  m'e9tro gd boulevard}cat: sightseeing paris utf.rtf: No such file or directory

Once it works with a file, i'll try you command on a folder.

Thank you Henk!!
+

Ah, I see what you're doing. Your command is good and it outputs the converted RTF. No syntax or other unix problems there. However, I see that the RTF format has an encoding of itself. For instance, it does not use é (in NFD or normal UTF-8 encoding), but 'e9 instead. I'm sure this is 'e9 in the source file as well. This is the way RTF apparently works with diacritical characters. What you're looking for is a converter from RTF-encoded files to UTF encoded files. mac2syn does NFD-UTF-8 to normal UTF-8. Cheers, Henk

…

On 23 Dec 2020, at 23:38, fguern ***@***.***> wrote: @fguern commented on this gist. Hey Henk, I tried your command, after installing the homebrew package + convmv (see : http://macappstore.org/convmv/ <http://macappstore.org/convmv/>). Because I understood that the script is based on these two package, right? When launching the script with your command, even by adding 'sightseeing paris utf.rtf' it doesn't do anything : the file is still not synchronized with my syno. ***@***.*** ~ % cd /Users/francois/Downloads/synology-scripts-master/mac-nfd-conversion ***@***.*** mac-nfd-conversion % ./mac2syn /Users/francois/Documents/01.\ Documents/2008_03_26\ -\ A\ voir\ à\ paris.rtf 'sightseeing paris utf.rtf' {\rtf1\ansi\ansicpg1252\cocoartf1265 \cocoascreenfonts1{\fonttbl\f0\fswiss\fcharset0 Helvetica;} {\colortbl;\red255\green255\blue255;} \paperw11900\paperh16840\margl1440\margr1440\vieww9000\viewh8400\viewkind0 \pard\tx566\tx1133\tx1700\tx2267\tx2834\tx3401\tx3968\tx4535\tx5102\tx5669\tx6236\tx6803\pardirnatural \f0\fs24 \cf0 * Caf'e9 Branly + mus'e9e m'e9tro alma marceau rer, pont de l'alma \ Caf'e9 des deux moulin _ Am'e9lie Poulain 15 rue Lepic _ Montmartre -> fait \ Mus'e9e Gr'e9vin, 10 boulevard Montmartre m'e9tro gd boulevard}cat: sightseeing paris utf.rtf: No such file or directory Once it works with a file, i'll try you command on a folder. Thank you Henk!! + — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://gist.github.com/8448512#gistcomment-3571663>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH35YYG2PSQAWCAVB6BGHVTSWJWNVANCNFSM4HRUSQ5Q>.

I did a test and it's only the title of the file which is the problem: if i create a copy and rename the file as "2008_03_26 - A voir a paris copy.rtf" instead of "2008_03_26 - A voir à paris.rtf" the file is synced with the synology

My goal is to rename all my files in a folder to adapt from NFD to NFC and in the future be sure all the accented files are sync with the synology.
And I thought the script mac2syn was to adapt the title file from NFD to NFC.

Was I wrong and did I miss something?

Thanks a lot for your help and time Henk.
Cheers, François

Yes, that explains. Yes, the script converts a file (or stdin) to the other encoding. If you want to just rename the file, (and leave the file's content alone), you basically have to pass the filename as text in stdin. This is how you do that (in bash): for f in directory-with-rtfs/*.rtf ; do mv -v -n "$f" "$(dirname "$f")/$(./mac2syn <<< "$(basename "$f")")" ; done dirname and basename are used to split the path to the rtf file in its directory and its filename. the simple form (if the directory path does not contain strange characters) would be: for f in directory-with-rtfs/*.rtf ; do mv -v -n "$f" "$(./mac2syn <<< "$f")" ; done

…

-v prints what mv is doing, -n prevents accidentally overwiting another file Cheers, Henk

On 24 Dec 2020, at 11:45, fguern ***@***.***> wrote: @fguern commented on this gist. I did a test and it's only the title of the file which is the problem: if i create a copy and rename the file as "2008_03_26 - A voir a paris copy.rtf" instead of "2008_03_26 - A voir à paris.rtf" the file is synced with the synology My goal is to rename all my files in a folder to adapt from NFD to NFC and in the future be sure all the accented files are sync with the synology. And I thought the script mac2syn was to adapt the title file from NFD to NFC. Was I wrong and did I miss something? Cheers, François — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://gist.github.com/8448512#gistcomment-3572013>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH35YYGWQU3SR4OYRKEDML3SWMLVNANCNFSM4HRUSQ5Q>.

hint: if you want to test/dry run what you're trying to do, just put echo in front of mv and it'll go through the motions and print the move instead of actually doing the move. hint: $(./mac2syn <<< "$f") does exactly the same as $(echo "$f" | ./mac2syn) then there are double quotes around it to preserve spaces in the file names. cheers, Henk

…

On 24 Dec 2020, at 17:42, Henk den Bok ***@***.***> wrote: Yes, that explains. Yes, the script converts a file (or stdin) to the other encoding. If you want to just rename the file, (and leave the file's content alone), you basically have to pass the filename as text in stdin. This is how you do that (in bash): for f in directory-with-rtfs/*.rtf ; do mv -v -n "$f" "$(dirname "$f")/$(./mac2syn <<< "$(basename "$f")")" ; done dirname and basename are used to split the path to the rtf file in its directory and its filename. the simple form (if the directory path does not contain strange characters) would be: for f in directory-with-rtfs/*.rtf ; do mv -v -n "$f" "$(./mac2syn <<< "$f")" ; done -v prints what mv is doing, -n prevents accidentally overwiting another file Cheers, Henk > On 24 Dec 2020, at 11:45, fguern ***@***.*** ***@***.***>> wrote: > > @fguern commented on this gist. > I did a test and it's only the title of the file which is the problem: if i create a copy and rename the file as "2008_03_26 - A voir a paris copy.rtf" instead of "2008_03_26 - A voir à paris.rtf" the file is synced with the synology > > My goal is to rename all my files in a folder to adapt from NFD to NFC and in the future be sure all the accented files are sync with the synology. > And I thought the script mac2syn was to adapt the title file from NFD to NFC. > > Was I wrong and did I miss something? > > Cheers, François > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub <https://gist.github.com/8448512#gistcomment-3572013>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH35YYGWQU3SR4OYRKEDML3SWMLVNANCNFSM4HRUSQ5Q>. >

Hello Henk,

I wish you a merry Christmas !

Thanks for the command. I tried it. At first the command worked but no file title where changed, and no sync with the syno happened. Then it display a dquote>.
Terminal stuff is definitely not for me.

To summarize :

• mac2syn script is in the mac download folder : /Users/francois/Downloads/synology-scripts-master/mac-nfd-conversion/mac2syn
• The folder and all its subfolders to change NFD to NFC is /Users/francois/Documents/01.\ Documents/25.\ Test/
• I created a "01. Test"sub folder in the "25. Test" Folder to simulate subfolders. (need it in the future as I will apply the command on all 01.\ Documents)
• The file "2008_03_26 - A voir à paris.rtf" is in the "01. Test" subfolder
• I CD the mac2syn foder (cd /Users/francois/Downloads/synology-scripts-master/mac-nfd-conversion/mac2syn)
• I enter your command without the */rtf to deal with all files : for f in /Users/francois/Documents/01.\ Documents/25.\ Test/ ; do mv -v -n ”$f" "$(dirname "$f")/$(./mac2syn <<< "$(basename "$f")")" ; done

Here is the result :
francois@Francoiss-MacBook-Air mac-nfd-conversion % for f in /Users/francois/Documents/01.\ Documents/25.\ Test/ ; do mv -v -n ”$f" "$(dirname "$f")/$(./mac2syn <<< "$(basename "$f")")" ; done
for dquote> for f in /Users/francois/Documents/01.\ Documents/25.\ Test/ ; do mv -v -n ”$f" "$(dirname "$f")/$(./mac2syn <<< "$(basename "$f")")" ; done
mv: rename ”/Users/francois/Documents/01. Documents/25. Test/ /Users/francois/Documents/01. to /Users/francois/Documents/01. Documents/25. Test/01.: No such file or directory
mv: rename Documents/25. to /Users/francois/Documents/01. Documents/25. Test/25.: No such file or directory
mv: rename Test ; done
for f in /Users/francois/Documents/01.\ Documents/25.\ Test/ ; do mv -v -n ”/Users/francois/Documents/01. Documents/25. Test/ to /Users/francois/Documents/01. Documents/25. Test/25. Test/: No such file or directory

Is the mac2syn and a rename command done for my need?

Thank you Henk,
François

Hi François, If you type for f in /Users/francois/Documents/01.\ Documents/25.\ Test/ ; do the variable f will only iterate through 1 value "/Users/francois/Documents/01.\ Documents/25.\ Test/", not the files within you need to type: for f in /Users/francois/Documents/01.\ Documents/25.\ Test/*.rft or for f in /Users/francois/Documents/01.\ Documents/25.\ Test/* if there are only rtf files there and no subdirectories and always use double quotes around "$f" because it contains spaces Cheers, Henk

…

On 26 Dec 2020, at 12:36, fguern ***@***.***> wrote: for f in /Users/francois/Documents/01.\ Documents/25.\ Test/

Got it.
I changed the double quotes and have an invalid argument this time:
francois@Francoiss-MacBook-Air mac-nfd-conversion % for f in /Users/francois/Documents/01.\ Documents/25.\ Test/ ; do mv -v -n "$f" "$(dirname "$f")/$(./mac2syn <<< "$(basename "$f")")" ; done
mv: rename /Users/francois/Documents/01. Documents/25. Test/ to /Users/francois/Documents/01. Documents/25. Test/25. Test/: Invalid argument

The command seems to repeat the last folder. Is it the Dirname+basename command?

However, I found another command which copy an entire folder and change the nfd, without script: rsync -a --iconv=utf-8-mac,utf-8 /Users/francois/Documents/01.\ Documents/25.\ Test/ /Users/francois/Documents/01.\ Documents/26.\ Test\ 2/
This one works. Even If it duplicate the files, I think it's a acceptable workaround. What do you think ?

Thank you,
François

the command doesnt' work because you didn't type the *.rtf behind the for f in /Users/francois/Documents/01.\ Documents/25.\ Test/ see previous email. You are passing a directory path into the mac2syn script, not a list of files within that directory. try again:

…

***@***.*** mac-nfd-conversion % for f in /Users/francois/Documents/01.\ Documents/25.\ Test/*.rtf ; do mv -v -n "$f" "$(dirname "$f")/$(./mac2syn <<< "$(basename "$f")")" ; done On 26 Dec 2020, at 19:40, fguern ***@***.***> wrote: @fguern commented on this gist. Got it. I changed the double quotes and have an invalid argument this time: ***@***.*** mac-nfd-conversion % for f in /Users/francois/Documents/01.\ Documents/25.\ Test/ ; do mv -v -n "$f" "$(dirname "$f")/$(./mac2syn <<< "$(basename "$f")")" ; done mv: rename /Users/francois/Documents/01. Documents/25. Test/ to /Users/francois/Documents/01. Documents/25. Test/25. Test/: Invalid argument The command seems to repeat the last folder. Is it the Dirname+basename command? However, I found another command which copy an entire folder and change the nfd, without script: rsync -a --iconv=utf-8-mac,utf-8 /Users/francois/Documents/01.\ Documents/25.\ Test/ /Users/francois/Documents/01.\ Documents/26.\ Test\ 2/ This one works. Even If it duplicate the files, I think it's a acceptable workaround. What do you think ? Thank you, François — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://gist.github.com/8448512#gistcomment-3573802>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH35YYFB6LK45I2UQDUNTALSWYUZFANCNFSM4HRUSQ5Q>.

Hello Henk.

At this stage I see three solutions:

/////1 - Your script

-> This time, it's "not overwritten":
francois@Francoiss-MacBook-Air mac-nfd-conversion % for f in /Users/francois/Documents/01.\ Documents/25.\ Test/*.rtf ; do mv -v -n "$f" "$(dirname "$f")/$(./mac2syn <<< "$(basename "$f")")" ; done
/Users/francois/Documents/01. Documents/25. Test/2008_03_26 - A voir à paris.rtf not overwritten
Is it possible to have an entire folder+subfolders rename with your script?

//// 2 - James CONVMV command
-> I also tried the command above "convmv -r -f utf8 -t utf8 --nfc --notest" and I got a "wrong/unknown encoding" :
francois@Francoiss-MacBook-Air 25. Test % convmv -r -f enc -t enc utf8 --nfc --notest
wrong/unknown "from" encoding!

///// 3 - Rsync local copy with NFC
-> No apparent problem, except the copy of 10Go
rsync -a --iconv=utf-8-mac,utf-8 /Users/francois/Documents/01.\ Documents/02.\ Administratif /Users/francois/Documents/01.\ Documents/02.\ Administratif\ nfc

What's your expert advise? Is it worth it to try to make the script or the convmv command work?
Thanks

Noe that the basics of script # 1 work, you can omit the -n option of the mv command Verstuurd vanaf mijn iPhone

…

Op 28 dec. 2020 om 00:27 heeft fguern ***@***.***> het volgende geschreven: ***@***.*** commented on this gist. Hello Henk. At this stage I see three solutions: /////1 - Your script -> This time, it's "not overwritten": ***@***.*** mac-nfd-conversion % for f in /Users/francois/Documents/01.\ Documents/25.\ Test/*.rtf ; do mv -v -n "$f" "$(dirname "$f")/$(./mac2syn <<< "$(basename "$f")")" ; done /Users/francois/Documents/01. Documents/25. Test/2008_03_26 - A voir à paris.rtf not overwritten Is it possible to have an entire folder+subfolders rename with your script? //// 2 - James CONVMV command -> I also tried the command above "convmv -r -f utf8 -t utf8 --nfc --notest" and I got a "wrong/unknown encoding" : ***@***.*** 25. Test % convmv -r -f enc -t enc utf8 --nfc --notest wrong/unknown "from" encoding! ///// 3 - Rsync local copy with NFC -> No apparent problem, except the copy of 10Go rsync -a --iconv=utf-8-mac,utf-8 /Users/francois/Documents/01.\ Documents/02.\ Administratif /Users/francois/Documents/01.\ Documents/02.\ Administratif\ nfc What's your expert advise? Is it worth it to try to make the script or the convmv command work? Thanks — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

and for the subdirectories, replace one or more levels of the directories in 'for i in ...' with */*.rtf, */*/*.rtf, */*/*/*.rtf etc. or use find <directory> -type f -name '*.rtf' -print | while read f ; do instead of the for i in ... ; do

…

On 28 Dec 2020, at 12:53, Henk den Bok ***@***.***> wrote: Noe that the basics of script # 1 work, you can omit the -n option of the mv command Verstuurd vanaf mijn iPhone > Op 28 dec. 2020 om 00:27 heeft fguern ***@***.***> het volgende geschreven: > > ***@***.*** commented on this gist. > Hello Henk. > > At this stage I see three solutions: > > /////1 - Your script > > -> This time, it's "not overwritten": > ***@***.*** mac-nfd-conversion % for f in /Users/francois/Documents/01.\ Documents/25.\ Test/*.rtf ; do mv -v -n "$f" "$(dirname "$f")/$(./mac2syn <<< "$(basename "$f")")" ; done > /Users/francois/Documents/01. Documents/25. Test/2008_03_26 - A voir à paris.rtf not overwritten > Is it possible to have an entire folder+subfolders rename with your script? > > //// 2 - James CONVMV command > -> I also tried the command above "convmv -r -f utf8 -t utf8 --nfc --notest" and I got a "wrong/unknown encoding" : > ***@***.*** 25. Test % convmv -r -f enc -t enc utf8 --nfc --notest > wrong/unknown "from" encoding! > > ///// 3 - Rsync local copy with NFC > -> No apparent problem, except the copy of 10Go > rsync -a --iconv=utf-8-mac,utf-8 /Users/francois/Documents/01.\ Documents/02.\ Administratif /Users/francois/Documents/01.\ Documents/02.\ Administratif\ nfc > > What's your expert advise? Is it worth it to try to make the script or the convmv command work? > Thanks > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub <https://gist.github.com/8448512#gistcomment-3574562>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH35YYFDO6SKYCDWF2537ZDSW67HBANCNFSM4HRUSQ5Q>. >

God bless you! 🙌

Great. Thanks a lot!

So i just upgraded to a new used intel-based macbook pro (aka mbp). I reformatted the internal SDD to use APFS and i'm seeing the exact symptoms defined here when i use VLC to view downloaded french video mp4 files with matching subtitle vtt filenames that have unicode characters in the filenames themselves! VLC plays the mp4 fine but it can't locate and autoload the vtt file with the same exact filename except for the extension! If there are no unicode "french" characters in the filenames it all works fine. But all files whether they had unicode "French" characters or not worked great on my old mbp with an older version of VLC and a NON-APFS file system.

In fact if i mount my external NTFS drive with a huge library of previously downloaded french video mp4 and vtt files on the new mbp (using an NTFS driver to mount the drive of course), the new VLC recognizes and plays these OLD files normally.

BUT on the new mbp, VLC will not autorecognzie the coinciding and same exact vtt filename as it's equivalent mp4 when the filename contains unicode french chars when i download new files via youtube-dl onto the internal SDD formatted as APFS

If i copy these new download files to the usb attached NTFS drive they then magically work they way i expect from VLC on the new mbp! :) If i then copy them back to the new mbp internal SDD with APFS, they also work the way i expect :) This seems to indicate that the NTFS filesystem changes the fileNAME encoding when the file is copied to it and that copying back to APFS somehow does NOT change that new encoding!???

I have a huge collection and am constantly adding to and maintaining it.

Is there any way to just set what your scripts are doing at the filesystem or even system level? Or going forward will i have to always run a post download script on every new filename to convert the unicode flavor used so that my mac / vlc can recognize them?!

Thanks

yup, that is exactly the madness with having two allowed but different character encodings for, say, the è (e-accent-grave)
the simple way to make sure both the media file and the vtt/srt file uses the same file name encoding is:

for i in *.mp4 ; do
mv -vn "$i" "$(syn2mac <<< "$i")"
done

and do the same for the subtitle files. You'll probably get a lot of "same file" warnings from mv on those files that were already in the target encoding.

Very nice tip!
On my Mac, I used brew install convmv, BTW.

You saved my day. Thank you!

Take a look at https://github.com/cr0sh/jaso for a faster alternative written in Rust.

$ brew install simnalamburt/x/jaso
$ jaso .
DONE; 100 files in 1.111529301 seconds

Just for reference, there is also NFCFN.py.