Using a high definition display on Mac OS X can be a pain to set up due to the fact that the system recognizes the monitor as a television and therefor forces the monitor in the YCbCr mode. There's a solution for the problem called EDID patching which forces the monitor into the RGB mode.
First of all, when are you using a version of Mac OS X which protects your system (>=El Captain), you have to disable the System Integry Protection before we can continue. To do so, you have to boot your Mac into the Recovery mode using the shortcut CMD + R right after the startup chime. Select a language, go to Utilities in the menu bar and open up the terminal. Now insert the following command and reboot your Mac:
csrutil disable
reboot
Please keep in mind we're disabling a security feature - make sure you know what you're doing
After your system has rebooted, log in as normal and download this script: http://embdev.net/attachment/168316/patch-edid.rb to your home folder.
curl http://embdev.net/attachment/168316/patch-edid.rb > patch-edid.rb
When you're using a MacBook, please connect a USB keyboard and close the display lid. The script creates profiles for all available displays and would create a monitor profile of your internal display as well, what we can prevent when we close the lid at first.
Now you can run the downloaded script and let it do the magic:
ruby patch-edid.rb
The script will create a new directoy in your home folder which contains the modified monitor profile. The directory name starts with DisplayVendorID-.
Now go to the folder /System/Library/Displays/Contents/Resources/Overrides (Finder -> Go -> Go to folder...) and drop the DisplayVendorID- folder into it.
Last but not least, restart your system and enjoy your new monitor.