The problem is the allowed bandwith for the headset is restricted for an unknow reason (when using the headset for a phone call for example).
The following commands fix the bandwith to the max allowed by osx. And it works fine.
defaults write com.apple.BluetoothAudioAgent "Apple Bitpool Max (editable)" 80
defaults write com.apple.BluetoothAudioAgent "Apple Bitpool Min (editable)" 80
defaults write com.apple.BluetoothAudioAgent "Apple Initial Bitpool (editable)" 80
defaults write com.apple.BluetoothAudioAgent "Apple Initial Bitpool Min (editable)" 80
defaults write com.apple.BluetoothAudioAgent "Negotiated Bitpool" 80
defaults write com.apple.BluetoothAudioAgent "Negotiated Bitpool Max" 80
defaults write com.apple.BluetoothAudioAgent "Negotiated Bitpool Min" 80
defaults read com.apple.BluetoothAudioAgent
The output will be something like
$ defaults read com.apple.BluetoothAudioAgent
{
"Apple Bitpool Max (editable)" = 80;
"Apple Bitpool Min (editable)" = 80;
"Apple Initial Bitpool (editable)" = 80;
"Apple Initial Bitpool Min (editable)" = 80;
"Negotiated Bitpool" = 80;
"Negotiated Bitpool Max" = 80;
"Negotiated Bitpool Min" = 80;
}