This is from man bash:
A pipeline is a sequence of one or more commands separated by one of the control operators | or |&. The format for a pipeline is:
[ time [ -p ]] [ ! ] command [ [ | | |& ] command2 ... ]
This is the only place in which ! can appear. It never prefixes commands, it prefixes pipelines.