Just added this into Apache2 config. E.g. /etc/apache2/sites-available/900-restrictDirectIP.conf and enable it afterwards.
<VirtualHost 192.168.0.1:80>
ServerName 192.168.0.1
Redirect 403 /
ErrorDocument 403 "Sorry, direct IP access not allowed."
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error_directIPAccess.log
CustomLog ${APACHE_LOG_DIR}/access_directIPAccess.log combined