Just added this into Apache2 config. E.g. /etc/apache2/sites-available/900-restrictDirectIP.conf and enable it afterwards.
<VirtualHost 192.168.0.1:80>
ServerName 192.168.0.1
Redirect 403 /
ErrorDocument 403 "Sorry, direct IP access not allowed."
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error_directIPAccess.log
CustomLog ${APACHE_LOG_DIR}/access_directIPAccess.log combined
</VirtualHost>
<VirtualHost 192.168.0.1:443>
ServerName 192.168.0.1
Redirect 403 /
ErrorDocument 403 "Sorry, direct IP access not allowed."
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error_directIPAccess.log
CustomLog ${APACHE_LOG_DIR}/access_directIPAccess.log combined
</VirtualHost>
And this script to cron if you have dynamic IP --> https://github.com/GAS85/cubietruck/blob/master/apache2IPUpdate.sh
You can aslo added accessed IPs direct to ban via fail2ban and iptables, by reading ${APACHE_LOG_DIR}/access_directIPAccess.log