Excluding common requests from your apache logs

areadme.md

Original article: http://www.blue-bag.com/blog/excluding-common-requests-your-apache-logs

Log files can get filled up with repeated calls to files such as favicon, robots.txt, images, css js etc

Mostly you want to log the initial request for a page and not all of the resources subsequently requested.

Troubleshooting other issues may mean you would log files such as favicon, images etc - but generally they needlessly fill up your logs.

Do it ocassionally to look for missing images etc.

Here is a way to exclude them from your log files. This method sets an environment variable for each type of file so that you can turn logging on and off for each as you see fit.

E.g. To turn logging back on for images - update the line:

SetEnvIf Request_URI "(\.gif|\.png|\.jpg)$" image-request=nolog

to

SetEnvIf Request_URI "(\.gif|\.png|\.jpg)$" image-request=log

Put the following section in your Apache config / Vhosts settings for the site (just before the closing VirtualHost tag):

logfiles

  ## flag robots.txt requests
  SetEnvIf Request_URI "^/robots\.txt$" robots-request=log
  ## flag favicon requests
  SetEnvIf Request_URI "^/favicon\.ico$" favicon-request=nolog

  ## flag image requests
  SetEnvIf Request_URI "(\.gif|\.png|\.jpg)$" image-request=nolog

  ## flag Css and JS requests
  SetEnvIf Request_URI \.css css-request=nolog
  SetEnvIf Request_URI \.js js-request=nolog

  ## flag cron calls
  SetEnvIf Request_URI "(^/cron\.php|^/bgp-start/)" cron-request=nolog

  ## set do_not_log if any of the above flags are set
  SetEnvIf robots-request nolog do_not_log
  SetEnvIf favicon-request nolog do_not_log
  SetEnvIf image-request nolog do_not_log
  SetEnvIf css-request nolog do_not_log
  SetEnvIf js-request nolog do_not_log
  SetEnvIf cron-request nolog do_not_log


  ## only log if do_not_log is not set
  CustomLog /var/www/log/general-access.log vcommon env=!do_not_log