How to monitor a logrotated file

gistfile1.txt

(Read the comments)

The scenario: some daemon is writing a log file, and you wish to monitor that log file (ie. read it and do something with the logged information). Independently, the log file is being rotated: ie. the current file gets renamed to log.1, and the daemon will write to a new version of the file.

We can simulate this scenario as follows:

Daemon

We simulate a daemon that writes data to a local file t.log at regular intervals:

while true; do fortune >> t.log; sleep 1; done

Logrotate

We configure logrotate to manage this file with the following local config file logrotate.conf:

# Test logrotate config file

"t.log" {
  rotate 3
}

To test logrotate on this local file without disturbing the system's logrotate, we use the -s option to specify a local status file. We use the -f option to force a logrotate now (by default logrotate will refuse to rotate files newer than a day old):

logrotate -f -s ./logrotate.status logrotate.conf

To rotate every 3 seconds, and list the collection of log files after each rotation:

while true; do logrotate -f -s ./logrotate.status logrotate.conf ; ls -l t.log*; sleep 3; done

Tail

We can monitor this rotated log file with tail -F, which will follow the file name. It periodically reports to stderr;

tail: `t.log' has become inaccessible: No such file or directory
tail: `t.log' has appeared;  following end of new file

strace will allow us to see the system calls that tail -F uses to track renames:

strace -o strace.out tail -F t.log

The resulting strace.out reveals:

open("t.log", O_RDONLY|O_NONBLOCK)      = -1 ENOENT (No such file or directory)
write(2, "tail: ", 6)                   = 6
write(2, "`t.log' has become inaccessible", 31) = 31
write(2, ": No such file or directory", 27) = 27
write(2, "\n", 1)                       = 1
close(3)                                = 0
read(4, "\1\0\0\0\0\1\0\0\0\0\0\0\20\0\0\0t.log\0\0\0\0\0\0\0\0\0\0\0", 44) = 32
inotify_add_watch(4, "t.log", IN_MODIFY|IN_ATTRIB|IN_DELETE_SELF|IN_MOVE_SELF) = 4
open("t.log", O_RDONLY|O_NONBLOCK)      = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
fstatfs(3, {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=230629351, f_bfree=35201598, f_bavail=23486296, f_files=58580992, f_ffree=56205924, f_fsid={946342991, 389165871}, f_namelen=255, f_frsize=4096}) = 0
write(2, "tail: ", 6)                   = 6
write(2, "`t.log' has appeared;  following"..., 48) = 48
write(2, "\n", 1)                       = 1