Yandex FreeBSD kernel panic stats. Feb/Apr.

dst00_mail_grep.txt

$ grep -h Subject /place/mail/textdumps.2013-03-* | perl -pe 's/Subject: \[sendcore\] panic on //' | sort -u | grep ^ws[0-9] | perl -pe 's/ws[0-9]+-[0-9]+: //' | perl -pe 's/0x[0-f]+/<pointer>/g' | sort | uniq -c | sort -n
   1 NMI indicates hardware failure
   1 ffs_write: type <pointer> 0 (0,161)
   1 initiate_write_inodeblock_ufs2: already started
   2 Bad effnlink fip <pointer>, fdp <pointer>, tdp <pointer>
   4 softdep_setup_inomapdep: dependency <pointer> for newinode already exists
   5 handle_written_inodeblock: live inodedep <pointer>
   5 softdep_deallocate_dependencies: unrecovered I/O error
   6 ffs_blkfree_cg: freeing free frag
   6 spin lock held too long
   6 ufs_dirrem: Bad link count 2 on parent
   7 ufs_dirbad: /: bad dir ino 486713 at offset 0: mangled entry
   8 handle_written_inodeblock: Invalid link count 65535 for inodedep <pointer>
   9 cam_periph_release_locked_buses: release of <pointer> when refcount is zero
   9 ffs_valloc: dup alloc
  10 ffs_blkfree_cg: freeing free block
  15 worklist_remove: <pointer> pagedep(<pointer>) not on list

$ grep -h Subject /place/mail/textdumps.2013-04-* | perl -pe 's/Subject: \[sendcore\] panic on //' | sort -u | grep ^ws[0-9] | perl -pe 's/ws[0-9]+-[0-9]+: //' | perl -pe 's/0x[0-f]+/<pointer>/g' | sort | uniq -c | sort -n
   1 NMI indicates hardware failure
   1 handle_written_inodeblock: Invalid link count 32946 for inodedep <pointer>
   1 initiate_write_inodeblock_ufs2: already started
   1 sleeping thread
   1 softdep_setup_freeblocks: inode busy
   1 ufs_dirbad: /place: bad dir ino 218737 at offset 512: mangled entry
   2 handle_written_inodeblock: Invalid link count 65535 for inodedep <pointer>
   3 indir_trunc: Bad indirdep 0 from buf <pointer>
   3 softdep_setup_inomapdep: dependency <pointer> for newinode already exists
   4 ffs_blkfree_cg: freeing free frag
   4 handle_written_inodeblock: live inodedep <pointer>
   5 softdep_deallocate_dependencies: unrecovered I/O error
   5 ufs_dirrem: Bad link count 2 on parent
   6 spin lock held too long
  10 worklist_remove: <pointer> pagedep(<pointer>) not on list
  11 ffs_blkfree_cg: freeing free block
  12 cam_periph_release_locked_buses: release of <pointer> when refcount is zero
  16 ffs_valloc: dup alloc
  49 Bad effnlink fip <pointer>, fdp <pointer>, tdp <pointer>

$ grep -h Subject /place/mail/textdumps.2013-05-* | perl -pe 's/Subject: \[sendcore\] panic on //' | sort -u | grep ^ws[0-9] | perl -pe 's/ws[0-9]+-[0-9]+: //' | perl -pe 's/0x[0-f]+/<pointer>/g' | sort | uniq -c | sort -n
   1 Bad effnlink fip <pointer>, fdp <pointer>, tdp <pointer>
   1 NMI indicates hardware failure
   1 initiate_write_bmsafemap: inode -646016512 marked free
   1 ufs_dirbad: /: bad dir ino 91115 at offset 512: mangled entry
   1 ufs_dirbad: /db: bad dir ino 13836 at offset 512: mangled entry
   1 ufs_dirbad: /db: bad dir ino 33541 at offset 0: mangled entry
   1 ufs_dirbad: /db: bad dir ino 43264 at offset 0: mangled entry
   1 ufs_dirbad: /place: bad dir ino 47141 at offset 512: mangled entry
   1 ufs_dirbad: /var: bad dir ino 17281 at offset 512: mangled entry
   1 ufs_dirrem: Bad link count 2 on parent
   1 vm_fault: fault on nofault entry, addr: ffffff84072d2000
   2 indir_trunc: Bad indirdep 0 from buf <pointer>
   4 ffs_blkfree_cg: freeing free frag
   4 handle_written_inodeblock: Invalid link count 65535 for inodedep <pointer>
   5 spin lock held too long
   7 ffs_valloc: dup alloc
   8 ffs_blkfree_cg: freeing free block
  10 worklist_remove: <pointer> pagedep(<pointer>) not on list
  14 cam_periph_release_locked_buses: release of <pointer> when refcount is zero
  43 empty