Debug

01.md

Problem

Survive between 10 and 15 minutes between bootloops.

Observation

Print 2 lines in System Event Log during the freeze/reboot:

b2 | 09/15/2018 | 08:43:47 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
b3 | 09/15/2018 | 08:43:47 | Unknown #0x1a |  | Asserted

I found nothing in the logs of my main operating system. PCI bus:00 device:05 function:0 refers to the PCI chipset that handles my LSI Controller Card that manages my hard drives. But it could be the cause or the consequence of our bug.

The system survived longer when I did a kexec (as and because ht kexec took a looooooooong time to execute). Even more, the system seems to work quite well on Linux kernel 4.12.0-2-amd64. The default kernel is 4.16.0-1-amd64.

Hypothesis

• It comes from a new version of a software (eg. linux) which is uncompatible with a broken firmware
• It comes from an application that try to access a non supported feature of our PCI bus / LSI logic controller
• The bus chipset is dead
• The LSI logic card controller is dead

Tried

• I removed every trace of OpenManage / OMSA.
• I tried to stop most of the services.
• I checked that the system fails AFTER the complete boot of the system as stated by systemd.

I tried to remove 00:05.0:

echo 1 > /sys/bus/pci/devices/0000\:00\:05.0/remove

But I loose my access to disks... which makes sense at least...

• I tried an old kernel:

kexec -l /boot/vmlinuz-4.12.0-2-amd64 --append="$( cat /proc/cmdline ) panic=10 console=tty0 console=ttyS0,115200n8" --initrd=/boot/initrd.img-4.12.0-2-amd64
kexec -e

Step 1 - The system is stable on a kexeced Linux 4.12.0-2-amd64 from Linux 4.16.0-1-amd64

• The system ran for 55 minutes without crashing

Problem: the only thing I know is the problem has been introduced / triggered by something added in Linux between 4.12.0 and 4.16.0.

I have the following installed kernels on my system, I took the date of the initrd as it is generated during the installation unlike vmlinuz and config which have the date of the packet creation:

• 4.12.0-2-amd64 2017-09-27
• 4.13.0-1-amd64 2017-12-24
• 4.14.0-2-amd64 2018-01-18
• 4.14.0-3-amd64 2018-03-27
• 4.15.0-2-amd64 2018-04-24
• 4.15.0-3-amd64 2018-04-29
• 4.16.0-1-amd64 2018-05-22

And we have a problem, as the server has been rebooted during august. So we were running the same kernel for a while. Nothing new here.

What's next ?

• Read the changelog of Linux 4.16
• Do a kexec on the same kernel to get some return from the Serial Over Lan proposed by IPMI
• Prevent kernel removal (at least 4.12)
• Install Linux 4.18 and test it

Could it be I am wrong? And be a coincidence? I have run a ipmi ... power reset command, might also have fixed the thing... Let's see...

Failure: The system crashed after 56 minutes

I was running ipmi ... sel list and it crashed just after. Trying ipmi ... power off and wait 5 minutes before ipmi ... power on. We will try to boot Linux 4.16 without running any ipmi command after ipmi ... power on.

And... that's a failure!

   1 | 09/15/2018 | 11:03:57 | Event Logging Disabled #0x72 | Log area reset/cleared | Asserted
   2 | 09/15/2018 | 11:11:54 | Memory #0x1b | Transition to Non-critical from OK | Asserted
   3 | 09/15/2018 | 11:14:29 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
   4 | 09/15/2018 | 11:14:29 | Unknown #0x1a |  | Asserted

We will try to reboot on kernel 4.12 with kexec and install latest kernel

• https://askubuntu.com/questions/668408/keep-old-kernels
• https://utcc.utoronto.ca/~cks/space/blog/linux/UbuntuKernelAutoremove

Check that the package is still here: apt info linux-image-4.12.0-2-generic

And fixed it in /etc/apt/apt.conf.d/01autoremove-kernels-stop:

APT::NeverAutoRemove
{
   "^linux-image-4\.12\.0-2-generic$";
}

And update only main kernel (to save time):

apt update
apt install linux-image-amd64

In fact, we need to fix many more packages. But any way, our 4.12 kernel should be gone a long time ago.

Ok it failed. It seems it failed at the very same moment I tried to write ipmi ... sol activate Let's forget IPMI

kexec -l /boot/vmlinuz-4.12.0-2-amd64 --append="$( cat /proc/cmdline ) panic=10" --initrd=/boot/initrd.img-4.12.0-2-amd64
kexec -e

And still try to upgrade

Step 3 - It worked for some hours on 4.18 but eventually failed

Try to get some info with lsiutil. No, there is absolutely no infomation on how to use this tool... There are some reset options which seems to be interesting, but I don't want to try them as I am pretty sure it can bite hard, they are not documented and as I don't know the origin of the problem, it's only speculation.

The log option returns three or four columns containing hexadecimal data that don't make sense to me. Linux crashed just after outputting the data...

Step 4: No solution yet, assuming it's a hardware error

Extracting data. Unable to do a pg_dump as it will crash before the end. Will need to copy the whole database manually contained in /var/lib/postgresql. Need to be careful with the architecture+postgre version we will use as it could fail to read it. To try to stabilize the system and limit the network usage, I disabled most of the services and all the containers.

After that, I have setup a rsync that support the server reboots:

 while true; do rsync -avz --partial --progress --timeout=60 -e 'ssh -p 110' [email protected]:/var/lib/lxc/db/rootfs/var/lib/postgresql/9.6 ./postgresql/9.6; done

But it's 17GB of (compressible) data to send over an ADSL uplink of 60 kB/s. At least with the compression I can achieve a bandwidth of ~150 to ~240 kB/s. Still, it should take around 24 hours to complete...

Backuping media:

while true; do rsync -avz --partial --progress --timeout=60 -e 'ssh -p 110' [email protected]:/var/lib/lxc/matrix/rootfs/var/lib/matrix-synapse ./; done

Scratchpad

To gain a Serial over Lan/Wan access (you need to configure Bios, Grub and Linux to have information appearing):

ipmitool -I lanplus -H xxxx -U xxxx -P xxxx sol activate

If you disconnected, you must deactivate it after opening a new session:

ipmitool -I lanplus -H xxxx -U xxxx -P xxxx sol deactivate

To list System Events logs :

ipmitool -I lanplus -H xxxx -U xxxx -P xxxx sel list

Slower but could query more information:

ipmitool -I lanplus -H xxxx -U xxxx -P xxxx sel elist

Clear the log as it has a limited number of entries:

ipmitool -I lanplus -H xxxx -U xxxx -P xxxx sel clear

Show information about a specific event:

ipmitool -I lanplus -H xxxx -U xxxx -P xxxx sel get 0xb7

Boot, reboot, reset:

ipmitool -I lanplus -H xxxx -U xxxx -P xxxx power off
ipmitool -I lanplus -H xxxx -U xxxx -P xxxx power on
ipmitool -I lanplus -H xxxx -U xxxx -P xxxx power cycle
ipmitool -I lanplus -H xxxx -U xxxx -P xxxx power reset

Some fun info:

ipmitool -I lanplus -H xxxx -U xxxx -P xxxx chassis info
ipmitool -I lanplus -H xxxx -U xxxx -P xxxx sdr info

10 ipmi sel list short.txt

  a8 | 09/15/2018 | 08:01:14 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  a9 | 09/15/2018 | 08:01:14 | Unknown #0x1a |  | Asserted
  aa | 09/15/2018 | 08:09:58 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  ab | 09/15/2018 | 08:09:58 | Unknown #0x1a |  | Asserted
  ac | 09/15/2018 | 08:17:05 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  ad | 09/15/2018 | 08:17:05 | Unknown #0x1a |  | Asserted
  ae | 09/15/2018 | 08:20:08 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  af | 09/15/2018 | 08:20:08 | Unknown #0x1a |  | Asserted
  b0 | 09/15/2018 | 08:29:32 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  b1 | 09/15/2018 | 08:29:32 | Unknown #0x1a |  | Asserted
  b2 | 09/15/2018 | 08:43:47 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  b3 | 09/15/2018 | 08:43:47 | Unknown #0x1a |  | Asserted

20 lspci.txt

root@havelock:~# lspci
00:00.0 Host bridge: Intel Corporation 5000X Chipset Memory Controller Hub (rev 12)
00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 2 (rev 12)
00:03.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 3 (rev 12)
00:04.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 4 (rev 12)
00:05.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 5 (rev 12)
00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 6-7 (rev 12)
00:07.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 7 (rev 12)
00:08.0 System peripheral: Intel Corporation 5000 Series Chipset DMA Engine (rev 12)
00:10.0 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev 12)
00:10.1 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev 12)
00:10.2 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev 12)
00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev 12)
00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev 12)
00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev 12)
00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev 12)
00:1c.0 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 1 (rev 09)
00:1d.0 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09)
00:1d.1 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2 (rev 09)
00:1d.2 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 (rev 09)
00:1d.3 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #4 (rev 09)
00:1d.7 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller (rev 09)
00:1f.1 IDE interface: Intel Corporation 631xESB/632xESB IDE Controller (rev 09)
00:1f.2 IDE interface: Intel Corporation 631xESB/632xESB/3100 Chipset SATA IDE Controller (rev 09)
01:00.0 PCI bridge: Intel Corporation 6702PXH PCI Express-to-PCI Bridge A (rev 09)
02:08.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068 PCI-X Fusion-MPT SAS (rev 01)
03:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c2)
04:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5708 Gigabit Ethernet (rev 11)
05:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01)
05:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge (rev 01)
06:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1 (rev 01)
06:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E2 (rev 01)
07:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c2)
08:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5708 Gigabit Ethernet (rev 11)
09:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
0a:01.0 SCSI storage controller: Adaptec AHA-3960D / AIC-7899A U160/m (rev 01)
0a:01.1 SCSI storage controller: Adaptec AHA-3960D / AIC-7899A U160/m (rev 01)

30 lspci -tv.txt

-[0000:00]-+-00.0  Intel Corporation 5000X Chipset Memory Controller Hub
           +-02.0-[05-0a]--+-00.0-[06-09]--+-00.0-[07-08]----00.0-[08]----00.0  Broadcom Limited NetXtreme II BCM5708 Gigabit Ethernet
           |               |               \-01.0-[09]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
           |               \-00.3-[0a]--+-01.0  Adaptec AHA-3960D / AIC-7899A U160/m
           |                            \-01.1  Adaptec AHA-3960D / AIC-7899A U160/m
           +-03.0-[0b]--
           +-04.0-[0c]--
           +-05.0-[01-02]----00.0-[02]----08.0  LSI Logic / Symbios Logic SAS1068 PCI-X Fusion-MPT SAS
           +-06.0-[0d]--
           +-07.0-[0e]--
           +-08.0  Intel Corporation 5000 Series Chipset DMA Engine
           +-10.0  Intel Corporation 5000 Series Chipset FSB Registers
           +-10.1  Intel Corporation 5000 Series Chipset FSB Registers
           +-10.2  Intel Corporation 5000 Series Chipset FSB Registers
           +-11.0  Intel Corporation 5000 Series Chipset Reserved Registers
           +-13.0  Intel Corporation 5000 Series Chipset Reserved Registers
           +-15.0  Intel Corporation 5000 Series Chipset FBD Registers
           +-16.0  Intel Corporation 5000 Series Chipset FBD Registers
           +-1c.0-[03-04]----00.0-[04]----00.0  Broadcom Limited NetXtreme II BCM5708 Gigabit Ethernet
           +-1d.0  Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1
           +-1d.1  Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2
           +-1d.2  Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3
           +-1d.3  Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #4
           +-1d.7  Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller
           +-1e.0-[0f]----0d.0  Advanced Micro Devices, Inc. [AMD/ATI] ES1000
           +-1f.0  Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller
           +-1f.1  Intel Corporation 631xESB/632xESB IDE Controller
           \-1f.2  Intel Corporation 631xESB/632xESB/3100 Chipset SATA IDE Controller

999 full SEL at 2018-09-15 11:04:00.txt

   1 | 08/28/2018 | 09:06:54 | Event Logging Disabled #0x72 | Log area reset/cleared | Asserted
   2 | 09/09/2018 | 03:41:02 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
   3 | 09/09/2018 | 03:41:02 | Unknown #0x1a |  | Asserted
   4 | 09/10/2018 | 22:56:41 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
   5 | 09/10/2018 | 22:56:41 | Unknown #0x1a |  | Asserted
   6 | 09/13/2018 | 11:44:23 | Memory #0x1b | Transition to Non-critical from OK | Asserted
   7 | 09/13/2018 | 11:45:05 | Memory #0x1b | Transition to Critical from less severe | Asserted
   8 | 09/14/2018 | 02:48:02 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
   9 | 09/14/2018 | 02:48:02 | Unknown #0x1a |  | Asserted
   a | 09/14/2018 | 00:51:09 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
   b | 09/14/2018 | 00:51:09 | Unknown #0x1a |  | Asserted
   c | 09/14/2018 | 00:54:27 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
   d | 09/14/2018 | 00:54:27 | Unknown #0x1a |  | Asserted
   e | 09/14/2018 | 00:57:46 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
   f | 09/14/2018 | 00:57:46 | Unknown #0x1a |  | Asserted
  10 | 09/14/2018 | 22:24:00 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  11 | 09/14/2018 | 22:24:00 | Unknown #0x1a |  | Asserted
  12 | 09/14/2018 | 20:30:53 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  13 | 09/14/2018 | 20:30:53 | Unknown #0x1a |  | Asserted
  14 | 09/14/2018 | 20:38:19 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  15 | 09/14/2018 | 20:38:19 | Unknown #0x1a |  | Asserted
  16 | 09/14/2018 | 20:45:26 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  17 | 09/14/2018 | 20:45:26 | Unknown #0x1a |  | Asserted
  18 | 09/14/2018 | 22:58:47 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  19 | 09/14/2018 | 22:58:47 | Unknown #0x1a |  | Asserted
  1a | 09/14/2018 | 21:00:43 | Memory #0x1b | Transition to Non-critical from OK | Asserted
  1b | 09/14/2018 | 23:12:30 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  1c | 09/14/2018 | 23:12:30 | Unknown #0x1a |  | Asserted
  1d | 09/14/2018 | 21:20:07 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  1e | 09/14/2018 | 21:20:08 | Unknown #0x1a |  | Asserted
  1f | 09/14/2018 | 23:40:31 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  20 | 09/14/2018 | 23:40:32 | Unknown #0x1a |  | Asserted
  21 | 09/14/2018 | 21:46:07 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  22 | 09/14/2018 | 21:46:07 | Unknown #0x1a |  | Asserted
  23 | 09/14/2018 | 21:48:03 | Memory #0x1b | Transition to Non-critical from OK | Asserted
  24 | 09/14/2018 | 21:48:03 | Memory #0x1b | Transition to Critical from less severe | Asserted
  25 | 09/14/2018 | 23:59:50 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  26 | 09/14/2018 | 23:59:50 | Unknown #0x1a |  | Asserted
  27 | 09/14/2018 | 22:06:13 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  28 | 09/14/2018 | 22:06:13 | Unknown #0x1a |  | Asserted
  29 | 09/15/2018 | 00:18:11 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  2a | 09/15/2018 | 00:18:12 | Unknown #0x1a |  | Asserted
  2b | 09/15/2018 | 00:33:10 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  2c | 09/15/2018 | 00:33:10 | Unknown #0x1a |  | Asserted
  2d | 09/15/2018 | 01:01:21 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  2e | 09/15/2018 | 01:01:21 | Unknown #0x1a |  | Asserted
  2f | 09/15/2018 | 01:20:11 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  30 | 09/15/2018 | 01:20:11 | Unknown #0x1a |  | Asserted
  31 | 09/15/2018 | 01:35:19 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  32 | 09/15/2018 | 01:35:20 | Unknown #0x1a |  | Asserted
  33 | 09/14/2018 | 23:39:48 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  34 | 09/14/2018 | 23:39:48 | Unknown #0x1a |  | Asserted
  35 | 09/15/2018 | 01:54:16 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  36 | 09/15/2018 | 01:54:16 | Unknown #0x1a |  | Asserted
  37 | 09/15/2018 | 02:18:57 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  38 | 09/15/2018 | 02:18:57 | Unknown #0x1a |  | Asserted
  39 | 09/15/2018 | 02:35:05 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  3a | 09/15/2018 | 02:35:05 | Unknown #0x1a |  | Asserted
  3b | 09/15/2018 | 02:42:30 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  3c | 09/15/2018 | 02:42:30 | Unknown #0x1a |  | Asserted
  3d | 09/15/2018 | 02:55:57 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  3e | 09/15/2018 | 02:55:57 | Unknown #0x1a |  | Asserted
  3f | 09/15/2018 | 03:13:31 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  40 | 09/15/2018 | 03:13:31 | Unknown #0x1a |  | Asserted
  41 | 09/15/2018 | 03:32:28 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  42 | 09/15/2018 | 03:32:28 | Unknown #0x1a |  | Asserted
  43 | 09/15/2018 | 03:55:33 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  44 | 09/15/2018 | 03:55:33 | Unknown #0x1a |  | Asserted
  45 | 09/15/2018 | 01:57:29 | Memory #0x1b | Transition to Non-critical from OK | Asserted
  46 | 09/15/2018 | 01:57:32 | Memory #0x1b | Transition to Critical from less severe | Asserted
  47 | 09/15/2018 | 04:11:23 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  48 | 09/15/2018 | 04:11:23 | Unknown #0x1a |  | Asserted
  49 | 09/15/2018 | 02:14:10 | Memory #0x1b | Transition to Non-critical from OK | Asserted
  4a | 09/15/2018 | 02:18:02 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  4b | 09/15/2018 | 02:18:02 | Unknown #0x1a |  | Asserted
  4c | 09/15/2018 | 04:28:54 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  4d | 09/15/2018 | 04:28:54 | Unknown #0x1a |  | Asserted
  4e | 09/15/2018 | 02:31:47 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  4f | 09/15/2018 | 02:31:47 | Unknown #0x1a |  | Asserted
  50 | 09/15/2018 | 02:35:05 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  51 | 09/15/2018 | 02:35:05 | Unknown #0x1a |  | Asserted
  52 | 09/15/2018 | 04:57:39 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  53 | 09/15/2018 | 04:57:39 | Unknown #0x1a |  | Asserted
  54 | 09/15/2018 | 05:11:19 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  55 | 09/15/2018 | 05:11:19 | Unknown #0x1a |  | Asserted
  56 | 09/15/2018 | 03:15:28 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  57 | 09/15/2018 | 03:15:28 | Unknown #0x1a |  | Asserted
  58 | 09/15/2018 | 05:25:52 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  59 | 09/15/2018 | 05:25:52 | Unknown #0x1a |  | Asserted
  5a | 09/15/2018 | 03:31:49 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  5b | 09/15/2018 | 03:31:49 | Unknown #0x1a |  | Asserted
  5c | 09/15/2018 | 03:39:49 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  5d | 09/15/2018 | 03:39:49 | Unknown #0x1a |  | Asserted
  5e | 09/15/2018 | 05:51:12 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  5f | 09/15/2018 | 05:51:12 | Unknown #0x1a |  | Asserted
  60 | 09/15/2018 | 06:00:21 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  61 | 09/15/2018 | 06:00:21 | Unknown #0x1a |  | Asserted
  62 | 09/15/2018 | 04:06:16 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  63 | 09/15/2018 | 04:06:16 | Unknown #0x1a |  | Asserted
  64 | 09/15/2018 | 06:47:18 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  65 | 09/15/2018 | 06:47:18 | Unknown #0x1a |  | Asserted
  66 | 09/15/2018 | 04:53:18 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  67 | 09/15/2018 | 04:53:18 | Unknown #0x1a |  | Asserted
  68 | 09/15/2018 | 04:58:27 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  69 | 09/15/2018 | 04:58:27 | Unknown #0x1a |  | Asserted
  6a | 09/15/2018 | 05:00:23 | Memory #0x1b | Transition to Non-critical from OK | Asserted
  6b | 09/15/2018 | 05:01:34 | Memory #0x1b | Transition to Critical from less severe | Asserted
  6c | 09/15/2018 | 05:03:39 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  6d | 09/15/2018 | 05:03:39 | Unknown #0x1a |  | Asserted
  6e | 09/15/2018 | 07:21:08 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  6f | 09/15/2018 | 07:21:08 | Unknown #0x1a |  | Asserted
  70 | 09/15/2018 | 05:24:46 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  71 | 09/15/2018 | 05:24:46 | Unknown #0x1a |  | Asserted
  72 | 09/15/2018 | 05:32:08 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  73 | 09/15/2018 | 05:32:08 | Unknown #0x1a |  | Asserted
  74 | 09/15/2018 | 07:43:50 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  75 | 09/15/2018 | 07:43:50 | Unknown #0x1a |  | Asserted
  76 | 09/15/2018 | 05:46:47 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  77 | 09/15/2018 | 05:46:47 | Unknown #0x1a |  | Asserted
  78 | 09/15/2018 | 05:49:58 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  79 | 09/15/2018 | 05:49:58 | Unknown #0x1a |  | Asserted
  7a | 09/15/2018 | 05:56:18 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  7b | 09/15/2018 | 05:56:18 | Unknown #0x1a |  | Asserted
  7c | 09/15/2018 | 06:03:20 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  7d | 09/15/2018 | 06:03:20 | Unknown #0x1a |  | Asserted
  7e | 09/15/2018 | 08:12:13 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  7f | 09/15/2018 | 08:12:13 | Unknown #0x1a |  | Asserted
  80 | 09/15/2018 | 06:15:02 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  81 | 09/15/2018 | 06:15:02 | Unknown #0x1a |  | Asserted
  82 | 09/15/2018 | 06:21:53 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  83 | 09/15/2018 | 06:21:53 | Unknown #0x1a |  | Asserted
  84 | 09/15/2018 | 06:29:51 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  85 | 09/15/2018 | 06:29:51 | Unknown #0x1a |  | Asserted
  86 | 09/15/2018 | 06:35:24 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  87 | 09/15/2018 | 06:35:24 | Unknown #0x1a |  | Asserted
  88 | 09/15/2018 | 06:42:47 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  89 | 09/15/2018 | 06:42:48 | Unknown #0x1a |  | Asserted
  8a | 09/15/2018 | 06:50:01 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  8b | 09/15/2018 | 06:50:01 | Unknown #0x1a |  | Asserted
  8c | 09/15/2018 | 06:56:30 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  8d | 09/15/2018 | 06:56:30 | Unknown #0x1a |  | Asserted
  8e | 09/15/2018 | 06:58:28 | Memory #0x1b | Transition to Non-critical from OK | Asserted
  8f | 09/15/2018 | 06:58:28 | Memory #0x1b | Transition to Critical from less severe | Asserted
  90 | 09/15/2018 | 09:06:17 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  91 | 09/15/2018 | 09:06:17 | Unknown #0x1a |  | Asserted
  92 | 09/15/2018 | 07:09:34 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  93 | 09/15/2018 | 07:09:34 | Unknown #0x1a |  | Asserted
  94 | 09/15/2018 | 07:15:32 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  95 | 09/15/2018 | 07:15:32 | Unknown #0x1a |  | Asserted
  96 | 09/15/2018 | 07:22:37 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  97 | 09/15/2018 | 07:22:37 | Unknown #0x1a |  | Asserted
  98 | 09/15/2018 | 07:25:15 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  99 | 09/15/2018 | 07:25:15 | Unknown #0x1a |  | Asserted
  9a | 09/15/2018 | 07:28:13 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  9b | 09/15/2018 | 07:28:14 | Unknown #0x1a |  | Asserted
  9c | 09/15/2018 | 07:31:04 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  9d | 09/15/2018 | 07:31:04 | Unknown #0x1a |  | Asserted
  9e | 09/15/2018 | 07:36:42 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  9f | 09/15/2018 | 07:36:42 | Unknown #0x1a |  | Asserted
  a0 | 09/15/2018 | 07:42:32 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  a1 | 09/15/2018 | 07:42:32 | Unknown #0x1a |  | Asserted
  a2 | 09/15/2018 | 07:46:49 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  a3 | 09/15/2018 | 07:46:49 | Unknown #0x1a |  | Asserted
  a4 | 09/15/2018 | 07:55:47 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  a5 | 09/15/2018 | 07:55:47 | Unknown #0x1a |  | Asserted
  a6 | 09/15/2018 | 07:58:29 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  a7 | 09/15/2018 | 07:58:29 | Unknown #0x1a |  | Asserted
  a8 | 09/15/2018 | 08:01:14 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  a9 | 09/15/2018 | 08:01:14 | Unknown #0x1a |  | Asserted
  aa | 09/15/2018 | 08:09:58 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  ab | 09/15/2018 | 08:09:58 | Unknown #0x1a |  | Asserted
  ac | 09/15/2018 | 08:17:05 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  ad | 09/15/2018 | 08:17:05 | Unknown #0x1a |  | Asserted
  ae | 09/15/2018 | 08:20:08 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  af | 09/15/2018 | 08:20:08 | Unknown #0x1a |  | Asserted
  b0 | 09/15/2018 | 08:29:32 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  b1 | 09/15/2018 | 08:29:32 | Unknown #0x1a |  | Asserted
  b2 | 09/15/2018 | 08:43:47 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  b3 | 09/15/2018 | 08:43:47 | Unknown #0x1a |  | Asserted
  b4 | 09/15/2018 | 08:54:46 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  b5 | 09/15/2018 | 08:54:46 | Unknown #0x1a |  | Asserted
  b6 | 09/15/2018 | 09:05:10 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  b7 | 09/15/2018 | 09:05:10 | Unknown #0x1a |  | Asserted
  b8 | 09/15/2018 | 11:01:05 | Critical Interrupt #0x18 | Bus Fatal Error ( PCI bus:00 device:05 function:0) | Asserted
  b9 | 09/15/2018 | 11:01:05 | Unknown #0x1a |  | Asserted