Skip to content

Instantly share code, notes, and snippets.

@rzezeski
Last active February 11, 2020 15:35
Show Gist options
  • Save rzezeski/226e9c7c2736b709105a823bd2f874e8 to your computer and use it in GitHub Desktop.
Save rzezeski/226e9c7c2736b709105a823bd2f874e8 to your computer and use it in GitHub Desktop.
check for i40e freeze

Step 1: Run mdb -k

# mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci ufs ip hook neti sockfs arp usba fctl stmf_sbd stmf zfs mm sd lofs idm sata crypto fcp random cpc logindmux ptm kvm sppp nsmb smbsrv nfs mr_sas i40e ]

Step 2: Find i40e mac_impl_t pointers.

> ::walk mac_impl_cache | ::printf "0x%p %s\n" mac_impl_t . mi_name ! grep i40
0xfffffe59416dfcc8 i40e1
0xfffffe59416e6848 i40e0

Step 3: For each of these pointers look at the Tx ring stats.

> 0xfffffe59416e6848::print mac_impl_t mi_driver | ::print i40e_t i40e_trqpairs | ::array i40e_trqpair_t 0x8 | ::printf "%p %p %u %u %u %u %u (%u)\n" i40e_trqpair_t itrq_i40e . itrq_txstat.itxs_packets.value.ui64 itrq_txstat.itxs_descriptors.value.ui64 itrq_txstat.itxs_recycled.value.ui64 itrq_txstat.itxs_err_nodescs.value.ui64 itrq_txstat.itxs_num_unblocked.value.ui64 itrq_desc_free
fffffe593ef1b000 fffffe5941634000 79 79 79 0 0 (1024)
fffffe593ef1b000 fffffe5941634770 6 6 6 0 0 (1024)
fffffe593ef1b000 fffffe5941634ee0 6 6 6 0 0 (1024)
fffffe593ef1b000 fffffe5941635650 0 0 0 0 0 (1024)
fffffe593ef1b000 fffffe5941635dc0 1236826 6044351 6044351 0 0 (1024)
fffffe593ef1b000 fffffe5941636530 12 12 12 0 0 (1024)
fffffe593ef1b000 fffffe5941636ca0 6194881 28952801 28951783 134 0 (6)
fffffe593ef1b000 fffffe5941637410 0 0 0 0 0 (1024)

Step 4: Look for rings where the number inside the parentheses is significantly less than 1024 (this is the number of free descriptors on that ring). E.g., in the example above ring fffffe5941636ca0 has the value 6. Then run the following command to create a raw dump of the descriptors. The ring address is the second column in the output above. You have to replace the address with your own in two places. I.e., replace both occurences of fffffe5941636ca0 with your own address.

fffffe5941636ca0::print i40e_trqpair_t itrq_desc_ring | ::array i40e_tx_desc_t 0t1024 | ::printf "0x%x\n" i40e_tx_desc_t cmd_type_offset_bsz ! xargs -L1 echo > /var/tmp/i40e_desc_ring_0xfffffe5941636ca0.out

Do this for each ring that has a low free descriptor count. Gather up these files and email them to support. There is no sensitive information contained in this output.

@rzezeski
Copy link
Author

./i40e_tx_dsec ~/Downloads/i40e_desc_ring_0xfffffe5941636ca0.out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment