Amazon Elastic Network Adapter (ENA) on CentOS 7

gistfile1.txt

sudo su

yum --enablerepo=extras install epel-release
yum -y install patch dkms kernel-devel perl
yum update

#Required for kernel num 5:
yum --enablerepo=elrepo-kernel -y install kernel-ml-devel

reboot

#After reboot:
sudo su

###############Update kernel START###################
## Update to latest kernel - optional
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/ elrepo-release-7.0-5.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install kernel-ml

awk -F\' /^menuentry/{print\$2} /etc/grub2.cfg
#Note: In the list of kernels, identify the latest and remeber it's index (index starts from 0)
#In case the first item in the list (index=0) is the kernel you want to activate, run:
grub2-set-default 0
grub2-mkconfig -o /boot/grub2/grub.cfg

 vi /etc/default/grub
#change line:  GRUB_DEFAULT=saved  to GRUB_DEFAULT=0  
reboot

#verify current kernel via:
uname -a

#remove old kernels:
rpm -qa | grep kernel | sort
#for the kernels in the list , that you want to remove:
yum remove REPLACE_WITH_KERNEL_NAME

###############Update kernel END  ###################

reboot

sudo su


//TODO fetch latest ena_driver from: https://github.com/amzn/amzn-drivers/releases, this script is based on 2.2.9
cd /tmp
curl -o ena_linux_2.2.9.tar.gz https://codeload.github.com/amzn/amzn-drivers/tar.gz/ena_linux_2.2.9
tar zxvf ena_linux_2.2.9.tar.gz
mv amzn-drivers-ena_linux_2.2.9 /usr/src/ena-2.2.9

#Copy the bellow and paste in command line (lines 52-63):
cat <<EOF > /usr/src/ena-2.2.9/dkms.conf
PACKAGE_NAME="ena"
PACKAGE_VERSION="2.2.9"
AUTOINSTALL="yes"
REMAKE_INITRD="yes"
BUILT_MODULE_LOCATION[0]="kernel/linux/ena"
BUILT_MODULE_NAME[0]="ena"
DEST_MODULE_LOCATION[0]="/updates"
DEST_MODULE_NAME[0]="ena"
CLEAN="cd kernel/linux/ena; make clean"
MAKE="cd kernel/linux/ena; make BUILD_KERNEL=\${kernelver}"
EOF

dkms add -m ena -v 2.2.9
dkms build -m ena -v 2.2.9
dkms install -m ena -v 2.2.9

dracut -f --add-drivers ena

##Verification: 
modinfo ena
##### You should get similar info:  #####
#filename:       /lib/modules/5.7.2-1.el7.elrepo.x86_64/extra/ena.ko
#version:        2.2.9g
#license:        GPL
#description:    Elastic Network Adapter (ENA)
#author:         Amazon.com, Inc. or its affiliates
#srcversion:     CBF3F17443FAB1A53E24D56
#alias:          pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
#alias:          pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
#alias:          pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
#alias:          pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
#alias:          pci:v00001D0Fd00000051sv*sd*bc*sc*i*
#depends:
#retpoline:      Y
#name:           ena
#vermagic:       5.7.2-1.el7.elrepo.x86_64 SMP mod_unload modversions
#parm:           debug:Debug level (0=none,...,16=all) (int)
#parm:           rx_queue_size:Rx queue size. The size should be a power of 2. Max value is 8K
# (int)
#parm:           force_large_llq_header:Increases maximum supported header size in LLQ mode to 224 bytes, while reducing the maximum TX queue size by half.
# (int)
#parm:           num_io_queues:Sets number of RX/TX queues to allocate to device. The maximum value depends on the device and number of online CPUs.
# (int)

# The below is relevant only if the instance did not have ENA enabled before :
#Stop the instance and from another AWS  instance run:
aws ec2 modify-instance-attribute --region THE_INSTANCE_REGION --instance-id THE_INSTANCE_ID --ena-support
#e.g: aws ec2 modify-instance-attribute --region ap-southeast-1 --instance-id i-0123456789abcdef --ena-support

#Create an image from the instance
#Open the image on ENA supportive instance type (e.g. M5 , R4,.. ) 
#Validate interface:
ifconfig -s -a
#get Iface name, e.g: ens5
ethtool -i ens5
##### You should get similar info:  #####
#driver: ena
#version: 2.1.1g   ################## the ENA version #############
#firmware-version:
#expansion-rom-version:
#bus-info: 0000:00:05.0
#supports-statistics: yes
#supports-test: no
#supports-eeprom-access: no
#supports-register-dump: no
#supports-priv-flags: no

######################################################################################
#  Find this gist useful ?? : Please share your love by clicking the "Star" icon above
######################################################################################

Just, Thank you, it works like a charm, just in the row 8, after the reboot I needed to execute the "sudo su" to be able to copy the ena_driver to /usr/src/ena-1.5.3

Finding errors in 1.5.3, appears others saw the same thing in later 1.4 but I'm seeing them in 1.5.3

CentOS 7.2

[Fri Sep 28 12:35:01 2018] ena 0000:00:05.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 93.
[Fri Sep 28 12:35:01 2018] ena 0000:00:05.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 94.
[Fri Sep 28 12:35:01 2018] ena 0000:00:05.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 95.

Affecting C5 AWS instances so far that I've found.

oldish kernel "3.10.0-693.11.6.el7.x86_64"

Any ideas, and or what can I provide?

Thanks

Finding errors in 1.5.3, appears others saw the same thing in later 1.4 but I'm seeing them in 1.5.3

CentOS 7.2

[Fri Sep 28 12:35:01 2018] ena 0000:00:05.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 93.
[Fri Sep 28 12:35:01 2018] ena 0000:00:05.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 94.
[Fri Sep 28 12:35:01 2018] ena 0000:00:05.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 95.

Affecting C5 AWS instances so far that I've found.

oldish kernel "3.10.0-693.11.6.el7.x86_64"

Any ideas, and or what can I provide?

Thanks

Script now updated to 1.6.0

Ray,

Have you tried this as a bash script in a Packer build?

I am getting this error : any idea what could be the reason.

_[root@ip-172-31-44-34 tmp]# dkms install -m ena -v 1.6.0

ena.ko.xz:
Running module version sanity check.

• Original module
  • Found /lib/modules/3.10.0-957.5.1.el7.x86_64/kernel/drivers/net/ethernet/amazon/ena/ena.ko.xz
  • Storing in /var/lib/dkms/ena/original_module/3.10.0-957.5.1.el7.x86_64/x86_64/
  • Archiving for uninstallation purposes
• Installation
  • Installing to /lib/modules/3.10.0-957.5.1.el7.x86_64/extra/
    Adding any weak-modules
    Failed to install module enanvme_core_

Ray,

Have you tried this as a bash script in a Packer build?

No.

Updated to ENA v. 2.0.3

For anyone that following this and updated the Kernel to 5.0.4 and is getting the error message "Your kernel headers for kernel 5.0.4... cannot be found at..." when running dkms build -m ena -v 2.0.3

You need to install the kernel-ml-devel package, to do this, run
yum --enablerepo=elrepo-kernel -y install kernel-ml-devel

And proceed with the command that failed.

yup just hit that @gmogames thanks for the heads up

Updated to ENA v. 2.1.1

For anyone that following this and updated the Kernel to 5.0.4 and is getting the error message "Your kernel headers for kernel 5.0.4... cannot be found at..." when running dkms build -m ena -v 2.0.3

You need to install the kernel-ml-devel package, to do this, run
yum --enablerepo=elrepo-kernel -y install kernel-ml-devel

And proceed with the command that failed.

Updated the script following your commnet.

FYI
Recent success result for modinfo ena
Represent as following

filename: /lib/modules/5.3.5-1.el7.elrepo.x86_64/extra/ena.ko
version: 2.1.1g
license: GPL
description: Elastic Network Adapter (ENA)
author: Amazon.com, Inc. or its affiliates
srcversion: F342F1BEE707726508E3DF8
alias: pci:v00001D0Fd0000EC21svsdbcsci*
alias: pci:v00001D0Fd0000EC20svsdbcsci*
alias: pci:v00001D0Fd00001EC2svsdbcsci*
alias: pci:v00001D0Fd00000EC2svsdbcsci*
depends:
retpoline: Y
name: ena
vermagic: 5.3.5-1.el7.elrepo.x86_64 SMP mod_unload modversions
parm: debug:Debug level (0=none,...,16=all) (int)
parm: rx_queue_size:Rx queue size. The size should be a power of 2. Max value is 8K
(int)
parm: force_large_llq_header:Increases maximum supported header size in LLQ mode to 224 bytes, while reducing the maximum TX queue size by half.
(int)

updated to version 2.2.9