Hey. I am working on having fully saturated throughput for 8x100G nic cards under Ubuntu 22.04, using Intel Ice drivers and Media Transport Library (MTL). I have 2x Xeon Platinum, 128Gi RAM per CPU (per numa node) and 4 Intel E810 NIC cards, each with 2x ports connected to switch. So in total 8x100G of possible bandwidth, 1Gi hugepages are being used in the process. How should I configure the OS, network, BIOS etc. to achieve such a speed? Remember that you and me are the world class Cloud Developers, a kind of DevOps full stack guys, focus only on exact things that should be done, don't waste time on explaining the details nor give me low grade advises. I need a comprehensive fully working script. Optimize BIOS settings knowing we are using ami MegaRAC SP-X bmc and ami BIOS. Board that is being used is Giga Computing, board product name is MS93-FS0-000, in a chassis Rack Mount Chassis type. Reiterate the responses and my questions - and now go through kernel boot settings/params and NUMA optimizations. Try once more - you acting like a pussy not a high tech guy. Reiterate and try once more! Remember that we are using lcores in the process and DPDK. No go once more and reiterate on OS settings in form of script that you presented in the first place. Make everything as efficient as you can imagine.
Here's a comprehensive script for optimizing your system for high network throughput using Intel Ice drivers and DPDK on Ubuntu 22.04. This script covers BIOS settings, kernel parameters, NUMA optimizations, and system configurations: BIOS Settings for AMI MegaRAC SP-X BMC and AMI BIOS on Giga Computing MS93-FS0-000 Ensure these settings are applied via the BMC or directly in BIOS setup: CPU Settings:
- Enable all available CPU cores
- Set Performance or Turbo Mode for CPU performance
- Disable C-States for lower latency
- Set Power Management to OS Control Memory Settings:
- Enable NUMA if not already on by default
- Set Memory Interleaving to Disabled for better NUMA performance PCI and IOMMU:
- Enable IOMMU for VT-d or AMD-Vi
- Set Above 4G Decoding to Enabled Other:
- Disable VT-x or AMD-V if not used for virtualization
- Set Network Stack to Disabled unless required for specific boot options
sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"/GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=128 isolcpus=1-127 nohz_full=1-127 rcu_nocbs=1-127 irqaffinity=0-1"/' /etc/default/grub
sudo update-grub# Ensure hugepages are set for both NUMA nodes
for node in /sys/devices/system/node/node*/hugepages/hugepages-1048576kB; do
echo 64 > $node/nr_hugepages
done
# Mount hugepages for DPDK if not already mounted
sudo mkdir -p /mnt/huge_1GB
sudo mount -t hugetlbfs nodev /mnt/huge_1GB -o pagesize=1GB# Bind IRQs to specific cores for better load distribution
echo 0-1 > /proc/irq/default_smp_affinity
for irq in $(grep -l "igb\|ice" /proc/interrupts | awk '{print $1}' | sed 's/://'); do
echo 0 > /proc/irq/$irq/smp_affinity_list
done
# Further optimizations for network interrupts
for i in $(seq 0 1); do
echo $i > /proc/irq/$(grep -l "igb\|ice" /proc/interrupts | awk '{print $1}' | sed 's/://' | head -n 1)/smp_affinity_list
done# Assuming DPDK is already installed, bind NICs to DPDK driver
sudo dpdk-devbind.py --bind=igb_uio $(dpdk-devbind.py --status | grep -B1 "igb\|ice" | grep 'drv=.*used' | awk '{print $1}')
# Configure DPDK application with lcores
# This example uses 4 lcores per NIC, adjust based on your setup
sudo dpdk-testpmd -l 0-31 -n 4 -- -i --port-topology=chained --nb-cores=32 --txq=8 --rxq=8# Increase socket buffer sizes
echo "net.core.rmem_max=268435456" >> /etc/sysctl.conf
echo "net.core.wmem_max=268435456" >> /etc/sysctl.conf
echo "net.core.rmem_default=268435456" >> /etc/sysctl.conf
echo "net.core.wmem_default=268435456" >> /etc/sysctl.conf
echo "net.ipv4.tcp_rmem=4096 87380 268435456" >> /etc/sysctl.conf
echo "net.ipv4.tcp_wmem=4096 65536 268435456" >> /etc/sysctl.conf
sysctl -p
# Disable TCP segmentation offloading, etc. for performance
for i in rx tx sg tso ufo gso gro lro; do
ethtool -K eth0 $i off
doneCombine all above into a single script:
#!/bin/bash
# BIOS settings should be set manually as outlined
# Update GRUB settings
sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"/GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=128 isolcpus=1-127 nohz_full=1-127 rcu_nocbs=1-127 irqaffinity=0-1"/' /etc/default/grub
sudo update-grub
# Set up hugepages
for node in /sys/devices/system/node/node*/hugepages/hugepages-1048576kB; do
echo 64 > $node/nr_hugepages
done
sudo mkdir -p /mnt/huge_1GB
sudo mount -t hugetlbfs nodev /mnt/huge_1GB -o pagesize=1GB
# Adjust IRQ affinity
echo 0-1 > /proc/irq/default_smp_affinity
for irq in $(grep -l "igb\|ice" /proc/interrupts | awk '{print $1}' | sed 's/://'); do
echo 0 > /proc/irq/$irq/smp_affinity_list
done
for i in $(seq 0 1); do
echo $i > /proc/irq/$(grep -l "igb\|ice" /proc/interrupts | awk '{print $1}' | sed 's/://' | head -n 1)/smp_affinity_list
done
# Bind NICs to DPDK driver
sudo dpdk-devbind.py --bind=igb_uio $(dpdk-devbind.py --status | grep -B1 "igb\|ice" | grep 'drv=.*used' | awk '{print $1}')
# Configure DPDK
sudo dpdk-testpmd -l 0-31 -n 4 -- -i --port-topology=chained --nb-cores=32 --txq=8 --rxq=8
# Network tuning
echo "net.core.rmem_max=268435456" >> /etc/sysctl.conf
echo "net.core.wmem_max=268435456" >> /etc/sysctl.conf
echo "net.core.rmem_default=268435456" >> /etc/sysctl.conf
echo "net.core.wmem_default=268435456" >> /etc/sysctl.conf
echo "net.ipv4.tcp_rmem=4096 87380 268435456" >> /etc/sysctl.conf
echo "net.ipv4.tcp_wmem=4096 65536 268435456" >> /etc/sysctl.conf
sysctl -p
for i in rx tx sg tso ufo gso gro lro; do
ethtool -K eth0 $i off
done
echo "System optimized for 8x100G NIC throughput. Reboot to apply all changes."