Skip to content

Instantly share code, notes, and snippets.

@FlorianHeigl

FlorianHeigl/01_VEP4600 Infos.md

Last active February 10, 2025 16:01
Show Gist options
  • Save FlorianHeigl/ca7fe009077437983a3dc927d12f339b to your computer and use it in GitHub Desktop.
Save FlorianHeigl/ca7fe009077437983a3dc927d12f339b to your computer and use it in GitHub Desktop.
Download ZIP
VEP4600
Raw
BIOS Passwort.md

Es gibt zwei Userlevel

User

  • das ist das Bootpasswort
  • wenn es gesetzt ist, startet sas System nur mit dem PW)

Admin

  • das ist das BIOS Passwort
  • Wird beim Aufruf des BIOS abgefragt
  • Wenn man nur "enter" drueckt, kommt man ins BIOS, hat aber nicht den "admin" Userlevel
  • Das PW sollte normalerweise SERVICETAG! sein
  • auf dem Auszieher mit den QR Codes oben rechts an der Gehaeusefront
  • das Service Tag findet man auch auf einem Sticker auf dem Deckel
Raw
BIOS Update via DiagOS.md

DiagOS selbst kann mittels einem .deb Package geupdated werden

FW Updates

updatetool --dev=BIOS --update -e filename.img
updatetool --dev=CPLD --update -e filename.jed
updatetool --dev=BMC --update -e filename.bmc

Wenn man am BMC Passwort scheitert (serieller Port) ist das BMC Update ein recht guter Loesungspfad

Das DiagOS zieht sich keine neue IP vom DHCP Server, wenn man den Lan-Port aus- und einsteckt.

Beispiele

Vorbereitung

  • Updates entpacken / Files identifizieren (s.u.)
  • Per TTY in DiagOS einloggen, aktiven LAN-Port identifizieren, IP auslesen.
  • Neue IP bei Bedarf via ifdown / ifup holen
  • Die Update Files per scp in das gebootete DiagOS kopieren

BMC Update

root@dellemc-diag-os:~# updatetool --dev=BMC --update -e VEP4600-BMC-v2.30.ima
Disable device protect 
Write image to BMC
INFO: Yafu INI Configuration File not found... Default options will not be applied...

Creating IPMI session via USB...Done

-------------------------------------------------
YAFUFlash - Firmware Upgrade Utility (Version 5.0.0)

-------------------------------------------------
(C)Copyright 2016, American Megatrends Inc.
Image To be updated is (Image-1)
The Module boot major or minor version is different from the one in the image
So,Type (Y/y) to do Full Firmware Upgrade or (N/n) to exit
Enter your Option : y
****************************************************************************
 WARNING!
        FIRMWARE UPGRADE MUST NOT BE INTERRUPTED ONCE IT IS STARTED.
        PLEASE DO NOT USE THIS FLASH TOOL FROM THE REDIRECTION CONSOLE.
****************************************************************************
Preserving Env Variables...             done
Uploading Firmware Image : 100%... done
Flashing [boot] Module ....
Flashing  Firmware Image : 100%... done
Verifying Firmware Image : 100%... done
Flashing [conf] Module ....
Flashing  Firmware Image : 100%... done
Verifying Firmware Image : 100%... done
Flashing [root] Module ....
...
Flashing [ast2500e] Module ....
Flashing  Firmware Image : 100%... done
Verifying Firmware Image : 100%... done
Setting Env variables...               done
Resetting the firmware..........
write BMC image success
Enable device protect 

Update BMC image success

Beim BMC Update geht die Lueftersteuerung kurz aus (Full)

CPLD

Das hier hat wohl nicht funktioniert

root@dellemc-diag-os:~# updatetool --dev=CPLD --update -e AZUL_CPLD_V12_20210520.jed 

-------------------------------------------------
YAFUFlash - Firmware Upgrade Utility (Version 5.0.0)

-------------------------------------------------
(C)Copyright 2016, American Megatrends Inc.
Beginning CPLD Update...
Uploading Image : 100%... done
Flashing  Firmware Image : 0%Error in ECFStatus 

Error in updating CPLD image
write CPLD image success
Enable device protect 

Update CPLD image success

BIOS

BIOS Update fragt nicht nach Yes/No

root@dellemc-diag-os:~# updatetool --dev=BIOS --update -e VEP4600-BIOS-3.41.0.9-23.BIN
Write image to BIOS
+---------------------------------------------------------------------------+
|            AMI Firmware Update Utility Manufacture for Dell Nv5.11.06.1854|
|      Copyright (C)2018 American Megatrends Inc. All Rights Reserved.      |
+---------------------------------------------------------------------------+
 Reading flash ............... done                
 - ME Data Size checking . ok
 - FFS checksums ......... ok
 - Check RomLayout ........ ok.
 Erasing Main Block .......... done
 Updating Main Block ......... 0x0028C000 (13%) 
 ...
 - Update success for MER                              
WARNING : System must power-off to have the changes take effect!
write BIOS image success
Update BIOS image success

Am Ende also das System mit poweroff runterfahren - oder irgendwann spaeter. Im Fall mit dem CPLD Update Fehler am besten ganz stromlos machen. (Funktionierte so, aber hab es nicht ohne stromlos machen versucht.)

Raw
BIOS update via LAN.md

Uebersicht / Beschreibung:

https://infohub.delltechnologies.com/p/dell-technologies-vep4600-bios-firmware-upgrade/

DiagOS und Unified FW Updater

https://www.dell.com/support/manuals/de-de/virtual-edge-platform-4600-8c/vep4600_os_install_pub/run-the-unified-firmware-updater-tool-from-a-linux-server?guid=guid-556d049c-9a4f-4784-99f2-8678a8eebed9&lang=en-us

DiagOS Partitioning issues

Fehlerbeschreibung

Repartitioning

Echte Loesung war: DiagOS mit Rufus auf frischen USB Stick. Diesen in UEFI Modus booten.

Anleitung dazu: https://www.dell.com/support/manuals/de-de/virtual-edge-platform-4600-8c/ucpe_diag_pub/restore-to-manufacture-diag-os?guid=guid-e0457706-c0c2-4407-8aad-f3b788f53217

Release Notes aktuelle Version 3.8

Release Notes VEP 4600 DiagOS

Download Link Treiber und Tools, UFW

System fuer Arbeiten:

  • Ubuntu 20: Geht
  • CentOS Stream: Geht nicht!

Software arbeitet normal in einer RAM Disk, also genug Ram fuer VM zuordnen.

Alle Zipfiles auspacken das ufw_v.x ist das binary, aufrufen mit

./ufwding 192.168.86.xx interactive

NOTE If an IPMI session error is seen while connecting to the BMC, run following command to reset the BMC prior to re-run of UFW.

Creating IPMI session via network with address 192.168.86.xx...Failed

root@ubs18-04:~/# ipmitool -I lanplus -H xxx.xx.xx.xxx -U admin -P admin mc reset cold

need reset!!!!!

[...]
    9. Reset BMC after CPLD update completion and system bootup
    q. Exit



    Enter your choice:9
Reset BMC, in case of BMC is not in correct status (e.g. BMC in Firmware Update Mode will cause firmware update that through CFU failed)
Sending reset to BMC to recover, need to wait for 180 seconds

Reset sieht man auf der BMC Konsole auch:

[ 9899.290059] ipmi_si IPI0001:00: Maybe ok, but ipmi might run very slowly.

Reset per Menu hilft NICHT

updating BMC image

****************************
* BMC version = 2.0       *
* Back-up Version = 2.0   *
****************************
updating primary image only..
INFO: Yafu INI Configuration File not found... Default options will not be applied...

Creating IPMI session via network with address 192.168.xx.100...Failed
Image update failed
Upgrade Failed & Exiting
/home/floh/VEP4600/temp
Raw
BMC netzwerk config.md

Kabel einstecken sucht scheinbar nach IP Plugging a cable triggers DHCP refresh.

~ # DHCP monitor: Re[  366.450000] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
newing eth0 interface for IPv6
RTNETLINK answers: File exists
Sending ICMPv6 packet: Cannot assign requested address
Renewing DNS for eth0 interface.
DHCP monitor: Renewing eth0 interface for IPv6
RTNETLINK answers: File exists
Sending ICMPv6 packet: Cannot assign requested address
Renewing DNS for eth0 interface.

kein funktionierenden DHCP beobachtet

https://www.dell.com/support/manuals/de-de/virtual-edge-platform-4600-8c/vep4600_os_install_pub/configure-bmc-management-address-in-bios?guid=guid-f6405c55-60af-45ad-8a28-0e0b7f46c4f2&lang=en-us

From the BMC console window, run the following commands to boot into the BIOS:

ipmitool -I lanplus -H 127.0.0.1 -U admin -P admin chassis bootparam set bootflag force_bios

~ # ipmitool -I lanplus -H 127.0.0.1 -U admin -P admin power reset
Chassis Power Control: Reset
~ # RESET CHASSIS
[  653.800000] LPC RESET
PDK LPC Reset is invoked

Seriell muss man dann bestimmt umstecken

Note: ipmi user/pw != bmc

Laut Handbuch muss man das alles per BIOS machen.

Geht aber auch mit ipmitool lokal

 ipmitool -I lanplus -H 127.0.0.1 -U admin -P admin lan set 1 ipaddr 10.0.0.1
41~ # ipmitool -I lanplus -H 127.0.0.1 -U admin -P admin lan set 1 netmask 255.255
.255.0

~ # ipmitool -I lanplus -H 127.0.0.1 -U admin -P admin lan set 1 defgw ipaddr 10.0.0.1
Raw
ipmi.md

https://www.dell.com/support/kbdoc/de-de/000112188/how-to-access-console-port-of-dell-emc-networking-vep4600

Management Port Overview

Port names are in clock-wise order

image

Port Description
Port A Serial Console Port to Access CPU
Port B Serial Console Port to Access BMC
Port C Management Port to Access CPU (igb4)
Port D Management Port to Access BMC
Port E USB-B Console port to Access CPU

If a cable is connected to Port E (referenced above) even if not in use, the CPU RJ45 (Port A) console port will not work.

Port C is an Intel i210 desktop NIC with reduced features intended for management purposes only. The other 1g Ports use i350 server NICs. Rarely will the i210 port be enumerated as the first NIC

Access and Default Settings

Accessing the console ports:

Port A/E -  Serial/USB-B Console Port to access CPU
                               User Name      : root
                                Password       : calvin

The above login credentials are only for the default installed DIAG OS When installed with any other OS, follows vendor directions.

Port B -  Serial Console Port to access BMC
                              User Name       : sysadmin
                              Password         : superuser

In bestimmten Versionen ist der BMC Login anders:

                              User Name       : sysadmin
                              Password         : SERVICETAG!

Accessing BMC through HTTPS
                             User Name        : admin
                             Password          : admin

Default Console Settings
Baud Rate : 115200
Data Bits   : 8
Stop Bits   : 1
Parity         : None
Flow Control : None

Login Beispiel:

BMCXXXX login: sysadmin
Password: 
[1446 : 1446 INFO]SERIAL Login from IP:127.0.0.1 user:sysadmin

IP Setup?

3: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether XXXX brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 brd 192.168.0.255 scope global eth0
Raw
LEDs.md

image

  1. Power LED
  2. Primary unit indicator LED
  3. System LED
  4. Locator LED
  5. Temperature LED
  6. Fan LED
  7. SFP+ indicator LED
  8. 10/100/1000 BaseT RJ-45 networking link (left) and activity (right) LEDs
  9. 10/100/1000 BaseT RJ-45 networking link (left) and activity (right) LEDs for the processor (left) and for the BMC (right)

LED codes im Dell EMC VEP4600 Installation Guide

Raw
NIC overview.md

Typical configuration without rNDC module

image

Ports assigment:

id/os name id/os name
"9"/igb4 bmc
id/os name id/os name
3/igb1 5/igb3
4/igb0 6/igb2
id/os name id/os name
1 ix0
2 ix1

Internal Ports

There are two more ports facing the internal backplane for 'service chaining' use cases. Those are the ones labeled "for backplane" in the overview

Raw
OS_install.md
Raw
Virtualisierung.md

Netzwerkkarten

SR-IOV ist im BIOS per Defaukt aktiv. VT-d etc. ebenfalls, sind verfuegbar

Linux

SR-IOV Settings

  • i210 onboard: Managementport und Chip supported es nicht
  • i350 onboard: SR-IOV deaktiviert und nicht im BIOS konfiguierbar
  • i7xx onboard: SR-IOV deaktiviert und nicht im BIOS konfiguierbar
  • i7xx auf rNDC: SR-IOV im BIOS konfigurierbar

Dazu gibt es eine Anleitung

Link?

PCI Passthrough

Bei nutzung von Passthrough wurde nur 1 device von 4 weitergereicht, die anderen waren dann "weg"

VFs

aus u.g. VMware Doku (running on KVM)

  • a. Set the spoofcheck off.
    # ip link set eth1 vf 0 spoofchk off
  • b. Set the Trusted mode on.
    # ip link set dev eth1 vf 0 trust on
  • c. Set the VLAN, if required.
    # ip link set eth1 vf 0 vlan 3500
  • d. disable GRO on host
    # ethtool –K <interface> gro off tx off

ESXi

zu Pruefen

  • Es gibt ein NIC FW Update - aber nur fuer VMware

Ist auch im VMware SD-WAN Guide angedeutet, sehr gut moeglich, dass das also helfen wuerde. https://docs.vmware.com/en/VMware-SD-WAN/4.0/vmware-sd-wan-partner-guide.pdf

Security Update

https://www.dell.com/support/kbdoc/en-us/000215718/dsa-2023-251-security-update-for-dell-networking-vep4600-for-intel-ethernet-controllers-and-adapters-advisory-intel-ta-00593

Product Software/Firmware Affected Versions Remediated Versions Link Dell Networking VEP4600 16-CORE Firmware Versions prior to 22.0.9 Version 22.0.9 or later https://www.dell.com/support/home/en-ca/drivers/driversdetails?driverid=9nppg

Relnotes: https://dl.dell.com/FOLDER09675541M/4/fw_release_x710.txt?uid=67a7d282-d9f6-4eb3-56c8-0e3fa7a02476&fn=fw_release_x710.txt

FW Update!!! (fuer die 710)) https://dl.dell.com/FOLDER10201251M/1/vep4600-x722-fw-rn.pdf?uid=d6f44b92-fd82-4244-40bf-353f8d1f97ae&fn=vep4600-x722-fw-rn.pdf

https://dl.dell.com/FOLDER10201235M/1/X722_V6_20_Linux.tar.gz

Siehe hier auch die resolved Issues, da passt so einiges...

Proxmox Thread bzgl. i350 Passthrough

https://forum.proxmox.com/threads/nic-card-passthrough-and-essential-things-to-think-about.80805/

@FlorianHeigl
Copy link
Author

Yafuflash doku: https://www.intel.com/content/www/us/en/download/19552/ami-yafu-yet-another-firmware-upgrade-flash-64-bit-utility-for-m10jnp2sb.html

@FlorianHeigl
Copy link
Author

FlorianHeigl commented Feb 10, 2025 •
edited
Loading

RAM:
It might come with UDIMMs, but does accept UDIMM and LRDIMM. Dual Rank seem to work perfectly, I have 4x32GB Samsung there.

        Speed: 2400 MT/s
        Manufacturer: Samsung
        Serial Number: lalalalatoolate
        Asset Tag: CPU1_DIMM_A1_AssetTag
        Part Number: M393A4K40BB1-CRC    
        Rank: 2
        Configured Memory Speed: 2400 MT/s
        Minimum Voltage: 1.2 V
        Maximum Voltage: 1.2 V
        Configured Voltage: 1.2 V

The CPU is meant for 2666MHz modules, and the BIOS even has overclocking up to 3200MHz (untested, don't care)

A likely working 2666MHz DIMM is M386A8K40BM2 - though those are quad rank so please don't simply take my word.

@FlorianHeigl
Copy link
Author

FlorianHeigl commented Feb 10, 2025 •
edited
Loading

NVMe: The BIOS has NVMe support, but due to a chipset limitation the onboard M.2 sockets do not support NVMe. The reason is that the XeonD CPU includes parts of a C620 chipset and in that chipset the 'HSIO' lanes don't support switching to SATA. There's some relevant posts on the Intel forums on it. I also tried it out and it just doesn't work. If someone can make an rNDC carrier board with a PEX PCIe switch and M.2 it would be nice, but failing that the best option is to use iSCSI via the network interfaces. Note that Dell seemingly also disabled the iSCSI offload features on the Intel NICs but that should not be too big of a loss.
What you can do if you need to get some reliability is to use the NIC partitioning (not SR-IOV) and accordingly have a virtual NIC with a fixed bandwidth reservation of, say, 60%.

@FlorianHeigl
Copy link
Author

SR-IOV can be enabled on the 10g interfaces on the rNDC cards (not the onboard), further docs are here:
https://www.kernel.org/doc/html/v5.0/networking/device_drivers/intel/i40e.html

For the record, I was not yet able to make it work in my VM host. Enabled or not, not working (yet)

@FlorianHeigl
Copy link
Author

Release notes for latest(afaik) firmware
https://dl.dell.com/FOLDER11749132M/1/rn_vep4600_Final.pdf?uid=1fba6b03-14bf-45e3-3740-66ffa5b10c4a&fn=rn_vep4600_Final.pdf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment