Skip to content

Instantly share code, notes, and snippets.

@timroster

timroster/f5-mgmtNIC-IBMCLoud.md

Last active May 11, 2022 01:02
Show Gist options
  • Save timroster/11f6d7de68b4920e9248a7e26ba5c3ab to your computer and use it in GitHub Desktop.
Save timroster/11f6d7de68b4920e9248a7e26ba5c3ab to your computer and use it in GitHub Desktop.
Download ZIP
Notes on switching F5 management to NIC2 on IBM Cloud VPC
Raw
f5-mgmtNIC-IBMCLoud.md

WIP - moving F5 management to eth2 on a 3 NIC VSI

Load balancing services on IBM Cloud VPC currently leverage access to the primary NIC associated with the instance. By default, F5 BIG-IP configures to run management traffic on the primary NIC (eth0) and data plane on other NICS. This can be manually switched as follows.

Note - applying a license to a BIG-IP creates a dependency on the mac address of the management interface. Perform all of these steps before applying the BIG-IP license.

Starting point - deployment of F5 BIG-IP using terraform code from https://github.com/f5devcentral/ibmcloud_schematics_bigip_multinic_declared . This can be performed from the command line or schematics. Since the later intent is to have the NIC referred to as management in the code be a public interface, it can be convenient to assign a floating IP address to this interface by setting bigip_management_floating_ip to true.

When created by the terraform code, there will be a root and admin user defined. The root user will only be accessible over ssh to the host using the ssh-key provided at deployment. For VNC Console access, on first login, set the root password to a strong value as the admin user is only able to sign in over the BIG-IP web UI.

After deployment - ssh to the management IP address and then issue the command to switch the management nic to eth2 (reference https://clouddocs.f5.com/cloud/public/v1/shared/change_mgmt_nic_google.html ):

tmsh modify sys db provision.managementeth value eth2
reboot

After the device reboots, testing seems to indicate that until configured, it will not be possible to ssh to the IP actually associated with eth2 in VPC, instead the inside instance IP seems to work - connect to this over ssh or use the VNC Console if the root user password was set before the reboot.

The following commands will reconfigure the now available NIC1 as a VLAN interface for TMM and finish setting up the new management interface. Define the following or substitute as necessary:

  • MANAGEMENT_IP - IP address on eth2 of VSI
  • MANAGEMENT_GW - .1 address of the subnet for the management interface
  • EXTERNAL_IP - IP address on eth0 of VSI
  • EXTERNAL_NM - Network mask bits for the subnet of the external interface
  • EXTERNAL_GW - .1 address of the subnet for the external interface
  • INTERNAL_RT (optional) - specific internal route to add to the internal (eth1) interface for application access, CIDR format
  • INTERNAL_GW (optional) - next hop (.1) for internal application access
tmsh delete net self selfip_1_2
tmsh delete net vlan net_1_2
tmsh delete sys management-route all
tmsh delete sys management-ip all
tmsh create sys management-ip ${MANAGEMENT_IP}/32
tmsh create sys management-route mgmt_gw network ${MANAGEMENT_GW}/32 type interface
tmsh create sys management-route default gateway ${MANAGEMENT_GW}
tmsh create net vlan net_1_0 interfaces add { 1.0 } mtu 1500
tmsh create net self selfip_1_0 address ${EXTERNAL_IP}/${EXTERNAL_NM} vlan net_1_0
tmsh create net route ext_gw_interface network ${EXTERNAL_GW}/32 interface net_1_0
tmsh create net route default gw ${EXTERNAL_GW}
# Optional - additional static internal routes
tmsh create net route torsl network ${INTERNAL_RT} gw ${INTERNAL_GW}
tmsh save /sys config

Before rebooting, back an the OS shell for the device, edit the configuration file (steps from: https://support.f5.com/csp/article/K12149#deleteint )

edit /config/bigip_base.conf: remove this entry:

net interface 1.2 {
    media-fixed 10000T-FD
}

remove interface from net stp

finally - save /config/bigip_base.conf

indicate that a config reload is needed and restart:

touch /service/mcpd/forceload
reboot

After the BIG-IP instance restarts, it will be possible to ssh to the management ip address and verify mac addresses, interfaces and routes:

root@(f5-ve-01)(cfg-sync Standalone)(NO LICENSE)(/Common)(tmos.sys.mac-address)# show

---------------------------------------------------------------------
Sys::MAC Address 
Entry              Component              Object-ID       Property
---------------------------------------------------------------------
02:00:00:12:d1:c5  net arp                172.25.64.1     mac-address
02:00:0c:12:d1:c5  cm device              f5-ve-01.local  base-mac
02:00:0c:12:d1:c5  net interface          1.0             mac-address
02:00:0c:12:d1:c5  sys hardware platform  n/a             base-mac
02:00:0c:12:d1:c5  net vlan               net_1_0         mac-true
02:00:0c:12:d3:eb  net interface          1.1             mac-address
02:00:0c:12:d3:eb  net vlan               net_1_1         mac-true
02:00:0e:12:d3:e4  net interface          mgmt            mac-address
root@(f5-ve-01)(cfg-sync Standalone)(NO LICENSE)(/Common)(tmos.net.interface)# show

---------------------------------------------------------------
Net::Interface
Name  Status   Bits   Bits  Pkts  Pkts  Drops   Errs      Media
                 In    Out    In   Out                         
---------------------------------------------------------------
1.0       up  50.1K  42.8K    75    72      0      0  10000T-FD
1.1       up   7.2K  15.3K    15    30      0      0  10000T-FD
mgmt      up  50.5E  54.7E  6.3E  6.3E  12.5E  12.5E   100TX-FD

Note that interface statistics appear broken and can't be reset (there's a KB issue on this)

root@(f5-ve-02)(cfg-sync Standalone)(NO LICENSE)(/Common)(tmos.net.route)# show
                  
----------------------------------------------------------------------------------
Net::Routes       
Name                Destination         Type       NextHop               Origin
----------------------------------------------------------------------------------
default             default             gw         172.25.64.1           static
ext_gw_interface    172.25.64.1/32      interface  /Common/net_1_0       static
torsl               10.115.0.0/16       gw         172.25.66.1           static
127.20.0.0/16       127.20.0.0/16       interface  tmm_bp                connected
172.25.64.0/24      172.25.64.0/24      interface  /Common/net_1_0       connected
172.25.66.0/24      172.25.66.0/24      interface  /Common/net_1_1       connected
127.1.1.0/24        127.1.1.0/24        interface  tmm                   connected
fe80::%vlan4094/64  fe80::%vlan4094/64  interface  /Common/net_1_0       connected
fe80::%vlan4093/64  fe80::%vlan4093/64  interface  /Common/net_1_1       connected
ff02:fff::/64       ff02:fff::/64       interface  tmm_bp                connected
fe80::%vlan4095/64  fe80::%vlan4095/64  interface  tmm_bp                connected
fe80::/64           fe80::/64           interface  /Common/socks-tunnel  connected
fe80::/64           fe80::/64           interface  /Common/http-tunnel   connected
fe80::%vlan4095/64  fe80::%vlan4095/64  interface  /Common/tmm_bp        connected
ff02:fff::/64       ff02:fff::/64       interface  /Common/tmm_bp        connected
ff02:ffd::/64       ff02:ffd::/64       interface  /Common/net_1_1       connected
ff02:ffe::/64       ff02:ffe::/64       interface  /Common/net_1_0       connected
ff02::/64           ff02::/64           interface  tmm                   connected
fe80::/64           fe80::/64           interface  tmm                   connected

It is safe to apply a license to the BIG-IP at this point. There can be cases (especially if there are network restrictions on outbound traffic on the public interface) where licensing may fail. If this occurs, consider setting up a route on the management interface for the license activation traffic per https://support.f5.com/csp/article/K97107542 . Verify routing to the license server at the Linux os with netstat -rn.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment