Non-OpenStack libvirt + kvm plugged into Calico as part of platform control plane. OpenStack is too complex to manage for this workload, and using containers walks the complexity line depending on the solution.
- CentOS 7.3
- libvirt 2.0.0
- OpenStack packages for Calico bits
- calico-common-2.0.2-1.el7.centos.x86_64
- calico-felix-2.0.2-1.el7.centos.x86_64
- dnsmasq-2.72_calico1.0.0-1.el7.x86_64
- dnsmasq-utils-2.72_calico1.0.0-1.el7.x86_64
- calicoctl v1.0.2
| Node Name | Node Description | Node Networks |
|---|---|---|
| router | linux machine playing the role of a router |
|
| compute | linux machine playing the role of a hypervisor |
|
| docker | linux machine playing the role of a docker container machine |
|
- Install Cumulus Quagga packages on router device
! /etc/quagga/bgpd.conf
!
router bgp 65001
bgp router-id 192.168.100.150
neighbor 192.168.100.160 remote-as 65002
neighbor 192.168.100.170 remote-as 65003
!
address-family ipv4 unicast
neighbor 192.168.100.160 activate
neighbor 192.168.100.170 activate
exit-address-family
!
line vty
!
- Install etcd, calico-felix, calicoctl, dnsmasq, libvirt+qemu on compute node
# /etc/calico/felix.cfg
[global]
EtcdEndpoints = "http://compute.cofront.local:4001"
FelixHostname = compute.cofront.local
# interface definition for libvirt
<interface type='ethernet'>
<mac address='6a:1a:c0:a8:fe:0a'/>
<target dev='calic0a8fe0a'/>
<model type='virtio'/>
<driver name='qemu'/>
<alias name='net0'/>
</interface>
cat << EOF | ./calicoctl apply -f -
apiVersion: v1
kind: node
metadata:
name: compute.cofront.local
spec:
bgp:
asNumber: 65002
ipv4Address: 192.168.100.160
EOF
cat << EOF | ./calicoctl apply -f -
- apiVersion: v1
kind: profile
metadata:
name: wide-open
tags: []
spec:
egress:
- action: allow
destination:
net: 0.0.0.0/0
ipVersion: 4
source: {}
- action: allow
destination:
net: ::/0
ipVersion: 6
source: {}
ingress:
- action: allow
destination: {}
ipVersion: 4
source:
net: 0.0.0.0/0
- action: allow
destination: {}
ipVersion: 6
source:
net: ::/0
EOF
cat << EOF | ./calicoctl apply -f -
apiVersion: v1
kind: workloadEndpoint
metadata:
name: abc123
workload: vm.cofront.local
orchestrator: cfnt
node: compute.cofront.local
labels: {}
spec:
interfaceName: calic0a8fe0a
mac: 6a:1a:c0:a8:fe:0a
ipNetworks:
- 192.168.254.10/32
profiles:
- wide-open
EOF
ip tuntap add dev calic0a8fe0a mode tap
ip link set dev calic0a8fe0a up
virsh start cirros
When bringing up an OpenStack backed VM OpenStack handles the creation of the tap interface which ensures it comes online in an UP state. When letting libvirt do this work it's observed that the interface is down and bring it up puts it in an UNKNOWN state. Using libvirt hooks it should be possible to work around this issue. A rudimentary hook is included below, but would need much more work to be production ready.
#!/usr/bin/python
import sys
from lxml import etree
from pyroute2 import IPRoute
IFPREFIXES = ['cali', 'tap']
def configure_interfaces(interfaces):
ip = IPRoute()
try:
for interface in interfaces:
ip.link('add', ifname=interface, kind='tuntap', mode='tap')
ipif = ip.link_lookup(ifname=interface)
ip.link('set', index=ipif[0], state='up')
finally:
ip.close()
def determine_interfaces(domain):
interfaces = []
difs = domain.findall('devices/interface')
for dif in difs:
dif_target = dif.find('target')
if dif_target is not None:
ifname = dif_target.attrib.get('dev', '')
for ifprefix in IFPREFIXES:
if ifname.startswith(ifprefix):
interfaces.append(ifname)
return interfaces
if __name__ == '__main__':
import time
timestamp = int(time.time())
with open('/tmp/hook.log', 'a') as log:
target, oper, sub_oper = sys.argv[1:4]
log.write('%s %s %s %s\n' % (timestamp, target, oper, sub_oper))
# only run at prepare begin
if oper != 'prepare' or sub_oper != 'begin':
sys.exit(0)
# read domain configuration
domain = etree.fromstring(sys.stdin.read())
# get interface list
interfaces = determine_interfaces(domain)
log.write('%s %s\n' % (timestamp, interfaces))
configure_interfaces(interfaces)
sys.exit(0)None