Skip to content

Instantly share code, notes, and snippets.

@markus-hentsch

markus-hentsch/openstack_devstack_aio_vm.md

Last active August 27, 2024 15:18
Show Gist options
  • Save markus-hentsch/a2487f69363b84aea5dc1fc602578507 to your computer and use it in GitHub Desktop.
Save markus-hentsch/a2487f69363b84aea5dc1fc602578507 to your computer and use it in GitHub Desktop.
Download ZIP
DevStack within an OpenStack VM (all-in-one)
Raw
openstack_devstack_aio_vm.md

DevStack within an OpenStack VM (all-in-one)

Host VM on OpenStack

source $OPENRC_FILE

openstack volume create --size 200 --image "Ubuntu 22.04 LTS x64" devstack-boot-volume

openstack security group create devstack-sg
openstack security group rule create --protocol tcp --dst-port 22 devstack-sg

openstack network create devstack-access-network
openstack subnet create --dhcp --network devstack-access-network --subnet-range 10.0.1.0/24 devstack-access-subnet

openstack router create --external-gateway shared-public-IPv4 devstack-access-router
openstack router add subnet devstack-access-router devstack-access-subnet

openstack network create devstack-provider-network
openstack subnet create --dhcp --network devstack-provider-network --subnet-range 10.0.2.0/24 devstack-provider-subnet

openstack router create --external-gateway shared-public-IPv4 devstack-provider-router
openstack router add subnet devstack-provider-router devstack-provider-subnet

openstack floating ip create --description "Access IP for DevStack" shared-public-IPv4
FIP=$(openstack floating ip list --long -f value -c Description -c "Floating IP Address" \
    | grep "Access IP for DevStack" | cut -d ' ' -f1)
echo "$FIP"

openstack server create \
--volume devstack-boot-volume \
--network devstack-access-network \
--network devstack-provider-network \
--security-group devstack-sg \
--flavor "XL.mem+" \
--key-name markus-cnh \
devstack-vm

# determine the IP on the primary interface as assigned by OpenStack's DHCP
PRIM_IP=$(openstack server show -f value -c addresses devstack-vm | grep -oE "10\.0\.1\.[0-9]+")
echo "$PRIM_IP"

# attach floating ip to the primary 10.0.1.X interface
# NOTE: the secondary one will be claimed by Neutron and would kill SSH
openstack server add floating ip --fixed-ip $PRIM_IP devstack-vm $FIP

echo "ssh ubuntu@$FIP"

DevStack Preparation

Execute the following on the Host VM created above:

sudo useradd -s /bin/bash -d /opt/stack -m stack
sudo chmod +x /opt/stack
echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack
sudo -u stack -i
git clone https://opendev.org/openstack/devstack
cd devstack
cp samples/local.conf local.conf

Now adjust the local.conf as shown below.

  • replace HOST_IP with the address of the first interface of the Host VM

NOTE: make sure that the Q_FLOATING_ALLOCATION_POOL range does not clash with the IP that the Host VM received via DHCP on the second interface!

WARNING: the DevStack setup as configured below will remove the IP from the second interface of the Host VM and establish a bridge instead.

local.conf

# HOST_IP will determine which IP (and by extension which interface) the APIs
# will bind to, e.g. Keystone API
# You can set this to localhost (127.0.0.1) if you want
#HOST_IP=127.0.0.1
HOST_IP=10.0.1.116
SERVICE_HOST=$HOST_IP

# WARNING: *do not* set PUBLIC_INTERFACE to any interface as this
# will kill the network connectivity of the host VM!
# The second interface of the host VM will automatically have its IP removed
# by Neutron regardless of this setting!
#PUBLIC_INTERFACE=
# The following IPs will be taken for the provider net.
PUBLIC_NETWORK_GATEWAY=10.0.2.1
FLOATING_RANGE=10.0.2.0/24
Q_FLOATING_ALLOCATION_POOL=start=10.0.2.100,end=10.0.2.150

# Setup OVN instead of OVS
# source: https://opendev.org/openstack/neutron/src/branch/master/devstack/ovn-local.conf.sample
Q_AGENT=ovn
Q_ML2_PLUGIN_MECHANISM_DRIVERS=ovn,logger
Q_ML2_PLUGIN_TYPE_DRIVERS=local,flat,vlan,geneve
Q_ML2_TENANT_NETWORK_TYPE="vlan"
enable_service ovn-northd
enable_service ovn-controller
enable_service q-ovn-metadata-agent
enable_service q-svc
# Disable Neutron agents not used with OVN.
disable_service q-agt
disable_service q-l3
disable_service q-dhcp
disable_service q-meta
# Neutron services
enable_service q-trunk
enable_service q-dns
enable_service q-port-forwarding
enable_service q-qos
enable_service neutron-segments
enable_service q-log

# Barbican plugin
enable_plugin barbican https://opendev.org/openstack/barbican

To allow external access to the API execute the following for the Host VM:

openstack security group rule create --protocol tcp --dst-port 80 --remote-ip $OFFICE_IP devstack-sg

(replace OFFICE_IP by the office's IP you are connecting from - you do not want to expose this to the public internet!)

DevStack Deploy & Teardown

Deploy

sudo -u stack -i
cd devstack/
./stack.sh       # deploy DevStack

Teardown

sudo -u stack -i
cd devstack/
./unstack.sh     # teardown DevStack

NOTE: Use the ./clean.sh script between ./unstack.sh and ./stack.sh if you run into errors after re-deploying. This will do deeper system cleanups on the host VM and wipe pretty much any remnants of previous configurations. It will take a bit longer but can help if switching between too many configurations leaves the host VM in a bad state.

DevStack Usage

sudo -u stack -i
cd devstack/

# authenticate
source openrc  # user: demo, project: demo
source openrc admin admin  # user: admin, project: admin

# use
openstack image list

VM connectivity

When the network is set up correctly as per the above local.conf snippet, the following networking functionality is available for VMs:

  • VMs will have internet access (egress direction) when the following is true: the VM is connected to a tenant network which is connected to the "public" network (provider network) via a router
  • VMs that have a Floating IP assigned to them will be reachable from the underlying Host VM (ingress direction) as soon as their security groups allow the ports/protocols (e.g. ICMP or 22 for SSH)

NOTE: due to the nature of this setup, the Floating IPs determined by FLOATING_RANGE and Q_FLOATING_ALLOCATION_POOL of local.conf are not actually public IPv4 addresses and will not be routed from outside of the Host VM. Those VMs can only be reached by ingress traffic either from the Host VM itself or tunneling techniques like sshuttle or WireGuard.

Volume Encryption

In case the Barbican plugin is enabled, an encrypted volume type can be created using:

openstack volume type create \
--property volume_backend_name='lvmdriver-1' \
--encryption-provider luks \
--encryption-cipher aes-xts-plain64 \
--encryption-key-size 256 \
--encryption-control-location front-end \
lvmdriver-1-LUKS

(This example assumes that Cinder is configured to use the LVM backend, look at your existing volume types with openstack volume type list/show as admin for reference)

Tips

Inspecting the logs

Some useful commands for inspecting logs:

# get the name of units
sudo systemctl list-units | grep devstack

# get logs for a specific unit
sudo journalctl -u devstack@c-api

# get logs starting with a specific date
sudo journalctl --since 2024-01-01
sudo journalctl --since 2024-01-01 -u devstack@c-api

Fix Glance image creation failing with Bad Gateway error

The DevStack is provisioned with Glance image quotas registered in Keystone per default. You can inspect them with:

openstack registered limit list

This can quickly lead to the following non-obvious error when creating images from files or volumes once that quota is exceeded:

HttpException: 502: Server Error for url: http://.../image/v2/images/.../file, :
502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response
from an upstream server.: Apache/2.4.52 (Ubuntu) Server at ... Port 80

To fix this, remove the oslo.limit section and set use_keystone_limits = False in /etc/glance/glance-api.conf and restart Glance:

sudo systemctl restart [email protected]

Fix connection to guest VMs Floating IPs from the Host VM

If the host has been restarted or any other network changes occured it might be that both the secondary network interface and the br-ex bridge have an IP address in the same range. In this case, connection to Floating IPs of OpenStack VMs from the Host VM itself may not be possible (e.g. ssh). To fix this, remove the IP address from the second interface.

Example:

$ ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    ...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    ...
    inet 10.0.2.4/24 brd 10.0.2.255 scope global eth1
        valid_lft forever preferred_lft forever
    ...
6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    ...
    inet 10.0.2.1/24 scope global br-ex
        valid_lft forever preferred_lft forever

$ sudo ip a d 10.0.2.4/24 dev eth1

(your individual IP addresses may differ)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment