OpenStack on ARM64 – LBaaS
In part 2 of this series about OpenStack on ARM64, we got to the point where our cloud is fully deployed with all the Compute (VMs), Software Defined Networking (SDN) and Software Defined Storage (SDS) up and running. One additional component that we want to add is a Load Balancer as a Service (LBaaS), which is a key requirement for pretty much any high available type of workload and a must-have feature in any cloud.
OpenStack’s current official LBaaS component is called Octavia, which replaced the older Neutron LBaaS v1 project, starting with the Liberty release. Deploying and configuring requires a few steps, which explains the need for a dedicated blog post.
Octavia’s reference implementation uses VM instances called Amphorae to perform the actual load balancing. The octavia-worker service takes care of communicating with the amphorae and to do that we need to generate a few X509 CAs and certificates used to secure the communications. The good news is that starting with the Victoria release, kolla-ansible simplifies a lot this task. Here’s how to:
# Change the following according to your organization
echo "octavia_certs_country: US" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_state: Oregon" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_organization: OpenStack" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_organizational_unit: Octavia" | sudo tee -a /etc/kolla/globals.yml
# This is the kolla-ansible virtual env created in the previous blog post
cd kolla
source venv/bin/activate
sudo chown $USER:$USER /etc/kolla
kolla-ansible octavia-certificates
The communication between Octavia and the Amphorae needs an isolated network, as we don’t want to share it with the tenant network for security reasons. A simple way to accomplish that is to create a provider network with a dedicated VLAN ID, which is why we enabled Neutron provider networks and OVS VLAN segmentation in the previous post. Again, starting with Victoria, this got much easier with kolla-ansible.
# This is a dedicated network, outside your management LAN address space, change as needed
OCTAVIA_MGMT_SUBNET=192.168.43.0/24
OCTAVIA_MGMT_SUBNET_START=192.168.43.10
OCTAVIA_MGMT_SUBNET_END=192.168.43.254
OCTAVIA_MGMT_HOST_IP=192.168.43.1/24
OCTAVIA_MGMT_VLAN_ID=107
sudo tee -a /etc/kolla/globals.yml << EOT
octavia_amp_network:
name: lb-mgmt-net
provider_network_type: vlan
provider_segmentation_id: $OCTAVIA_MGMT_VLAN_ID
provider_physical_network: physnet1
external: false
shared: false
subnet:
name: lb-mgmt-subnet
cidr: "$OCTAVIA_MGMT_SUBNET"
allocation_pool_start: "$OCTAVIA_MGMT_SUBNET_START"
allocation_pool_end: "$OCTAVIA_MGMT_SUBNET_END"
gateway_ip: "$OCTAVIA_MGMT_HOST_IP"
enable_dhcp: yes
Unless there is a dedicated network adapter, a virtual ethernet one can be used. This needs to be configured at boot and added to the OVS br-ex switch.
# This sets up the VLAN veth interface
# Netplan doesn't have support for veth interfaces yet
sudo tee /usr/local/bin/veth-lbaas.sh << EOT
#!/bin/bash
sudo ip link add v-lbaas-vlan type veth peer name v-lbaas
sudo ip addr add $OCTAVIA_MGMT_HOST_IP dev v-lbaas
sudo ip link set v-lbaas-vlan up
sudo ip link set v-lbaas up
sudo chmod 744 /usr/local/bin/veth-lbaas.sh
sudo tee /etc/systemd/system/veth-lbaas.service << EOT
[Unit]
After=network.service
[Service]
ExecStart=/usr/local/bin/veth-lbaas.sh
[Install]
WantedBy=default.target
sudo chmod 644 /etc/systemd/system/veth-lbaas.service
sudo systemctl daemon-reload
sudo systemctl enable veth-lbaas.service
sudo systemctl start veth-lbaas.service
docker exec openvswitch_vswitchd ovs-vsctl add-port \
br-ex v-lbaas-vlan tag=$OCTAVIA_MGMT_VLAN_ID
A few more Octavia kolla-ansible configurations…
echo "enable_octavia: \"yes\"" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_network_interface: v-lbaas" | sudo tee -a /etc/kolla/globals.yml
# Flavor used when booting an amphora, change as needed
sudo tee -a /etc/kolla/globals.yml << EOT
octavia_amp_flavor:
name: "amphora"
is_public: no
vcpus: 1
ram: 1024
disk: 5
sudo mkdir /etc/kolla/config/octavia
# Use a config drive in the Amphorae for cloud-init
sudo tee /etc/kolla/config/octavia/octavia-worker.conf << EOT
[controller_worker]
user_data_config_drive = true
…and we can finally tell kolla-ansible to deploy Octavia:
kolla-ansible -i all-in-one deploy --tags common,horizon,octavia
Octavia uses a special VM image for the Amphorae, which needs to be built for ARM64. We prepared Dockerfiles for building either an Ubuntu or a CentOS image, you can choose either one in the following snippets. We use containers to perform the build in order to isolate the requirements and be independent from the host OS.
git clone https://github.com/cloudbase/openstack-kolla-arm64-scripts
cd openstack-kolla-arm64-scripts/victoria
# Choose either Ubuntu or CentOS (not both!)
# Ubuntu
docker build amphora-image-arm64-docker -f amphora-image-arm64-docker/Dockerfile.Ubuntu \
-t amphora-image-build-arm64-ubuntu
# Centos
docker build amphora-image-arm64-docker -f amphora-image-arm64-docker/Dockerfile.Centos \
-t amphora-image-build-arm64-centos
ARM64 needs a trivial patch in the diskimage-create.sh build script (we also submitted it upstream):
git clone https://opendev.org/openstack/octavia -b stable/victoria
# Use latest branch Octavia to create Ubuntu image
cd octavia
# diskimage-create.sh includes armhf but not arm64
git apply ../0001-Add-arm64-in-diskimage-create.sh.patch
Build the image (this will take a bit):
# Again, choose either Ubuntu or CentOS (not both!)
# Note the mount of /mnt and /proc in the docker container
# BEWARE!!!!! Without mounting /proc, the diskimage-builder fails to find mount points and deletes the host's /dev,
# making the host unusable
docker run --privileged -v /dev:/dev -v /proc:/proc -v /mnt:/mnt \
-v $(pwd)/octavia/:/octavia -ti amphora-image-build-arm64-ubuntu
# Create CentOS 8 Amphora image
docker run --privileged -v /dev:/dev -v $(pwd)/octavia/:/octavia \
-ti amphora-image-build-arm64-centos
Add the image to Glance, using the octavia user in the service project. The amphora tag is used by Octavia to find the image.
. /etc/kolla/admin-openrc.sh
# Switch to the octavia user and service project
export OS_USERNAME=octavia
export OS_PASSWORD=$(grep octavia_keystone_password /etc/kolla/passwords.yml | awk '{ print $2}')
export OS_PROJECT_NAME=service
export OS_TENANT_NAME=service
openstack image create amphora-x64-haproxy.qcow2 \
--container-format bare \
--disk-format qcow2 \
--private \
--tag amphora \
--file octavia/diskimage-create/amphora-x64-haproxy.qcow2
# We can now delete the image file
rm -f octavia/diskimage-create/amphora-x64-haproxy.qcow2
Currently, we need a small patch in Octavia to properly render the userdata for the Amphorae:
# Patch the user_data_config_drive_template
cd octavia
git apply ../0001-Fix-userdata-template.patch
# For now just update the octavia-worker container, no need to restart it
docker cp octavia/common/jinja/templates/user_data_config_drive.template \
octavia_worker:/usr/lib/python3/dist-packages/octavia/common/jinja/templates/user_data_config_drive.template
Finally, let’s create a load balancer to make sure everything works fine:
# To create the loadbalancer
. /etc/kolla/admin-openrc.sh
openstack loadbalancer create --name loadbalancer1 --vip-subnet-id public1-subnet
# Check the status until it's marked as ONLINE
openstack loadbalancer list
Congratulations! You have a working LBaaS in your private cloud!!
Troubleshooting
In case something goes wrong, finding the root cause might be tricky. Here are a few suggestions to ease up the process.
# Check for errors
sudo tail -f /var/log/kolla/octavia/octavia-worker.log
# SSH into amphora
# Get amphora VM IP either from the octavia-worker.log or from:
openstack server list --all-projects
ssh ubuntu@<amphora_ip> -i octavia_ssh_key #ubuntu
ssh cloud-user@<amphora_ip> -i octavia_ssh_key #centos
# Instances stuck in pending create cannot be deleted
# Password: grep octavia_database_password /etc/kolla/passwords.yml
docker exec -ti mariadb mysql -u octavia -p octavia
update load_balancer set provisioning_status = 'ERROR' where provisioning_status = 'PENDING_CREATE';
exit;
