Automated Kickstart Install of RHEL/Clones

There are two approaches to making your life more automated about installing multiple instances of operating systems. You can either maintain up-to-date templates for them or you can have automated/scripted installations. In this article I will share how to generate ISO image and Kickstart configuration to install Red Hat Enterprise Linux (and its clones such as Alma/Rocky/CentOS/Scientific/…) in easy and fast way. For the process I will use 8.5 version of RHEL.

Here is the Table of Contents for this article.

• Logo
• Possibilities
• Environment
  • FreeBSD – www
  • RHEL Client – kickme
• Validate
• Generation
  • kickstart.config
  • kickstart.skel
  • kickstart.sh
  • ISO
• Result
• Alternatives
• Summary

Shortly after IBM acquisition Red Hat started to use kinda boring ‘just a red hat’ logo – but its earlier logo – shown below – was more interesting.

If you stare long enough you will see two dinosaurs. The Tyrannosaurus (red) punching a Triceratops (white) in the head. Once you see it you will not be able to unsee it

Possibilities

There are many ways to do that automated Kickstart installation. You can use NFS/HTTP/FTP/HTTPS or your own generated DVD media … or use ‘stock’ DVD and Kickstart config available somewhere on the network.

I will use the following method that I find currently is best suited for my needs:

• FreeBSD host with NGINX serving RHEL 8.5 DVD content (rhel-8.5-x86_64-dvd.iso) over HTTP.
• Generate small (less then 1 MB in size) ISO with Kickstart config on it.
• Boot from small rhel-8.5-x86_64-boot.iso ISO and also with generated Kickstart ISO.

Environment

I will use VirtualBox for this demo with NAT Network configuration for the virtual machines network adapters. The nat0 VirtualBox network is defined as 10.0.10.0/24 and I use Port Forwarding to access these machines from the FreeBSD host system.

Machines:

• 10.0.10.210 - www – FreeBSD system with NGINX to serve RHEL 8.5 DVD contents
• 10.0.10.199 - kickme – RHEL machine that would be installed with automated Kickstart install

FreeBSD – www

Below you will find the FreeBSD machine configuration as seen on VirtualBox.

Its default FreeBSD ZFS install on single disk. Nothing fancy here to be honest. Below you will find its configuration from /etc/rc.conf file. I also installed the nginx package but the only thing I did with NGINX was to enable it to start automatically. I used the default stock config that points at /usr/local/www/nginx place. I later copied the RHEL 8.5 DVD contents to the /usr/local/www/nginx/rhel-8.5 directory. It takes about 10 GB.

www # cat /etc/rc.conf
hostname=www
ifconfig_em0="inet 10.0.10.210 netmask 255.255.255.0"
defaultrouter=10.0.10.1
sshd_enable=YES
nginx_enable=YES
zfs_enable=YES
dumpdev=AUTO
sendmail_enable=NO
sendmail_submit_enable=NO
sendmail_outbound_enable=NO
sendmail_msp_queue_enable=NO
update_motd=NO

Here is the unmodified NGINX config but with comments non displayed.

www # grep -v '#' /usr/local/etc/nginx/nginx.conf | grep '^[^#]'
worker_processes  1;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        server_name  localhost;
        location / {
            root   /usr/local/www/nginx;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }
    }
}

Here are the contents of the /usr/local/www/nginx/rhel-8.5 directory after copying here contents of RHEL 8.5 DVD.

www:~ # ls -l /usr/local/www/nginx/rhel-8.5/
total 71
-r--r--r--  1 root  wheel     60 Apr  6 21:34 .discinfo
-r--r--r--  1 root  wheel   1560 Apr  6 21:34 .treeinfo
dr-xr-xr-x  4 root  wheel      4 Apr  6 21:50 AppStream
dr-xr-xr-x  4 root  wheel      4 Apr  6 21:53 BaseOS
dr-xr-xr-x  3 root  wheel      3 Apr  6 21:53 EFI
-r--r--r--  1 root  wheel   8154 Apr  6 21:53 EULA
-r--r--r--  1 root  wheel  18092 Apr  6 21:53 GPL
-r--r--r--  1 root  wheel   1669 Apr  6 21:53 RPM-GPG-KEY-redhat-beta
-r--r--r--  1 root  wheel   5135 Apr  6 21:53 RPM-GPG-KEY-redhat-release
-r--r--r--  1 root  wheel   1796 Apr  6 21:53 TRANS.TBL
-r--r--r--  1 root  wheel   1455 Apr  6 21:53 extra_files.json
dr-xr-xr-x  3 root  wheel      6 Apr  6 21:54 images
dr-xr-xr-x  2 root  wheel     16 Apr  6 21:54 isolinux
-r--r--r--  1 root  wheel    103 Apr  6 21:54 media.repo

RHEL Client – kickme

Below you will find the RHEL machine that will be used for the automated Kickstart installation – also as seen on VirtualBox.

To make that example more interesting (and more corporate) I added second NIC for the backup network – so we will also have to generate additional route for it in the Kickstart config.

The kickme RHEL machine needs to have two (2) CD-ROM drives. In the PRIMARY (the one to boot from) we will load the rhel-8.5-x86_64-boot.iso ISO file. In the SECONDARY one we will load our generated kickme.oemdrv.iso ISO file containing Kickstart config file.

Validate

There also exists pykickstart package which offers ksvalidator tool. Theoretically it allows you to make sure that your Kickstart config has proper syntax and that it would work – but only in theory. Here is what it RHEL documentation states about its accuracy.

It means that you will not know if your Kickstart config will work until you really try it – thus I do not cared that much about this tool as is not available on FreeBSD.

Generation

Now to the most important part – Kickstart generation and ISO generation. Probably the easiest way to create new Kickstart config is to install ‘by hand’ new RHEL system under virtual machine and then take the generated by Anaconda /root/anaconda-ks.cfg file as a starting point. This is what I also did.

When you would like to install next host you will have to edit the hostname and IP addresses for next host in that Kickstart file – which seems not very convenient to say the least. In order to make that less PITA I created a kickstart.sh script that will read values from kickstart.config file and then put them into the kickstart.skel file that would be base for our future installations. Then the kickstart.sh will copy the generated Kickstart file and used Kickstart config with that hostname as a backup or for future reference – or for example for documentation purposes.

kickstart.config

Here is how such kickstart.config file looks like.

# cat kickstart.config
  SYSTEM_NAME=kickme
  REPO_SERVER_IP=10.0.10.210
  INTERFACE1=enp0s3
  IP_ADDRESS1=10.0.10.199
  NETMASK1=255.255.255.0
  INTERFACE2=enp0s8
  IP_ADDRESS2=10.0.90.199
  NETMASK2=255.255.255.0
  GATEWAY=10.0.10.1
  NAMESERVER1=1.1.1.1
  NAMESERVER2=9.9.9.9
  NTP1=132.163.97.6
  NTP2=216.239.35.0
  ROUTE_NET=10.0.40.10/24
  ROUTE_VIA=10.0.20.1

kickstart.skel

The kickstart.skel file is little longer – this is our barebone for the Kickstart configs.

# cat kickstart.skel
#version=RHEL8

# USE sda DISK
ignoredisk --only-use=sda

# CLEAR DISK PARTITIONS BEFORE INSTALL
clearpart --all --initlabel

# USE text INSTALL
text

# USE ONLINE INSTALLATION MEDIA
url --url=http://REPO_SERVER_IP/rhel-8.5/BaseOS --noverifyssl

# KEYBOARD LAYOUTS
keyboard --vckeymap=us --xlayouts='us','pl'

# LANGUAGE
lang en_US.UTF-8

# NETWORK INFORMATION
network --bootproto=static --device=INTERFACE1 --ip=IP_ADDRESS1 --netmask=NETMASK1 --gateway=GATEWAY --nameserver=NAMESERVER1,NAMESERVER2 --noipv6 --activate
network --bootproto=static --device=INTERFACE2 --ip=IP_ADDRESS2 --netmask=NETMASK2 --noipv6 --activate --onboot=on
network --hostname=SYSTEM_NAME

# REPOS
repo --name="AppStream" --baseurl=http://REPO_SERVER_IP/rhel-8.5/AppStream

# ROOT PASSWORD
rootpw --plaintext asd

# DISABLE Setup Agent ON FIRST BOOT
firstboot --disable

# DISABLE SELinux AND FIREWALL
selinux --disabled
firewall --disabled

# OMIT X11
skipx

# REBOOT AND EJECT BOOT MEDIUM
reboot --eject

# TIMEZONE
timezone Europe/Warsaw --isUtc --nontp

# PARTITIONS
part   /boot/efi --fstype="efi"   --size=600  --ondisk=sda --label=EFI  --fsoptions="umask=0077,shortname=winnt"
part   /boot     --fstype="xfs"   --size=1024 --ondisk=sda --label=BOOT --fsoptions="rw,noatime,nodiratime"
part   pv.475    --fstype="lvmpv" --size=1    --ondisk=sda --grow

# LVM
volgroup rootvg --pesize=4096 pv.475
logvol /         --fstype="xfs"   --size=1024 --name=root --label="ROOT" --vgname=rootvg --fsoptions="rw,noatime,nodiratime"
logvol /usr      --fstype="xfs"   --size=5120 --name=usr  --label="USR"  --vgname=rootvg --fsoptions="rw,noatime,nodiratime"
logvol /var      --fstype="xfs"   --size=3072 --name=var  --label="VAR"  --vgname=rootvg --fsoptions="rw,noatime,nodiratime"
logvol /tmp      --fstype="xfs"   --size=1024 --name=tmp  --label="TMP"  --vgname=rootvg --fsoptions="rw,noatime,nodiratime"
logvol /opt      --fstype="xfs"   --size=1024 --name=opt  --label="OPT"  --vgname=rootvg --fsoptions="rw,noatime,nodiratime"
logvol /home     --fstype="xfs"   --size=1024 --name=home --label="HOME" --vgname=rootvg --fsoptions="rw,noatime,nodiratime"
logvol swap      --fstype="swap"  --size=4096 --name=swap --label="SWAP" --vgname=rootvg

# RPM PACKAGES
%packages
@^minimal-environment
kexec-tools
nfs-utils
nfs4-acl-tools
perl
chrony
%end

# KDUMP
%addon com_redhat_kdump --enable --reserve-mb='auto'
%end

# POST INSTALL COMMANDS TO EXECUTE
%post --log=/root/ks-post.log --interpreter=/usr/bin/bash

  # POST: route
  echo ROUTE_NET via ROUTE_VIA > /etc/sysconfig/network-scripts/route-INTERFACE2

  # POST: chrony CONFIG
  cat << TIME > /etc/chrony.conf
server NTP1 iburst
server NTP2 iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony
keyfile /etc/chrony.keys
leapsectz right/UTC
TIME

  # POST: chrony SERVICE
  systemctl enable chronyd

%end

# PASSWORD REQUIREMENTS
%anaconda
pwpolicy root --minlen=6 --minquality=1 --notstrict --nochanges --notempty
pwpolicy user --minlen=6 --minquality=1 --notstrict --nochanges --emptyok
pwpolicy luks --minlen=6 --minquality=1 --notstrict --nochanges --notempty
%end

kickstart.sh

… and last but not least – the kickstart.sh script. It does not take any arguments – it just loads the kickstart.config file variables and then replaces all config options in kickstart.skel with sed(1) to generate new Kickstart file as files/${SYSTEM_NAME}.cfg file. It also copies that config into files/${SYSTEM_NAME}.config for convenience.

# cat kickstart.sh
#! /bin/sh

if [ ! -f kickstart.config ]
then
  echo "ERROR: file 'kickstart.config' not available"
  exit 1
fi

if [ ! -f kickstart.skel ]
then
  echo "ERROR: file 'kickstart.skel' not available"
  exit 1
fi

. "$( pwd )/kickstart.config"

mkdir -p files ksfloppy iso

cp kickstart.config files/${SYSTEM_NAME}.config

if [ ${?} -eq 0 ]
then
  echo "INFO: kickstart config copied to 'files/${SYSTEM_NAME}.config' location"
else
  echo "ERROR: could not copy config to 'files/${SYSTEM_NAME}.config' location"
  exit 1
fi

sed                                           \
  -e s@SYSTEM_NAME@${SYSTEM_NAME}@g           \
  -e s@SALT_MINION_NAME@${SALT_MINION_NAME}@g \
  -e s@SALT_MASTER_IP@${SALT_MASTER_IP}@g     \
  -e s@REPO_SERVER_IP@${REPO_SERVER_IP}@g     \
  -e s@RHEL_MAJOR@${RHEL_MAJOR}@g             \
  -e s@RHEL_VERSION@${RHEL_VERSION}@g         \
  -e s@RHEL_ARCH@${RHEL_ARCH}@g               \
  -e s@INTERFACE1@${INTERFACE1}@g             \
  -e s@IP_ADDRESS1@${IP_ADDRESS1}@g           \
  -e s@NETMASK1@${NETMASK1}@g                 \
  -e s@INTERFACE2@${INTERFACE2}@g             \
  -e s@IP_ADDRESS2@${IP_ADDRESS2}@g           \
  -e s@NETMASK2@${NETMASK2}@g                 \
  -e s@GATEWAY@${GATEWAY}@g                   \
  -e s@NAMESERVER1@${NAMESERVER1}@g           \
  -e s@NAMESERVER2@${NAMESERVER2}@g           \
  -e s@NTP1@${NTP1}@g                         \
  -e s@NTP2@${NTP2}@g                         \
  -e s@ROUTE_NET@${ROUTE_NET}@g               \
  -e s@ROUTE_VIA@${ROUTE_VIA}@g               \
  kickstart.skel > files/${SYSTEM_NAME}.cfg

if [ ${?} -eq 0 ]
then
  echo "INFO: kickstart file 'files/${SYSTEM_NAME}.cfg' generated"
else
  echo "ERROR: failed to generate 'files/${SYSTEM_NAME}.cfg' kickstart file"
  exit 1
fi

echo "INFO: mkisofs(8) output BEGIN"
echo "-----------------------------"

mkisofs -J -R -l -graft-points -V "OEMDRV" \
        -input-charset utf-8 \
        -o iso/${SYSTEM_NAME}.oemdrv.iso \
        ks.cfg=files/${SYSTEM_NAME}.cfg ksfloppy

echo "-----------------------------"
echo "INFO: mkisofs(8) output ENDED"

if [ ${?} -eq 0 ]
then
  echo "INFO: ISO image 'iso/${SYSTEM_NAME}.oemdrv.iso' generated"
else
  echo "ERROR: failed to generate 'iso/${SYSTEM_NAME}.oemdrv.iso' ISO image"
  exit 1
fi

It finishes its work in less then a second. Here is its output.

… and the generated ISO file.

# ls -lh iso/kickme.oemdrv.iso
-rw-r--r--  1 root  wheel   366K Apr 10 21:57 iso/kickme.oemdrv.iso

Result

Now – when you boot the kickme VirtualBox virtual machine with CD-ROM devices loaded you will end up with RHEL system installed according to your generated Kickstart config. Here are some files from the kickme RHEL installed system.

Filesystems with LABELs such as BOOT or VAR defined.

# lsblk -i -f
NAME            FSTYPE      LABEL UUID                                   MOUNTPOINT
sda
|-sda1          xfs         BOOT  b5c66ea5-b38a-4072-b1a8-0d5882ace179   /boot
|-sda2          vfat        EFI   BB7A-4BFD                              /boot/efi
`-sda3          LVM2_member       e9BwIq-4I2W-zX6y-As42-f9N2-WTTR-WfHKdC
  |-rootvg-root xfs         ROOT  dbf8dd30-51cc-408a-9d05-b1ae67c0637c   /
  |-rootvg-swap swap        SWAP  c8a016b5-f43d-4510-9703-e9c68f02ae64   [SWAP]
  |-rootvg-usr  xfs         USR   c6694796-a5bd-4833-9a6b-a740e8bf83bf   /usr
  |-rootvg-home xfs         HOME  5264afe6-5d9c-4dc3-9d8d-e19078864aea   /home
  |-rootvg-opt  xfs         OPT   039e9575-3af8-4a8b-95c0-54fb8a515f70   /opt
  |-rootvg-tmp  xfs         TMP   11709a48-a64f-4b93-86a3-e943ff9ecf01   /tmp
  `-rootvg-var  xfs         VAR   ed20343c-b915-4234-b13e-0c6c94e03edc   /var
sr0
sr1

The /etc/fstab file with rw,noatime,nodiratime mount options.

# grep '^[^#] /etc/fstab
/dev/mapper/rootvg-root /                       xfs     rw,noatime,nodiratime 0 0
UUID=b5c66ea5-b38a-4072-b1a8-0d5882ace179 /boot xfs     rw,noatime,nodiratime 0 0
UUID=BB7A-4BFD          /boot/efi               vfat    defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2
/dev/mapper/rootvg-home /home                   xfs     rw,noatime,nodiratime 0 0
/dev/mapper/rootvg-opt  /opt                    xfs     rw,noatime,nodiratime 0 0
/dev/mapper/rootvg-tmp  /tmp                    xfs     rw,noatime,nodiratime 0 0
/dev/mapper/rootvg-usr  /usr                    xfs     rw,noatime,nodiratime 0 0
/dev/mapper/rootvg-var  /var                    xfs     rw,noatime,nodiratime 0 0
/dev/mapper/rootvg-swap none                    swap    defaults        0 0

Networking on two network interfaces and additional route generated.

# cat /etc/sysconfig/network-scripts/ifcfg-enp0s3
# Generated by parse-kickstart
TYPE=Ethernet
DEVICE=enp0s3
UUID=e1fddd41-f398-4c1d-8bef-e28ef705d568
ONBOOT=yes
IPADDR=10.0.10.199
NETMASK=255.255.255.0
GATEWAY=10.0.10.1
IPV6INIT=no
DNS1=1.1.1.1
DNS2=9.9.9.9
PROXY_METHOD=none
BROWSER_ONLY=no
PREFIX=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
NAME="System enp0s3"

# cat /etc/sysconfig/network-scripts/ifcfg-enp0s8
# Generated by parse-kickstart
TYPE=Ethernet
DEVICE=enp0s8
UUID=5454b587-8c29-41a7-93f8-532c814865de
ONBOOT=yes
IPADDR=10.0.90.199
NETMASK=255.255.255.0
IPV6INIT=no
PROXY_METHOD=none
BROWSER_ONLY=no
PREFIX=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
NAME="System enp0s8"

# cat /etc/sysconfig/network-scripts/route-enp0s8
10.0.40.10/24 via 10.0.20.1

# cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 1.1.1.1
nameserver 9.9.9.9

Alternatives

Its also possible to use the livemedia-creator from the lorax package … but I would omit it. To be honest I tried it – and it did not worked at all. I started the following process … and it run for more then a DAY and produced NOTHING.

# livemedia-creator \
    --make-iso \
    --ram 4096 \
    --vcpus 4 \
    --iso=/mnt/rhel-8.5-x86_64-boot.iso \
    --ks=/mnt/mykick.cfg

The log file for the operation was also EMPTY. At least that was the run when using the virt-install option with creating everything under virtual machine. This also intrigue me a lot. Why use virtual machines just to create installation media? Its just a bunch of files. There are better options available such as chroot(8) for example … or even so glorified containers such as Docker or Podman. Why use fully fledged virtual machine just to create ISO image? This is a big mystery for me.

Seems that livemedia-creator also has --no-virt option available … but as documentation states – it can “render the entire system unusable” – not very production ready solution for my taste. Below is a screenshot from the official RHEL documentation.

Pity that livemedia-creator did not worked for me – but I already have a working process for that.

Summary

I am not the best at summaries so I will just write here that the article has ended successfully

Regards.