keepalived static link build

之前那篇 ipvs svc 的文章，内部已经上生产了，客户的环境可能完全内网，包管理安装 keepalived 不现实，所以 keepalived 是部署容器里的。在容灾测试的时候，例如 3 台机器部署好业务，然后跑压测脚本模拟用户使用，发现关台机器的时候故障时间很短，但是这个机器开机的期间，还是很大概率故障时间很长，体现在接口的错误数量很多。大概看了下，是 keepalived 启动慢，先试启动 docker daemon，然后容器启动是顺序不固定，可能 keepalived 很后起来，于是就想着看看能不能 keepalived 拿出来，也就是静态编译。

buildx 使用

见文章 buildx 使用

在官方仓库提了 issue is there any way to static build 后，和开发者沟通尝试过不少姿势都不行，然后有个大佬 hack 下编译成功了。开发者参照改了下后我试了下最新源码试可以整出来了。

主要是在 alpine 容器构建:

git clone https://github.com/acassen/keepalived.git
cd keepalived
docker run -v $PWD:/opt --workdir /opt --rm -ti alpine

构建依赖参考仓库里的 Dockerfile.in , static 库之类的可以 alpinelinux 上去搜索

if [ -f /etc/apk/repositories ];then sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories; fi && \
    if [ -f /etc/apt/sources.list ];then sed -ri 's/(deb|security).debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list; fi && \
    if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns myhostname' > /etc/nsswitch.conf; fi

apk --no-cache add \
    binutils \
    file \
    file-dev \
    gcc \
    glib \
    glib-dev \
    ipset \
    ipset-dev \
    iptables \
    iptables-dev \
    libmnl-dev \
    libnftnl-dev \
    libnl3 \
    libnl3-dev \
    make \
    musl-dev \
    net-snmp-dev \
    openssl \
    openssl-dev \
    openssl-libs-static \
    pcre2 \
    pcre2-dev \
    autoconf \
    automake zlib-static  alpine-sdk linux-headers  libmnl-static

当然，你会发现没有 configure 脚本，可以参照 INSTALL 里执行 ./autogen.sh 生成，INSTALL 里可以参考下，有些包名在 os 上可能换了名字。同时如果要折腾的话，建议看下 --help 的内容。

./autogen.sh
./configure --help

静态编译的配置：

CFLAGS='-static -s' LDFLAGS=-static ./configure  --disable-dynamic-linking \
    --prefix=/usr \
    --exec-prefix=/usr \
    --bindir=/usr/bin \
    --sbindir=/usr/sbin \
    --sysconfdir=/etc \
    --datadir=/usr/share \
    --localstatedir=/var \
    --mandir=/usr/share/man \
    --enable-bfd \
    --enable-snmp \
    --enable-snmp-rfc \
    --enable-nftables \
    --enable-regex \
    --enable-json  --with-init=systemd --enable-vrrp --enable-libnl-dynamic

配置信息：

Keepalived version       : 2.2.7
Compiler                 : gcc gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027
Preprocessor flags       : -D_GNU_SOURCE -DNETSNMP_NO_INLINE
Compiler flags           : -g -static -s -Wall -Wextra -Wunused -Wstrict-prototypes -Wabi -Wabsolute-value -Waddress-of-packed-member -Walloca -Walloc-zero -Warith-conversion -Warray-bounds=2 -Wattribute-alias=2 -Wbad-function-cast -Wc11-c2x-compat -Wcast-align -Wcast-qual -Wdate-time -Wdisabled-optimization -Wdouble-promotion -Wduplicated-branches -Wduplicated-cond -Wfloat-conversion -Wfloat-equal -Wformat-overflow -Wformat-security -Wformat-signedness -Wformat-truncation -Wframe-larger-than=5120 -Wimplicit-fallthrough=3 -Winit-self -Winline -Winvalid-pch -Wjump-misses-init -Wlogical-op -Wmissing-declarations -Wmissing-field-initializers -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wnormalized -Wnull-dereference -Wold-style-definition -Woverlength-strings -Wpointer-arith -Wredundant-decls -Wshadow -Wshift-overflow=2 -Wstack-protector -Wstrict-overflow=4 -Wstringop-overflow=2 -Wstringop-truncation -Wsuggest-attribute=cold -Wsuggest-attribute=const -Wsuggest-attribute=format -Wsuggest-attribute=malloc -Wsuggest-attribute=noreturn -Wsuggest-attribute=pure -Wsync-nand -Wtrampolines -Wundef -Wuninitialized -Wunknown-pragmas -Wunsafe-loop-optimizations -Wunsuffixed-float-constants -Wunused-const-variable=2 -Wvariadic-macros -Wwrite-strings -fPIE -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -O2
Linker flags             : -static -pie -Wl,-z,relro -Wl,-z,now -L/usr/lib
Extra Lib                : -lm -lssl -lcrypto -lnftnl -lmnl -lpcre2-8 -lnetsnmpmibs -lnetsnmpagent -lnetsnmp -lcrypto
Use IPVS Framework       : Yes
IPVS use libnl           : No
IPVS syncd attributes    : Yes
IPVS 64 bit stats        : Yes
HTTP_GET regex support   : Yes
fwmark socket support    : Yes
Use VRRP Framework       : Yes
Use VRRP VMAC            : Yes
Use VRRP authentication  : Yes
With track_process       : Yes
With linkbeat            : Yes
Use BFD Framework        : Yes
SNMP vrrp support        : Yes
SNMP checker support     : Yes
SNMP RFCv2 support       : Yes
SNMP RFCv3 support       : Yes
SNMP send V3 for V2      : Yes
DBUS support             : No
Use JSON output          : Yes
libnl version            : None
Use IPv4 devconf         : Yes
Use iptables             : No
Use nftables             : Yes
init type                : systemd
systemd notify           : No
Strict config checks     : No
Build documentation      : No
Default runtime options  : -D

*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.

libnl/libnl-3 这个我试了下加不进去，apk add libnl-dev 可以加上去，但是我看了 alpine 里 keepalived -v 的 configure 里也没我上面开的多。强开我试了 CPPFLAGS='-I/usr/include/libnl3' LDLIBS='-lnl3 -lnl-genl-3' 和下载 libnl-3 编译安装后都不行，想折腾和传递参数啥的话，多看看 configure 文件里的内容。

编译和安装：

make && make install

/opt # keepalived -v
Keepalived v2.2.7 (02/23,2022), git commit v2.2.7-22-geb533a93

Copyright(C) 2001-2022 Alexandre Cassen, <[email protected]>

Built with kernel headers for Linux 5.10.41
Running on Linux 5.4.0-99-generic #112-Ubuntu SMP Thu Feb 3 13:50:55 UTC 2022
Distro: Alpine Linux v3.15

configure options: --disable-dynamic-linking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/share/man --enable-bfd --enable-snmp --enable-snmp-rfc --enable-nftables --enable-regex --enable-json --with-init=systemd --enable-vrrp --enable-libnl-dynamic CFLAGS=-static -s LDFLAGS=-static

Config options:  NFTABLES LVS REGEX VRRP VRRP_AUTH VRRP_VMAC JSON BFD OLD_CHKSUM_COMPAT SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 INIT=systemd

System options:  VSYSLOG MEMFD_CREATE IPV6_MULTICAST_ALL IPV4_DEVCONF RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION NETINET_LINUX_IF_ETHER_H_COLLISION IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS IPVS_TUN_TYPE IPVS_TUN_CSUM IPVS_TUN_GRE VRRP_IPVLAN IFLA_LINK_NETNSID INET6_ADDR_GEN_MODE VRF SO_MARK
/opt # file `which keepalived`
/usr/sbin/keepalived: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
/opt # ldd `which keepalived`
/lib/ld-musl-x86_64.so.1: /usr/sbin/keepalived: Not a valid dynamic program

buildx 一步到位

FROM alpine as build
RUN if [ -f /etc/apk/repositories ];then sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories; fi && \
    if [ -f /etc/apt/sources.list ];then sed -ri 's/(deb|security).debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list; fi && \
    if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns myhostname' > /etc/nsswitch.conf; fi  && \
    apk --no-cache add \
        binutils \
        file \
        file-dev \
        gcc \
        glib \
        glib-dev \
        ipset \
        ipset-dev \
        iptables \
        iptables-dev \
        libmnl-dev \
        libnftnl-dev \
        libnl3 \
        libnl3-dev \
        make \
        musl-dev \
        net-snmp-dev \
        openssl \
        openssl-dev \
        openssl-libs-static \
        pcre2 \
        pcre2-dev \
        autoconf \
        automake zlib-static  alpine-sdk linux-headers  libmnl-static git
WORKDIR /opt
RUN git clone https://github.com/acassen/keepalived.git

RUN set -ex && \
    cd /opt/keepalived && \
    ./autogen.sh && \
    CFLAGS='-static -s' LDFLAGS=-static ./configure  --disable-dynamic-linking \
    --prefix=/usr \
    --exec-prefix=/usr \
    --bindir=/usr/bin \
    --sbindir=/usr/sbin \
    --sysconfdir=/etc \
    --datadir=/usr/share \
    --localstatedir=/var \
    --mandir=/usr/share/man \
    --enable-bfd \
    --enable-snmp \
    --enable-snmp-rfc \
    --enable-nftables \
    --enable-regex \
    --enable-json  --with-init=systemd --enable-vrrp --enable-libnl-dynamic
RUN set -ex && \
    cd /opt/keepalived && \
    make && \
    make DESTDIR=/install_root install && \
    find /install_root && \
# delete the docs
    rm -rf /install_root/usr/share

FROM scratch AS bin
COPY --from=build /install_root /

docker buildx build  . --platform linux/amd64,linux/arm64 \
    --target bin --output .

$ ./usr/sbin/keepalived -v
Keepalived v2.2.7 (02/23,2022), git commit v2.2.7-22-geb533a93

Copyright(C) 2001-2022 Alexandre Cassen, <[email protected]>

Built with kernel headers for Linux 5.10.41
Running on Linux 5.4.0-99-generic #112-Ubuntu SMP Thu Feb 3 13:50:55 UTC 2022
Distro: Ubuntu 20.04.3 LTS

configure options: --disable-dynamic-linking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/share/man --enable-bfd --enable-snmp --enable-snmp-rfc --enable-nftables --enable-regex --enable-json --with-init=systemd --enable-vrrp --enable-libnl-dynamic CFLAGS=-static -s LDFLAGS=-static

Config options:  NFTABLES LVS REGEX VRRP VRRP_AUTH VRRP_VMAC JSON BFD OLD_CHKSUM_COMPAT SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 INIT=systemd

System options:  VSYSLOG MEMFD_CREATE IPV6_MULTICAST_ALL IPV4_DEVCONF RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION NETINET_LINUX_IF_ETHER_H_COLLISION IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS IPVS_TUN_TYPE IPVS_TUN_CSUM IPVS_TUN_GRE VRRP_IPVLAN IFLA_LINK_NETNSID INET6_ADDR_GEN_MODE VRF SO_MARK

$ ldd ./usr/sbin/keepalived 
	not a dynamic executable
$ file ./usr/sbin/keepalived
./usr/sbin/keepalived: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
$ ls -lh ./usr/sbin/keepalived
-rwxr-xr-x 1 root root 4.4M Feb 24 17:56 ./usr/sbin/keepalived

试了下 arm64的也可以构建 --platform linux/amd64,linux/arm64 即可。

buildx 使用

buildx 一步到位

Recommend

network-opt

Timing a web request the wrong way: with `time curl`

M3U8-Downloader

MIUI第356周更新预告：一键换机支持iPhone日历和备忘录

MIUI第356周更新：小米Note 3公交支持杭州通

1252万人次：大年初六创铁路春运单日客流高峰

“11.11”4亿新品、211送达如何做到？答案是京东物流一体化供应链物流服务

微信公布复工最快行业：美容美发美甲第一支付笔数暴增356%

中国联通回应美方撤销其214牌照：将积极维护合法正当权益

展锐唐古拉T770 5G手机获权威认证：36个月不卡、App启动快69％

About Joyk