6

54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例

 1 year ago
source link: https://blog.51cto.com/mooreyxia/6020828
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例

推荐 原创

VRRP：Virtual Router Redundancy Protocol

虚拟路由冗余协议,解决静态网关单点风险

  • 物理层:路由器、三层交换机
  • 软件层:keepalived

VRRP 网络层硬件实现

https://support.huawei.com/enterprise/zh/doc/EDOC1000141382/19258d72/basicconcepts-of-vrrp



54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例_VRRP



VRRP 相关术语



  • 虚拟路由器：Virtual Router
  • 虚拟路由器标识：VRID(0-255)，唯一标识虚拟路由器
  • VIP：Virtual IP
  • VMAC：Virutal MAC (00-00-5e-00-01-VRID)
  • 物理路由器：




  • master：主设备
  • backup：备用设备
  • priority：优先级



VRRP 相关技术



  • 通告：心跳，优先级等；周期性
  • 工作方式：抢占式，非抢占式
  • 安全认证：



  • 简单字符认证：预共享密钥



  • 工作模式：



  • 主/备：单虚拟路由器
  • 主/主：主/备（虚拟路由器1），备/主（虚拟路由器2）






Keepalived 架构



54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例_VRRP_02



  • vrrp 协议的软件实现，原生设计目的为了高可用 ipvs服务
  • keepalived 是高可用集群的通用无状态应用解决方案



  • 基于vrrp协议完成地址流动
  • 为vip地址所在的节点生成ipvs规则(在配置文件中预先定义)
  • 为ipvs集群的各RS做健康状态检测
  • 基于脚本调用接口完成脚本中定义的功能，进而影响集群事务，以此支持nginx、haproxy等服务



Keepalived 环境准备



54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例_通用高可用解决方案_03



包安装



[root@ubuntu2204 ~]#apt list keepalived
正在列表... 完成
keepalived/jammy 1:2.2.4-0.2build1 amd64
[root@ubuntu2204 ~]#apt update;apt -y install keepalived
#默认缺少配置，服务无法启动，提示/etc/keepalived/keepalived.conf 不存在
[root@ubuntu2204 ~]#systemctl status keepalived.service
○ keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor
preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Thu 2023-01-12 15:28:42 CST; 4s ago
└─ ConditionFileNotEmpty=/etc/keepalived/keepalived.conf was not
met
1月 12 15:26:47 ubuntu2204.wang.org systemd[1]: Condition check resulted in
Keepalive Daemon (LVS and VRRP) being skipped.
1月 12 15:28:42 ubuntu2204.wang.org systemd[1]: Condition check resulted in
Keepalive Daemon (LVS and VRRP) being skipped.
#利用模板生成生配置文件
[root@ubuntu2204 ~]#cp /usr/share/doc/keepalived/samples/keepalived.conf.sample
/etc/keepalived/keepalived.conf
[root@ubuntu2204 ~]#systemctl start keepalived.service
[root@ubuntu2204 ~]#systemctl status keepalived.service
● keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor
preset: enabled)
Active: active (running) since Thu 2023-01-12 15:29:54 CST; 1s ago
Main PID: 2174 (keepalived)
Tasks: 3 (limit: 2196)
Memory: 4.5M
CPU: 40ms
CGroup: /system.slice/keepalived.service
├─2174 /usr/sbin/keepalived --dont-fork
├─2175 /usr/sbin/keepalived --dont-fork
└─2176 /usr/sbin/keepalived --dont-fork



案例：实现master/slave的 Keepalived 单主架构



配置主从vrrp虚拟路由设备，同时加设心跳检测专用网卡



#202 主设备
[root@ka1 ~]#cat /etc/keepalived/keepalived.conf 
global_defs {
   router_id ka1.mooreyxia.org #每个keepalived主机唯一标识，建议使用当前主机名
   vrrp_mcast_group4 224.1.1.1 #指定组播IP地址
}

vrrp_instance VI_1 { #vrrp的实例名,一般为业务名称
    state MASTER #BACKUP#当前节点在此虚拟路由器上的初始状态，状态为MASTER或者BACKUP，当priority相同时，先启动的节点优先获取VIP
    interface eth1 #绑定为当前VRRP虚拟路由器使用的物理接口做集群内心跳检测.注意：如果和虚拟IP一样，有脑裂现象
    virtual_router_id 88 #每个虚拟路由器唯一标识，同一集群内公用
    priority 100 #当前物理节点在此虚拟路由器的优先级
    advert_int 1 #vrrp通告的时间间隔，默认1s
    authentication { #认证机制
        auth_type PASS #AH为IPSEC认证(不推荐),PASS为简单密码(建议使用)
        auth_pass 123456 #预共享密钥，仅前8位有效，同一个虚拟路由器的多个keepalived节点必须一样
    }
    virtual_ipaddress { #虚拟IP,生产环境可能指定几十上百个VIP地址
        10.0.0.200/24 dev eth0 lable eth0:1 #指定VIP的网卡，建议和interface指令指定的网卡不在一个网卡 #指定VIP的网卡label
    }
[root@ka1 ~]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:f6:07:67 brd ff:ff:ff:ff:ff:ff
    altname enp2s1
    altname ens33
    inet 10.0.0.202/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.200/24 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fef6:767/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:f6:07:71 brd ff:ff:ff:ff:ff:ff
    altname enp2s5
    altname ens37
    inet 192.168.10.202/24 brd 192.168.10.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fef6:771/64 scope link 
       valid_lft forever preferred_lft forever

[root@ka1 ~]#systemctl restart keepalived.service
[root@ka1 ~]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202

#203 从设备
[root@ka2 ~]#cat /etc/keepalived/keepalived.conf 
global_defs {
   router_id ka2.mooreyxia.org
   vrrp_mcast_group4 224.1.1.1
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth1
    virtual_router_id 88
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.200/24 dev eth0 lable eth0:1
    }
}
[root@ka2 ~]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:bb:ef:c8 brd ff:ff:ff:ff:ff:ff
    altname enp2s1
    altname ens33
    inet 10.0.0.203/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:febb:efc8/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:bb:ef:d2 brd ff:ff:ff:ff:ff:ff
    altname enp2s5
    altname ens37
    inet 192.168.10.203/24 brd 192.168.10.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:febb:efd2/64 scope link 
       valid_lft forever preferred_lft forever
[root@ka2 ~]#systemctl restart keepalived.service
[root@ka2 ~]#hostname -I
10.0.0.203 192.168.10.203 

#测试 - 抓包查看数据发送 
[root@ka2 ~]#tcpdump -i eth1 -nn host 224.1.1.1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:47:55.724918 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:47:56.726320 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:47:57.726704 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20

#停止202主设备服务
[root@ka1 ~]#systemctl stop keepalived.service
#数据包内容发生变化，从设备上线  注意prio值变化 100 --> 0 --> 80
[root@ka2 ~]#tcpdump -i eth1 -nn host 224.1.1.1
20:48:55.768656 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:48:56.768491 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:48:57.567124 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 0, authtype simple, intvl 1s, length 20
20:48:58.255613 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:48:59.257236 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:49:00.257956 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:49:01.258241 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20

#此时主设备上的vrrp ip地址消失，在从设备上出现
[root@ka1 ~]#hostname -I
10.0.0.202 192.168.10.202 
[root@ka2 ~]#hostname -I
10.0.0.203 10.0.0.200 192.168.10.203 

#恢复主设备服务
[root@ka1 ~]#systemctl restart keepalived.service
#数据包内容发生变化，从设备上线  注意prio值变化 80 --> 100
[root@ka2 ~]#tcpdump -i eth1 -nn host 224.1.1.1
20:53:06.452921 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:53:07.453781 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:53:08.454422 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:53:09.454788 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:53:10.455557 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:53:10.522302 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:53:11.524605 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:53:12.525119 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:53:13.525798 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20

#此时主设备上的vrrp ip地址恢复，在从设备上vrrp ip地址消失
[root@ka1 ~]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202 
[root@ka2 ~]#hostname -I
10.0.0.203 192.168.10.203



查看心跳线间的传输协议



*虽然集群间通信密码没有加密，还是可以被看到，有大于无



54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例_通用高可用解决方案_04



启用 Keepalived 日志功能



[root@ka2 ~]#cat /lib/systemd/system/keepalived.service |grep EnvironmentFile
EnvironmentFile=-/apps/keepalived/etc/sysconfig/keepalived
[root@ka2 ~]#vim /apps/keepalived/etc/sysconfig/keepalived
[root@ka2 ~]#cat /apps/keepalived/etc/sysconfig/keepalived
# Options for keepalived. See `keepalived --help' output and keepalived(8) and
# keepalived.conf(5) man pages for a list of all options. Here are the most
# common ones :
#
# --vrrp               -P    Only run with VRRP subsystem.
# --check              -C    Only run with Health-checker subsystem.
# --dont-release-vrrp  -V    Dont remove VRRP VIPs & VROUTEs on daemon stop.
# --dont-release-ipvs  -I    Dont remove IPVS topology on daemon stop.
# --dump-conf          -d    Dump the configuration data.
# --log-detail         -D    Detailed log messages.
# --log-facility       -S    0-7 Set local syslog facility (default=LOG_DAEMON)
#

KEEPALIVED_OPTIONS="-D -S 6"

[root@ka2 ~]#vim /etc/rsyslog.
rsyslog.conf  rsyslog.d/    
[root@ka2 ~]#vim /etc/rsyslog.conf 
[root@ka2 ~]#cat /etc/rsyslog.conf |grep local6
local6.* /var/log/keepalived.log
[root@ka2 ~]#systemctl restart keepalived rsyslog
#观察日志
[root@ka2 ~]#tail -f /var/log/keepalived.log
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: Registering Kernel netlink command channel
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: (/etc/keepalived/keepalived.conf: Line 17) Unknown configuration entry 'lable' for ip address - ignoring
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: (/etc/keepalived/keepalived.conf: Line 17) Unknown configuration entry 'eth0:1' for ip address - ignoring
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: Assigned address 192.168.10.203 for interface eth1
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: Assigned address fe80::20c:29ff:febb:efd2 for interface eth1
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: Registering gratuitous ARP shared channel
Jan 20 11:46:33 ka2 Keepalived[1467]: Startup complete
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: (VI_1) removing VIPs.
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: (VI_1) Entering BACKUP STATE (init)
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: VRRP sockpool: [ifindex(  3), family(IPv4), proto(112), fd(12,13) multicast, address(224.1.1.1)]



实现 Keepalived 独立子配置文件



  • 当生产环境复杂时， /etc/keepalived/keepalived.conf 文件中保存所有集群的配置会导致内容过多，不易管理，可以将不同集群的配置，比如：不同集群的VIP配置放在独立的子配置文件中



[root@ka1 ~]#vim /etc/keepalived/keepalived.conf
[root@ka1 ~]#cat /etc/keepalived/keepalived.conf 
global_defs {
   router_id ka1.mooreyxia.org
   vrrp_mcast_group4 224.1.1.1
}

include /etc/keepalived/conf.d/*.conf


[root@ka1 ~]#mkdir /etc/keepalived/conf.d/
[root@ka1 ~]#vim /etc/keepalived/conf.d/mooreyxia.org.conf
[root@ka1 ~]#cat /etc/keepalived/conf.d/mooreyxia.org.conf
vrrp_instance VI_1 {
    state MASTER
    interface eth1
    virtual_router_id 88
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.200/24 dev eth0 lable eth0:1
    }

[root@ka1 ~]#systemctl restart keepalived
[root@ka1 ~]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202



脑裂



主备节点同时拥有同一个VIP，此时为脑裂理象



脑裂现象原因



  • 心跳线故障
  • 防火墙错误配置
  • Keepalived 配置错误




  • interface错误，virtual_router_id不一致，密码不一致，优先级相同



范例: 发现脑裂



#在从节点服务器执行iptables -A INPUT -s 主服务心跳网卡IP -j DROP 进行模拟
[root@ka2 ~]#iptables -A INPUT -s 192.168.10.202  -j DROP
#主从节点都出现vrrp地址
[root@ka1 ~]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202 
[root@ka2 ~]#hostname -I
10.0.0.203 10.0.0.200 192.168.10.203 

#arping 发现两个MAC对外广播vrrp地址
[root@rocky8 ~]#arping 10.0.0.200
ARPING 10.0.0.200 from 10.0.0.8 eth0
Unicast reply from 10.0.0.200 [00:0C:29:BB:EF:C8]  1.081ms
Unicast reply from 10.0.0.200 [00:0C:29:F6:07:67]  1.104ms



非抢占模式 nopreempt



  • 默认为抢占模式 preempt，即当高优先级的主机恢复在线后，会抢占低先级的主机的master角色，造成网络抖动，建议设置为非抢占模式 nopreempt ，即高优先级主机恢复后，并不会抢占低优先级主机的master 角色



  • 注意: 非抢占模式下,如果原主机down机, VIP迁移至的新主机, 后续新主机也发生down（（keepalived服务down））时,VIP还会迁移回修复好的原主机
  • 但如果新主机的服务down掉（keepalived服务正常），原主机也不会接管VIP，仍会由新主机拥有VIP即非抢占式模式，只是适合当主节点宕机，切换到从节点的一次性的高可用性，后续即使当原主节点修复好，仍无法再次起到高用功能



注意：要关闭 VIP抢占，必须将各 Keepalived 服务器 state 配置为 BACKUP



#ka1
[root@ka1 ~]#vim /etc/keepalived/conf.d/mooreyxia.org.conf 
[root@ka1 ~]#cat /etc/keepalived/conf.d/mooreyxia.org.conf 
vrrp_instance VI_1 {
    state BACKUP #都为BACKUP
    interface eth1
    virtual_router_id 88
    priority 100
    advert_int 1
    nopreempt #添加此行，设为nopreempt
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.200/24 dev eth0 lable eth0:1
    }
[root@ka1 ~]#systemctl restart keepalived

#ka2 - 从服务器不需要添加nopreempt
[root@ka2 ~]#cat /etc/keepalived/conf.d/mooreyxia.org.conf 
vrrp_instance VI_1 {
    state BACKUP
    interface eth1
    virtual_router_id 88
    priority 80    #优先级低
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.200/24 dev eth0 lable eth0:1
    }
}

#测试
#正常情况
[root@ka1 ~]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202 
[root@ka2 ~]#hostname -I
10.0.0.203 192.168.10.203 
#停止ka1服务，主服务vrrp漂移至从服务
[root@ka1 ~]#systemctl stop keepalived
[root@ka1 ~]#hostname -I
10.0.0.202 192.168.10.202 
[root@ka2 ~]#hostname -I
10.0.0.203 10.0.0.200 192.168.10.203 
#修复ka1服务，vrrp未漂移
[root@ka1 ~]#systemctl start keepalived
[root@ka1 ~]#hostname -I
10.0.0.202 192.168.10.202 
[root@ka2 ~]#hostname -I
10.0.0.203 10.0.0.200 192.168.10.203 

#如果此时从节点服务dwon机，vrrp依然会漂移至主节点
[root@ka2 ~]#systemctl stop keepalived
[root@ka2 ~]#hostname -I
10.0.0.203 192.168.10.203 
[root@ka1 ~]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202



抢占延迟模式 preempt_delay



  • 抢占延迟模式，即优先级高的主机恢复后，不会立即抢回VIP，而是延迟一段时间（默认300s）再抢回VIP



#更改主节点抢占模式
[root@ka1 ~]#vim /etc/keepalived/conf.d/mooreyxia.org.conf 
[root@ka1 ~]#cat /etc/keepalived/conf.d/mooreyxia.org.conf 
vrrp_instance VI_1 {
    state BACKUP #都为BACKUP
    interface eth1
    virtual_router_id 88
    priority 100
    advert_int 1
   preempt_delay 60 #添加此行，设为延迟10秒再抢占
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.200/24 dev eth0 lable eth0:1
    }
[root@ka1 ~]#systemctl restart keepalived



VIP 单播配置



  • 默认keepalived主机之间利用多播相互通告消息，会造成网络拥塞，可以替换成单播，减少网络流量
  • 另外：有些公有云不支持多播，可以利用单播实现



注意：启用 vrrp_strict 时，不能启用单播



#设置心跳线网络为单播模式
#主节点
[root@ka1 ~]#vim /etc/keepalived/conf.d/mooreyxia.org.conf 
[root@ka1 ~]#cat /etc/keepalived/conf.d/mooreyxia.org.conf 
vrrp_instance VI_1 {
    state BACKUP
    interface eth1
    virtual_router_id 88
    priority 100
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.200/24 dev eth0 lable eth0:1
    }
    unicast_src_ip 192.168.10.202 #本机IP #单播优先于多播
    unicast_peer{
        192.168.10.203 #如果有多个keepalived,再加其它节点的IP
    }
}
[root@ka1 ~]#systemctl restart keepalived
#从节点
[root@ka2 ~]#vim /etc/keepalived/conf.d/mooreyxia.org.conf 
[root@ka2 ~]#cat /etc/keepalived/conf.d/mooreyxia.org.conf 
vrrp_instance VI_1 {
    state BACKUP
    interface eth1
    virtual_router_id 88
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.200/24 dev eth0 lable eth0:1
    }
    unicast_src_ip 192.168.10.203
    unicast_peer{
        192.168.10.202
    }
}

[root@ka2 ~]#systemctl restart keepalived



Keepalived 通知脚本配置



  • 通知脚本类型



#当前节点成为主节点时触发的脚本
notify_master <STRING>|<QUOTED-STRING>

#当前节点转为备节点时触发的脚本
notify_backup <STRING>|<QUOTED-STRING>

#当前节点转为“失败”状态时触发的脚本
notify_fault <STRING>|<QUOTED-STRING>

#通用格式的通知触发机制，一个脚本可完成以上三种状态的转换时的通知
notify <STRING>|<QUOTED-STRING>

#当停止VRRP时触发的脚本
notify_stop <STRING>|<QUOTED-STRING>

#脚本的调用方法 - 在 vrrp_instance VI_1 语句块的末尾加下面行
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"



  • 实现 Keepalived 状态切换的通知脚本



#在所有 keepalived节点配置如下
[root@ka1 ~]#cat /etc/keepalived/notify.sh
#!/bin/bash
contact='[email protected]'
email_send='[email protected]'
email_passwd='xxxxx'
email_smtp_server='smtp.qq.com'
. /etc/os-release
msg_error() {
  echo -e "\033[1;31m$1\033[0m"
}
msg_info() {
  echo -e "\033[1;32m$1\033[0m"
}
msg_warn() {
  echo -e "\033[1;33m$1\033[0m"
}
color () {
  RES_COL=60
  MOVE_TO_COL="echo -en \\033[${RES_COL}G"
  SETCOLOR_SUCCESS="echo -en \\033[1;32m"
  SETCOLOR_FAILURE="echo -en \\033[1;31m"
  SETCOLOR_WARNING="echo -en \\033[1;33m"
  SETCOLOR_NORMAL="echo -en \E[0m"
  echo -n "$1" && $MOVE_TO_COL
  echo -n "["
    if [ $2 = "success" -o $2 = "0" ] ;then
      ${SETCOLOR_SUCCESS}
  echo -n $" OK "
    elif [ $2 = "failure" -o $2 = "1" ] ;then
      ${SETCOLOR_FAILURE}
  echo -n $"FAILED"
    else
      ${SETCOLOR_WARNING}
      echo -n $"WARNING"
    fi
  ${SETCOLOR_NORMAL}
  echo -n "]"
  echo
  }
install_sendemail () {
if [[ $ID =~ rhel|centos|rocky ]];then
rpm -q sendemail &> /dev/null || yum install -y sendemail
elif [ $ID = 'ubuntu' ];then
dpkg -l |grep -q sendemail || { apt update; apt install -y libiosocket-
ssl-perl libnet-ssleay-perl sendemail ; }
else
color "不支持此操作系统，退出!" 1
exit
fi
}
send_email () {
local email_receive="$1"
local email_subject="$2"
local email_message="$3"
sendemail -f $email_send -t $email_receive -u $email_subject -m
$email_message -s $email_smtp_server -o message-charset=utf-8 -o tls=yes -xu
$email_send -xp $email_passwd
[ $? -eq 0 ] && color "邮件发送成功!" 0 || color "邮件发送失败!" 1
}
notify() {
if [[ $1 =~ ^(master|backup|fault)$ ]];then
mailsubject="$(hostname) to be $1, vip floating"
mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be
$1"
send_email "$contact" "$mailsubject" "$mailbody"
else
echo "Usage: $(basename $0) {master|backup|fault}"
exit 1
fi
}
install_sendemail
notify $1
[root@ka1 ~]#chmod a+x /etc/keepalived/notify.sh
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
......
virtual_ipaddress {
10.0.0.200/24 dev eth0 label eth0:1
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
#模拟master故障
[root@ka1 ~]#killall keepalived



实现 Master/Master 的 Keepalived 双主架构



54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例_VRRP_05



  • 将两个或以上VIP分别运行在不同的keepalived服务器，以实现服务器并行提供web访问的目的，提高服务器资源利用率
  • 注意：多主模型在发生VIP迁移时会使另一个服务器负载加大，注意承压阈值



#在拥有了一个vip的基础上再添加一组vip，与上一个vip设置主从关系相反
#ka1
[root@ka1 ~]#vim /etc/keepalived/conf.d/mooreyxia.com.conf 
[root@ka1 ~]#cat /etc/keepalived/conf.d/mooreyxia.com.conf 
vrrp_instance VI_2 {
    state BACKUP
    interface eth1 
    virtual_router_id 89
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 654321
    }
    virtual_ipaddress {
        10.0.0.100/24 dev eth0 lable eth0:2
    }
    unicast_src_ip 192.168.10.202
    unicast_peer{
        192.168.10.203
    }
}
[root@ka1 ~]#systemctl restart keepalived
#ka2
[root@ka2 ~]#vim /etc/keepalived/conf.d/mooreyxia.com.conf 
[root@ka2 ~]#cat /etc/keepalived/conf.d/mooreyxia.com.conf 
vrrp_instance VI_2 {
    state MASTER
    interface eth1
    virtual_router_id 89
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 654321
    }
    virtual_ipaddress {
        10.0.0.100/24 dev eth0 lable eth0:2
    }
    unicast_src_ip 192.168.10.203
    unicast_peer{
        192.168.10.202
    }
}
[root@ka2 ~]#systemctl restart keepalived

#测试
[root@ka1 conf.d]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202 
[root@ka2 conf.d]#hostname -I
10.0.0.203 10.0.0.100 192.168.10.203



实现多主模型架构



  • 三个节点的三主六从架构实现
  • 实际上超出三个节点的情况时每个服务器一主两从也足够应对
  • 注意：多主模型在发生VIP迁移时会使另一个服务器负载加大，注意承压阈值



54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例_通用高可用解决方案_06



#第一个节点ka1配置：
virtual_router_id 1 , Vrrp instance 1 , MASTER，优先级100
virtual_router_id 2 , Vrrp instance 2 , BACKUP，优先级80
virtual_router_id 3 , Vrrp instance 3 , BACKUP，优先级60
#第二个节点ka2配置：
virtual_router_id 1 , Vrrp instance 1 , BACKUP，优先级60
virtual_router_id 2 , Vrrp instance 2 , MASTER，优先级100
virtual_router_id 3 , Vrrp instance 3 , BACKUP，优先级80
#第三个节点ka3配置：
virtual_router_id 1 , Vrrp instance 1 , BACKUP，优先级80
virtual_router_id 2 , Vrrp instance 2 , BACKUP，优先级60
virtual_router_id 3 , Vrrp instance 3 , MASTER，优先级100



同步组



  • LVS NAT 模型VIP和DIP需要同步，需要同步组



vrrp_sync_group VG_1 {
group {
  VI_1 # name of vrrp_instance (below)
  VI_2 # One for each moveable IP
  }
}
vrrp_instance VI_1 {
  eth0
  vip
}
vrrp_instance VI_2 {
  eth1
  dip
}






实现单主的 LVS-DR 模式



  • 多主LVS-DR模型的实现也可参照，更改keepalived为互为主从策略即可



虚拟服务器配置



virtual_server IP port { #VIP和PORT
delay_loop <INT> #检查后端服务器的时间间隔
lb_algo rr|wrr|lc|wlc|lblc|sh|dh #定义调度方法
lb_kind NAT|DR|TUN #集群的类型,注意要大写
persistence_timeout <INT> #持久连接时长
protocol TCP|UDP|SCTP #指定服务协议,一般为TCP
sorry_server <IPADDR> <PORT> #所有RS故障时，备用服务器地址
real_server <IPADDR> <PORT> { #RS的IP和PORT
weight <INT> #RS权重
notify_up <STRING>|<QUOTED-STRING> #RS上线通知脚本
notify_down <STRING>|<QUOTED-STRING> #RS下线通知脚本
HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK { ... } #定义当前主机健康状态检测方法
}
}
#注意:括号必须分行写,两个括号写在同一行,如: }} 会出错



#准备两台后端RS主机
[root@rocky8 ~]#curl 10.0.0.204
web01  www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl 10.0.0.28
web02 www.mooreyxia.org 10.0.0.28

#使用脚本绑定VIP至web服务器lo网卡
[root@web01 ~]#cat lvs_dr_rs.sh 
#!/bin/bash
#Author:mooreyxia
#Date:2023-01-20
vip=10.0.0.200
mask='255.255.255.255'
dev=lo:1

#rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
#service httpd start &> /dev/null && echo "The httpd Server is Ready!"
#echo "`hostname -I`" > /var/www/html/index.html

case $1 in
start)
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    ifconfig $dev $vip netmask $mask #broadcast $vip up
    #route add -host $vip dev $dev
    echo "The RS Server is Ready!"
    ;;
stop)
    ifconfig $dev down
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "The RS Server is Canceled!"
    ;;
*) 
    echo "Usage: $(basename $0) start|stop"
    exit 1
    ;;
esac

#web01
[root@web01 ~]#bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@web01 ~]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.0.0.200/32 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:e3:99:33 brd ff:ff:ff:ff:ff:ff
    altname enp2s1
    altname ens33
    inet 10.0.0.204/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fee3:9933/64 scope link 
       valid_lft forever preferred_lft forever

#web02
[root@web02 ~]#bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@web02 ~]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.0.0.200/32 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:d3:0d:20 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.28/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fed3:d20/64 scope link 
       valid_lft forever preferred_lft forever



配置keepalived



#ka1节点的配置
[root@ka1 conf.d]#vim /etc/keepalived/conf.d/mooreyxia.org.conf 
[root@ka1 conf.d]#cat /etc/keepalived/conf.d/mooreyxia.org.conf 
vrrp_instance VI_1 {
    state MASTER
    interface eth1
    virtual_router_id 88
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.200/24 dev eth0 lable eth0:1
    }
    unicast_src_ip 192.168.10.202
    unicast_peer{
        192.168.10.203
    }
}

virtual_server 10.0.0.200 80 {
    delay_loop 3
    lb_algo wrr
    lb_kind DR
    protocol TCP
    sorry_server 127.0.0.1 80
    real_server 10.0.0.204 80 {
        weight 1
        HTTP_GET { 
            url {
                path /
                status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
    real_server 10.0.0.28 80 {
        weight 2
        TCP_CHECK { 
            connect_timeout 5
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}
[root@ka1 conf.d]#systemctl restart keepalived
[root@ka1 conf.d]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:f6:07:67 brd ff:ff:ff:ff:ff:ff
    altname enp2s1
    altname ens33
    inet 10.0.0.202/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.200/24 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fef6:767/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:f6:07:71 brd ff:ff:ff:ff:ff:ff
    altname enp2s5
    altname ens37
    inet 192.168.10.202/24 brd 192.168.10.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fef6:771/64 scope link 
       valid_lft forever preferred_lft forever


#ka2节点的配置
[root@ka2 conf.d]#vim mooreyxia.org.conf 
[root@ka2 conf.d]#cat mooreyxia.org.conf 
vrrp_instance VI_1 {
    state BACKUP
    interface eth1
    virtual_router_id 88
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.200/24 dev eth0 lable eth0:1
    }
    unicast_src_ip 192.168.10.203
    unicast_peer{
        192.168.10.202
    }
}

virtual_server 10.0.0.200 80 {
    delay_loop 3
    lb_algo wrr
    lb_kind DR
    protocol TCP
    sorry_server 127.0.0.1 80
    real_server 10.0.0.204 80 {
        weight 1
        HTTP_GET { 
            url {
                path /
                status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
    real_server 10.0.0.28 80 {
        weight 2
        TCP_CHECK { 
            connect_timeout 5
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}
[root@ka2 conf.d]#systemctl restart keepalived

#观察lvs调度模型
[root@ka1 conf.d]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.200:80 wrr
  -> 10.0.0.28:80                 Route   2      0          0         
  -> 10.0.0.204:80                Route   1      0          0  

#测试 - 权重策略成功
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web01  www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web01  www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28

#测试健康检查功能
#关闭后端服务后会自动调度到另一台服务上
[root@web01 ~]#systemctl stop nginx
#调度策略自动撤下204
[root@ka1 conf.d]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.200:80 wrr
  -> 10.0.0.28:80                 Route   2      0          0    
#访问调度到28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
#再度恢复204服务
[root@web01 ~]#systemctl start nginx
#调度策略自动加入204
[root@ka1 conf.d]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.200:80 wrr
  -> 10.0.0.28:80                 Route   2      0          4         
  -> 10.0.0.204:80                Route   1      0          0   
#访问恢复调度策略
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web01  www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web01  www.mooreyxia.org 10.0.0.204

#当后端服务都失效，keepalived会将自身加入策略组
[root@web01 ~]#systemctl stop nginx
[root@web02 ~]#systemctl stop nginx
[root@ka1 conf.d]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.200:80 wrr
  -> 127.0.0.1:80                 Route   1      0          1         

#本机服务可以做个失效页面或者做服务转发 --> 比如本机安装nginx 做失效html提醒



实现单主的 LVS-DR 模式，利用FWM绑定成多个服务为一个集群服务



  • 将http 80和https 443绑定到一个集群服务上



#在单主的 LVS-DR 模式的基础上配置后端服务https认证
[root@web01 conf.d]#vim certificate.sh 
[root@web01 conf.d]#cat certificate.sh 
#!/bin/bash

SITE_NAME=www.mooreyxia.org
CA_SUBJECT="/O=mooreyxia/CN=ca.mooreyxia.org"
SUBJECT="/C=CN/ST=huanggang/L=hubei/O=mooreyxia/CN=$SITE_NAME"
SERIAL=34
EXPIRE=202002
FILE=$SITE_NAME

openssl req  -x509 -newkey rsa:2048 -subj $CA_SUBJECT -keyout ca.key -nodes -days 202002 -out ca.crt

openssl req -newkey rsa:2048 -nodes -keyout ${FILE}.key  -subj $SUBJECT -out ${FILE}.csr

openssl x509 -req -in ${FILE}.csr  -CA ca.crt -CAkey ca.key -set_serial $SERIAL  -days $EXPIRE -out ${FILE}.crt

chmod 600 ${FILE}.key ca.key

[root@web01 conf.d]#bash certificate.sh 
.+....+......+..+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.........+......+.........+..+....+.....+...+.......+...+......+..+...+......+....+.........+.....+......+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+...+..+.........+....+..+.............+..+...+.+.........+...+...+.....+.+...+...........+.+.....+...+..........+...+.........+.......................+...+............+..........+...+...+.....+....+..+...+......+......+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
..+...+......+.......+.....+.........+......+...+....+..+..........+...+..+...+...+.......+.....+.+...........+.+..+.+......+.....+.+..+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+..+...+...+....+...+.....+.+...............+..+...+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.....+.........+......+......+...+..........+.....+....+...+..+...+............+..........+...+...+......+.....+........................+............+....+...+...........+......+.+..............+.........+.......+........+......+...+.+......+.....+.+.....+...+.+......+.........+..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...+.+.....+...+.+......+...+...+..+......+......+.+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+..........+..+.+..................+..+.......+.................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...........+......+.........+....+..+.+.........+...+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
Certificate request self-signature ok
subject=C = CN, ST = huanggang, L = hubei, O = mooreyxia, CN = www.mooreyxia.org
[root@web01 conf.d]#ls
ca.crt  ca.key  certificate.sh  www.mooreyxia.org.crt  www.mooreyxia.org.csr  www.mooreyxia.org.key
#合并ca.crt文件及www.mooreyxia.org.crt
[root@web01 conf.d]#mv www.mooreyxia.org.crt  www.mooreyxia.org.pem
[root@web01 conf.d]#ls
ca.crt  ca.key  certificate.sh  www.mooreyxia.org.crt  www.mooreyxia.org.csr  www.mooreyxia.org.key  www.mooreyxia.org.pem

#nginx启用https
[root@web01 conf.d]#vim /etc/nginx/sites-enabled/default 
[root@web01 conf.d]#cat /etc/nginx/sites-enabled/default 
...
server {
    listen 80 default_server;
    listen [::]:80 default_server;

# SSL configuration
    #
     listen 443 ssl default_server; --> 开放443
    # listen [::]:443 ssl default_server;
    #
    # Note: You should disable gzip for SSL traffic.
    # See: https://bugs.debian.org/773332
    #
    # Read up on ssl_ciphers to ensure a secure configuration.
    # See: https://bugs.debian.org/765782
    #
    # Self signed certs generated by the ssl-cert package
    # Don't use them in a production server!
    #
    # include snippets/snakeoil.conf;

root /var/www/html;

# Add index.php to the list if you are using PHP
    index index.html index.htm index.nginx-debian.html;
    ssl_certificate /etc/nginx/conf.d/www.mooreyxia.org.pem; --> 添加证书
    ssl_certificate_key /etc/nginx/conf.d/www.mooreyxia.org.key;

[root@web01 conf.d]#nginx -s reload
[root@web01 conf.d]#ss -nlt
State                Recv-Q               Send-Q                             Local Address:Port                             Peer Address:Port               Process               
LISTEN               0                    511                                      0.0.0.0:80                                    0.0.0.0:*                                        
LISTEN               0                    4096                               127.0.0.53%lo:53                                    0.0.0.0:*                                        
LISTEN               0                    128                                      0.0.0.0:22                                    0.0.0.0:*                                        
LISTEN               0                    128                                    127.0.0.1:6010                                  0.0.0.0:*                                        
LISTEN               0                    511                                      0.0.0.0:443                                   0.0.0.0:*                                        
LISTEN               0                    128                                    127.0.0.1:6011                                  0.0.0.0:*                                        
LISTEN               0                    511                                         [::]:80                                       [::]:*                                        
LISTEN               0                    128                                         [::]:22                                       [::]:*                                        
LISTEN               0                    128                                        [::1]:6010                                     [::]:*                                        
LISTEN               0                    128                                        [::1]:6011                                     [::]:*                         

#web02上的nginx也开启443
[root@web02 conf.d]#cat /etc/nginx/nginx.conf
...
    server {
        listen       443 ssl http2 default_server;
#        listen       [::]:443 ssl http2 default_server;
        server_name  _;
        root         /usr/share/nginx/html;
#
        ssl_certificate "/etc/nginx/conf.d/www.mooreyxia.org.pem";
        ssl_certificate_key "/etc/nginx/conf.d/www.mooreyxia.org.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers PROFILE=SYSTEM;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
        location / {
        }
#
        error_page 404 /404.html;
            location = /40x.html {
        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
    }

}
[root@web02 conf.d]#nginx -s reload
[root@web02 conf.d]#ss -nlt
State                Recv-Q               Send-Q                             Local Address:Port                             Peer Address:Port               Process               
LISTEN               0                    128                                      0.0.0.0:80                                    0.0.0.0:*                                        
LISTEN               0                    128                                      0.0.0.0:22                                    0.0.0.0:*                                        
LISTEN               0                    100                                    127.0.0.1:25                                    0.0.0.0:*                                        
LISTEN               0                    128                                      0.0.0.0:443                                   0.0.0.0:*                                        
LISTEN               0                    128                                         [::]:80                                       [::]:*                                        
LISTEN               0                    128                                         [::]:22                                       [::]:*                                        
LISTEN               0                    100                                        [::1]:25                                       [::]:*                                        

#测试
[root@rocky8 ~]#curl -k https://10.0.0.204
web01  www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl -k https://10.0.0.28
web02 www.mooreyxia.org 10.0.0.28



合并http和https两个集群



#两个节点都执行以下操作 - 将80和443作为相同的一组通讯 通过iptables贴标签的方式
[root@ka1 conf.d]#iptables -t mangle -A PREROUTING -d 10.0.0.200 -p tcp -m multiport --dports 80,443 -j MARK --set-mark 6
[root@ka2 conf.d]#iptables -t mangle -A PREROUTING -d 10.0.0.200 -p tcp -m multiport --dports 80,443 -j MARK --set-mark 6

#在keepalived中添加一组防火墙标签的服务
#ka1
[root@ka1 conf.d]#vim mooreyxia.org.conf 
[root@ka1 conf.d]#cat mooreyxia.org.conf 
vrrp_instance VI_1 {
    state MASTER
    interface eth1
    virtual_router_id 88
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.200/24 dev eth0 lable eth0:1
    }
    unicast_src_ip 192.168.10.202
    unicast_peer{
        192.168.10.203
    }
}

virtual_server fwmark 6 { #指定FWM为6
    delay_loop 3
    lb_algo wrr
    lb_kind DR
    protocol TCP
    sorry_server 127.0.0.1 80 #注意端口必须指定
    real_server 10.0.0.204 80 {
        weight 1
        HTTP_GET { 
            url {
                path /
                status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
    real_server 10.0.0.28 80 {
        weight 2
        TCP_CHECK { 
            connect_timeout 5
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}
[root@ka1 conf.d]#systemctl restart keepalived
[root@ka1 conf.d]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  6 wrr
  -> 10.0.0.28:0                  Route   2      0          0         
  -> 10.0.0.204:0                 Route   1      0          0   
#ka2
[root@ka2 conf.d]#vim mooreyxia.org.conf 
[root@ka2 conf.d]#cat mooreyxia.org.conf 
vrrp_instance VI_1 {
    state BACKUP
    interface eth1
    virtual_router_id 88
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.200/24 dev eth0 lable eth0:1
    }
    unicast_src_ip 192.168.10.203
    unicast_peer{
        192.168.10.202
    }
}

virtual_server fwmark 6 {
    delay_loop 3
    lb_algo wrr
    lb_kind DR
    protocol TCP
    sorry_server 127.0.0.1 80
    real_server 10.0.0.204 80 {
        weight 1
        HTTP_GET { 
            url {
                path /
                status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
    real_server 10.0.0.28 80 {
        weight 2
        TCP_CHECK { 
            connect_timeout 5
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}
[root@ka2 conf.d]#systemctl restart keepalived
[root@ka2 conf.d]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  6 wrr
  -> 10.0.0.28:0                  Route   2      0          0         
  -> 10.0.0.204:0                 Route   1      0          0



测试 - http与https都会调度到同一集群，只要一个服务器调度到http,另一个服务器必定调度到https，实现负载均衡策略



[root@rocky8 ~]#curl http://www.mooreyxia.org  -k https://www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
web01  www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl http://www.mooreyxia.org  -k https://www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28  --> 这里设置了权重导致
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl http://www.mooreyxia.org  -k https://www.mooreyxia.org
web01  www.mooreyxia.org 10.0.0.204
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl http://www.mooreyxia.org  -k https://www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
web01  www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl http://www.mooreyxia.org  -k https://www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28  --> 这里设置了权重导致
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl http://www.mooreyxia.org  -k https://www.mooreyxia.org
web01  www.mooreyxia.org 10.0.0.204
web02 www.mooreyxia.org 10.0.0.28






VRRP Script



  • keepalived利用 VRRP Script 技术，可以调用外部的辅助脚本进行资源监控，并根据监控的结果实现优先动态调整，从而实现其它应用的高可用性功能



案例：实现单主模式的 Nginx 反向代理的高可用



#在两个节点都配置nginx反向代理
[root@ka1-centos8 ~]#vim /etc/nginx/nginx.conf
http {
upstream websrvs {
server 10.0.0.7:80 weight=1;
server 10.0.0.17:80 weight=1;
}
server {
listen 80;
location /{
proxy_pass http://websrvs/;
}
}
}
#在两个节点都配置实现nginx反向代理高可用
[root@ka1-centos8 ~]#cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from kaadmin@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.mooreyxia.org #在另一个节点为ka2.mooreyxia.org
vrrp_mcast_group4 224.0.100.100
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
#script "/usr/bin/killall -0 nginx" 此写法支持
#script "/usr/bin/killall -0 nginx &>/dev/null" 不支持&>此写法
interval 1
weight -30
fall 3
rise 5
timeout 2
}
vrrp_instance VI_1 {
state MASTER #在另一个节点为BACKUP
interface eth0
virtual_router_id 66
priority 100 #在另一个节点为80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.10/24 dev eth0 label eth0:1
}
track_interface {
eth0
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
track_script {
check_nginx
}
}
[root@ka1-centos8 ~]# yum install psmisc -y
[root@ka1-centos8 ~]# cat /etc/keepalived/check_nginx.sh
#!/bin/bash
/usr/bin/killall -0 nginx || systemctl restart nginx --> killall -0 service 可以检查僵尸进程
[root@ka1-centos8 ~]# chmod a+x /etc/keepalived/check_nginx.sh



我是moore，大家一起加油！！！

        

        

            

                report
            

        

    







        
Recommend

        



                

    
About Joyk

    
 





Aggregate valuable and interesting links.

Joyk means Joy of geeK