7
微服务网关 APISIX 群集配置指南
source link: https://blog.51cto.com/waringid/5845100
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
微服务网关 APISIX 群集配置指南
精选 原创APISIX 通过 ETCD 存储数据，利用 ETCD 本身的功能实现群集管理和控制。有关 APISIX 的安装配置详情参考“ APISIX 网关 CentOS 7 环境安装配置”。
前面文章描述的是单个 ETCD 节点的配置操作，本文通过配置 ETCD 3节点群集的方式实现 APISIX 的群集操作。
1.1.环境
操作系统：CentOS7，APISIX：2.10.1 ，Dashboard：2.9，ETCD：3.5
192.168.112.98 | etcd1 | |
192.168.112.99 | etcd2 | |
192.168.112.100 | etcd3 |
1.2.补充内容
ApiSix 的前置相关组件安装参考“ nstall-dependencies.md”
curl https://raw.githubusercontent.com/apache/apisix/master/utils/install-dependencies.sh -sL | bash - |
更详细的安装配置可以参考“ how-to-build.md”
2.环境配置
2.1.配置主机
在三台机器上分别执行以下操作完成主机名称设置
cat >> /etc/hosts <<EOF 192.168.112.98 etcd1 192.168.112.99 etcd2 192.168.112.100 etcd3 EOF |
2.2.安全配置
关闭系统的 Selinux 功能和操作系统防火墙
systemctl stop firewalld systemctl disable firewalld setenforce 0 vi /etc/selinux/config SELINUX=disabled |
3.TLS 密钥和证书
使用TLS证书对通信进行加密，并开启即与CA根证书签名的双向认证
3.1.建立证书目录和数据目录
mkdir -p /etc/ssl/etcd/ssl/ mkdir -p /var/lib/etcd |
3.2.创建根 CA 证书
可以通过 openssl 和 cfssl 来创建证书，这里通过 openssl 的方式创建
cd /etc/ssl/etcd/ssl openssl genrsa -out ca.key 2048 openssl req -x509 -new -nodes -key ca.key -subj "/CN=apisix" -days 3650 -out ca.crt |
3.3.创建 ETCD 证书和私钥
1、创建配置文件
cat > ./etcd-ca.conf <<EOF [ req ] default_bits = 2048 prompt = no default_md = rsa req_extensions = req_ext distinguished_name = dn [ dn ] C = CN ST = GuangDong L = DongGuan O = etcd OU = LingFang CN = etcd [ req_ext ] subjectAltName = @alt_names [ alt_names ] DNS.1 = localhost DNS.2 = etcd1 DNS.3 = etcd2 DNS.4 = etcd3 IP.1 = 127.0.0.1 IP.2 = 192.168.112.98 IP.3 = 192.168.112.99 IP.4 = 192.168.112.100 [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=keyEncipherment,dataEncipherment extendedKeyUsage=serverAuth,clientAuth subjectAltName=@alt_names EOF |
2、生成密钥
openssl genrsa -out etcd.key 2048 |
3、生成证书签发请求(certificate signing request)
openssl req -new -key etcd.key -out etcd.csr -config etcd-ca.conf |
4、生成证书
openssl x509 -req -in etcd.csr -CA /etc/ssl/etcd/ca.crt -CAkey /etc/ssl/etcd/ca.key -CAcreateserial -out etcd.crt -days 3650 -extensions v3_ext -extfile etcd-ca.conf |
5、验证证书
openssl verify -CAfile ca.crt etcd.crt openssl x509 -text -noout -in /etc/ssl/etcd/ssl/etcd.crt |
4.安装 ETCD
4.1.安装证书
将CA证书ca.crt，etcd证书etcd.crt和秘钥etcd.key，拷贝到各节点的/etc/ssl/etcd/ssl/目录中
4.2.下载并安装 ETCD
将其中的 etcd 和 etcdctl 两个可执行文件复制到各节点的/usr/local/bin目录
tar zxvf etcd-v3.5.1-linux-amd64.tar.gz cp ./etcd-v3.5.1-linux-amd64/{etcd,etcdctl} /usr/local/bin/ |
4.3.以 systemd 形式管理 etcd 服务
1、创建 ETCD 配置文件
mkdir -p /etc/etcd cat > /etc/etcd/etcd.conf <<EOF # [Member Flags] # ETCD_ELECTION_TIMEOUT=1000 # ETCD_HEARTBEAT_INTERVAL=100 # 指定etcd的数据目录 ETCD_NAME=etcd1 ETCD_DATA_DIR=/var/lib/etcd/ # [Cluster Flags] # ETCD_AUTO_COMPACTION_RETENTIO:N=0 ETCD_INITIAL_CLUSTER_STATE=new ETCD_ADVERTISE_CLIENT_URLS=https://192.168.112.98:2379 ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.112.98:2380 ETCD_LISTEN_CLIENT_URLS=https://192.168.112.98:2379,https://127.0.0.1:2379 ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster ETCD_LISTEN_PEER_URLS=https://192.168.112.98:2380 ETCD_INITIAL_CLUSTER=etcd1=https://192.168.112.98:2380,etcd2=https://192.168.112.99:2380,etcd3=https://192.168.112.100:2380 # [Proxy Flags] ETCD_PROXY=off # [Security flags] # ETCD_CLIENT_CERT_AUTH= # ETCD_PEER_CLIENT_CERT_AUTH= # 指定etcd的公钥证书和私钥 ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.crt ETCD_CERT_FILE=/etc/ssl/etcd/ssl/etcd.crt ETCD_KEY_FILE=/etc/ssl/etcd/ssl/etcd.key # 指定etcd的Peers通信的公钥证书和私钥 ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.crt ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/etcd.crt ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/etcd.key #ETCD_PEER_CLIENT_CERT_AUTH=true #ETCD_CLIENT_CERT_AUTH=true # [Profiling flags] # ETCD_METRICS={{ etcd_metrics }} EOF |
2、创建 systemd 服务启用文件
cat > /usr/lib/systemd/system/etcd.service <<EOF [Unit] Description=etcd server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify WorkingDirectory=/var/lib/etcd/ EnvironmentFile=-/etc/etcd/etcd.conf ExecStart=/usr/local/bin/etcd NotifyAccess=all Restart=always RestartSec=5s LimitNOFILE=40000 [Install] WantedBy=multi-user.target EOF |
其中 ETCD2 和 ETCD3 主机配置文件中的内容需要对应修改
4.4.启动 ETCD
只有一个节点启动etcd服务后，etcd会去访问其他两个节点的2380端口，如果一定时间内无法访问，则会服务启动失败。所以应当迅速启动至少两个节点，服务才会成功启动。
systemctl daemon-reload systemctl enable etcd systemctl start etcd systemctl status etcd |
4.5.检查集群是否健康
etcdctl --cacert=/etc/ssl/etcd/ssl/ca.crt --cert=/etc/ssl/etcd/ssl/etcd.crt \ --key=/etc/ssl/etcd/ssl/etcd.key \ --endpoints="https://192.168.112.98:2379,https://192.168.112.99:2379,https://192.168.112.100:2379" \ endpoint health |
curl --cacert /etc/ssl/etcd/ssl/ca.crt -i https://192.168.112.98:2379/version \ --cert /etc/ssl/etcd/ssl/etcd.crt --key /etc/ssl/etcd/ssl/etcd.key |
etcdctl --cacert=/etc/ssl/etcd/ssl/ca.crt --cert=/etc/ssl/etcd/ssl/etcd.crt \ --key=/etc/ssl/etcd/ssl/etcd.key --endpoints="https://192.168.112.99:2379" \ get / --prefix --keys-only openssl s_client -connect 192.168.112.98:2379 -key /etc/ssl/etcd/etcd.key \ -cert /etc/ssl/etcd/etcd.crt -CAfile /etc/ssl/etcd/ssl/ca.crt |
4.6.ETCD 参考指南
https://jimmysong.io/kubernetes-handbook/practice/etcd-cluster-installation.html
5.APISix 配置
主要涉及配置文件和 ETCD 连接部分的内容
5.1.config.yaml
apisix: admin_key: - name: admin key: 3cea9e6d29e70e98c09d29ca98bd452a43d057bb3e6a748f3d835492 role: admin port_admin: 9180 node_listen: - 80 allow_admin: - 127.0.0.0/24 ssl: enable: true listen: - 443 ssl_trusted_certificate: /etc/ssl/etcd/ssl/ca.crt #ssl_protocols: SSLv3 TLSv1.2 TLSv1.3 etcd: host: - "https://127.0.0.1:2379" - "https://192.168.112.98:2379" - "https://192.168.112.99:2379" - "https://192.168.112.100:2379" prefix: /apisix timeout: 30 resync_delay: 5 health_check_timeout: 10 tls: cert: /etc/ssl/etcd/ssl/etcd.crt key: /etc/ssl/etcd/ssl/etcd.key verify: true |
5.2.conf.yaml
conf: listen: # host: 127.0.0.1 port: 9000 # ssl: # host: 127.0.0.1 # the address on which the `Manager API` should listen for HTTPS. # The default value is 0.0.0.0, if want to specify, please enable it. # port: 9001 # The port on which the `Manager API` should listen for HTTPS. # cert: "/tmp/cert/example.crt" # Path of your SSL cert. # key: "/tmp/cert/example.key" # Path of your SSL key. allow_list: # If we don't set any IP list, then any IP access is allowed by default. - 127.0.0.1 # The rules are checked in sequence until the first match is found. - ::1 - 0.0.0.0/0 etcd: endpoints: - 127.0.0.1:2379 - 192.168.112.98:2379 - 192.168.112.99:2379 - 192.168.112.100:2379 mtls: key_file: "/etc/ssl/etcd/ssl/etcd.key" # Path of your self-signed client side key cert_file: "/etc/ssl/etcd/ssl/etcd.crt" # Path of your self-signed client side cert ca_file: "/etc/ssl/etcd/ssl/ca.crt" # Path of your self-signed ca cert, the CA is used to sign callers' certificates # prefix: /apisix # apisix config's prefix in etcd, /apisix by default log: error_log: level: warn # supports levels, lower to higher: debug, info, warn, error, panic, fatal file_path: logs/error.log # supports relative path, absolute path, standard output access_log: file_path: logs/access.log # supports relative path, absolute path, standard output max_cpu: 0 authentication: secret: secret # secret for jwt token generation. expire_time: 3600 # jwt token expire time, in second users: # yamllint enable rule:comments-indentation - username: admin # username and password for login `manager api` password: admin - username: user password: user plugins: # plugin list (sorted in alphabetical order) - api-breaker - authz-keycloak - basic-auth - batch-requests - consumer-restriction - cors # - dubbo-proxy - echo # - error-log-logger # - example-plugin - fault-injection - grpc-transcode - hmac-auth - http-logger - ip-restriction - jwt-auth - kafka-logger - key-auth - limit-conn - limit-count - limit-req # - log-rotate # - node-status - openid-connect - prometheus - proxy-cache - proxy-mirror - proxy-rewrite - redirect - referer-restriction - request-id - request-validation - response-rewrite - serverless-post-function - serverless-pre-function # - skywalking - sls-logger - syslog - tcp-logger - udp-logger - uri-blocker - wolf-rbac - zipkin - server-info - traffic-split |
6.防火墙配置
受 ApiSix 在匹配 ETCD 群集 mtls 的限制，ETCD 群集不采用证书配置模式，考虑安全通过系统自带的防火墙功能实现对源 IP 地址对目标端口的访问控制。
systemctl start firewalld firewall-cmd --add-port=80/tcp --permanent firewall-cmd --add-port=443/tcp --permanent firewall-cmd --add-port=9000/tcp --permanent firewall-cmd --add-port=22/tcp --permanent firewall-cmd --list-rich-rules firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.112.100" port port="2379" protocol=tcp accept' firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.112.100" port port="2380" protocol=tcp accept' firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.112.98" port port="2379" protocol=tcp accept' firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.112.98" port port="2380" protocol=tcp accept' |
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK