26

统信USO 20 hostPort 无法访问

 3 years ago
source link: https://zhangguanzhang.github.io/2020/10/30/uos20-nftables/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

我自己环境和客户的环境都遇到了无法访问，我自己测试的机器信息是:

$ cat /etc/os-release
PRETTY_NAME="Uniontech OS Server 20 Enterprise"
NAME="Uniontech OS Server 20 Enterprise"
VERSION_ID="20"
VERSION="20"
ID=UOS
HOME_URL="https://www.chinauos.com/"
BUG_REPORT_URL="http://bbs.chinauos.com"
VERSION_CODENAME=fou
$ uname -a
Linux xxx-PC 4.19.0-arm64-server #1760 SMP Tue Jun 30 19:51:30 CST 2020 aarch64 GNU/Linux

客户信息是:

$ cat /etc/os-release
PRETTY_NAME="uos 20 SP1"
NAME="uos"
VERSION_ID="20 SP1"
VERSION="20 SP1"
ID=uos
HOME_URL="https://www.chinauos.com/"
BUG_REPORT_URL="http://bbs.chinauos.com"
$ uname -a
Linux kunpeng-PC 4.19.0-arm64-server #1707 SMP Thu Mar 26 17:43:52 CST 2020 aarch64 GNU/Linux

k8s 版本信息:

$ kubectl version -o json
{
  "clientVersion": {
    "major": "1",
    "minor": "15",
    "gitVersion": "v1.15.12",
    "gitCommit": "e2a822d9f3c2fdb5c9bfbe64313cf9f657f0a725",
    "gitTreeState": "clean",
    "buildDate": "2020-05-06T05:17:59Z",
    "goVersion": "go1.12.17",
    "compiler": "gc",
    "platform": "linux/arm64"
  },
  "serverVersion": {
    "major": "1",
    "minor": "15",
    "gitVersion": "v1.15.12",
    "gitCommit": "e2a822d9f3c2fdb5c9bfbe64313cf9f657f0a725",
    "gitTreeState": "clean",
    "buildDate": "2020-05-06T05:09:48Z",
    "goVersion": "go1.12.17",
    "compiler": "gc",
    "platform": "linux/arm64"
  }
}

理论上 v1.15 以后的大版本号都不会有这种问题

问题现象和解决手段

问题现象

我们业务入口是一个用的 hostPort 的 nginx ，部署好后无法访问，在宿主机上 curl 也会无法访问，同时 iptables 的条目会异常（下面的-m mark各个系统的先后可能顺序不同）: 

$ iptables -S
...
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
.....
$ iptables -S | grep 'KUBE-FIREWALL' | wc -l
11054

实际上这个问题就是因为统信系统基于 ubuntu 改的，很多新发行版系统包括centos8都是开始使用 nf_tables 作为rule规则管理，默认的 iptables 是 nf_tables ，低版本 kube-proxy 对它的兼容性不好，需要我们切换下 iptables 到老版本。

防止相关进程更新iptables，每台机器都得这样操作，先停掉相关进程

systemctl stop docker kubelet kube-proxy

清空iptables

iptables -F
iptables -t nat -F
iptables -t mangle -F
iptables -X
kube-proxy --cleanup

切换到老的iptables，apt系列的 update-alternatives 是系统自带的软连接管理，下面是把老的itables做成系统PATH的软连 

update-alternatives --set iptables /usr/sbin/iptables-legacy

然后启动相关的，恢复正常

systemctl start docker kubelet kube-proxy

report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK