日期：2024.12.07
前言：配置pxe启动环境时跨网段的tftp服务器访问不到，记录下排查过程及解决方案。
参照：
- https://unix.stackexchange.com/questions/579508/iptables-rules-to-forward-tftp-via-nat
- https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-4.7.y&id=3bb398d925ec73e42b778cf823c8f4aecae359ea
- https://home.regit.org/netfilter-en/secure-use-of-helpers/
- https://wiki.nftables.org/wiki-nftables/index.php/Conntrack_helpers
先看下目前tftp-client和tftp-server是不是互通的，然后再tftp-client上每隔10秒ping一次tftp-server，由于待会测试tftp-clinet的ssh连接会断掉，所以用nohup运行。
[root@tftp-client ~]# man ping
NAMEping - send ICMP ECHO_REQUEST to network hosts-c countStop after sending count ECHO_REQUEST packets. With deadline option, ping waits for count ECHO_REPLY packets, untilthe timeout expires.-i intervalWait interval seconds between sending each packet. Real number allowed with dot as a decimal separator (regardlesslocale setup). The default is to wait for one second between each packet normally, or not to wait in flood mode. Onlysuper-user may set interval to values less than 0.2 seconds.[root@tftp-client ~]# ping -c 2 192.168.5.253
PING 192.168.5.253 (192.168.5.253) 56(84) bytes of data.
64 bytes from 192.168.5.253: icmp_seq=1 ttl=63 time=0.345 ms
64 bytes from 192.168.5.253: icmp_seq=2 ttl=63 time=0.349 ms
--- 192.168.5.253 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1060ms
rtt min/avg/max/mdev = 0.345/0.347/0.349/0.002 ms[root@tftp-client ~]# nohup ping -i 10 192.168.5.253 &
[1] 7670[root@tftp-client ~]# ps aux | grep ping
root 7670 0.0 0.2 7352 2432 pts/0 S 23:57 0:00 ping -i 10 192.168.5.253
root 7672 0.0 0.2 6408 2176 pts/0 S+ 23:57 0:00 grep --color=auto ping
在router-nft上确认下防火墙是开启的，规则里有SNAT的转发规则
[root@router-nft ~]# systemctl status nftables.service
● nftables.service - Netfilter TablesLoaded: loaded (/usr/lib/systemd/system/nftables.service; enabled; vendor preset: disabled)Active: active (exited) since Mon 2024-12-09 23:53:16 CST; 5min agoDocs: man:nft(8)Process: 17617 ExecStop=/sbin/nft flush ruleset (code=exited, status=0/SUCCESS)Process: 2208 ExecReload=/sbin/nft flush ruleset; include "/etc/sysconfig/nftables.conf"; (code=exited, status=0/SUCCESS)Process: 17618 ExecStart=/sbin/nft -f /etc/sysconfig/nftables.conf (code=exited, status=0/SUCCESS)Main PID: 17618 (code=exited, status=0/SUCCESS)Tasks: 0 (limit: 5904)Memory: 0BCGroup: /system.slice/nftables.serviceDec 09 23:53:16 router-nft systemd[1]: nftables.service: Succeeded.
Dec 09 23:53:16 router-nft systemd[1]: Stopped Netfilter Tables.
Dec 09 23:53:16 router-nft systemd[1]: Starting Netfilter Tables...
Dec 09 23:53:16 router-nft systemd[1]: Started Netfilter Tables.[root@router-nft ~]# nft list chain inet NFT_NAT NFC_POST
table inet NFT_NAT {chain NFC_POST {type nat hook postrouting priority srcnat; policy accept;iifname "enp2s0" oifname "enp1s0" ip saddr 172.31.0.0/24 masqueradeiifname "enp3s0" oifname "enp1s0" ip saddr 10.31.0.0/24 masquerade}
}
在router-nft的enp3s0和enp1s0两张网卡上监听从tftp-client发送给tftp-server的icmp协议包
可以看到snat生效，由enp3s0网卡进来的源地址为10.31.0.1的icmp包源地址被转换为192.168.5.254
[root@router-nft ~]# tcpdump dst host 192.168.5.253 -i enp3s0 and icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:19:59.305372 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 135, length 64
00:20:09.545385 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 136, length 64[root@router-nft ~]# tcpdump dst host 192.168.5.253 -i enp1s0 and icmp -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:19:59.305392 IP 192.168.5.254 > 192.168.5.253: ICMP echo request, id 30, seq 135, length 64
00:20:09.545407 IP 192.168.5.254 > 192.168.5.253: ICMP echo request, id 30, seq 136, length 64
tftp-server收到源地址为192.168.5.254的icmp包后发起回应，回应包的目标地址为192.168.5.254，router-nft主机再将enp1s0网卡收到的目标地址为192.168.5.254的icmp包转换目标地址为10.31.0.1后通过enp3s网卡发送给tftp-client，最后tftp-client收到源地址是tftp-server目标地址为自身的icmp回应包。这就是SNAT（Source Network Address）源地址转换的工作过程。
[root@tftp-server ~]# tcpdump -i WANbridge icmp -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on WANbridge, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:39:36.905506 IP 192.168.5.254 > 192.168.5.253: ICMP echo request, id 30, seq 250, length 64
00:39:36.905529 IP 192.168.5.253 > 192.168.5.254: ICMP echo reply, id 30, seq 250, length 64[root@router-nft ~]# tcpdump -i enp1s0 icmp -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:39:36.905387 IP 192.168.5.254 > 192.168.5.253: ICMP echo request, id 30, seq 250, length 64
00:39:36.905461 IP 192.168.5.253 > 192.168.5.254: ICMP echo reply, id 30, seq 250, length 64[root@router-nft ~]# tcpdump -i enp3s0 icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:39:36.905366 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 250, length 64
00:39:36.905466 IP 192.168.5.253 > 10.31.0.1: ICMP echo reply, id 30, seq 250, length 64[root@tftp-client ~]# tcpdump -i enp1s0 icmp -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:39:36.905051 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 250, length 64
00:39:36.905345 IP 192.168.5.253 > 10.31.0.1: ICMP echo reply, id 30, seq 250, length 64
现在关闭router-nft上的防火墙，相关的snat也会停止工作，然后再观察下tftp-client ping ftfp-server的icmp包
[root@router-nft ~]# systemctl stop nftables.service
[root@router-nft ~]# systemctl status nftables.service
● nftables.service - Netfilter TablesLoaded: loaded (/usr/lib/systemd/system/nftables.service; enabled; vendor preset: disabled)Active: inactive (dead) since Tue 2024-12-10 00:51:30 CST; 9s agoDocs: man:nft(8)Process: 972 ExecStop=/sbin/nft flush ruleset (code=exited, status=0/SUCCESS)Process: 728 ExecStart=/sbin/nft -f /etc/sysconfig/nftables.conf (code=exited, status=0/SUCCESS)Main PID: 728 (code=exited, status=0/SUCCESS)Dec 10 00:47:56 router-nft systemd[1]: Starting Netfilter Tables...
Dec 10 00:47:56 router-nft systemd[1]: Started Netfilter Tables.
Dec 10 00:51:30 router-nft systemd[1]: Stopping Netfilter Tables...
Dec 10 00:51:30 router-nft systemd[1]: nftables.service: Succeeded.
Dec 10 00:51:30 router-nft systemd[1]: Stopped Netfilter Tables.
可以看到tftp-server收到了来自10.31.0.1的icmp包，但是回应给10.31.0.1的icmp包却没有原路返回再经过router-nft
[root@tftp-server ~]# tcpdump -i WANbridge icmp -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on WANbridge, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:54:46.217559 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 339, length 64
00:54:46.217588 IP 192.168.5.253 > 10.31.0.1: ICMP echo reply, id 30, seq 339, length 64
00:54:56.457590 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 340, length 64
00:54:56.457626 IP 192.168.5.253 > 10.31.0.1: ICMP echo reply, id 30, seq 340, length 64[root@router-nft ~]# tcpdump -i enp1s0 icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:54:46.217725 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 339, length 64
00:54:56.457429 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 340, length 64[root@router-nft ~]# tcpdump -i enp3s0 icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:54:46.217712 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 339, length 64
00:54:56.457413 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 340, length 64
检查tftp-server的路由表可知，tftp-server只有一条默认路由，非同网段（192.168.4.0/23）的数据包都会走默认路由发送给192.168.4.1，并不知道发送给10.31.0.1的数据包需要先发给192.168.5.254，所以10.31.0.1的数据包也发送给了默认路由192.168.4.1，没有发送给192.168.5.254，因此192.168.5.254并没有收到来自tftp-server的返回包。
[root@tftp-server ~]# ip route show
default via 192.168.4.1 dev WANbridge proto static metric 428
192.168.4.0/23 dev WANbridge proto kernel scope link src 192.168.5.253 metric 428
给tftp-server新增一条路由，发往10.31.0.0/24网段的数据包要先发送给192.168.5.254
[root@tftp-server ~]# nmcli connection modify WANbridge +ipv4.routes "10.31.0.0/24 192.168.5.254"
[root@tftp-server ~]# nmcli connection up WANbridge
Connection successfully activated (master waiting for slaves) (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/16)
[root@tftp-server ~]# ip route show
default via 192.168.4.1 dev WANbridge proto static metric 428
10.31.0.0/24 via 192.168.5.254 dev WANbridge proto static metric 428
192.168.4.0/23 dev WANbridge proto kernel scope link src 192.168.5.253 metric 428
再观察icmp包在各个主机直接的发送和接受过程，可见tftp-server（192.168.5.253）接收到了tftp-client（10.31.0.1）发送的icmp包后，回应的icmp包成功通过router-nft原路发送回去最终被tftp-client收到。
[root@tftp-server ~]# tcpdump -i WANbridge icmp -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on WANbridge, link-type EN10MB (Ethernet), snapshot length 262144 bytes
01:26:48.777618 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 528, length 64
01:26:48.777648 IP 192.168.5.253 > 10.31.0.1: ICMP echo reply, id 30, seq 528, length 64[root@router-nft ~]# tcpdump -i enp1s0 icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
01:26:48.777372 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 528, length 64
01:26:48.777460 IP 192.168.5.253 > 10.31.0.1: ICMP echo reply, id 30, seq 528, length 64[root@router-nft ~]# tcpdump -i enp3s0 icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
01:26:48.777359 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 528, length 64
01:26:48.777465 IP 192.168.5.253 > 10.31.0.1: ICMP echo reply, id 30, seq 528, length 64[root@tftp-client ~]# tcpdump -i enp1s0 icmp -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
01:26:48.777062 IP 10.31.0.1 > 192.168.5.253: ICMP echo request, id 30, seq 528, length 64
01:26:48.777448 IP 192.168.5.253 > 10.31.0.1: ICMP echo reply, id 30, seq 528, length 64
可见通过静态路由和防火墙nftables的SNAT功能这两种方式都能实现tftp-client和tftp-server两台主机的icmp包互通。现在来看下tftp协议，先确保tftp-server上的tftp服务开启。
[root@tftp-server ~]# systemctl status tftp.socket
● tftp.socket - Tftp Server Activation SocketLoaded: loaded (/usr/lib/systemd/system/tftp.socket; enabled; preset: disabled)Active: active (listening) since Mon 2024-12-02 15:36:00 CST; 1 week 0 days agoUntil: Mon 2024-12-02 15:36:00 CST; 1 week 0 days agoTriggers: ● tftp.serviceListen: [::]:69 (Datagram)Tasks: 0 (limit: 101263)Memory: 8.0KCPU: 410usCGroup: /system.slice/tftp.socketDec 02 15:36:00 tftp-server systemd[1]: Listening on Tftp Server Activation Socket.[root@tftp-server ~]# ss -upln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 127.0.0.1:323 0.0.0.0:* users:(("chronyd",pid=790,fd=5))
UNCONN 0 0 *:69 *:* users:(("systemd",pid=1,fd=55))
UNCONN 0 0 [::1]:323 [::]:* users:(("chronyd",pid=790,fd=6)) [root@tftp-server ~]# lsof -i:69
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 55u IPv6 22816 0t0 UDP *:tftp
在tftp默认的根目录里创建一个文件用于测试，大小1K，先本地get一下确认能够获取文件。可以看到69端口原本由systemd监听，发起连接后，in.tftpd启动提供服务。
[root@tftp-server ~]# dd if=/dev/zero of=/var/lib/tftpboot/666 bs=1K count=1
1+0 records in
1+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 9.1442e-05 s, 11.2 MB/s[root@tftp-server ~]# ll /var/lib/tftpboot/666
-rw-r--r-- 1 root root 1024 Dec 11 11:59 /var/lib/tftpboot/666[root@tftp-server ~]# ss -upln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 127.0.0.1:323 0.0.0.0:* users:(("chronyd",pid=790,fd=5))
UNCONN 0 0 *:69 *:* users:(("systemd",pid=1,fd=97))
UNCONN 0 0 [::1]:323 [::]:* users:(("chronyd",pid=790,fd=6)) [root@tftp-server ~]# lsof -i:69
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 97u IPv6 287681 0t0 UDP *:tftp [root@tftp-server ~]# tftp 192.168.5.253
tftp> get 666
tftp> quit[root@tftp-server ~]# ll 666
-rw-r--r-- 1 root root 1024 Dec 11 12:00 666[root@tftp-server ~]# ss -upln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 127.0.0.1:323 0.0.0.0:* users:(("chronyd",pid=790,fd=5))
UNCONN 0 0 *:69 *:* users:(("in.tftpd",pid=53921,fd=0),("systemd",pid=1,fd=97))
UNCONN 0 0 [::1]:323 [::]:* users:(("chronyd",pid=790,fd=6)) [root@tftp-server ~]# lsof -i:69
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 97u IPv6 287681 0t0 UDP *:tftp
in.tftpd 53921 root 0u IPv6 287681 0t0 UDP *:tftp
先观察一下tftp-client目前监听的端口，再去连接192.168.5.253，连上后新开一个终端再查看监听端口，可以看到tftp-client随机选取的用于同tftp-server通信的50789端口打开了
[root@tftp-client ~]# ss -upln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 0.0.0.0:67 0.0.0.0:* users:(("dhcpd",pid=1034,fd=7))
UNCONN 0 0 0.0.0.0:123 0.0.0.0:* users:(("chronyd",pid=654,fd=7))
UNCONN 0 0 127.0.0.1:323 0.0.0.0:* users:(("chronyd",pid=654,fd=5))
UNCONN 0 0 [::1]:323 [::]:* users:(("chronyd",pid=654,fd=6)) [root@tftp-client ~]# tftp 192.168.5.253
tftp> [root@tftp-client ~]# ss -upln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 0.0.0.0:67 0.0.0.0:* users:(("dhcpd",pid=1034,fd=7))
UNCONN 0 0 0.0.0.0:50789 0.0.0.0:* users:(("tftp",pid=8816,fd=3))
UNCONN 0 0 0.0.0.0:123 0.0.0.0:* users:(("chronyd",pid=654,fd=7))
UNCONN 0 0 127.0.0.1:323 0.0.0.0:* users:(("chronyd",pid=654,fd=5))
UNCONN 0 0 [::1]:323 [::]:* users:(("chronyd",pid=654,fd=6))
在三台主机的四张网卡上监听69和50789两个端口，然后get 666文件，观察整个通信过程:
tfpt-client请求文件：tfpt-client---->router-nft enp3s0---->router-nft enp1s0---->tftp-server
tftp-server发送文件：tftp-server---->router-nft enp1s0---->router-nft enp3s0---->tfpt-client
可以看到1KB的666文件被分成两个包发送过去
[root@tftp-client ~]# tftp 192.168.5.253
tftp> get 666[root@tftp-client ~]# tcpdump -i enp1s0 -n port 69 or 50789
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:03:23.280456 IP 10.31.0.1.50789 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
12:03:23.282303 IP 192.168.5.253.47430 > 10.31.0.1.50789: UDP, length 516
12:03:23.282340 IP 10.31.0.1.50789 > 192.168.5.253.47430: UDP, length 4
12:03:23.282507 IP 192.168.5.253.47430 > 10.31.0.1.50789: UDP, length 516
12:03:23.282524 IP 10.31.0.1.50789 > 192.168.5.253.47430: UDP, length 4
12:03:23.282614 IP 192.168.5.253.47430 > 10.31.0.1.50789: UDP, length 4
12:03:23.282641 IP 10.31.0.1.50789 > 192.168.5.253.47430: UDP, length 4[root@router-nft ~]# tcpdump -i enp3s0 -n port 69 or 50789
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:03:23.280575 IP 10.31.0.1.50789 > 192.168.5.253.tftp: 15 RRQ "666" netascii
12:03:23.282257 IP 192.168.5.253.47430 > 10.31.0.1.50789: UDP, length 516
12:03:23.282440 IP 10.31.0.1.50789 > 192.168.5.253.47430: UDP, length 4
12:03:23.282492 IP 192.168.5.253.47430 > 10.31.0.1.50789: UDP, length 516
12:03:23.282565 IP 10.31.0.1.50789 > 192.168.5.253.47430: UDP, length 4
12:03:23.282601 IP 192.168.5.253.47430 > 10.31.0.1.50789: UDP, length 4
12:03:23.282685 IP 10.31.0.1.50789 > 192.168.5.253.47430: UDP, length 4[root@router-nft ~]# tcpdump -i enp1s0 -n port 69 or 50789
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:03:23.280587 IP 10.31.0.1.50789 > 192.168.5.253.tftp: 15 RRQ "666" netascii
12:03:23.282249 IP 192.168.5.253.47430 > 10.31.0.1.50789: UDP, length 516
12:03:23.282446 IP 10.31.0.1.50789 > 192.168.5.253.47430: UDP, length 4
12:03:23.282488 IP 192.168.5.253.47430 > 10.31.0.1.50789: UDP, length 516
12:03:23.282569 IP 10.31.0.1.50789 > 192.168.5.253.47430: UDP, length 4
12:03:23.282599 IP 192.168.5.253.47430 > 10.31.0.1.50789: UDP, length 4
12:03:23.282694 IP 10.31.0.1.50789 > 192.168.5.253.47430: UDP, length 4[root@tftp-server ~]# tcpdump -i WANbridge -n port 69 or 50789
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on WANbridge, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:03:23.280632 IP 10.31.0.1.50789 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
12:03:23.282197 IP 192.168.5.253.47430 > 10.31.0.1.50789: UDP, length 516
12:03:23.282464 IP 10.31.0.1.50789 > 192.168.5.253.47430: UDP, length 4
12:03:23.282489 IP 192.168.5.253.47430 > 10.31.0.1.50789: UDP, length 516
12:03:23.282586 IP 10.31.0.1.50789 > 192.168.5.253.47430: UDP, length 4
12:03:23.282600 IP 192.168.5.253.47430 > 10.31.0.1.50789: UDP, length 4
12:03:23.282710 IP 10.31.0.1.50789 > 192.168.5.253.47430: UDP, length 4[root@tftp-client ~]# ll 666
-rw-r--r-- 1 root root 1024 Dec 11 12:03 666
由此可见，通过路由方式进行tftp传输是可行的，下面打开防火墙使用SNAT功能尝试下
[root@router-nft ~]# systemctl status nftables.service
● nftables.service - Netfilter TablesLoaded: loaded (/usr/lib/systemd/system/nftables.service; enabled; vendor preset: disa>Active: inactive (dead) since Tue 2024-12-10 01:57:44 CST; 1 day 10h agoDocs: man:nft(8)Process: 1139 ExecStop=/sbin/nft flush ruleset (code=exited, status=0/SUCCESS)Process: 1079 ExecStart=/sbin/nft -f /etc/sysconfig/nftables.conf (code=exited, status=>Main PID: 1079 (code=exited, status=0/SUCCESS)Dec 10 01:45:40 router-nft systemd[1]: Starting Netfilter Tables...
Dec 10 01:45:40 router-nft systemd[1]: Started Netfilter Tables.
Dec 10 01:57:44 router-nft systemd[1]: Stopping Netfilter Tables...
Dec 10 01:57:44 router-nft systemd[1]: nftables.service: Succeeded.
Dec 10 01:57:44 router-nft systemd[1]: Stopped Netfilter Tables.[root@router-nft ~]# systemctl start nftables.service
[root@router-nft ~]# systemctl status nftables.service
● nftables.service - Netfilter TablesLoaded: loaded (/usr/lib/systemd/system/nftables.service; enabled; vendor preset: disa>Active: active (exited) since Wed 2024-12-11 12:27:36 CST; 1s agoDocs: man:nft(8)Process: 1139 ExecStop=/sbin/nft flush ruleset (code=exited, status=0/SUCCESS)Process: 4471 ExecStart=/sbin/nft -f /etc/sysconfig/nftables.conf (code=exited, status=>Main PID: 4471 (code=exited, status=0/SUCCESS)Dec 11 12:27:36 router-nft systemd[1]: Starting Netfilter Tables...
Dec 11 12:27:36 router-nft systemd[1]: Started Netfilter Tables.
tftp-client连接tftp-server，现在tftp-client的随机通信端口为57964
[root@tftp-client ~]# tftp 192.168.5.253
tftp>
[root@tftp-client ~]# ss -upln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 0.0.0.0:67 0.0.0.0:* users:(("dhcpd",pid=1034,fd=7))
UNCONN 0 0 0.0.0.0:57964 0.0.0.0:* users:(("tftp",pid=8913,fd=3))
UNCONN 0 0 0.0.0.0:123 0.0.0.0:* users:(("chronyd",pid=654,fd=7))
UNCONN 0 0 127.0.0.1:323 0.0.0.0:* users:(("chronyd",pid=654,fd=5))
UNCONN 0 0 [::1]:323 [::]:* users:(("chronyd",pid=654,fd=6))
tftp-client获取666文件，观察通讯过程,tftp-client提示Transfer timed out，传输超时，观察抓包结果可知：
当tftp-client发起获取文件请求到tftp-server时，tfpt-client---->router-nft enp3s0---->router-nft enp1s0---->tftp-server，防火墙nftables的SNAT工作正常
当tftp-server向tftp-client发送文件时，tftp-server---->router-nft enp1s0---->没有进行SNAT的转换,后续router-nft enp3s0---->tfpt-client自然收不到回应的包
[root@tftp-client ~]# tftp 192.168.5.253
tftp> get 666
Transfer timed out.tftp> [root@tftp-client ~]# tcpdump -i enp1s0 -n port 69 or 57964
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:32:26.522134 IP 10.31.0.1.57964 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
12:32:31.522290 IP 10.31.0.1.57964 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
12:32:36.522445 IP 10.31.0.1.57964 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
12:32:41.522601 IP 10.31.0.1.57964 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
12:32:46.522742 IP 10.31.0.1.57964 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
^C
5 packets captured
7 packets received by filter
0 packets dropped by kernel[root@router-nft ~]# tcpdump -i enp3s0 -n port 69 or 57964
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:32:26.522285 IP 10.31.0.1.57964 > 192.168.5.253.tftp: 15 RRQ "666" netascii
12:32:31.522428 IP 10.31.0.1.57964 > 192.168.5.253.tftp: 15 RRQ "666" netascii
12:32:36.522588 IP 10.31.0.1.57964 > 192.168.5.253.tftp: 15 RRQ "666" netascii
12:32:41.522769 IP 10.31.0.1.57964 > 192.168.5.253.tftp: 15 RRQ "666" netascii
12:32:46.522882 IP 10.31.0.1.57964 > 192.168.5.253.tftp: 15 RRQ "666" netascii
^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel[root@router-nft ~]# tcpdump -i enp1s0 -n port 69 or 57964
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:32:26.522303 IP 192.168.5.254.57964 > 192.168.5.253.tftp: 15 RRQ "666" netascii
12:32:26.535244 IP 192.168.5.253.41194 > 192.168.5.254.57964: UDP, length 516
12:32:27.536336 IP 192.168.5.253.41194 > 192.168.5.254.57964: UDP, length 516
12:32:29.538424 IP 192.168.5.253.41194 > 192.168.5.254.57964: UDP, length 516
12:32:31.522449 IP 192.168.5.254.57964 > 192.168.5.253.tftp: 15 RRQ "666" netascii
12:32:31.524122 IP 192.168.5.253.56444 > 192.168.5.254.57964: UDP, length 516
12:32:32.525254 IP 192.168.5.253.56444 > 192.168.5.254.57964: UDP, length 516
12:32:33.542545 IP 192.168.5.253.41194 > 192.168.5.254.57964: UDP, length 516
12:32:34.527306 IP 192.168.5.253.56444 > 192.168.5.254.57964: UDP, length 516
12:32:36.522607 IP 192.168.5.254.57964 > 192.168.5.253.tftp: 15 RRQ "666" netascii
12:32:36.524280 IP 192.168.5.253.reachout > 192.168.5.254.57964: UDP, length 516
12:32:37.524770 IP 192.168.5.253.reachout > 192.168.5.254.57964: UDP, length 516
12:32:38.530618 IP 192.168.5.253.56444 > 192.168.5.254.57964: UDP, length 516
12:32:39.526872 IP 192.168.5.253.reachout > 192.168.5.254.57964: UDP, length 516
... ...[root@tftp-server ~]# tcpdump -i WANbridge -n port 69 or 57964
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on WANbridge, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:32:26.522348 IP 192.168.5.254.57964 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
12:32:26.535146 IP 192.168.5.253.41194 > 192.168.5.254.57964: UDP, length 516
12:32:27.536268 IP 192.168.5.253.41194 > 192.168.5.254.57964: UDP, length 516
12:32:29.538387 IP 192.168.5.253.41194 > 192.168.5.254.57964: UDP, length 516
12:32:31.522493 IP 192.168.5.254.57964 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
12:32:31.524067 IP 192.168.5.253.56444 > 192.168.5.254.57964: UDP, length 516
12:32:32.525185 IP 192.168.5.253.56444 > 192.168.5.254.57964: UDP, length 516
12:32:33.542507 IP 192.168.5.253.41194 > 192.168.5.254.57964: UDP, length 516
12:32:34.527247 IP 192.168.5.253.56444 > 192.168.5.254.57964: UDP, length 516
12:32:36.522655 IP 192.168.5.254.57964 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
12:32:36.524221 IP 192.168.5.253.reachout > 192.168.5.254.57964: UDP, length 516
12:32:37.524695 IP 192.168.5.253.reachout > 192.168.5.254.57964: UDP, length 516
12:32:38.530553 IP 192.168.5.253.56444 > 192.168.5.254.57964: UDP, length 516
12:32:39.526814 IP 192.168.5.253.reachout > 192.168.5.254.57964: UDP, length 516
... ...
原因如下，引用下面文档的话：
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-4.7.y&id=3bb398d925ec73e42b778cf823c8f4aecae359ea
netfilter: nf_ct_helper: disable automatic helper assignment
Four years ago we introduced a new sysctl knob to disable automatic helper assignment in 72110dfaa907 ("netfilter: nf_ct_helper: disable automatic helper assignment"). This knob kept this behaviour enabled by default to remain conservative.
四年前我们引入了一个新的sysctl控制开关来关闭 automatic helper assignment，这个开关保持这个功能默认是开启的
This measure was introduced to provide a secure way to configure iptables and connection tracking helpers through explicit rules.
这个方式被引入是为了提供一种安全的途径，通过明确声明相关的规则来配置iptables和connection tracking helpers
Give the time we have waited for this, let's turn off this by default now, worse case users still have a chance to recover the former
behaviour by explicitly enabling this back through sysctl.
过了这么久时间终于等到了现在，让我们默认关闭这个功能，最坏的情况下用户仍有机会通过sysctl声明开启来恢复之前的习惯用法。
总结下就是2016年默认关闭了自动功能的nf_conntrack_helper，不过可以通过sysctl开启，也可以通过声明防火墙相关规则来开启，后者更加安全
那么具体什么是nf_conntrack_helper？引用下面文档的话：
https://home.regit.org/netfilter-en/secure-use-of-helpers/
Principle of helpers
Some protocols use different flows for signaling and data transfers. This is the case for FTP, SIP and H.323 among many others. In the setup stage, it is common that the signaling flow is used to negotiate the configuration parameters for the establishment of the data flow, i.e. the IP address and port that are used to establish the data flow. These sort of protocols are particularly harder to filter by firewalls since they violate layering by introducing OSI layer 3/4 parameters in the OSI layer 7.
有些协议传输信号和传输数据分别使用不同的“流”，例如FTP, SIP and H.323等。在传输刚开始的设定阶段，通常使用信号流协商配置参数来建立数据流，即协商好IP地址和端口号来建立数据流。自从强行将OSI7层模型中的参数引入OSI3/4层中后，这类协议尤其难以被防火墙跟踪。
In order to overcome this situation in the iptables firewall, Netfilter provides the Connection Tracking helpers, which are modules that are able to assist the firewall in tracking these protocols. These helpers create the so-called expectations, as defined by the Netfilter project jargon. An expectation is similar to a connection tracking entry, but it is stored in a separate table and generally with a limited duration. Expectations are used to signal the kernel that in the coming seconds, if a packet with corresponding parameters reaches the firewall, then this packet is RELATED to the previous connection.
为了在iptables防火墙中应对这种情况，Netfilter提供了链接跟踪助手（Connection Tracking helpers），是个可以帮助防火墙跟踪这些协议的模组。链接跟踪助手会创建expectations，一个expectation就类似于一个链接跟踪记录，不过是在另一个表中存放且通常存在时间限制。Expectations用于在数据包刚进入时通知内核，如果一个数据包携带和之前的链接相对应的参数，那么这个包和之前的链接就是RELATED的关系。
These kind of packets can then be authorized thanks to modules like state or conntrack which can match RELATED packets.
这类的包可以通过模组授权成为状态或者链接匹配RELATED的包。
This system relies on parsing of data coming either from the user or the server.It is therefore vulnerable to attack and great care must be taken when using connection tracking helpers.
这套系统依赖于解析数据，既可以是服务器端的数据也可以是用户端的数据。因此很脆弱容易被攻击，使用链接跟踪助手（connection tracking helpers）时要非常小心。
nftables官方维基也有解释：
https://wiki.nftables.org/wiki-nftables/index.php/Conntrack_helpers
Some internet protocols use multiple ports that are negotiated between endpoints during the initial connection. Netfilter's connection tracking system uses protocol helpers that look inside these negotiation packets to determine which ports will be part of the connection. The ct helper tells conntrack to expect packets to these ports; when such packets arrive conntrack assigns them related status.
一些网络协议在初始建立连接时在两端之间协商使用不同的端口，Netfilter's connection tracking system 使用协议助手观察这些协商包内部来决定哪些端口是链接的一部分，链接跟踪助手告知链接跟踪来“期待”这些端口的包，当这些被期待的包到来时链接跟踪将他们划分为related的状态。
我理解就是，conntack helper可以跟踪数据包，将一些在发送和接受来回之间ip和端口相对应的包定义成related状态，由此防火墙就可以通过放行related来放行这些包。之前这个功能是各种协议全自动开启的，现在为了安全考虑，改为要手动指定单独的协议了。由于我没有开启这个功能，所以我的tftp-client的请求包和tftp-server的发送包不是related关系，所以防火墙没有放行tftp-server的发送包。
根据上述文档得知，有两种解决方案，sysctl开启全自动 或者 防火墙手动配置协议规则，我测试的结果是无论那种都需要 nf_nat_tftp 模组的支持。目前手头的操作系统都是最小化安装，9.4默认安装了，8.10默认没装
[root@tftp-server ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 9.4 (Plow)
[root@tftp-server ~]# lsmod | grep tftp
nf_nat_tftp 16384 0
nf_conntrack_tftp 20480 3 nf_nat_tftp
nf_nat 61440 2 nf_nat_tftp,nft_chain_nat
nf_conntrack 217088 5 nf_nat,nf_conntrack_tftp,nft_ct,nf_nat_tftp,nf_conntrack_netlink[root@router-nft ~]# lsmod | grep tftp
[root@router-nft ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.10 (Ootpa)
安装 nf_nat_tftp 后会自动安装依赖的 nf_conntrack_tftp
[root@router-nft ~]# modprobe nf_nat_tftp
[root@router-nft ~]# lsmod | grep tftp
nf_nat_tftp 16384 0
nf_conntrack_tftp 16384 1 nf_nat_tftp
nf_nat 49152 4 nft_nat,nf_nat_tftp,nft_masq,nft_chain_nat
nf_conntrack 176128 7 nf_nat,nf_conntrack_tftp,nft_ct,nft_nat,nf_nat_tftp,nf_conntrack_netlink,nft_masq
卸载 nf_nat_tftp 后，nf_conntrack_tftp 不再被任何模组依赖，也同样会被卸载，而 nf_nat 和 nf_conntrack 由于仍被其他模组依赖，所以不会被卸载
[root@router-nft ~]# modprobe -r nf_nat_tftp
[root@router-nft ~]# lsmod | grep tftp
[root@router-nft ~]# lsmod | grep nf_nat
nf_nat 49152 3 nft_nat,nft_masq,nft_chain_nat
nf_conntrack 176128 5 nf_nat,nft_ct,nft_nat,nf_conntrack_netlink,nft_masq
libcrc32c 16384 4 nf_conntrack,nf_nat,nf_tables,xfs
要使模组开机被自动加载，需要在 /etc/modules-load.d/ 创建 .conf 为后缀的文件，并将需要加载的模组名写入进文件
[root@router-nft ~]# ls /etc/modules-load.d/
[root@router-nft ~]# echo nf_nat_tftp > /etc/modules-load.d/nf_nat_tftp.conf
[root@router-nft ~]# cat /etc/modules-load.d/nf_nat_tftp.conf
nf_nat_tftp
方案一，通过sysctl控制/net/netfilter/nf_conntrack_helper
将默认 0 关闭改为 1 开启。同理，为保证开机生效，需要在 /etc/sysctl.d/ 目录下创建 .conf 后缀配置文件并将 net.netfilter.nf_conntrack_helper=1 写入文件中
[root@router-nft ~]# cat /proc/sys/net/netfilter/nf_conntrack_helper
0
[root@router-nft ~]# echo 1 >| /proc/sys/net/netfilter/nf_conntrack_helper
[root@router-nft ~]# cat /proc/sys/net/netfilter/nf_conntrack_helper
1[root@router-nft ~]# ls /etc/sysctl.d/
99-sysctl.conf ipforward.conf
[root@router-nft ~]# echo "net.netfilter.nf_conntrack_helper=1" > /etc/sysctl.d/cthelper.conf
[root@router-nft ~]# cat /etc/sysctl.d/cthelper.conf
net.netfilter.nf_conntrack_helper=1[root@router-nft ~]# echo 0 >| /proc/sys/net/netfilter/nf_conntrack_helper
[root@router-nft ~]# cat /proc/sys/net/netfilter/nf_conntrack_helper
0
[root@router-nft ~]# sysctl -p /etc/sysctl.d/cthelper.conf
net.netfilter.nf_conntrack_helper = 1
[root@router-nft ~]# cat /proc/sys/net/netfilter/nf_conntrack_helper
1
在tftp-client上发起连接到tftp-server，tftp-client用于通信的55476端口打开
[root@tftp-client ~]# ss -upln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 0.0.0.0:67 0.0.0.0:* users:(("dhcpd",pid=1034,fd=7))
UNCONN 0 0 0.0.0.0:123 0.0.0.0:* users:(("chronyd",pid=654,fd=7))
UNCONN 0 0 127.0.0.1:323 0.0.0.0:* users:(("chronyd",pid=654,fd=5))
UNCONN 0 0 [::1]:323 [::]:* users:(("chronyd",pid=654,fd=6))[root@tftp-client ~]# tftp 192.168.5.253
tftp> [root@tftp-client ~]# ss -upln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 0.0.0.0:67 0.0.0.0:* users:(("dhcpd",pid=1034,fd=7))
UNCONN 0 0 0.0.0.0:123 0.0.0.0:* users:(("chronyd",pid=654,fd=7))
UNCONN 0 0 0.0.0.0:55476 0.0.0.0:* users:(("tftp",pid=11088,fd=3))
UNCONN 0 0 127.0.0.1:323 0.0.0.0:* users:(("chronyd",pid=654,fd=5))
UNCONN 0 0 [::1]:323 [::]:* users:(("chronyd",pid=654,fd=6))
tftp-client尝试获取文件，观察各主机端口抓包结果
tfpt-client---->router-nft enp3s0---->router-nft enp1s0---->tftp-server
成功获取文件
[root@tftp-client ~]# tftp 192.168.5.253
tftp> get 666[root@tftp-client ~]# tcpdump -i enp1s0 port 69 or 55476 -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:21:49.379792 IP 10.31.0.1.55476 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
11:21:49.381621 IP 192.168.5.253.56140 > 10.31.0.1.55476: UDP, length 516
11:21:49.381660 IP 10.31.0.1.55476 > 192.168.5.253.56140: UDP, length 4
11:21:49.381905 IP 192.168.5.253.56140 > 10.31.0.1.55476: UDP, length 516
11:21:49.381923 IP 10.31.0.1.55476 > 192.168.5.253.56140: UDP, length 4
11:21:49.382035 IP 192.168.5.253.56140 > 10.31.0.1.55476: UDP, length 4
11:21:49.382052 IP 10.31.0.1.55476 > 192.168.5.253.56140: UDP, length 4[root@router-nft ~]# tcpdump -i enp3s0 port 69 or 55476 -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:21:49.380171 IP 10.31.0.1.55476 > 192.168.5.253.tftp: 15 RRQ "666" netascii
11:21:49.381846 IP 192.168.5.253.56140 > 10.31.0.1.55476: UDP, length 516
11:21:49.382048 IP 10.31.0.1.55476 > 192.168.5.253.56140: UDP, length 4
11:21:49.382160 IP 192.168.5.253.56140 > 10.31.0.1.55476: UDP, length 516
11:21:49.382277 IP 10.31.0.1.55476 > 192.168.5.253.56140: UDP, length 4
11:21:49.382318 IP 192.168.5.253.56140 > 10.31.0.1.55476: UDP, length 4
11:21:49.382390 IP 10.31.0.1.55476 > 192.168.5.253.56140: UDP, length 4[root@router-nft ~]# tcpdump -i enp1s0 port 69 or 55476 -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:21:49.380202 IP 192.168.5.254.55476 > 192.168.5.253.tftp: 15 RRQ "666" netascii
11:21:49.381827 IP 192.168.5.253.56140 > 192.168.5.254.55476: UDP, length 516
11:21:49.382060 IP 192.168.5.254.55476 > 192.168.5.253.56140: UDP, length 4
11:21:49.382151 IP 192.168.5.253.56140 > 192.168.5.254.55476: UDP, length 516
11:21:49.382282 IP 192.168.5.254.55476 > 192.168.5.253.56140: UDP, length 4
11:21:49.382314 IP 192.168.5.253.56140 > 192.168.5.254.55476: UDP, length 4
11:21:49.382395 IP 192.168.5.254.55476 > 192.168.5.253.56140: UDP, length 4[root@tftp-server ~]# tcpdump -i WANbridge port 69 or 55476 -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on WANbridge, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:21:49.380061 IP 192.168.5.254.55476 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
11:21:49.381636 IP 192.168.5.253.56140 > 192.168.5.254.55476: UDP, length 516
11:21:49.381938 IP 192.168.5.254.55476 > 192.168.5.253.56140: UDP, length 4
11:21:49.381964 IP 192.168.5.253.56140 > 192.168.5.254.55476: UDP, length 516
11:21:49.382139 IP 192.168.5.254.55476 > 192.168.5.253.56140: UDP, length 4
11:21:49.382156 IP 192.168.5.253.56140 > 192.168.5.254.55476: UDP, length 4
11:21:49.382252 IP 192.168.5.254.55476 > 192.168.5.253.56140: UDP, length 4[root@tftp-client ~]# ll 666
-rw-r--r-- 1 root root 1024 Dec 13 11:21 666
在router-nft上用conntrack观察，可以看到开启 conntrack helper 之后跟踪到的链接最后出现了 helper=tftp
[root@router-nft ~]# dnf provides conntrack
Failed to set locale, defaulting to C.UTF-8
Updating Subscription Management repositories.
Last metadata expiration check: 2:20:09 ago on Fri Dec 13 09:07:34 2024.
conntrack-tools-1.4.4-9.el8.x86_64 : Manipulate netfilter connection tracking table and run High Availability
Repo : rhel-8-for-x86_64-baseos-rpms
Matched from:
Provide : conntrack = 1.0-1[root@router-nft ~]# man conntrack
NAMEconntrack - command line interface for netfilter connection trackingCOMMANDSThese options specify the particular operation to perform. Only one of them can be specified at any given time.-L --dumpList connection tracking or expectation table-E, --eventDisplay a real-time event log.FILTER PARAMETERS-s, --src, --orig-src IP_ADDRESSMatch only entries whose source address in the original direction equals the one specified as argument. Implies "--mask-src" when CIDR notation is used.-d, --dst, --orig-dst IP_ADDRESSMatch only entries whose destination address in the original direction equals the one specified as argument. Implies "--mask-dst" when CIDR notation isused.#没开启 conntrack helper 之前
[root@router-nft ~]# conntrack -E -s 10.31.0.1[NEW] udp 17 30 src=10.31.0.1 dst=192.168.5.253 sport=55476 dport=69 [UNREPLIED] src=192.168.5.253 dst=192.168.5.254 sport=69 dport=55476
[DESTROY] udp 17 src=10.31.0.1 dst=192.168.5.253 sport=55476 dport=69 [UNREPLIED] src=192.168.5.253 dst=192.168.5.254 sport=69 dport=55476#开启 conntrack helper 之后
[root@router-nft ~]# conntrack -E -s 10.31.0.1[NEW] udp 17 30 src=10.31.0.1 dst=192.168.5.253 sport=55476 dport=69 [UNREPLIED] src=192.168.5.253 dst=192.168.5.254 sport=69 dport=55476 helper=tftp
[DESTROY] udp 17 src=10.31.0.1 dst=192.168.5.253 sport=55476 dport=69 [UNREPLIED] src=192.168.5.253 dst=192.168.5.254 sport=69 dport=55476
回滚方案一的操作，准备测试方案二，配置文件也一并修改回去
[root@router-nft ~]# echo 0 >| /proc/sys/net/netfilter/nf_conntrack_helper
[root@router-nft ~]# cat /proc/sys/net/netfilter/nf_conntrack_helper
0[root@router-nft ~]# sed -En '/.*/p' /etc/sysctl.d/cthelper.conf
net.netfilter.nf_conntrack_helper=1
[root@router-nft ~]# sed -Ei 's/(.*)/#\1/' /etc/sysctl.d/cthelper.conf
[root@router-nft ~]# sed -En '/.*/p' /etc/sysctl.d/cthelper.conf
#net.netfilter.nf_conntrack_helper=1
方案二，配置防火墙
#创建一个名为 STATEFUL_HELPER 的表用于存放helper声明
[root@router-nft ~]# nft add table inet STATEFUL_HELPER
[root@router-nft ~]# nft list tables
table inet NFT_FILTER
table inet NFT_NAT
table inet STATEFUL_HELPER#声明名称为 TFTP_HELPER 的类型为 tftp 协议为 udp
[root@router-nft ~]# nft add ct helper inet STATEFUL_HELPER TFTP_HELPER '{ type "tftp" protocol udp ; }'
[root@router-nft ~]# nft list ct helpers table inet STATEFUL_HELPER
table inet STATEFUL_HELPER {ct helper TFTP_HELPER {type "tftp" protocol udpl3proto inet}
}#在 STATEFUL_HELPER 表中添加一个 TFTP_POST 用于存放关于 TFTP_HELPER 的规则
#链的类型我选择与之前的snat相同，在出路由最后的时候使用 helper
[root@router-nft ~]# nft add chain inet STATEFUL_HELPER TFTP_POST '{ type nat hook postrouting priority srcnat ; }'#在 TFTP_POST 链中添加规则对协议 udp 目标端口 dport 69 的链接使用 TFTP_HELPER
[root@router-nft ~]# nft add rule inet STATEFUL_HELPER TFTP_POST udp dport 69 ct helper set "TFTP_HELPER" # STATEFUL_HELPER 全表
[root@router-nft ~]# nft list table inet STATEFUL_HELPER
table inet STATEFUL_HELPER {ct helper TFTP_HELPER {type "tftp" protocol udpl3proto inet}chain TFTP_POST {type nat hook postrouting priority srcnat; policy accept;udp dport 69 ct helper set "TFTP_HELPER"}
}
保存修改后的规则到配置文件 /etc/nftables/rhel8.nft
[root@router-nft ~]# nft -a list ruleset >| /etc/nftables/rhel8.nft
[root@router-nft ~]# cat /etc/nftables/rhel8.nft
table inet NFT_FILTER { # handle 14chain NFC_INPUT { # handle 1type filter hook input priority filter; policy drop;ct state established,related accept # handle 2meta l4proto icmp accept # handle 3iifname "lo" accept # handle 4iifname "enp1s0" ip saddr 2.57.122.117 reject with icmp port-unreachable # handle 5iifname "enp1s0" tcp dport 22 accept # handle 6iifname "enp2s0" udp dport 67 accept # handle 7}
}
table inet NFT_NAT { # handle 15chain NFC_PRE { # handle 1type nat hook prerouting priority dstnat; policy accept;iifname "enp1s0" tcp dport 65523 dnat ip to 10.31.0.1:22 # handle 3}chain NFC_POST { # handle 2type nat hook postrouting priority srcnat; policy accept;iifname "enp2s0" oifname "enp1s0" ip saddr 172.31.0.0/24 masquerade # handle 4iifname "enp3s0" oifname "enp1s0" ip saddr 10.31.0.0/24 masquerade # handle 5}
}
table inet STATEFUL_HELPER { # handle 18ct helper TFTP_HELPER { # handle 2type "tftp" protocol udpl3proto inet}chain TFTP_POST { # handle 10type nat hook postrouting priority srcnat; policy accept;udp dport 69 ct helper set "TFTP_HELPER" # handle 11}
}
tftp-client获取文件，观察结果
tfpt-client---->router-nft enp3s0---->router-nft enp1s0---->tftp-server
成功
[root@tftp-client ~]# tftp 192.168.5.253
tftp> get 666
tftp> get 666[root@tftp-client ~]# tcpdump -i enp1s0 port 69 or 55476 -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:04:46.130001 IP 10.31.0.1.55476 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
15:04:46.131895 IP 192.168.5.253.38549 > 10.31.0.1.55476: UDP, length 516
15:04:46.131945 IP 10.31.0.1.55476 > 192.168.5.253.38549: UDP, length 4
15:04:46.132168 IP 192.168.5.253.38549 > 10.31.0.1.55476: UDP, length 516
15:04:46.132188 IP 10.31.0.1.55476 > 192.168.5.253.38549: UDP, length 4
15:04:46.132343 IP 192.168.5.253.38549 > 10.31.0.1.55476: UDP, length 4
15:04:46.132360 IP 10.31.0.1.55476 > 192.168.5.253.38549: UDP, length 4[root@router-nft ~]# tcpdump -i enp3s0 port 69 or 55476 -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:04:46.130225 IP 10.31.0.1.55476 > 192.168.5.253.tftp: 15 RRQ "666" netascii
15:04:46.131921 IP 192.168.5.253.38549 > 10.31.0.1.55476: UDP, length 516
15:04:46.132138 IP 10.31.0.1.55476 > 192.168.5.253.38549: UDP, length 4
15:04:46.132227 IP 192.168.5.253.38549 > 10.31.0.1.55476: UDP, length 516
15:04:46.132352 IP 10.31.0.1.55476 > 192.168.5.253.38549: UDP, length 4
15:04:46.132447 IP 192.168.5.253.38549 > 10.31.0.1.55476: UDP, length 4
15:04:46.132506 IP 10.31.0.1.55476 > 192.168.5.253.38549: UDP, length 4[root@router-nft ~]# tcpdump -i enp1s0 port 69 or 55476 -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:04:46.130258 IP 192.168.5.254.55476 > 192.168.5.253.tftp: 15 RRQ "666" netascii
15:04:46.131901 IP 192.168.5.253.38549 > 192.168.5.254.55476: UDP, length 516
15:04:46.132149 IP 192.168.5.254.55476 > 192.168.5.253.38549: UDP, length 4
15:04:46.132219 IP 192.168.5.253.38549 > 192.168.5.254.55476: UDP, length 516
15:04:46.132360 IP 192.168.5.254.55476 > 192.168.5.253.38549: UDP, length 4
15:04:46.132442 IP 192.168.5.253.38549 > 192.168.5.254.55476: UDP, length 4
15:04:46.132512 IP 192.168.5.254.55476 > 192.168.5.253.38549: UDP, length 4[root@tftp-server ~]# tcpdump -i WANbridge port 69 or 55476 -n
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on WANbridge, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:04:46.130302 IP 192.168.5.254.55476 > 192.168.5.253.tftp: TFTP, length 15, RRQ "666" netascii
15:04:46.131872 IP 192.168.5.253.38549 > 192.168.5.254.55476: UDP, length 516
15:04:46.132170 IP 192.168.5.254.55476 > 192.168.5.253.38549: UDP, length 4
15:04:46.132223 IP 192.168.5.253.38549 > 192.168.5.254.55476: UDP, length 516
15:04:46.132404 IP 192.168.5.254.55476 > 192.168.5.253.38549: UDP, length 4
15:04:46.132448 IP 192.168.5.253.38549 > 192.168.5.254.55476: UDP, length 4
15:04:46.132533 IP 192.168.5.254.55476 > 192.168.5.253.38549: UDP, length 4[root@router-nft ~]# conntrack -E -s 10.31.0.1[NEW] udp 17 30 src=10.31.0.1 dst=192.168.5.253 sport=55476 dport=69 [UNREPLIED] src=192.168.5.253 dst=192.168.5.254 sport=69 dport=55476 helper=tftp
[DESTROY] udp 17 src=10.31.0.1 dst=192.168.5.253 sport=55476 dport=69 [UNREPLIED] src=192.168.5.253 dst=192.168.5.254 sport=69 dport=55476
^Cconntrack v1.4.4 (conntrack-tools): 2 flow events have been shown.[root@tftp-client ~]# ll 666
-rw-r--r-- 1 root root 1024 Dec 13 15:04 666
