一、目标
-
利用DPVS部署一个基于OSPF/ECMP的提供HTTP服务的多活高可用的测试环境。
-
本次部署仅用于验证功能，不提供性能验证。
-
配置两台DPVS组成集群、两台REAL SERVER提供实际HTTP服务。
注：在虚拟环境里面，通过在一台虚拟服务器上面安装FRRouting来模拟支持OSPF的路由器。
二、组网架构
角色及配置说明：
CLIENT: 透过DPVS集群访问REAL SERVER提供的相关服务。
CLIENT1的配置： 网卡名称： enp0s8
ip地址： 192.168.2.100/24
静态路由： ip route add 192.168.0.0/16 via 192.168.2.254 dev enp0s8将所有和192.168.0.0/16的请求指向路由器CLIENT2的配置： 网卡名称： enp0s8
ip地址： 192.168.2.101/24
静态路由： ip route add 192.168.0/16 via 192.168.2.254 dev enp0s8将所有和192.168.0.0/16的请求指向路由器
ROUTER: 提供OSPF/ECMP功能的路由器（这里用frrouting模拟）。
网卡1名称： enp0s8
网卡1的ip地址： 192.168.2.254/24网卡2名称： enp0s9
网卡2的ip地址： 192.168.3.254/24DPVS: dpvs slb集群。
DPVS1服务器的配置： 网卡1名称： enp0s8 -> dpdk0 dpdk0.kni
网卡1的ip地址： 192.168.3.1/24192.168.0.1/24 (VIP)网卡2名称： enp0s9 -> dpdk1 dpdk1.kni
网卡2的ip地址： 192.168.1.1/24DPVS1服务器2的配置： 网卡1名称： enp0s8 -> dpdk0 dpdk0.kni
网卡1的ip地址： 192.168.3.2/24192.168.0.1/24 (VIP)网卡2名称： enp0s9 -> dpdk1 dpdk1.kni
网卡2的ip地址： 192.168.1.2/24REAL SERVER: 提供HTTP SERVER服务。
real server1的配置:网卡名称： enp0s8
ip地址： 192.168.1.100/24real server2的配置:网卡名称： enp0s8
ip地址： 192.168.1.101/24
三、部署过程
以下以dpvs1为例, dpvs2和dpvs1的配置类似。
1. ROUTER的部署
1.1 安装frrouting服务
对于centos7执行以下命令
curl -O https://rpm.frrouting.org/repo/$FRRVER-repo-1-0.el7.noarch.rpm
sudo yum install ./$FRRVER*# install FRR
sudo yum install frr frr-pythontools
对于centos8执行以下命令
curl -O https://rpm.frrouting.org/repo/$FRRVER-repo-1-0.el8.noarch.rpm
sudo yum install ./$FRRVER*# install FRR
sudo yum install frr frr-pythontools
1.2 配置frr服务
1.2.1 开启ospf服务
在启动前，先要开启linux服务器的路由转发功能
echo "net.ipv4.ip_forward=1" > /etc/sysctl.conf
sysctl -p
在/etc/frr/daemons文件中开启：
ospfd=yes （对于ipv4)
ospf6d=yes (对于ipv6)
cd /etc/frr
cat daemons
----------------------
# This file tells the frr package which daemons to start.
#
# Sample configurations for these daemons can be found in
# /usr/share/doc/frr/examples/.
#
# ATTENTION:
#
# When activating a daemon for the first time, a config file, even if it is
# empty, has to be present *and* be owned by the user and group "frr", else
# the daemon will not be started by /etc/init.d/frr. The permissions should
# be u=rw,g=r,o=.
# When using "vtysh" such a config file is also needed. It should be owned by
# group "frrvty" and set to ug=rw,o= though. Check /etc/pam.d/frr, too.
#
# The watchfrr, zebra and staticd daemons are always started.
#
bgpd=no
ospfd=yes
ospf6d=yes
ripd=no
ripngd=no
isisd=no
pimd=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
sharpd=no
pbrd=no
bfdd=no
fabricd=no
vrrpd=no
pathd=no1.2.2 配置ospf服务
cd /etc/frr
cat frr.conf
----------------------frr version 8.1
frr defaults traditional
hostname router
log syslog informational
service integrated-vtysh-config
!
interface enp0s8ip address 192.168.2.254/24
exit
!
interface enp0s9ip address 192.168.3.254/24ip ospf 100 area 0ip ospf dead-interval 40
exit
!
router ospf 100ospf router-id 192.168.2.254
exit
!
1.3 启动FRR服务
service frr start
用ps -ef|grep frr检查frr,ospfd,zebra进程是否已经启动
ps -ef|grep frr
-----------------
root 3635 1 0 18:11 ? 00:00:01 /usr/lib/frr/watchfrr -d -F traditional zebra ospfd-100 ospfd-200 ospfd-300 staticd
frr 3657 1 0 18:11 ? 00:00:00 /usr/lib/frr/zebra -d -F traditional --daemon -A 127.0.0.1 -s 90000000
frr 3662 1 0 18:11 ? 00:00:02 /usr/lib/frr/ospfd -d -F traditional -n 100 --daemon -A 127.0.0.1
frr 3665 1 0 18:11 ? 00:00:00 /usr/lib/frr/ospfd -d -F traditional -n 200 --daemon -A 127.0.0.1
frr 3668 1 0 18:11 ? 00:00:00 /usr/lib/frr/ospfd -d -F traditional -n 300 --daemon -A 127.0.0.1
frr 3671 1 0 18:11 ? 00:00:00 /usr/lib/frr/staticd -d -F traditional --daemon -A 127.0.0.1
1.4 frr的常用命令
# 查看邻居列表
vtyshdo show ip ospf neighbor# 查看路由表
vtyshshow ip route # 写入配置
do write
do write mem
2. DPVS的部署
2.1 安装frrouting服务
2.1.1 Centos7执行以下命令
curl -O https://rpm.frrouting.org/repo/$FRRVER-repo-1-0.el7.noarch.rpm
sudo yum install ./$FRRVER*# install FRR
sudo yum install frr frr-pythontools
对于centos8执行以下命令
curl -O https://rpm.frrouting.org/repo/$FRRVER-repo-1-0.el8.noarch.rpm
sudo yum install ./$FRRVER*# install FRR
sudo yum install frr frr-pythontools
2.1.2 配置frr服务
2.1.2.1 开启ospf服务
在/etc/frr/daemons文件中开启：
ospfd=yes （对于ipv4)
ospf6d=yes (对于ipv6)
cd /etc/frr
cat daemons
----------------------
# This file tells the frr package which daemons to start.
#
# Sample configurations for these daemons can be found in
# /usr/share/doc/frr/examples/.
#
# ATTENTION:
#
# When activating a daemon for the first time, a config file, even if it is
# empty, has to be present *and* be owned by the user and group "frr", else
# the daemon will not be started by /etc/init.d/frr. The permissions should
# be u=rw,g=r,o=.
# When using "vtysh" such a config file is also needed. It should be owned by
# group "frrvty" and set to ug=rw,o= though. Check /etc/pam.d/frr, too.
#
# The watchfrr, zebra and staticd daemons are always started.
#
bgpd=no
ospfd=yes
ospf6d=yes
ripd=no
ripngd=no
isisd=no
pimd=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
sharpd=no
pbrd=no
bfdd=no
fabricd=no
vrrpd=no
pathd=no2.1.2.2 配置ospf服务
cd /etc/frr
cat frr.conf
----------------------frr version 8.2.2
frr defaults traditional
hostname localhost.localdomain
log syslog informational
no ip forwarding
no ipv6 forwarding
!
interface dpdk0.kniip ospf dead-interval 10ip ospf dead-interval 40
exit
!
router ospfospf router-id 192.168.3.1 # 如果是dpvs2, 则修改为192.168.3.2log-adjacency-changesauto-cost reference-bandwidth 1000network 192.168.0.0/24 area 0network 192.168.3.0/24 area 0
exit
!
2.1.3 启动FRR服务
service frr start
用ps -ef|grep frr检查frr,ospfd,zebra进程是否已经启动
ps -ef|grep frr
-----------------
root 3635 1 0 18:11 ? 00:00:01 /usr/lib/frr/watchfrr -d -F traditional zebra ospfd-100 ospfd-200 ospfd-300 staticd
frr 3657 1 0 18:11 ? 00:00:00 /usr/lib/frr/zebra -d -F traditional --daemon -A 127.0.0.1 -s 90000000
frr 3662 1 0 18:11 ? 00:00:02 /usr/lib/frr/ospfd -d -F traditional -n 100 --daemon -A 127.0.0.1
frr 3665 1 0 18:11 ? 00:00:00 /usr/lib/frr/ospfd -d -F traditional -n 200 --daemon -A 127.0.0.1
frr 3668 1 0 18:11 ? 00:00:00 /usr/lib/frr/ospfd -d -F traditional -n 300 --daemon -A 127.0.0.1
frr 3671 1 0 18:11 ? 00:00:00 /usr/lib/frr/staticd -d -F traditional --daemon -A 127.0.0.1
2.2 部署dpvs服务
2.2.1 编译dpvs源码
# 拉取dpvs和dpdk的代码
git clone http://10.0.171.10/networkService/l4slb-dpvs-1.9.git
git checkout feature_ospf# 编译dpdk 20.11.1
cd l4slb-dpvs-1.9/dpdk-stable-20.11.1
meson -Denable_kmods=true -Dexamples=l2fwd,l3fwd -Ddisable_drivers=net/af_xdp,event/dpaa,,event/dpaa2 -Dprefix=`pwd`/dpdklib ./buildninja -C build
ninja -C build installexport PKG_CONFIG_PATH=`pwd`/dpdk-stable-20.11.1/dpdklib/lib64/pkgconfig# 编译dpvs
make -j8#如果编译DEBUG版本
make DEBUG=1 -j82.2.2 初始化dpdk运行环境
# 配置大页内存
echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
[ -d /mnt/huge ] || mkdir /mnt/hugemount -t hugetlbfs nodev /mnt/huge# 加载uio内核模块
modprobe uio_pci_generic# 加载rte_kni内核模块
insmod dpdk-stable-20.11.1/build/kernel/linux/kni/rte_kni.ko# 加载dpvs使用的网卡
dpdk-stable-20.11.1/usertools/dpdk-devbind.py -b uio_pci_generic enp0s8
dpdk-stable-20.11.1/usertools/dpdk-devbind.py -b uio_pci_generic enp0s9
2.2.3 配置dpvs:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! This is dpvs default configuration file.
!
! The attribute "<init>" denotes the configuration item at initialization stage. Item of
! this type is configured oneshoot and not reloadable. If invalid value configured in the
! file, dpvs would use its default value.
!
! Note that dpvs configuration file supports the following comment type:
! * line comment: using '#" or '!'
! * inline range comment: using '<' and '>', put comment in between
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! global config
global_defs {log_level INFO! log_file /var/log/dpvs.log! log_async_mode on! pdump off
}! netif config
netif_defs {<init> pktpool_size 65536<init> pktpool_cache 256<init> fdir_mode perfect<init> device dpdk0 {rx {queue_number 1descriptor_number 1024rss all}tx {queue_number 1descriptor_number 1024}mtu 1500!promisc_modekni_name dpdk0.kni}<init> device dpdk1 {rx {queue_number 1descriptor_number 1024rss all}tx {queue_number 1descriptor_number 1024}mtu 1500!promisc_modekni_name dpdk1.kni}
}! worker config (lcores)
! notes:
! 1. rx(tx) queue ids MUST start from 0 and continous
! 2. cpu ids and rx(tx) queue ids MUST be unique, repeated ids is forbidden
! 3. cpu ids identify dpvs workers only, and not correspond to physical cpu cores.
! If you are to specify cpu cores on which to run dpvs, please use dpdk eal options,
! such as "-c", "-l", "--lcores". Use "dpvs -- --help" for supported eal options.
worker_defs {<init> worker cpu0 {type mastercpu_id 0}<init> worker cpu1 {type slavecpu_id 1port dpdk0 {rx_queue_ids 0tx_queue_ids 0! isol_rx_cpu_ids 9! isol_rxq_ring_sz 1048576}port dpdk1 {rx_queue_ids 0tx_queue_ids 0! isol_rx_cpu_ids 9! isol_rxq_ring_sz 1048576}}
}! timer config
timer_defs {# cpu job loops to schedule dpdk timer managementschedule_interval 500
}! dpvs neighbor config
neigh_defs {<init> unres_queue_length 128timeout 60
}! dpvs ipset config
ipset_defs {<init> ipset_hash_pool_size 131072
}! dpvs ipv4 config
ipv4_defs {forwarding off<init> default_ttl 64fragment {<init> bucket_number 4096<init> bucket_entries 16<init> max_entries 4096<init> ttl 1}
}! dpvs ipv6 config
ipv6_defs {disable offforwarding offroute6 {<init> method hlistrecycle_time 10}
}! control plane config
ctrl_defs {lcore_msg {<init> ring_size 4096sync_msg_timeout_us 20000priority_level low}ipc_msg {<init> unix_domain /var/run/dpvs_ctrl}
}! ipvs config
ipvs_defs {conn {<init> conn_pool_size 65536<init> conn_pool_cache 256conn_init_timeout 3! expire_quiescent_template! fast_xmit_close! <init> redirect off}udp {! defence_udp_dropuoa_mode ipo <opp|ipo>uoa_max_trail 3timeout {normal 300last 3}}tcp {! defence_tcp_droptimeout {none 2established 90syn_sent 3syn_recv 30fin_wait 7time_wait 7close 3close_wait 7last_ack 7listen 120synack 30last 2}synproxy {synack_options {mss 1452ttl 63sack! wscale! timestamp}close_client_window! defer_rs_synrs_syn_max_retry 3ack_storm_thresh 10max_ack_saved 3conn_reuse_state {closetime_wait! fin_wait! close_wait! last_ack}}}
}! sa_pool config
sa_pool {pool_hash_size 16flow_enable off
}2.2.4 启动dpvs:
bin/dpvs
2.2.5 初始化dpvs的服务配置,创建端口为8080的HTTP负载均衡服务：
VIP=192.168.0.1 # dpvs虚IP
OSPFIP=192.168.3.1 # dpvs的ospf心跳IP, 如果是dpvs2，改成192.168.3.2
GATEWAY=192.168.3.254 # dpvs的出口网关（指向ROUTER)
VIP_PREFIX=192.168.0.0 # VIP的掩码前缀
LIP_PREFIX=192.168.1.0 # LOCAL IP的掩码前缀
LIP=("192.168.1.1") # LOCAL IP地址列表, 如果是client2，改成192.168.1.2
SERVICE=$VIP:8080 # 开启的服务# REAL SERVER列表
RS_SERVER=("192.168.1.100:8080" "192.168.1.101:8080")# 配置IP地址
./tools/dpip/build/dpip addr add $VIP/24 dev dpdk0
./tools/dpip/build/dpip addr add $OSPFIP/24 dev dpdk0
# 定义默认出口网关指向ROUTER
./tools/dpip/build/dpip route add default via $GATEWAY dev dpdk0# 配置kni虚拟网卡的IP以及路由（和dpdk网卡中的配置一致）
ifconfig dpdk0.kni $VIP netmask 255.255.255.0
ip addr add $OSPFIP/24 dev dpdk0.kni
ip route add default via $GATEWAY dev dpdk0.kni
ifconfig dpdk1.kni ${LIP[0]} netmask 255.255.255.0# 添加后端IP
./tools/dpip/build/dpip route add $LIP_PREFIX/24 dev dpdk1# 添加服务
./tools/ipvsadm/ipvsadm -A -t $SERVICE -s rr# 添加RS
for rs in ${RS_SERVER[*]}
do./tools/ipvsadm/ipvsadm -a -t $SERVICE -r $rs -b
done# 配置VIP监听器使用的DPVS local_ip
for lip in ${LIP[*]}
do./tools/ipvsadm/ipvsadm -P -z $lip -t $SERVICE -F dpdk1
done2.2.6 部署real server的健康检查
#!/bin/bash
#
VIP=192.168.0.1 # DPVS的vip
CPORT=8080 # DPVS的服务端口
RS=("192.168.1.100" "192.168.1.101") # REAL SERVER列表
declare -a RSSTATUS # 存储 REAL SERVER的状态
RW=("2" "1") # REAL SERVER的权重
RPORT=8080 # REAL SERVER的HTTP服务端口
RSURL="/index.html" # REAL SERVER的测试URL路径
TYPE=b # FULLNAT模式
CHKLOOP=3 # 错误检查失败尝试次数
LOG=/var/log/ipvsmonitor.log # 日志存放路径# 添加realserver
addrs() {ipvsadm -a -t $VIP:$CPORT -r $1:$RPORT -$TYPE -w $2[ $? -eq 0 ] && return 0 || return 1
}# 删除realserver
delrs() {ipvsadm -d -t $VIP:$CPORT -r $1:$RPORT[ $? -eq 0 ] && return 0 || return 1
}# 通过http接口对realserver的状态进行探测
checkrs() {local I=1while [ $I -le $CHKLOOP ]; doif curl --connect-timeout 1 http://${1}:${RPORT}${RSURL} &> /dev/null; thenreturn 0filet I++donereturn 1
}# 状态初始化
initstatus() {local Ilocal COUNT=0;for I in ${RS[*]}; doif ipvsadm -L -n | grep "$I:$RPORT" && > /dev/null ; thenRSSTATUS[$COUNT]=1elseRSSTATUS[$COUNT]=0filet COUNT++done
}initstatus# 状态检测循环
while :; dolet COUNT=0for I in ${RS[*]}; doif checkrs $I; thenif [ ${RSSTATUS[$COUNT]} -eq 0 ]; thenaddrs $I ${RW[$COUNT]}[ $? -eq 0 ] && RSSTATUS[$COUNT]=1 && echo "`date +'%F %H:%M:%S'`, $I is back." >> $LOGfielse if [ ${RSSTATUS[$COUNT]} -eq 1 ]; thendelrs $I[ $? -eq 0 ] && RSSTATUS[$COUNT]=0 && echo "`date +'%F %H:%M:%S'`, $I is gone." >> $LOGfifilet COUNT++donesleep 5
done3. REALSERVER的部署
3.1 部署nginx服务
3.1.1 编译nginx
# 安装nginx服务
wget https://nginx.org/download/nginx-1.22.1.tar.gz
tar xzvf /nginx-1.22.1.tar.gz
cd nginx-1.22.1
./configure --prefix=/opt/nginx
make -j4
make install3.1.2 配置nginx服务
# 安装nginx服务
wget https://nginx.org/download/nginx-1.22.1.tar.gz
tar xzvf /nginx-1.22.1.tar.gz
cd nginx-1.22.1
./configure --prefix=/opt/nginx
make -j4
make install3.1.3 启动nginx服务
cd /opt/nginx/
./sbin/nginx
四、测试验证
-
连通性测试
# 在client服务器上面ping VIP是否通 ping 192.168.0.1# 在dpvs服务器上面ping client是否通 ping 192.168.2.100 ping 192.168.2.101# 在client服务器上面测试8080 tcp端口是否通 telnet 192.168.0.1 8080 -
http请求测试
# 在client服务器上面请求HTTP curl "http://192.168.0.1:8080/index.html" -
模拟单dpvs宕机测试
随机选择一台dpvs，譬如dvps1，将dpvs进程kill 查看http请求测试依旧可以成功 -
模拟单dpvs宕机后恢复测试
将以上kill的dpvs进程重新启动 通过查看real server上的访问日志，查看http请求测试是否可以重新从这台dpvs服务器恢复。 -
模拟单realserver宕机测试
将以上kill的dpvs进程重新启动
通过查看real server上的访问日志，查看http请求测试是否可以重新从这台dpvs服务器恢复。
- 模拟单realserver宕机后恢复测试
将以上kill的dpvs进程重新启动
通过查看real server上的访问日志，查看http请求测试是否可以重新从这台dpvs服务器恢复。
