Cilium Gateway API 特性(转载)
一、环境信息
主机 | IP |
---|---|
ubuntu | 10.0.0.234 |
软件 | 版本 |
---|---|
docker | 26.1.4 |
helm | v3.15.0-rc.2 |
kind | 0.18.0 |
kubernetes | 1.23.4 |
ubuntu os | Ubuntu 22.04.6 LTS |
kernel | 5.15.0-106 |
二、Cilium
Gateway API
流程图
Cilium 现在提供完全一致的 Gateway API 实现。 Gateway API 是 Kubernetes 集群中南北负载均衡和流量路由的新标准,并且是 Ingress API 的长期继承者。 Gateway API 代表了流量管理的未来。
GatewaAPI 从设计之初就是为了解决 Ingress API 的局限性
- Gateway API 的创建是源于 Ingress API 存在一些局限性:首先,它不提供用户需要定义的高级负载均衡功能。它本身仅支持简单的基于路径的 HTTP 流量请求路由
- 其次,用户管理变得不切实际:产品供应商通过注释来解决 Ingress API 中功能缺乏的问题。但是注释虽然非常强大,但最终会导致一个 Ingress 与另一个 Ingress 之间出现不一致
- 第三,由于 Ingress API 是单一 API 资源,因此它操作受限:不适合具有共享负载均衡基础设施的多团队集群
在 Cilium 1.13 中,Cilium Gateway API 通过了所有 Gateway API 一致性测试 (v0.5.1)[https://isovalent.com/blog/post/cilium-release-113/]
参考官方链接
三、Cilium
Gateway API
模式环境搭建
安装须知
- 配置 Cilium 时,必须启用 NodePort,使用 nodePort.enabled=true 或使用 kubeProxyReplacement 将 kube-proxy 替换为部分或严格。
- 配置 Cilium 时,必须使用 --enable-l7-proxy 标志(默认已启用)启用 L7 代理。
- 必须预先安装以下来自 Gateway API v0.5.1 的 CRD。(安装步骤)(https://docs.cilium.io/en/v1.13/network/servicemesh/gateway-api/gateway-api/#prerequisites)
- 与 Ingress 类似,Gateway API 控制器会创建一个 LoadBalancer 类型的服务,因此您的环境需要支持该服务。
kind
配置文件信息
root@KinD:~# cat install.sh#!/bin/bash
date
set -v# 1.prep noCNI env
cat <<EOF | kind create cluster --name=cilium-gatewayapi-http --image=kindest/node:v1.23.4 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:# kind 默认使用 rancher cni,cni 我们需要自己创建disableDefaultCNI: true# 此处使用 cilium 代替 kube-proxy 功能kubeProxyMode: "none"
nodes:- role: control-plane- role: worker- role: workercontainerdConfigPatches:
- |-[plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.evescn.com"]endpoint = ["https://harbor.evescn.com"]
EOF# 2.install necessary tools
for i in $(docker ps -a --format "table {{.Names}}" | grep cilium)
doecho $idocker cp /usr/bin/ping $i:/usr/bin/pingdocker exec -it $i bash -c "sed -i -e 's/jp.archive.ubuntu.com\|archive.ubuntu.com\|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list"docker exec -it $i bash -c "apt-get -y update >/dev/null && apt-get -y install net-tools tcpdump lrzsz bridge-utils >/dev/null 2>&1"
done
- 安装
k8s
集群和cilium
服务
root@KinD:~# ./install.shCreating cluster "cilium-gatewayapi-http" ...✓ Ensuring node image (kindest/node:v1.23.4) 🖼✓ Preparing nodes 📦 📦 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing StorageClass 💾 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-cilium-gatewayapi-http"
You can now use your cluster with:kubectl cluster-info --context kind-cilium-gatewayapi-httpThanks for using kind! 😊
- 安装
gateway api
root@KinD:~# kubectl apply -f https://gh.api.99988866.xyz/https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.5.1/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created
root@KinD:~# kubectl apply -f https://gh.api.99988866.xyz/https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.5.1/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created
root@KinD:~# kubectl apply -f https://gh.api.99988866.xyz/https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.5.1/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created
root@KinD:~# kubectl apply -f https://gh.api.99988866.xyz/https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.5.1/config/crd/experimental/gateway.networking.k8s.io_referencegrants.yaml
customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created
- 安装
cilium
服务
#/bin/bash
set -v
controller_node_ip=`kubectl get node -o wide --no-headers | grep -E "control-plane|bpf1" | awk -F " " '{print $6}'`helm repo add cilium https://helm.cilium.io > /dev/null 2>&1
helm repo update > /dev/null 2>&1helm install cilium cilium/cilium \--set k8sServiceHost=$controller_node_ip \--set k8sServicePort=6443 \--version 1.13.0-rc5 \--namespace kube-system \--set debug.enabled=true \--set debug.verbose=datapath \--set monitorAggregation=none \--set ipam.mode=cluster-pool \--set cluster.name=cilium-gatewayapi-http \--set kubeProxyReplacement=strict \--set bpf.masquerade=true \--set gatewayAPI.enabled=true
--set
参数解释
-
--set kubeProxyReplacement=strict
- 含义: 启用 kube-proxy 替代功能,并以严格模式运行。
- 用途: Cilium 将完全替代 kube-proxy 实现服务负载均衡,提供更高效的流量转发和网络策略管理。
-
--set bpf.masquerade
- 含义: 启用 eBPF 功能。
- 用途: 使用 eBPF 实现数据路由,提供更高效和灵活的网络地址转换功能。
-
--set gatewayAPI.enable=true
:- 启用 Gateway API 支持,Cilium 将支持并管理 Gateway API 资源。
- 查看安装的服务
root@KinD:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cilium-55g65 1/1 Running 0 103s
kube-system cilium-bb4s7 1/1 Running 0 103s
kube-system cilium-operator-58959f76d6-46b62 1/1 Running 0 103s
kube-system cilium-operator-58959f76d6-92kwx 1/1 Running 0 103s
kube-system cilium-r5v9v 1/1 Running 0 103s
kube-system coredns-64897985d-l5rqq 1/1 Running 0 131m
kube-system coredns-64897985d-qx7bl 1/1 Running 0 131m
kube-system etcd-cilium-gatewayapi-http-control-plane 1/1 Running 0 132m
kube-system kube-apiserver-cilium-gatewayapi-http-control-plane 1/1 Running 0 132m
kube-system kube-controller-manager-cilium-gatewayapi-http-control-plane 1/1 Running 0 132m
kube-system kube-scheduler-cilium-gatewayapi-http-control-plane 1/1 Running 0 132m
local-path-storage local-path-provisioner-5ddd94ff66-7t65r 1/1 Running 0 131m
安装 metallb
服务,提供 LoadBanlencer
功能
- 安装
metallb
服务
root@KinD:~# kubectl apply -f https://gh.api.99988866.xyz/https://raw.githubusercontent.com/metallb/metallb/v0.13.10/config/manifests/metallb-native.yamlnamespace/metallb-system created
customresourcedefinition.apiextensions.k8s.io/addresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created
customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created
customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created
serviceaccount/controller created
serviceaccount/speaker created
role.rbac.authorization.k8s.io/controller created
role.rbac.authorization.k8s.io/pod-lister created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/controller created
rolebinding.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
configmap/metallb-excludel2 created
secret/webhook-server-cert created
service/webhook-service created
deployment.apps/controller created
daemonset.apps/speaker created
validatingwebhookconfiguration.admissionregistration.k8s.io/metallb-webhook-configuration created
- 创建 metalLb layer2 的 IPAddressPool
root@KinD:~# cat metallb-l2-ip-config.yaml
---
# metallb 分配给 loadbanlencer 的 ip 地址段定义
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:name: ippoolnamespace: metallb-system
spec:addresses:- 172.18.0.200-172.18.0.210---
# 创建 L2Advertisement 进行 IPAddressPool 地址段
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:name: ippoolnamespace: metallb-system
spec:ipAddressPools:- ippoolroot@KinD:~# kubectl apply -f metallb-l2-ip-config.yaml
ipaddresspool.metallb.io/ippool unchanged
l2advertisement.metallb.io/ippool createdroot@KinD:~# kubectl get ipaddresspool -n metallb-system
NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES
ippool true false ["172.18.0.200-172.18.0.210"]
root@KinD:~# kubectl get l2advertisement -n metallb-system
NAME IPADDRESSPOOLS IPADDRESSPOOL SELECTORS INTERFACES
ippool ["ippool"]
k8s
集群安装 Pod
和 Service
root@KinD:~# cat cni.yaml
---
apiVersion: apps/v1
kind: DaemonSet
#kind: Deployment
metadata:labels:app: cniname: cni
spec:#replicas: 1selector:matchLabels:app: cnitemplate:metadata:labels:app: cnispec:containers:- image: harbor.dayuan1997.com/devops/nettool:0.9name: nettoolboxsecurityContext:privileged: true---
apiVersion: v1
kind: Service
metadata:name: cni
spec:ports:- port: 80targetPort: 80protocol: TCPtype: ClusterIPselector:app: cni
root@KinD:~# kubectl apply -f cni.yaml
daemonset.apps/cni created
service/serversvc created
- 查看安装服务信息
root@KinD:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cni-9c8h5 1/1 Running 0 7s 10.0.2.2 cilium-gatewayapi-http-worker <none> <none>
cni-j9bzr 1/1 Running 0 7s 10.0.1.69 cilium-gatewayapi-http-worker2 <none> <none>root@KinD:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cni ClusterIP 10.96.145.109 <none> 80/TCP 20s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 134m
四、测试 Gateway
HTTP
特性
配置 Gateway
HTTP
root@KinD:~# cat gateway.yamnl
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:name: basic-gateway
spec:gatewayClassName: ciliumlisteners:- protocol: HTTPport: 80name: web-gatewayallowedRoutes:namespaces:from: Same
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:name: http-app-cni
spec:parentRefs:- name: basic-gatewaynamespace: defaultrules:- matches:- path:type: PathPrefixvalue: /backendRefs:- name: cniport: 80root@KinD:~# kubectl apply -f gateway.yaml
gateway.gateway.networking.k8s.io/basic-gateway created
httproute.gateway.networking.k8s.io/http-app-cni createdroot@KinD:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cilium-gateway-basic-gateway LoadBalancer 10.96.30.37 172.18.0.200 80:30594/TCP 21s
cni ClusterIP 10.96.145.109 <none> 80/TCP 4m9s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 138mroot@KinD:~# kubectl get gateway
NAME CLASS ADDRESS READY AGE
basic-gateway cilium 172.18.0.200 True 37sroot@KinD:~# kubectl get HTTPRoute
NAME HOSTNAMES AGE
http-app-cni 54s
创建
Gateway
后basic-gateway
在定义的地址池中自动获取了一个IP
地址,并且自动创建了一个cilium-gateway-basic-gateway
svc
服务,服务的EXTERNAL-IP
为basic-gateway
从地址池中自动获取的IP
地址
测试 Gateway
HTTP
root@KinD:~# GATEWAY=$(kubectl get gateway basic-gateway -o jsonpath='{.status.addresses[0].value}')
root@KinD:~# curl -v http://"$GATEWAY"/
* Trying 172.18.0.200:80...
* Connected to 172.18.0.200 (172.18.0.200) port 80 (#0)
> GET / HTTP/1.1
> Host: 172.18.0.200
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< server: envoy
< date: Tue, 16 Jul 2024 06:14:57 GMT
< content-type: text/html
< content-length: 46
< last-modified: Tue, 16 Jul 2024 06:09:16 GMT
< etag: "66960e8c-2e"
< accept-ranges: bytes
< x-envoy-upstream-service-time: 0
<
PodName: cni-j9bzr | PodIP: eth0 10.0.1.69/32
* Connection #0 to host 172.18.0.200 left intact
在宿主机
172.18.0.1
主机上访问业务正常访问,使用 ip 地址为172.18.0.200
五、测试 Gateway
HTTPS
特性
创建 tls 证书,此处使用了一个测试证书
root@KinD:~# kubectl create secret tls demo-cert --cert=tls.crt --key=tls.key
secret/demo-cert created
配置 Gateway
HTTPS
root@KinD:~# cat gateway-https.yamnl
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:name: tls-gateway
spec:gatewayClassName: ciliumlisteners:- name: httpsprotocol: HTTPSport: 443hostname: "cni.evescn.com"tls:certificateRefs:- kind: Secretname: test-cert
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:name: https-app-cni
spec:parentRefs:- name: tls-gatewayhostnames:- "cni.evescn.com"rules:- matches:- path:type: PathPrefixvalue: /backendRefs:- name: cniport: 80root@KinD:~# kubectl apply -f gateway-https.yaml
gateway.gateway.networking.k8s.io/tls-gateway created
httproute.gateway.networking.k8s.io/https-app-cni createdroot@KinD:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cilium-gateway-basic-gateway LoadBalancer 10.96.30.37 172.18.0.200 80:30594/TCP 7m21s
cilium-gateway-tls-gateway LoadBalancer 10.96.5.61 172.18.0.201 443:30197/TCP 19s
cni ClusterIP 10.96.145.109 <none> 80/TCP 11m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 145mroot@KinD:~# kubectl get gateway
NAME CLASS ADDRESS READY AGE
basic-gateway cilium 172.18.0.200 True 7m47s
tls-gateway cilium 172.18.0.201 True 45sroot@KinD:~# kubectl get HTTPRoute
NAME HOSTNAMES AGE
http-app-cni 7m56s
https-app-cni ["cni.evescn.com"] 54s
创建
gateway
后tls-gateway
在定义的地址池中自动获取了一个IP
地址,并且自动创建了一个cilium-gateway-tls-gateway
svc
服务,服务的EXTERNAL-IP
为tls-gateway
从地址池中自动获取的IP
地址
测试 Gateway
HTTPS
/etc/hosts
新增dns
解析
root@KinD:~# cat /etc/hosts
......
172.18.0.201 cni.evescn.com
- 测试
Gateway
HTTPS
root@KinD:~# curl --cacert /root/evescn.com.pem -v https://cni.evescn.com/
* Trying 172.18.0.201:443...
* Connected to cni.evescn.com (172.18.0.201) port 443 (#0)
> GET / HTTP/1.1
> Host: cni.evescn.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< server: envoy
< date: Tue, 16 Jul 2024 06:25:27 GMT
< content-type: text/html
< content-length: 46
< last-modified: Tue, 16 Jul 2024 06:09:16 GMT
< etag: "66960e8c-2e"
< accept-ranges: bytes
< x-envoy-upstream-service-time: 0
<
PodName: cni-j9bzr | PodIP: eth0 10.0.1.69/32
* Connection #0 to host cni.evescn.com left intact
在宿主机
172.18.0.1
主机上访问业务正常访问,使用 ip 地址为172.18.0.201
六、转载博客
https://bbs.huaweicloud.com/blogs/418684