K8S中ingress-nginx通过header路由到不同后端

背景

• 公司使用ingress-nginx作为网关的项目，需要在相同域名、uri，根据header将请求转发到不同的后端中
• 在稳定发布的情况下，ingress-nginx是没有语法直接支持根据header做转发的。但是这个可以利用灰度发布的特性实现header路由功能

准备

• 准备两个后端，后端代码如下，路由均为 /app
  • main.go

package mainimport "github.com/gin-gonic/gin"func main() {r := gin.Default()r.GET("/app", func(context *gin.Context) {context.JSON(200, gin.H{"message": "app1"})})r.Run(":8080")
}

• 使用Dockerfile构建镜像
  • 这里构建 goapp1:v1，goapp2:v1两个镜像（goapp2请将main.go修改 “message”: “app2”）

FROM golang:1.17.13
RUN mkdir -p /go/app/; \cd /go/app/; \go mod init app1;\GOPROXY="https://goproxy.cn,direct" go get github.com/gin-gonic/gin@v1.6.3
WORKDIR /go/app/
COPY main.go /go/app
EXPOSE 8080
CMD go run main.go

使用灰度发布的特性进行header的路由

• 此解决方案参考：https://v2-1.docs.kubesphere.io/docs/zh-CN/quick-start/ingress-canary/
• 注：本人使用低版本ingress-nginx，高版本的请大家自行修改不同之处
• 首先部署goapp1:v1 和 goapp2:v1 的deployment和service
  • 此为goapp1。goapp2请自行修改

apiVersion: apps/v1
kind: Deployment
metadata:name: goapp1namespace: default
spec:replicas: 1selector:matchLabels:app: goapp1template:metadata:labels:app: goapp1spec:containers:- image: goapp1:v1imagePullPolicy: IfNotPresentname: goapp1ports:- containerPort: 80protocol: TCP
---
apiVersion: v1
kind: Service
metadata:name: goapp1namespace: default
spec:ports:- port: 8080protocol: TCPtargetPort: 8080selector:app: goapp1

• 部署稳定发布版本的ingress，路由至goapp1

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: goapp1 namespace: defaultannotations:kubernetes.io/ingress.class: nginx
spec:rules:- host: test.comhttp:paths:- path: /app pathType: Prefixbackend:service:name: goapp1port: number: 8080

• 部署canary版本的ingress，路由至goapp2
  • 这里可见 域名都是 test.com，uri都是 /app
  • 注解：
    • nginx.ingress.kubernetes.io/canary: “true” # 启用canary灰度发布特性
    • nginx.ingress.kubernetes.io/canary-by-header: canary # 通过header可选择是否转发至canary版本的后端

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: goapp2namespace: defaultannotations:kubernetes.io/ingress.class: nginxnginx.ingress.kubernetes.io/canary: "true"nginx.ingress.kubernetes.io/canary-by-header: canary
spec:rules:- host: test.comhttp:paths:- path: /app pathType: Prefixbackend:service:name: goapp2port: number: 8080

• 进行测试

for i in {1..20};# ingress-nginx的NodePort请自行查看，替换下面的端口do curl test.com:31132/app -H "canary: never"; # 路由至稳定版本的goapp1echo -e "";
done

for i in {1..20};do curl test.com:31132/app -H "canary: always"; # 路由至canary版本的goapp2echo -e "";
done

• 效果如下，可以看到可以通过header控制发送请求到不同后端，能够满足需求
  

通过nginx进行转发

• 第二种方法可通过在k8s集群部署一个nginx， 通过nginx进行分流
  • 流量路径如下： ingress-nginx --> nginx --> goapp1或goapp2
• 这里nginx写法有比较多，我选择最简单的通过if判断$http_my_header
• 在使用$http_my_header之前，需要对ingress-nginx和nginx添加参数，允许header中存在下划线
  • ingress-nginx

kubectl edit cm ingress-nginx-controller
------------------
apiVersion: v1
data:allow-snippet-annotations: "true"# 添加下面这两个参数enable-underscores-in-headers: "true"ignore-invalid-headers: "false"
kind: ConfigMap

• 部署nginx，nginx中开启允许header下划线的参数：underscores_in_headers on;

apiVersion: apps/v1
kind: Deployment
metadata:name: nginxnamespace: default
spec:replicas: 1selector:matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:containers:- name: nginximage: nginx:1.24.0ports:- containerPort: 80volumeMounts:- name: nginxmountPath: /etc/nginx/nginx.confsubPath: nginx.confvolumes:- name: nginxconfigMap:name: nginxitems:- key: nginx.confpath: nginx.conf
---
apiVersion: v1
kind: Service
metadata:name: nginxnamespace: default
spec:selector:app: nginxports:- protocol: TCPport: 80targetPort: 80
---
apiVersion: v1
data:nginx.conf: |user  nginx;worker_processes  auto;error_log  /var/log/nginx/error.log notice;pid        /var/run/nginx.pid;events {worker_connections  1024;}http {upstream upstream_server1 {server goapp1:8080;}upstream upstream_server2 {server goapp2:8080;}include       /etc/nginx/mime.types;default_type  application/octet-stream;log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for" "$http_my_header"';access_log  /var/log/nginx/access.log  main;sendfile        on;keepalive_timeout  65;server {underscores_in_headers on; listen 80;server_name test.com;location /app {if ($http_my_header = "value1") {proxy_pass http://upstream_server1;}if ($http_my_header = "value2") {proxy_pass http://upstream_server2;}}}}kind: ConfigMap
metadata:name: nginxnamespace: default

• 上面的配置判断 $http_my_header是 value1 还是 value2，再转发到不同的upstream
• 测试

curl test.com/app -H "my_header:value1"
curl test.com/app -H "my_header:value2"