文章目录
- 环境
- 部署ingress
- 可以访问 registry.k8s.io 的环境
- 不能访问 registry.k8s.io 的环境
- 手工部署
- 总结
- 测试ingress
- 参考
环境
- RHEL 9.3
- Docker Community 24.0.7
- minikube v1.32.0
部署ingress
可以访问 registry.k8s.io 的环境
通过minikube启用 ingress ：
minikube addons enable ingress
注：接下来在不能访问 registry.k8s.io 的环境里，用该环境作为“标准环境”，以便做对比。
不能访问 registry.k8s.io 的环境
通过minikube启用 ingress 会报错：
$ minikube addons enable ingress
💡 ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS▪ Using image registry.k8s.io/ingress-nginx/controller:v1.9.4▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
🔎 Verifying ingress addon...❌ Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: context deadline exceeded]╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ Please also attach the following file to the GitHub issue: │
│ - /tmp/minikube_addons_bbcace35d8e0350ae758b20ff8fff60714ee7a80_0.log │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
ingress使用的是 ingress-nginx namespace：
$ kubectl get namespace
NAME STATUS AGE
default Active 42h
ingress-nginx Active 31h
kube-node-lease Active 42h
kube-public Active 42h
kube-system Active 42h
kubernetes-dashboard Active 41h
查看 ingress-nginx namespace：
$ kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-274rs 0/1 ImagePullBackOff 0 51s
pod/ingress-nginx-admission-patch-tg887 0/1 ContainerCreating 0 51s
pod/ingress-nginx-controller-7c6974c4d8-lq5q4 0/1 ContainerCreating 0 51sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.111.104.1 <none> 80:31322/TCP,443:31156/TCP 51s
service/ingress-nginx-controller-admission ClusterIP 10.100.55.232 <none> 443/TCP 51sNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 0/1 1 0 51sNAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-7c6974c4d8 1 1 0 51sNAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 0/1 51s 51s
job.batch/ingress-nginx-admission-patch 0/1 51s 51s
查看deployment：
$ kubectl describe deployment ingress-nginx-controller -n ingress-nginx
......Image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
......Volumes:webhook-cert:Type: Secret (a volume populated by a Secret)SecretName: ingress-nginx-admissionOptional: false
......
NewReplicaSet: ingress-nginx-controller-7c6974c4d8 (1/1 replicas created)
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal ScalingReplicaSet 3m22s deployment-controller Scaled up replica set ingress-nginx-controller-7c6974c4d8 to 1
$ kubectl describe replicaset ingress-nginx-controller-7c6974c4d8 -n ingress-nginx
......
Controlled By: Deployment/ingress-nginx-controller
......Containers:controller:Image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
......Volumes:webhook-cert:Type: Secret (a volume populated by a Secret)SecretName: ingress-nginx-admissionOptional: false
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal SuccessfulCreate 11m replicaset-controller Created pod: ingress-nginx-controller-7c6974c4d8-lq5q4
$ kubectl describe pod ingress-nginx-controller-7c6974c4d8-lq5q4 -n ingress-nginx
......
Controlled By: ReplicaSet/ingress-nginx-controller-7c6974c4d8
Containers:controller:Container ID: Image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
......
Volumes:webhook-cert:Type: Secret (a volume populated by a Secret)SecretName: ingress-nginx-admissionOptional: false
......
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Scheduled 13m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-7c6974c4d8-lq5q4 to minikubeWarning FailedMount 65s (x14 over 13m) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
查看job：
$ kubectl describe job ingress-nginx-admission-create -n ingress-nginx
......Containers:create:Image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
......
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal SuccessfulCreate 22m job-controller Created pod: ingress-nginx-admission-create-274rs
$ kubectl describe pod ingress-nginx-admission-create-274rs -n ingress-nginx
......
Controlled By: Job/ingress-nginx-admission-create
Containers:create:Container ID: Image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
......
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Scheduled 16m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-admission-create-274rs to minikubeNormal Pulling 12m (x4 over 16m) kubelet Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"Warning Failed 12m (x4 over 15m) kubelet Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": dial tcp 108.177.97.82:443: i/o timeoutWarning Failed 12m (x4 over 15m) kubelet Error: ErrImagePullWarning Failed 11m (x6 over 15m) kubelet Error: ImagePullBackOffNormal BackOff 56s (x44 over 15m) kubelet Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"
$ kubectl describe job ingress-nginx-admission-patch -n ingress-nginx
......Containers:patch:Image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
......
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal SuccessfulCreate 25m job-controller Created pod: ingress-nginx-admission-patch-tg887
$ kubectl describe pod ingress-nginx-admission-patch-tg887 -n ingress-nginx
......
Controlled By: Job/ingress-nginx-admission-patch
Containers:patch:Container ID: Image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
......
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Scheduled 17m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-admission-patch-tg887 to minikubeNormal Pulling 14m (x4 over 17m) kubelet Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"Warning Failed 13m (x4 over 16m) kubelet Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": dial tcp 108.177.97.82:443: i/o timeoutWarning Failed 13m (x4 over 16m) kubelet Error: ErrImagePullWarning Failed 12m (x7 over 16m) kubelet Error: ImagePullBackOffNormal BackOff 7m43s (x23 over 16m) kubelet Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"Warning Failed 2m32s kubelet Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": dial tcp 142.251.8.82:443: i/o timeout
所以，归根到底，还是 registry.k8s.io 的网络问题。从 minikube addons enable ingress 的输出可知，使用到的image是：
registry.k8s.io/ingress-nginx/controller:v1.9.4registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
注：从上述 kubectl describe xxx 命令也可以看出，是这两个image pull不下来。
所以，接下来我们用手工方式来部署ingress。
手工部署
在可以访问 registry.k8s.io 的地方，找到上述两个image，pull下来，tag一下，push到DockerHub自己的repository里。
比如：我的image是：
kaiding1/controller:v1.9.4kaiding1/kube-webhook-certgen:v20231011-8b53cabe0
打开浏览器，访问 https://github.com/kubernetes/ingress-nginx ，其中“Supported Versions table”如下：
查看Kubernetes版本：
$ kubectl version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3
由对照表可知，k8s v1.28.3支持的Ingress-Nginx版本有v1.9.0到v1.9.5。我们就用v1.9.4，和 registry.k8s.io/ingress-nginx/controller:v1.9.4 保持一致（也和“标准环境”保持一致）。
把 https://github.com/kubernetes/ingress-nginx/blob/controller-v1.9.4/deploy/static/provider/kind/deploy.yaml 下载到本地。
注：有kind、cloud、baremetal等多个版本，我比较了一下，kind里的版本最接近“标准环境”。
然后修改 deploy.yaml ：
- 把：
image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
替换为：
image: kaiding1/controller:v1.9.4
- 把：
image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
替换为：
image: kaiding1/kube-webhook-certgen:v20231011-8b53cabe0
注意有两处 kube-webhook-certgen 要替换。
注：删掉了 @sha256:xxxxx ，因为跟原来的digest不一样，如果想加上，需要使用新的digest。
部署该yaml文件：
$ kubectl apply -f deploy.yaml
namespace/ingress-nginx unchanged
serviceaccount/ingress-nginx configured
serviceaccount/ingress-nginx-admission configured
role.rbac.authorization.k8s.io/ingress-nginx configured
role.rbac.authorization.k8s.io/ingress-nginx-admission configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission configured
rolebinding.rbac.authorization.k8s.io/ingress-nginx configured
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission configured
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx configured
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission configured
configmap/ingress-nginx-controller configured
service/ingress-nginx-controller configured
service/ingress-nginx-controller-admission configured
deployment.apps/ingress-nginx-controller configured
ingressclass.networking.k8s.io/nginx configured
networkpolicy.networking.k8s.io/ingress-nginx-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/component\":\"admission-webhook\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"1.9.4\"},\"name\":\"ingress-nginx-admission-create\",\"namespace\":\"ingress-nginx\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"admission-webhook\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"1.9.4\"},\"name\":\"ingress-nginx-admission-create\"},\"spec\":{\"containers\":[{\"args\":[\"create\",\"--host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc\",\"--namespace=$(POD_NAMESPACE)\",\"--secret-name=ingress-nginx-admission\"],\"env\":[{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}}],\"image\":\"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"create\",\"securityContext\":{\"allowPrivilegeEscalation\":false}}],\"nodeSelector\":{\"kubernetes.io/os\":\"linux\"},\"restartPolicy\":\"OnFailure\",\"securityContext\":{\"fsGroup\":2000,\"runAsNonRoot\":true,\"runAsUser\":2000},\"serviceAccountName\":\"ingress-nginx-admission\"}}}}\n"},"labels":{"app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.4"}},"spec":{"template":{"metadata":{"labels":{"app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.4"}},"spec":{"$setElementOrder/containers":[{"name":"create"}],"containers":[{"image":"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0","name":"create"}],"nodeSelector":{"minikube.k8s.io/primary":null},"securityContext":{"fsGroup":2000}}}}}
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "ingress-nginx-admission-create", Namespace: "ingress-nginx"
for: "deploy.yaml": error when patching "deploy.yaml": Job.batch "ingress-nginx-admission-create" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"ingress-nginx-admission-create", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app.kubernetes.io/component":"admission-webhook", "app.kubernetes.io/instance":"ingress-nginx", "app.kubernetes.io/name":"ingress-nginx", "app.kubernetes.io/part-of":"ingress-nginx", "app.kubernetes.io/version":"1.9.4", "batch.kubernetes.io/controller-uid":"4dc1f43f-04d8-4e43-9469-addcd7a96f8c", "batch.kubernetes.io/job-name":"ingress-nginx-admission-create", "controller-uid":"4dc1f43f-04d8-4e43-9469-addcd7a96f8c", "job-name":"ingress-nginx-admission-create"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"create", Image:"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0", Command:[]string(nil), Args:[]string{"create", "--host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc", "--namespace=$(POD_NAMESPACE)", "--secret-name=ingress-nginx-admission"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"POD_NAMESPACE", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00b2e2720)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(0xc00b40c2a0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc00d6fdd40), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"kubernetes.io/os":"linux"}, ServiceAccountName:"ingress-nginx-admission", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc00b52e480), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/component\":\"admission-webhook\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"1.9.4\"},\"name\":\"ingress-nginx-admission-patch\",\"namespace\":\"ingress-nginx\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"admission-webhook\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"1.9.4\"},\"name\":\"ingress-nginx-admission-patch\"},\"spec\":{\"containers\":[{\"args\":[\"patch\",\"--webhook-name=ingress-nginx-admission\",\"--namespace=$(POD_NAMESPACE)\",\"--patch-mutating=false\",\"--secret-name=ingress-nginx-admission\",\"--patch-failure-policy=Fail\"],\"env\":[{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}}],\"image\":\"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"patch\",\"securityContext\":{\"allowPrivilegeEscalation\":false}}],\"nodeSelector\":{\"kubernetes.io/os\":\"linux\"},\"restartPolicy\":\"OnFailure\",\"securityContext\":{\"fsGroup\":2000,\"runAsNonRoot\":true,\"runAsUser\":2000},\"serviceAccountName\":\"ingress-nginx-admission\"}}}}\n"},"labels":{"app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.4"}},"spec":{"template":{"metadata":{"labels":{"app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.4"}},"spec":{"$setElementOrder/containers":[{"name":"patch"}],"containers":[{"image":"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0","name":"patch"}],"nodeSelector":{"minikube.k8s.io/primary":null},"securityContext":{"fsGroup":2000}}}}}
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "ingress-nginx-admission-patch", Namespace: "ingress-nginx"
for: "deploy.yaml": error when patching "deploy.yaml": Job.batch "ingress-nginx-admission-patch" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"ingress-nginx-admission-patch", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app.kubernetes.io/component":"admission-webhook", "app.kubernetes.io/instance":"ingress-nginx", "app.kubernetes.io/name":"ingress-nginx", "app.kubernetes.io/part-of":"ingress-nginx", "app.kubernetes.io/version":"1.9.4", "batch.kubernetes.io/controller-uid":"1033bcb2-b2f4-4a6a-b43c-15ad9351d6ec", "batch.kubernetes.io/job-name":"ingress-nginx-admission-patch", "controller-uid":"1033bcb2-b2f4-4a6a-b43c-15ad9351d6ec", "job-name":"ingress-nginx-admission-patch"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"patch", Image:"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0", Command:[]string(nil), Args:[]string{"patch", "--webhook-name=ingress-nginx-admission", "--namespace=$(POD_NAMESPACE)", "--patch-mutating=false", "--secret-name=ingress-nginx-admission", "--patch-failure-policy=Fail"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"POD_NAMESPACE", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00bb53f80)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(0xc00bb5d440), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc00bb82c00), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"kubernetes.io/os":"linux"}, ServiceAccountName:"ingress-nginx-admission", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc00bb75d40), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable
有些配置跟原来已有的配置冲突，为了简单以前，把原来的namespace整个删掉：
kubectl delete namespace ingress-nginx
所有的deployment、pod都资源都没了，然后再重新部署：
$ kubectl apply -f deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx unchanged
networkpolicy.networking.k8s.io/ingress-nginx-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
再次查看资源：
$ kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-sbrf8 0/1 Completed 0 40s
pod/ingress-nginx-admission-patch-ftgxj 0/1 Completed 1 40s
pod/ingress-nginx-controller-6d5bdfb648-wkrdr 0/1 Pending 0 40sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.103.85.240 <none> 80:30507/TCP,443:32553/TCP 40s
service/ingress-nginx-controller-admission ClusterIP 10.96.189.167 <none> 443/TCP 40sNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 0/1 1 0 40sNAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-6d5bdfb648 1 1 0 40sNAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 4s 40s
job.batch/ingress-nginx-admission-patch 1/1 5s 40s
可见， /ingress-nginx-controller 的pod没有启起来，处于“Pending”状态。
查看该pod：
$ kubectl describe pod ingress-nginx-controller-6d5bdfb648-wkrdr -n ingress-nginx
......
Events:Type Reason Age From Message---- ------ ---- ---- -------Warning FailedScheduling 97s default-scheduler 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
和“标准环境”的 ingress-nginx-controller deployment做对比，发现需要注释掉这一行：
apiVersion: apps/v1
kind: Deployment
......nodeSelector:# ingress-ready: "true" # 把这一行注释掉
还有一些别的不同之处，但是貌似不影响，只有这一行是有影响的。
然后再次删掉 ingress-nginx namespace重建：
kubectl delete namespace ingress-nginxkubectl apply -f deploy.yaml
查看资源：
$ kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-pk7jj 0/1 Completed 0 13s
pod/ingress-nginx-admission-patch-2twvg 0/1 Completed 0 13s
pod/ingress-nginx-controller-68db9c8cbf-zvsmr 1/1 Running 0 13sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.101.105.112 <none> 80:31854/TCP,443:31461/TCP 13s
service/ingress-nginx-controller-admission ClusterIP 10.98.77.135 <none> 443/TCP 13sNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 13sNAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-68db9c8cbf 1 1 1 13sNAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 4s 13s
job.batch/ingress-nginx-admission-patch 1/1 4s 13s
现在所有资源都OK了。
总结
deploy.yaml 需要修改之处：
image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3：替换为可访问的imageimage: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80：替换为可访问的image，注意有两处ingress-ready: "true"：注释掉这一行
测试ingress
创建 ingress-example.yaml 文件如下：
kind: Pod
apiVersion: v1
metadata:name: foo-applabels:app: foo
spec:containers:- name: foo-appimage: 'kicbase/echo-server:1.0'
---
kind: Service
apiVersion: v1
metadata:name: foo-service
spec:selector:app: fooports:- port: 8080
---
kind: Pod
apiVersion: v1
metadata:name: bar-applabels:app: bar
spec:containers:- name: bar-appimage: 'kicbase/echo-server:1.0'
---
kind: Service
apiVersion: v1
metadata:name: bar-service
spec:selector:app: barports:- port: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: example-ingress
spec:rules:- http:paths:- pathType: Prefixpath: /foobackend:service:name: foo-serviceport:number: 8080- pathType: Prefixpath: /barbackend:service:name: bar-serviceport:number: 8080
---
部署：
$ kubectl apply -f ingress-example.yaml
pod/foo-app created
service/foo-service created
pod/bar-app created
service/bar-service created
ingress.networking.k8s.io/example-ingress created
（注意：部署完以后查看一下这些pod，有时会遇到pull不下来的情况，报错是“You have reached your pull rate limit”。解决方法是 docker login 。但是今天好像login了也不行……我没有细看这个问题，直接用老办法，把它pull下来然后push到个人的repository了。）
查看ingress：
$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
example-ingress <none> * localhost 80 39s
但是，不能用localhost作为host：
$ curl http://localhost/foo
curl: (7) Failed to connect to localhost port 80: Connection refused
不知道是不是我哪里配置的有问题。
而要用 192.168.49.2 （注： 192.168.49.2 是minikube的internal IP）：
$ curl http://192.168.49.2/foo
Request served by foo-appHTTP/1.1 GET /fooHost: 192.168.49.2
Accept: */*
User-Agent: curl/7.76.1
X-Forwarded-For: 192.168.49.1
X-Forwarded-Host: 192.168.49.2
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Scheme: http
X-Real-Ip: 192.168.49.1
X-Request-Id: 47a53f8a53335fd2e412c08d52918966
X-Scheme: http
同理：
$ curl http://192.168.49.2/bar
Request served by bar-appHTTP/1.1 GET /barHost: 192.168.49.2
Accept: */*
User-Agent: curl/7.76.1
X-Forwarded-For: 192.168.49.1
X-Forwarded-Host: 192.168.49.2
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Scheme: http
X-Real-Ip: 192.168.49.1
X-Request-Id: c471ed7ba12a154cafe052107d553480
X-Scheme: http
查看 exmaple-ingress ，如下：
$ kubectl describe ingress example-ingress
Name: example-ingress
Labels: <none>
Namespace: default
Address: localhost
Ingress Class: <none>
Default backend: <default>
Rules:Host Path Backends---- ---- --------* /foo foo-service:8080 (10.244.0.34:8080)/bar bar-service:8080 (10.244.0.35:8080)
Annotations: <none>
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Sync 52m (x2 over 53m) nginx-ingress-controller Scheduled for sync
参考
https://minikube.sigs.k8s.io/docs/starthttps://blog.csdn.net/alwaysbefine/article/details/129201448
