Kubernetes 集群内部使用kube-dns实现服务发现的功能;
Kubernetes 集群中的应用使用 NodePort 和 LoadBlancer 类型的 Service 可以实现把应用暴露给外部用户使用,NodePort要维护端口信息,适合小规模应用;
Kubernetes提供另一个非常重要的资源对象可用来暴露服务给外部用户,即ingress
Ingress
是外部访问kuberenets集群的一个入口,将外部的请求转发到集群内不同的 Service 上,相当于 nginx、haproxy 等负载均衡代理服务器。
Ingress controller 类似监听器,通过kube-apiserver实时感知service、pod变化,当得到这些变化信息后,Ingress controlle结合 Ingress的配置,更新反向代理负载均衡器,达到服务发现的作用。现在用得较多的是ingress controller包括:traefik和nginx-controller,traefik性能较nginx-controller差,但配置简单;本文具体讨论通过helm安装nginx-controller
步骤1:在应用商店添加栏,添加应用商店ingress-nginx,执行刷新,等待状态变成active,商店url地址:
https://github.com/kubernetes/ingress-nginx.git

点击应用商店栏下的启动,选择ingress-nginx,进入如下页面,在应答下设置controller.service.type(默认是LoadBalancer)为NodePort。
当controller.service.type为LoadBalancer类型,适于购买云厂商托管提供的Kubernetes 集群,会自动分配外网 IP 作为集群的EXTERNAL-IP,否则本地部署的Kubernetes 集群安装后会出现nginx-ingress service的EXTERNAL-IP 一直是<Pending>状态
也可手动指定EXTERNAL-IP,命令如下:
helm install --name nginx-ingress --namespace kube-system --set "rbac.create=true,controller.service.externalIPs[0]=192.168.0.123,controller.service.externalIPs[1]=192.168.0.122,controller.service.externalIPs[2]=192.168.0.121" stable/nginx-ingress
但是因为指定的EXTERNAL-IP是内网 IP,所以不能接入外部网络。
同时为了能使Prometheus能够通过/metrics采集监控数据,ingress-nginx需要添加额外设置如下:

get svc -n ingress-nginx,得到如下的资源
ingress-nginx-controlleringress-nginx-controller-admissioningress-nginx-controller-metrics
从ingress-nginx-controller容器中,访问ingress-nginx-controller-metrics的ClusterIP:9913/metrics
curl 10.102.67.233:9913/metrics# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.# TYPE go_gc_duration_seconds summarygo_gc_duration_seconds{quantile="0"} 6.704e-06go_gc_duration_seconds{quantile="0.25"} 2.4994e-05go_gc_duration_seconds{quantile="0.5"} 3.4973e-05go_gc_duration_seconds{quantile="0.75"} 0.000186418go_gc_duration_seconds{quantile="1"} 0.001107024go_gc_duration_seconds_sum 0.002850017go_gc_duration_seconds_count 20# HELP go_goroutines Number of goroutines that currently exist.# TYPE go_goroutines gaugego_goroutines 97# HELP go_info Information about the Go environment.# TYPE go_info gaugego_info{version="go1.14.2"} 1# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.# TYPE go_memstats_alloc_bytes gaugego_memstats_alloc_bytes 5.687784e+06# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.# TYPE go_memstats_alloc_bytes_total countergo_memstats_alloc_bytes_total 6.4473584e+07.......
在prometheus的configmap中添加job如下,其中targets是通过get svc -n ingress-nginx获取metrics对应的资源名称.namespace.svc.cluster.local:metrics暴露的端口
- job_name: 'ingress-nginx-controller-metrics'static_configs:- targets: ['ingress-nginx-controller-metrics.ingress-nginx.svc.cluster.local:9913']
kubectl delete -f prometheus-configmap.yaml
kubectl create -f prometheus-configmap.yaml
kubectl get svc -n kube-ops
curl -X POST "http://NodeIP:NodePort/-/reload" # 外部能够访问该服务的IP及端口
ingress-nginx chart 的配置参数及默认值如下:
镜像默认地址:quay.io/kubernetes-ingress-controller/nginx-ingress-controller
| 参数 | 描述 | 默认值 |
|---|---|---|
| controller.image.repository | 镜像地址 | |
controller.image.tag | 镜像tag | 0.30.0 |
controller.image.digest | 镜像描述 | "" |
controller.image.pullPolicy | 拉取镜像的策略 | IfNotPresent |
controller.image.runAsUser | User ID | 101 |
controller.containerPort.http | 监听http请求的端口 | 80 |
controller.containerPort.https | 监听https请求的端口 | 443 |
controller.config | nginx的ConfigMap对象 | none |
controller.configAnnotations | {} | |
controller.hostNetwork | 当controller.service.externalIPs被设置且使用 kube-proxy时,不要设置该值为true,否则端口冲突 80 | false |
controller.dnsPolicy | 当hostNetwork=true其值变为ClusterFirstWithHostNet | ClusterFirst |
controller.dnsConfig | pod's dns config 自定义dns配置 | {} |
controller.electionID | election ID用于 status更新 | ingress-controller-leader |
controller.extraEnvs | pods额外设置的环境变量 | {} |
controller.extraContainers | pods额外设置的容器 | {} |
controller.extraVolumeMounts | volumeMounts挂载点 | {} |
controller.extraVolumes | 数据卷 | {} |
controller.extraInitContainers | 初始化容器Containers, app容器启动前的容器 | [] |
controller.healthCheckPath | health检测路径 | /healthz" |
controller.ingressClass | ingress路由类名 | nginx |
controller.maxmindLicenseKey | Maxmind许可证密钥,用于下载GeoLite2数据库。请参阅访问和使用GeoLite2数据库 | "" |
controller.scope.enabled | false | |
controller.scope.namespace | 命名空间 | "" |
controller.extraArgs | 额外的容器启动参数 | {} |
controller.kind | 部署方式Deployment或DaemonSet | Deployment |
controller.annotations | 部署时添加的注释 | {} |
controller.autoscaling.enabled | 为true会创建Horizontal Pod Autoscaler | false |
controller.autoscaling.minReplicas | 自动缩放的最小副本数 | 2 |
controller.autoscaling.maxReplicas | autoscaling是能的情况下设置最大副本数 | 11 |
controller.autoscaling.targetCPUUtilizationPercentage | 触发扩容的cpu利用率 | "50" |
controller.autoscaling.targetMemoryUtilizationPercentage | 触发扩容的内存利用率 | "50" |
controller.hostPort.enabled | 使能端口映射TCP/80及TCP/443 | false |
controller.hostPort.ports.http | controller.hostPort.enabled为true的http端口 | "80" |
controller.hostPort.ports.https | controller.hostPort.enabled为true的https端口 | "443" |
controller.tolerations | node taints to tolerate (requires Kubernetes >=1.6) | [] |
controller.affinity | node/pod affinities (requires Kubernetes >=1.6) | {} |
controller.terminationGracePeriodSeconds | 终止pod的等待时间 | 60 |
controller.minReadySeconds | 允许pod更新时kill掉其他pod的就绪时间 | 0 |
controller.nodeSelector | 标签选择 | {} |
controller.podAnnotations | pod注释 | {} |
controller.podLabels | 元素据标签 | {} |
controller.podSecurityContext | Security context 策略 | {} |
controller.replicaCount | 副本数 | 1 |
controller.minAvailable | 分布式场景下最小可用数 | 1 |
controller.resources | resource管理requests & limits | {} |
controller.priorityClassName | controller priorityClassName | nil |
controller.lifecycle | 钩子 | {} |
controller.service.annotations | service注释 | {} |
controller.service.labels | service标签 | {} |
controller.publishService.enabled | false | |
controller.publishService.pathOverride | 重写默认的publish-service名称 | "" |
controller.service.enabled | disabled情况下service不会被创建,在controller.kind为 DaemonSet且 controller.hostPorts.enabled为 true 时很有用 | true |
controller.service.clusterIP | service IP ( "-"表示传递空) | nil |
controller.service.externalIPs | service的外部IP列表,当 controller.hostNetwork为 true且使用了 kube-proxy会导致端口冲突 80 | [] |
controller.service.externalTrafficPolicy | controller.service.type为 NodePortor LoadBalancer, 当设置为 Local使能 source IP preservation | "Cluster" |
controller.service.sessionAffinity | 需确保 ClientIP或 None | "" |
controller.service.healthCheckNodePort | controller.service.type为 NodePort或 LoadBalancer和 controller.service.externalTrafficPolicy设置为 Local, 将暴露该健康监测端口. 如果为空, 会随机生成port | "" |
controller.service.loadBalancerIP | 负载均衡器的IP | "" |
controller.service.loadBalancerSourceRanges | 允许访问负载均衡器IP CIDRs列表 | [] |
controller.service.enableHttp | 80 | true |
controller.service.enableHttps | 443 | true |
controller.service.targetPorts.http | 设置targetPort映射到Ingress的80端口 | 80 |
controller.service.targetPorts.https | 设置targetPort映射到Ingress的端443口 | 443 |
controller.service.ports.http | 设置service的http端口 | 80 |
controller.service.ports.https | 设置service的https端口 | 443 |
controller.service.type | service创建的类型 | LoadBalancer |
controller.service.nodePorts.http | 设置nodePort通过http映射到Ingress的80端口 | "" |
controller.service.nodePorts.https | 设置nodePort通过https映射到Ingress的80端口 | "" |
controller.service.nodePorts.tcp | 设置nodePort通过tcp映射到Ingress的80端口 | {} |
controller.service.nodePorts.udp | 设置nodePort通过udp映射到Ingress的80端口 | {} |
controller.livenessProbe.initialDelaySeconds | liveness probe初始化延迟时间 | 10 |
controller.livenessProbe.periodSeconds | 隔多久执行一次探活 | 10 |
controller.livenessProbe.timeoutSeconds | 探活超时时间 | 5 |
controller.livenessProbe.successThreshold | 探活失败后被认为探活成功的最少连续成功次数 | 1 |
controller.livenessProbe.failureThreshold | 探活成功后被认为探活失败的最少连续失败次数 | 3 |
controller.livenessProbe.port | liveness probe监听的端口 | 10254 |
controller.readinessProbe.initialDelaySeconds | readiness probe首次等待时间 | 10 |
controller.readinessProbe.periodSeconds | 隔多久执行一次可读性检测 | 10 |
controller.readinessProbe.timeoutSeconds | 可读性检测超时时间 | 1 |
controller.readinessProbe.successThreshold | 可读性失败后被认为可读性成功的最少连续成功次数 | 1 |
controller.readinessProbe.failureThreshold | 可读性成功后被认为可读性失败的最少连续失败次数 | 3 |
controller.readinessProbe.port | readiness probe监听的端口号 | 10254 |
controller.metrics.enabled | true使能Prometheus metrics | false |
controller.metrics.service.annotations | Prometheus metrics service注释 | {} |
controller.metrics.service.clusterIP | Prometheus metrics service的cluster IP | nil |
controller.metrics.service.externalIPs | Prometheus metrics service 的external IP | [] |
controller.metrics.service.labels | metrics service标签 | {} |
controller.metrics.service.loadBalancerIP | 指定负载均衡的IP地址 | "" |
controller.metrics.service.loadBalancerSourceRanges | 允许访问负载均衡的IP CIDRs列表 | [] |
controller.metrics.service.servicePort | Prometheus metrics service 端口 | 9913 |
controller.metrics.service.type | Prometheus metrics service类型 | ClusterIP |
controller.metrics.serviceMonitor.enabled | true 创建 ServiceMonitor用于Prometheus operator | false |
controller.metrics.serviceMonitor.additionalLabels | 用于发现ServiceMonitor的标签 | {} |
controller.metrics.serviceMonitor.honorLabels | false | false |
controller.metrics.serviceMonitor.namespace | servicemonitor resource的命名空间 | nginx ingress同名空间 |
controller.metrics.serviceMonitor.namespaceSelector | namespaceSelector指定scrape的namespaces | helm release namespace |
controller.metrics.serviceMonitor.scrapeInterval | Prometheus scraping的间隔 | 30s |
controller.metrics.prometheusRule.enabled | true创建prometheusRules用于Prometheus operator | false |
controller.metrics.prometheusRule.additionalLabels | 用于发现prometheusRules的标签 | {} |
controller.metrics.prometheusRule.namespace | prometheusRules resource的命名空间 | nginxingress同命名空间 |
controller.metrics.prometheusRule.rules | prometheus中的rulesYAML格式 | [] |
controller.admissionWebhooks.enabled | 创建Ingress admission webhooks. 验证ingress 语法 | true |
controller.admissionWebhooks.failurePolicy | admission webhooks失败策略 | Fail |
controller.admissionWebhooks.port | Admission webhook端口 | 8443 |
controller.admissionWebhooks.service.annotations | admission webhook service注解 | {} |
controller.admissionWebhooks.service.clusterIP | admission webhook service 的cluster IP | nil |
controller.admissionWebhooks.service.externalIPs | admission webhook service 的外部IP | [] |
controller.zadmissionWebhooks.service.loadBalancerIP | 指定负载均衡的IP地址 | "" |
controller.admissionWebhooks.service.loadBalancerSourceRanges | 允许访问负载均衡的IP CIDRs列表 | [] |
controller.admissionWebhooks.service.servicePort | Admission webhook service 端口 | 443 |
controller.admissionWebhooks.service.type | admission webhook service类型 | ClusterIP |
controller.admissionWebhooks.patch.enabled | patch使能 | true |
controller.admissionWebhooks.patch.image.repository | webhook integration jobs镜像地址 | jettech/kube-webhook-certgen |
controller.admissionWebhooks.patch.image.tag | webhook integration jobs的镜像tag | v1.2.0 |
controller.admissionWebhooks.patch.image.digest | webhook integration jobs镜像描述 | "" |
controller.admissionWebhooks.patch.image.pullPolicy | webhook integration jobs镜像的策略 | IfNotPresent |
controller.admissionWebhooks.patch.priorityClassName | Priority class for the webhook integration jobs | "" |
controller.admissionWebhooks.patch.podAnnotations | webhook job pods注释 | {} |
controller.admissionWebhooks.patch.nodeSelector | 运行admission hook patch jobs的节点选择器 | {} |
controller.admissionWebhooks.patch.tolerations | Node taints/tolerations for running admission hook patch jobs | [] |
controller.customTemplate.configMapName | configMap containing a custom nginx template | "" |
controller.customTemplate.configMapKey | configMap key携带nginx模板 | "" |
controller.addHeaders | configMap key:value对携带响应客户端的头部信息 | {} |
controller.proxySetHeaders | configMap key:value对携带请求后端服务的头部信息 | {} |
controller.updateStrategy | 允许设置RollingUpdate策略 | {} |
controller.configMapNamespace | nginx-configmap命名空间 | "" |
controller.tcp.configMapNamespace | tcp-services-configmap命名空间 | "" |
controller.tcp.annotations | tcp configmap | {} |
controller.udp.configMapNamespace | udp-services-configmap命名空间 | "" |
controller.udp.annotations | udp configmap注释 | {} |
defaultBackend.enabled | 使用default backend组件 | false |
defaultBackend.image.repository | 镜像地址 | k8s.gcr.io/defaultbackend-amd64 |
defaultBackend.image.tag | 镜像tag | 1.5 |
defaultBackend.image.digest | 镜像描述 | "" |
defaultBackend.image.pullPolicy | 拉取镜像的策略 | IfNotPresent |
defaultBackend.image.runAsUser | User ID,默认使用nobody | 65534 |
defaultBackend.extraArgs | 额外参数 | {} |
defaultBackend.extraEnvs | 额外环境变量 | [] |
defaultBackend.port | Http port | 8080 |
defaultBackend.livenessProbe.initialDelaySeconds | liveness probe初始化延迟时间 | 30 |
defaultBackend.livenessProbe.periodSeconds | 隔多久执行一次探活 | 10 |
defaultBackend.livenessProbe.timeoutSeconds | 探活超时时间 | 5 |
defaultBackend.livenessProbe.successThreshold | 探活失败后被认为探活成功的最少连续成功次数 | 1 |
defaultBackend.livenessProbe.failureThreshold | 探活成功后被认为探活失败的最少连续失败次数 | 3 |
defaultBackend.readinessProbe.initialDelaySeconds | readiness probe首次等待时间 | 0 |
defaultBackend.readinessProbe.periodSeconds | 隔多久执行一次可读性检测 | 5 |
defaultBackend.readinessProbe.timeoutSeconds | 可读性检测超时时间 | 5 |
defaultBackend.readinessProbe.successThreshold | 可读性失败后被认为可读性成功的最少连续成功次数 | 1 |
defaultBackend.readinessProbe.failureThreshold | 可读性成功后被认为可读性失败的最少连续失败次数 | 6 |
defaultBackend.tolerations | [] | |
defaultBackend.affinity | node/pod affinities (requires Kubernetes >=1.6) | {} |
defaultBackend.nodeSelector | 节点标签 | {} |
defaultBackend.podAnnotations | 注释 | {} |
defaultBackend.podLabels | 标签 | {} |
defaultBackend.replicaCount | 默认副本数 | 1 |
defaultBackend.minAvailable | 最小可用pod | 1 |
defaultBackend.resources | default backend pod resource requests & limits | {} |
defaultBackend.priorityClassName | default backend priorityClassName | nil |
defaultBackend.podSecurityContext | 默认添加到后端的Security context policies | {} |
defaultBackend.service.annotations | 默认的后端服务注释 | {} |
defaultBackend.service.clusterIP | 后端service的cluster IP | nil |
defaultBackend.service.externalIPs | 默认后端服务的外部IP列表 | [] |
defaultBackend.service.loadBalancerIP | 指定负载均衡的IP地址 | "" |
defaultBackend.service.loadBalancerSourceRanges | 允许访问负载均衡的IP CIDRs列表 | [] |
defaultBackend.service.type | 默认的后端服务创建类型 | ClusterIP |
defaultBackend.serviceAccount.create | true, 创建后端服务账号名称. 只有让pod security policy 运行在后端才有效 | true |
defaultBackend.serviceAccount.name | 后端服务账号名称 | |
imagePullSecrets | nil | |
rbac.create | true, 创建并使用RBAC resources | true |
rbac.scope | itrue, 不要创建使用clusterrole and -binding 结合controller.scope.enabled = true设置为true可以禁用负载均衡器状态更新并完全限制入口范围 | false |
podSecurityPolicy.enabled | true, 创建并使用Pod Security Policy 资源 | false |
serviceAccount.create | true, 创建 service account | true |
serviceAccount.name | service 账号 | |
revisionHistoryLimit | 维护的允许回滚的历史版本数 | 10 |
安装后生成3个pod,ingress-nginx-controller-admission、ingress-nginx-controller及ingress-nginx-controller-metrics,对应yaml文件分别如下,本文未详细给出rbac相关资源对应yaml
ingress-nginx-controller-admission文件:
apiVersion: v1kind: Servicemetadata:creationTimestamp: nulllabels:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/managed-by: Tillerapp.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: 0.32.0helm.sh/chart: ingress-nginx-2.3.0io.cattle.field/appId: ingress-nginxmanagedFields:- apiVersion: v1fieldsType: FieldsV1fieldsV1:f:metadata:f:labels:.: {}f:app.kubernetes.io/component: {}f:app.kubernetes.io/instance: {}f:app.kubernetes.io/managed-by: {}f:app.kubernetes.io/name: {}f:app.kubernetes.io/version: {}f:helm.sh/chart: {}f:io.cattle.field/appId: {}f:spec:f:ports:.: {}k:{"port":443,"protocol":"TCP"}:.: {}f:name: {}f:port: {}f:protocol: {}f:targetPort: {}f:selector:.: {}f:app.kubernetes.io/component: {}f:app.kubernetes.io/instance: {}f:app.kubernetes.io/name: {}f:sessionAffinity: {}f:type: {}manager: Go-http-clientoperation: Updatetime: "2020-05-23T06:28:02Z"name: ingress-nginx-controller-admissionselfLink: /api/v1/namespaces/ingress-nginx/services/ingress-nginx-controller-admissionspec:ports:- name: https-webhookport: 443protocol: TCPtargetPort: webhookselector:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/name: ingress-nginxsessionAffinity: Nonetype: ClusterIPstatus:loadBalancer: {}
ingress-nginx-controller-metrics.yaml
apiVersion: v1kind: Servicemetadata:creationTimestamp: nulllabels:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/managed-by: Tillerapp.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: 0.32.0helm.sh/chart: ingress-nginx-2.3.0io.cattle.field/appId: ingress-nginxmanagedFields:- apiVersion: v1fieldsType: FieldsV1fieldsV1:f:metadata:f:labels:.: {}f:app.kubernetes.io/component: {}f:app.kubernetes.io/instance: {}f:app.kubernetes.io/managed-by: {}f:app.kubernetes.io/name: {}f:app.kubernetes.io/version: {}f:helm.sh/chart: {}f:io.cattle.field/appId: {}f:spec:f:ports:.: {}k:{"port":9913,"protocol":"TCP"}:.: {}f:name: {}f:port: {}f:protocol: {}f:targetPort: {}f:selector:.: {}f:app.kubernetes.io/component: {}f:app.kubernetes.io/instance: {}f:app.kubernetes.io/name: {}f:sessionAffinity: {}f:type: {}manager: Go-http-clientoperation: Updatetime: "2020-05-24T02:27:16Z"name: ingress-nginx-controller-metricsselfLink: /api/v1/namespaces/ingress-nginx/services/ingress-nginx-controller-metricsspec:ports:- name: metricsport: 9913protocol: TCPtargetPort: metricsselector:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/name: ingress-nginxsessionAffinity: Nonetype: ClusterIPstatus:loadBalancer: {}
ingress-nginx-controller.yaml文件:
apiVersion: v1kind: Servicemetadata:annotations:field.cattle.io/publicEndpoints: '[{"port":30390,"protocol":"TCP","serviceName":"ingress-nginx:ingress-nginx-controller","allNodes":true},{"port":31792,"protocol":"TCP","serviceName":"ingress-nginx:ingress-nginx-controller","allNodes":true}]'creationTimestamp: nulllabels:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/managed-by: Tillerapp.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: 0.32.0helm.sh/chart: ingress-nginx-2.3.0io.cattle.field/appId: ingress-nginxmanagedFields:- apiVersion: v1fieldsType: FieldsV1fieldsV1:f:metadata:f:annotations:.: {}f:field.cattle.io/publicEndpoints: {}f:labels:.: {}f:app.kubernetes.io/component: {}f:app.kubernetes.io/instance: {}f:app.kubernetes.io/managed-by: {}f:app.kubernetes.io/name: {}f:app.kubernetes.io/version: {}f:helm.sh/chart: {}f:io.cattle.field/appId: {}f:spec:f:externalTrafficPolicy: {}f:ports:.: {}k:{"port":80,"protocol":"TCP"}:.: {}f:name: {}f:port: {}f:protocol: {}f:targetPort: {}k:{"port":443,"protocol":"TCP"}:.: {}f:name: {}f:port: {}f:protocol: {}f:targetPort: {}f:selector:.: {}f:app.kubernetes.io/component: {}f:app.kubernetes.io/instance: {}f:app.kubernetes.io/name: {}f:sessionAffinity: {}f:type: {}manager: Go-http-clientoperation: Updatetime: "2020-05-23T07:32:13Z"name: ingress-nginx-controllerselfLink: /api/v1/namespaces/ingress-nginx/services/ingress-nginx-controllerspec:externalTrafficPolicy: Clusterports:- name: httpport: 80protocol: TCPtargetPort: http- name: httpsport: 443protocol: TCPtargetPort: httpsselector:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/name: ingress-nginxsessionAffinity: Nonetype: NodePortstatus:loadBalancer: {}




