暂无图片
暂无图片
暂无图片
暂无图片
暂无图片

同一kubernetes部署多个Nginx Ingress Controller

4243


01

写在之前


nginx在反向代理领域占有很大的份额,因为它让你可以快速上手,并且配置相对简单。目前也有不少的中小型企业会针对传统的http服务,使用nginx做一个简单的API网关(有的说法叫接入层,都属于一类东西),访问的流量都经过此nginx,在此基础上进行一些日志收集、访问控制waf、限流、隔离、traceID生成等等,并且根据公司服务安全级别不同、应用是否对外提供服务、对外提供的是80还是443、是web网站还是app服务、是线上环境还是测试环境等等,通常情况下,为了分类别进行管理,会存在多套nginx,每套只负责专一类型的服务接口,例如内网Web API网关、外网Web API网关、APP专用入口等;


要满足类似上面的需求,在kubernetes集群中应该如何实现呢?Nginx Ingress Controller的使用默认安装方式是行不通的,通过查看配置,可以发现nginx ingress controller 提供了ingress标识,即INGRESS_CLASS:Ingress 标识,需要注意的是不能使用nginx这个字符串,因为已经预留给集群默认IngressController服务。INGRESS_CLASS标识不属于任何名称空间(namespace),它属于集群资源,但ingress规则是有名称空间的,把规则应用到(注册到)NGRESS_CLASS标识指向的nginx ingress controller中。


02

重要概念或标识


ingress规则:定义时需要说明规则属于哪一个名称空间中,属于名称空间资源,ingress规则关联service的时候,要确保此名称空间有这个serviceName;

ingress_class:不属于任何名称空间,属于集群资源;

Nginx Ingress Controller:是ingress-controller控制器的一种,还有一种比较常用的traefik,它们都是承载ingress功能的Pod,还有一些与之关联的configmap等资源,这些资源是需要定义在一个名称空间下的;


03

架构图




04

安装部署


安装说明

Nginx Ingress Controller安装方式有多种多样,这里采用helm v3,首先下载chart并解压,自行修改values.yaml,然后通过helm template生成yaml,最后再apply,可以查看yaml中的内容、更新镜像、或者对yaml文件进行备份等;


这里只是为了验证,创建两个ingress_class标识,一个是inroute,另一个是outroute,把nginx-ingress都放在同一个名称空间kubeops中;


安装准备

mkdir /data/ingresstest/ && cd /data/ingresstest/
helm pull stable/nginx-ingress # 会下载最新稳定版
tar -zxf nginx-ingress-1.30.0.tgz


配置文件

为inrouter和outroute ingress-class创建两个values.yaml文件。


cd /data/ingresstest/

# 创建两个配置文件如下:

inroute_values.yaml文件配置如下

controller:
  replicaCount: 2
  hostNetwork: true
  ingressClass: 'inroute'
  nodeSelector:
    node-role.kubernetes.io/edge: 'inroute'
  affinity:
    podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - nginx-ingress
            - key: component
              operator: In
              values:
              - controller
          topologyKey: kubernetes.io/hostname
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule

defaultBackend:
  nodeSelector:
    node-role.kubernetes.io/edge: 'inroute'
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule


#

outroute_values.yaml文件配置如下

controller:
  replicaCount: 2
  hostNetwork: true
  ingressClass: 'outroute'
  nodeSelector:
    node-role.kubernetes.io/edge: 'outroute'
  affinity:
    podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - nginx-ingress
            - key: component
              operator: In
              values:
              - controller
          topologyKey: kubernetes.io/hostname
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule

defaultBackend:
  nodeSelector:
    node-role.kubernetes.io/edge: 'outroute'
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule


说明:

hostNetwork:true,使用宿主机的端口,这个也是关键点,我们通过node节点的80、443服务,这样一来就要求不同的ingree-class需要部署在不同的机器上面,供LB来访问;

nodeSelector:由于Pod创建时,不一定创建到哪台宿主机上,提供规划,使用nodeSelector进行选择宿主;

ingressClass:ingress-controller的唯一标识,ingress规则通过注解标识使用哪一个ingress-controller;


生成yaml文件及apply 

利用刚才自定义的values.yaml文件,再结合helm template 生成 nginx-ingress的yaml文件如下

[root@master01 nginx-ingress]# helm template --namespace=kubeops -f ../inroute_values.yaml . >../inroute.yaml
[root@master01 nginx-ingress]# helm template --namespace=kubeops -f ../outroute_values.yaml . >../outroute.yaml


以下这步在helm v2中,RELEASE-NAME会替换,但在helm v3中不会,这个有一个helm的知识点,需要再深入学习下,现在我简单的进行下替换,用于不同的标识

[root@master01 ingresstest]# sed -i 's/RELEASE-NAME/inroute/g' inroute.yaml
[root@master01 ingresstest]# sed -i 's/RELEASE-NAME/outroute/g' outroute.yaml


为node节点打标签

[root@master01 ingresstest]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master01.k8s.vip Ready <none> 46d v1.17.0
master02 Ready <none> 46d v1.17.0
master03 Ready <none> 46d v1.17.0
node01 Ready <none> 46d v1.17.0
[root@master01 ingresstest]# kubectl label nodes master01.k8s.vip node-role.kubernetes.io/edge=outroute
node/master01.k8s.vip labeled
[root@master01 ingresstest]# kubectl label nodes master02 node-role.kubernetes.io/edge=outroute
node/master02 labeled
[root@master01 ingresstest]# kubectl label nodes master03 node-role.kubernetes.io/edge=inroute
node/master03 labeled
[root@master01 ingresstest]# kubectl label nodes node01 node-role.kubernetes.io/edge=inroute
node/node01 labeled
[root@master01 ingresstest]# kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
master01.k8s.vip Ready edge 46d v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master01.k8s.vip,kubernetes.io/os=linux,node-role.kubernetes.io/edge=outroute
master02 Ready edge 46d v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master02,kubernetes.io/os=linux,node-role.kubernetes.io/edge=outroute
master03 Ready edge 46d v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master03,kubernetes.io/os=linux,node-role.kubernetes.io/edge=inroute
node01 Ready edge 46d v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,node-role.kubernetes.io/edge=inroute
[root@master01 ingresstest]#


增:kubectl label nodes <node-name> <label-key>=<label-value> 

删:kubectl label nodes <node-name> <label-key>-

改:kubectl label nodes <node-name> <label-key>=<label-value> --overwrite

查:kubectl get nodes --show-labels  


创建承载ingress-controller的Pod的名称空间

[root@master01 ingresstest]# kubectl create ns kubeops
namespace/kubeops created
[root@master01 ingresstest]#


创建nginx-ingress 

[root@master01 ingresstest]# pwd
/data/ingresstest
[root@master01 ingresstest]#
[root@master01 ingresstest]# ls
inroute_values.yaml inroute.yaml nginx-ingress nginx-ingress-1.30.0.tgz outroute_values.yaml outroute.yaml
[root@master01 ingresstest]# kubectl apply -f inroute.yaml -n kubeops
poddisruptionbudget.policy/inroute-nginx-ingress-controller created
serviceaccount/inroute-nginx-ingress created
serviceaccount/inroute-nginx-ingress-backend created
clusterrole.rbac.authorization.k8s.io/inroute-nginx-ingress created
clusterrolebinding.rbac.authorization.k8s.io/inroute-nginx-ingress created
role.rbac.authorization.k8s.io/inroute-nginx-ingress created
rolebinding.rbac.authorization.k8s.io/inroute-nginx-ingress created
service/inroute-nginx-ingress-controller created
service/inroute-nginx-ingress-default-backend created
deployment.apps/inroute-nginx-ingress-controller created
deployment.apps/inroute-nginx-ingress-default-backend created
[root@master01 ingresstest]#
[root@master01 ingresstest]#
[root@master01 ingresstest]#
[root@master01 ingresstest]# kubectl apply -f outroute.yaml -n kubeops
poddisruptionbudget.policy/outroute-nginx-ingress-controller created
serviceaccount/outroute-nginx-ingress created
serviceaccount/outroute-nginx-ingress-backend created
clusterrole.rbac.authorization.k8s.io/outroute-nginx-ingress created
clusterrolebinding.rbac.authorization.k8s.io/outroute-nginx-ingress created
role.rbac.authorization.k8s.io/outroute-nginx-ingress created
rolebinding.rbac.authorization.k8s.io/outroute-nginx-ingress created
service/outroute-nginx-ingress-controller created
service/outroute-nginx-ingress-default-backend created
deployment.apps/outroute-nginx-ingress-controller created
deployment.apps/outroute-nginx-ingress-default-backend created
[root@master01 ingresstest]#


查看创建结果

[root@master01 ingresstest]# kubectl get pods -n kubeops -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
inroute-nginx-ingress-controller-fcdf4cbc9-6hztr 1/1     Running 0          9m58s 100.73.16.111   node01 <none>           <none>
inroute-nginx-ingress-controller-fcdf4cbc9-xs4pp 1/1     Running 0          3m49s 100.73.16.110   master03 <none>           <none>
inroute-nginx-ingress-default-backend-665848c956-h8h6w 1/1     Running 0          9m58s 172.19.56.4     node01 <none>           <none>
outroute-nginx-ingress-controller-5f9bf984f4-6gvbb 1/1     Running 0          9m47s 100.73.16.108   master01.k8s.vip <none>           <none>
outroute-nginx-ingress-controller-5f9bf984f4-lg5b8 1/1     Running 0          9m47s 100.73.16.109   master02 <none>           <none>
outroute-nginx-ingress-default-backend-f55fdd9cb-hnqch 1/1     Running 0          9m47s 172.19.184.3    master02 <none>           <none>
[root@master01 ingresstest]#


05

不同ingress测试


创建测试demo(deployment、service、ingress)并应用

[root@master01 ingresstest]# cat demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-deployment-nginx
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      run: default-deployment-nginx
  template:
    metadata:
      labels:
        run: default-deployment-nginx
    spec:
      containers:
      - name: default-deployment-nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: default-svc-nginx
  namespace: default
spec:
  selector:
    run: default-deployment-nginx
  type: ClusterIP
  ports:
    - name: nginx-port
      port: 80
      targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: default-ingress-nginx
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "inroute"
    nginx.ingress.kubernetes.io/proxy-body-size: 1000m
    nginx.ingress.kubernetes.io/rewrite-target:
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
  - host: a.k8s.vip
    http:
      paths:
      - path:
        backend:
          serviceName: default-svc-nginx
          servicePort: nginx-port
[root@master01 ingresstest]#
[root@master01 ingresstest]# kubectl apply -f demo.yaml
deployment.apps/default-deployment-nginx created
service/default-svc-nginx created
ingress.extensions/default-ingress-nginx created
[root@master01 ingresstest]#


ingress规则通过annotations注册到了kubernetes.io/ingress.class: "inroute" 上面;


测试

[root@master01 ingresstest]# curl -H "HOST: a.k8s.vip" http://node01/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@master01 ingresstest]# curl -H "HOST: a.k8s.vip" http://master03/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@master01 ingresstest]#
[root@master01 ingresstest]# curl -H "HOST: a.k8s.vip" http://master02/
default backend - 404[root@master01 ingresstest]#
[root@master01 ingresstest]#
[root@master01 ingresstest]# curl -H "HOST: a.k8s.vip" http://master01/
default backend - 404[root@master01 ingresstest]#
[root@master01 ingresstest]#


相同的服务(service)可以通过定义ingress规则,在metadata.annotations 指定不同的kubernetes.io/ingress.class,就会注册到不同的Nginx Ingress Controller中;

一定要注意一点,相同名称空间下ingress规则的名称一定不能相同,如果相同,规则即会被覆盖;


06

总结


Nginx Ingress Controller是通过 ingress-class来区分不同的nginx-ingress,在创建ingress规则中,metadata.annotations 指定不同的kubernetes.io/ingress.class,这样一来就可以向传统使用nginx一样,创建和区别同一类别的服务;


有可能有人疑问,为什么不使用一个默认的kubernetes.io/ingress.class: nginx,明确说明下,是可以的,只是从安全(有些接口不能暴露到外网、有些需要外网访问)、管理(虚拟主机、app服务、不同部门的运维)、可操作、适用性角度来简单说明下。




您的关注是我写作的动力



往期分享


通俗易懂理解Kubernetes核心组件及原理

kubernetes v1.17.x 二进制安装文档 

Harbor镜像仓库高可用配置与使用


文章转载自Linux点滴运维实践,如果涉嫌侵权,请发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。

评论