暂无图片
暂无图片
暂无图片
暂无图片
暂无图片
首页 / 同一kubernetes部署多个Nginx Ingress Controller

同一kubernetes部署多个Nginx Ingress Controller

Linux点滴运维实践 2021-04-12
4398


01

写在之前


nginx在反向代理领域占有很大的份额，因为它让你可以快速上手，并且配置相对简单。目前也有不少的中小型企业会针对传统的http服务，使用nginx做一个简单的API网关（有的说法叫接入层，都属于一类东西），访问的流量都经过此nginx，在此基础上进行一些日志收集、访问控制waf、限流、隔离、traceID生成等等，并且根据公司服务安全级别不同、应用是否对外提供服务、对外提供的是80还是443、是web网站还是app服务、是线上环境还是测试环境等等，通常情况下，为了分类别进行管理，会存在多套nginx，每套只负责专一类型的服务接口，例如内网Web API网关、外网Web API网关、APP专用入口等；


要满足类似上面的需求，在kubernetes集群中应该如何实现呢？Nginx Ingress Controller的使用默认安装方式是行不通的，通过查看配置，可以发现nginx ingress controller 提供了ingress标识，即INGRESS_CLASS：Ingress 标识，需要注意的是不能使用nginx这个字符串，因为已经预留给集群默认IngressController服务。INGRESS_CLASS标识不属于任何名称空间(namespace)，它属于集群资源，但ingress规则是有名称空间的，把规则应用到（注册到）NGRESS_CLASS标识指向的nginx ingress controller中。


02

重要概念或标识


ingress规则：定义时需要说明规则属于哪一个名称空间中，属于名称空间资源，ingress规则关联service的时候，要确保此名称空间有这个serviceName；

ingress_class：不属于任何名称空间，属于集群资源；

Nginx Ingress Controller：是ingress-controller控制器的一种，还有一种比较常用的traefik，它们都是承载ingress功能的Pod，还有一些与之关联的configmap等资源，这些资源是需要定义在一个名称空间下的；


03

架构图




04

安装部署


安装说明

Nginx Ingress Controller安装方式有多种多样，这里采用helm v3，首先下载chart并解压，自行修改values.yaml，然后通过helm template生成yaml，最后再apply，可以查看yaml中的内容、更新镜像、或者对yaml文件进行备份等；


这里只是为了验证，创建两个ingress_class标识，一个是inroute，另一个是outroute，把nginx-ingress都放在同一个名称空间kubeops中；


安装准备

mkdir /data/ingresstest/ && cd /data/ingresstest/
helm pull stable/nginx-ingress # 会下载最新稳定版
tar -zxf nginx-ingress-1.30.0.tgz


配置文件

为inrouter和outroute ingress-class创建两个values.yaml文件。


cd /data/ingresstest/

# 创建两个配置文件如下：

inroute_values.yaml文件配置如下

controller:
  replicaCount: 2
  hostNetwork: true
  ingressClass: 'inroute'
  nodeSelector:
    node-role.kubernetes.io/edge: 'inroute'
  affinity:
    podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - nginx-ingress
            - key: component
              operator: In
              values:
              - controller
          topologyKey: kubernetes.io/hostname
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule

defaultBackend:
  nodeSelector:
    node-role.kubernetes.io/edge: 'inroute'
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule


#

outroute_values.yaml文件配置如下

controller:
  replicaCount: 2
  hostNetwork: true
  ingressClass: 'outroute'
  nodeSelector:
    node-role.kubernetes.io/edge: 'outroute'
  affinity:
    podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - nginx-ingress
            - key: component
              operator: In
              values:
              - controller
          topologyKey: kubernetes.io/hostname
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule

defaultBackend:
  nodeSelector:
    node-role.kubernetes.io/edge: 'outroute'
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule


说明：

hostNetwork：true，使用宿主机的端口，这个也是关键点，我们通过node节点的80、443服务，这样一来就要求不同的ingree-class需要部署在不同的机器上面，供LB来访问；

nodeSelector：由于Pod创建时，不一定创建到哪台宿主机上，提供规划，使用nodeSelector进行选择宿主；

ingressClass：ingress-controller的唯一标识，ingress规则通过注解标识使用哪一个ingress-controller；


生成yaml文件及apply 

利用刚才自定义的values.yaml文件，再结合helm template 生成 nginx-ingress的yaml文件如下

[root@master01 nginx-ingress]# helm template --namespace=kubeops -f ../inroute_values.yaml . >../inroute.yaml
[root@master01 nginx-ingress]# helm template --namespace=kubeops -f ../outroute_values.yaml . >../outroute.yaml


以下这步在helm v2中，RELEASE-NAME会替换，但在helm v3中不会，这个有一个helm的知识点，需要再深入学习下，现在我简单的进行下替换，用于不同的标识

[root@master01 ingresstest]# sed -i 's/RELEASE-NAME/inroute/g' inroute.yaml
[root@master01 ingresstest]# sed -i 's/RELEASE-NAME/outroute/g' outroute.yaml


为node节点打标签

[root@master01 ingresstest]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master01.k8s.vip Ready <none> 46d v1.17.0
master02 Ready <none> 46d v1.17.0
master03 Ready <none> 46d v1.17.0
node01 Ready <none> 46d v1.17.0
[root@master01 ingresstest]# kubectl label nodes master01.k8s.vip node-role.kubernetes.io/edge=outroute
node/master01.k8s.vip labeled
[root@master01 ingresstest]# kubectl label nodes master02 node-role.kubernetes.io/edge=outroute
node/master02 labeled
[root@master01 ingresstest]# kubectl label nodes master03 node-role.kubernetes.io/edge=inroute
node/master03 labeled
[root@master01 ingresstest]# kubectl label nodes node01 node-role.kubernetes.io/edge=inroute
node/node01 labeled
[root@master01 ingresstest]# kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
master01.k8s.vip Ready edge 46d v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master01.k8s.vip,kubernetes.io/os=linux,node-role.kubernetes.io/edge=outroute
master02 Ready edge 46d v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master02,kubernetes.io/os=linux,node-role.kubernetes.io/edge=outroute
master03 Ready edge 46d v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master03,kubernetes.io/os=linux,node-role.kubernetes.io/edge=inroute
node01 Ready edge 46d v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,node-role.kubernetes.io/edge=inroute
[root@master01 ingresstest]#


增：kubectl label nodes <node-name> <label-key>=<label-value> 

删：kubectl label nodes <node-name> <label-key>-

改：kubectl label nodes <node-name> <label-key>=<label-value> --overwrite

查：kubectl get nodes --show-labels  


创建承载ingress-controller的Pod的名称空间

[root@master01 ingresstest]# kubectl create ns kubeops
namespace/kubeops created
[root@master01 ingresstest]#


创建nginx-ingress 

[root@master01 ingresstest]# pwd
/data/ingresstest
[root@master01 ingresstest]#
[root@master01 ingresstest]# ls
inroute_values.yaml inroute.yaml nginx-ingress nginx-ingress-1.30.0.tgz outroute_values.yaml outroute.yaml
[root@master01 ingresstest]# kubectl apply -f inroute.yaml -n kubeops
poddisruptionbudget.policy/inroute-nginx-ingress-controller created
serviceaccount/inroute-nginx-ingress created
serviceaccount/inroute-nginx-ingress-backend created
clusterrole.rbac.authorization.k8s.io/inroute-nginx-ingress created
clusterrolebinding.rbac.authorization.k8s.io/inroute-nginx-ingress created
role.rbac.authorization.k8s.io/inroute-nginx-ingress created
rolebinding.rbac.authorization.k8s.io/inroute-nginx-ingress created
service/inroute-nginx-ingress-controller created
service/inroute-nginx-ingress-default-backend created
deployment.apps/inroute-nginx-ingress-controller created
deployment.apps/inroute-nginx-ingress-default-backend created
[root@master01 ingresstest]#
[root@master01 ingresstest]#
[root@master01 ingresstest]#
[root@master01 ingresstest]# kubectl apply -f outroute.yaml -n kubeops
poddisruptionbudget.policy/outroute-nginx-ingress-controller created
serviceaccount/outroute-nginx-ingress created
serviceaccount/outroute-nginx-ingress-backend created
clusterrole.rbac.authorization.k8s.io/outroute-nginx-ingress created
clusterrolebinding.rbac.authorization.k8s.io/outroute-nginx-ingress created
role.rbac.authorization.k8s.io/outroute-nginx-ingress created
rolebinding.rbac.authorization.k8s.io/outroute-nginx-ingress created
service/outroute-nginx-ingress-controller created
service/outroute-nginx-ingress-default-backend created
deployment.apps/outroute-nginx-ingress-controller created
deployment.apps/outroute-nginx-ingress-default-backend created
[root@master01 ingresstest]#


查看创建结果

[root@master01 ingresstest]# kubectl get pods -n kubeops -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
inroute-nginx-ingress-controller-fcdf4cbc9-6hztr 1/1     Running 0          9m58s 100.73.16.111   node01 <none>           <none>
inroute-nginx-ingress-controller-fcdf4cbc9-xs4pp 1/1     Running 0          3m49s 100.73.16.110   master03 <none>           <none>
inroute-nginx-ingress-default-backend-665848c956-h8h6w 1/1     Running 0          9m58s 172.19.56.4     node01 <none>           <none>
outroute-nginx-ingress-controller-5f9bf984f4-6gvbb 1/1     Running 0          9m47s 100.73.16.108   master01.k8s.vip <none>           <none>
outroute-nginx-ingress-controller-5f9bf984f4-lg5b8 1/1     Running 0          9m47s 100.73.16.109   master02 <none>           <none>
outroute-nginx-ingress-default-backend-f55fdd9cb-hnqch 1/1     Running 0          9m47s 172.19.184.3    master02 <none>           <none>
[root@master01 ingresstest]#


05

不同ingress测试


创建测试demo（deployment、service、ingress）并应用

[root@master01 ingresstest]# cat demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-deployment-nginx
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      run: default-deployment-nginx
  template:
    metadata:
      labels:
        run: default-deployment-nginx
    spec:
      containers:
      - name: default-deployment-nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: default-svc-nginx
  namespace: default
spec:
  selector:
    run: default-deployment-nginx
  type: ClusterIP
  ports:
    - name: nginx-port
      port: 80
      targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: default-ingress-nginx
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "inroute"
    nginx.ingress.kubernetes.io/proxy-body-size: 1000m
    nginx.ingress.kubernetes.io/rewrite-target: 
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
  - host: a.k8s.vip
    http:
      paths:
      - path: 
        backend:
          serviceName: default-svc-nginx
          servicePort: nginx-port
[root@master01 ingresstest]#
[root@master01 ingresstest]# kubectl apply -f demo.yaml
deployment.apps/default-deployment-nginx created
service/default-svc-nginx created
ingress.extensions/default-ingress-nginx created
[root@master01 ingresstest]#


ingress规则通过annotations注册到了kubernetes.io/ingress.class: "inroute" 上面；


测试

[root@master01 ingresstest]# curl -H "HOST: a.k8s.vip" http://node01/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@master01 ingresstest]# curl -H "HOST: a.k8s.vip" http://master03/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@master01 ingresstest]#
[root@master01 ingresstest]# curl -H "HOST: a.k8s.vip" http://master02/
default backend - 404[root@master01 ingresstest]#
[root@master01 ingresstest]#
[root@master01 ingresstest]# curl -H "HOST: a.k8s.vip" http://master01/
default backend - 404[root@master01 ingresstest]#
[root@master01 ingresstest]#


相同的服务（service）可以通过定义ingress规则，在metadata.annotations 指定不同的kubernetes.io/ingress.class，就会注册到不同的Nginx Ingress Controller中；

一定要注意一点，相同名称空间下ingress规则的名称一定不能相同，如果相同，规则即会被覆盖；


06

总结


Nginx Ingress Controller是通过 ingress-class来区分不同的nginx-ingress，在创建ingress规则中，metadata.annotations 指定不同的kubernetes.io/ingress.class，这样一来就可以向传统使用nginx一样，创建和区别同一类别的服务；


有可能有人疑问，为什么不使用一个默认的kubernetes.io/ingress.class: nginx，明确说明下，是可以的，只是从安全（有些接口不能暴露到外网、有些需要外网访问）、管理（虚拟主机、app服务、不同部门的运维）、可操作、适用性角度来简单说明下。




您的关注是我写作的动力



往期分享


通俗易懂理解Kubernetes核心组件及原理

kubernetes v1.17.x 二进制安装文档 

Harbor镜像仓库高可用配置与使用


数据库
文章转载自Linux点滴运维实践，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

评论