这就是k8s系列之五（使用ingress-nginx暴露服务）

埋头过坎 2020-12-01

1371

上一篇k8s系列文章介绍了使用nginx来给kube-apiserver作反向代理，除了这个作用， nginx也是常用的ingress controller，本文介绍如何使用ingress-nginx进行服务暴露（先介绍操作篇，原理后续再做介绍）。

笔者的datalab k8s集群是v1.19版本的，使用kubespary安装时，在addon上是打开了ingress-nginx开关的，安装后的版本如下：

[root@host01~]# kubectl exec -n ingress-nginx  -it ingress-nginx-controller-8c9qm   -- nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v0.40.2
  Build:         fc4ccc5eb0e41be2436a978b01477fc354f31643
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.3


-------------------------------------------------------------------------------

安装后，查看pod状态如下：

[root@host01~]# kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --watch
NAME                             READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-8c9qm   1/1     Running   0          26h
ingress-nginx-controller-nq9qj   1/1     Running   0          26h
ingress-nginx-controller-r5xxj   1/1     Running   0          26h
ingress-nginx-controller-tdnnv   1/1     Running   0          26h
ingress-nginx-controller-wzhm7   1/1     Running   0          26h

可以看到，在v.40.2版本下， 5个节点都各自部署了1个实例。笔者按照官方的教程(https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/exposing-tcp-udp-services.md ) 配置了一下并没有生效，从官网的另一篇安装指南文章（https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal) 看到当前版本已经更新到v0.41.2，于是决定重新安装配置ingress-nginx。

删除掉原来的安装分两部分进行操作，首先是直接删除整个ingress-nginx的namespace，这样里面的pod等资源就会一起删除，然后是从default namespace中删除ingress相关的ClusterRole、ClusterRoleBinding即可。下面介绍重新安装的具体过程和遇到的一些问题。

step 1. 创建刚才删除掉的namespace

kubectl create ns  ingress-nginx

step 2. 按照上面的安装指南链接，采用bare-metal场景下node方式安装:



root@host01 ingress-nginx]# kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/baremetal/deploy.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
namespace/ingress-nginx configured
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created


clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
# 查看安装结果
[root@host01 ~]# kubectl get all  -n ingress-nginx
NAME                                            READY   STATUS      RESTARTS   AGE
pod/ingress-nginx-admission-create-fvssz        0/1     Completed   0          147m
pod/ingress-nginx-admission-patch-b47rm         0/1     Completed   0          147m
pod/ingress-nginx-controller-5c4dfb5c7b-84wpl   1/1     Running     0          29m


NAME                                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             NodePort   ip01 <none>        80:30179/TCP,443:32752/TCP   147m
service/ingress-nginx-controller-admission   ClusterIP   ip02 <none>        443/TCP                      147m


NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   1/1     1            1           147m


NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-5c4dfb5c7b   1         1         1       29m
replicaset.apps/ingress-nginx-controller-5dbd9649d4   0         0         0       147m
replicaset.apps/ingress-nginx-controller-b98cb849d    0         0         0       143m
replicaset.apps/ingress-nginx-controller-bd947bc79    0         0         0       132m


NAME                                       COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   1/1           3s         147m
job.batch/ingress-nginx-admission-patch    1/1           3s         147m


# 查看版本
[root@host01 ~]#  kubectl exec -n ingress-nginx  -it  ingress-nginx-controller-5c4dfb5c7b-84wpl   -- nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v0.41.2
  Build:         d8a93551e6e5798fc4af3eb910cef62ecddc8938
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.4


-------------------------------------------------------------------------------

step 3: 创建tcp-services的configmap, 结果如下：

# 这里配置了mogodb的服务及dashboard服务
(base) bash-4.1$ cat ConfigMap-tcp-services.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
data:
  8998: "kube-system/kubernetes-dashboard:443"
  8717: "mongodb/bitnami-mongodb-mongodb-sharded:27017"
  
  # 部署configmap
  kubectl apply -f ConfigMap-tcp-services.yaml 
  
  # 查看结果
  [root@host01 ~]#  kubectl get configmap  -n ingress-nginx
NAME                              DATA   AGE
ingress-controller-leader-nginx   0      3h5m
ingress-nginx-controller          0      169m
tcp-services                      2      177m

这里补充一下，用于验证的mongodb服务（如何安装的后续再介绍），该服务的状态如下：

[root@host01 ~]# kubectl get all -n mongodb
NAME                                                          READY   STATUS    RESTARTS   AGE
pod/bitnami-mongodb-mongodb-sharded-configsvr-0               1/1     Running   0          8h
pod/bitnami-mongodb-mongodb-sharded-mongos-77758696fc-6t4kr   1/1     Running   0          8h
pod/bitnami-mongodb-mongodb-sharded-shard0-data-0             1/1     Running   0          8h
pod/bitnami-mongodb-mongodb-sharded-shard1-data-0             1/1     Running   0          8h


NAME                                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
service/bitnami-mongodb-mongodb-sharded            ClusterIP   ip01  <none>        27017/TCP   8h
service/bitnami-mongodb-mongodb-sharded-headless   ClusterIP   None            <none>        27017/TCP   8h


NAME                                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/bitnami-mongodb-mongodb-sharded-mongos   1/1     1            1           8h


NAME                                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/bitnami-mongodb-mongodb-sharded-mongos-77758696fc   1         1         1       8h


NAME                                                           READY   AGE
statefulset.apps/bitnami-mongodb-mongodb-sharded-configsvr     1/1     8h
statefulset.apps/bitnami-mongodb-mongodb-sharded-shard0-data   1/1     8h
statefulset.apps/bitnami-mongodb-mongodb-sharded-shard1-data   1/1     8h
[root@host01 ~]# 


[root@host01 ~]# kubectl get pods -n mongodb -o wide
NAME                                                      READY   STATUS    RESTARTS   AGE   IP              NODE                             NOMINATED NODE   READINESS GATES
bitnami-mongodb-mongodb-sharded-configsvr-0               1/1     Running   0          8h    ip01   host01   <none>           <none>
bitnami-mongodb-mongodb-sharded-mongos-77758696fc-6t4kr   1/1     Running   0          8h    ip04  host04   <none>           <none>
bitnami-mongodb-mongodb-sharded-shard0-data-0             1/1     Running   0          8h    ip04  host04   <none>           <none>
bitnami-mongodb-mongodb-sharded-shard1-data-0             1/1     Running   0          8h    ip03    host03   <none>           <none>
[root@host01 ~]#

可以看到mogodb服务bitnami-mongodb-mongodb-sharded的类型为ClusterIP，对应的后端的POD位于host04, 这个信息后面将用于验证是ingress nginx配置是否生效。

step 4: 修改ingress nginx的配置文件，主要包括容器程序启动程序添加上面的configmap为参数，另外是要打开主机网络的开关，如下：

root@host01 ingress-nginx]# kubectl edit deployment ingress-nginx-controller -n ingress-nginx
      containers:
      - args:
        - nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
        - --election-id=ingress-controller-leader
        - --ingress-class=nginx
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
       ......
       
      dnsPolicy: ClusterFirst
      hostNetwork: true   # 添加该配置项， 不然无法生效
      nodeSelector:
      
   # 查看配置结果
   
Ports:       80/TCP, 443/TCP, 8443/TCP
Host Ports:  80/TCP, 443/TCP, 8443/TCP

step 5: 验证配置是否生效，主要是通过查看ingress nginx controller所在的机器是否已经开启了相应的端口以及通过mogodb服务进行验证，如下：

[root@host04 ~]# netstat -antp |grep 8717
tcp        0      0 0.0.0.0:8717            0.0.0.0:*               LISTEN      34685/nginx: master 
tcp6       0      0 :::8717                 :::*                    LISTEN      34685/nginx: master


# mongo验证
[root@host01 ~]#  mongo --host host04:8717
MongoDB shell version v4.0.3
connecting to: mongodb://host04:8717/
Implicit session: session { "id" : UUID("ad0399da-6444-47a4-9449-5bb30f3156d2") }
MongoDB server version: 4.4.2
WARNING: shell and server versions do not match
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
  http://docs.mongodb.org/
Questions? Try the support group
  http://groups.google.com/group/mongodb-user
mongos>

可以看到mogodb客户端成功连接了服务，说明服务已经成功通过ingress nginx controller对外暴露了。

从上面看到，还配置了dashboard服务，但验证是失败的，主要原因是https，这个问题后面文章再介绍如何解决。

数据库

文章转载自埋头过坎，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

这就是k8s系列之五（使用ingress-nginx暴露服务）

评论