第一部分 Service发布Pod容器服务
一、service
1.1 service与kube-proxy的关系
service对象就是工作节点上的一组iptables或者ipvs规则,用于将到达service对象IP地址上的流量调度至endpoints指向的IP或者端口上;
每个工作节点上的kube-proxy通过API server持续监控各service及其关联Pod对象的变化,将其变化实时表现在iptables或者ipvs上。
1.2 service代理模型
1.2.1 userspace代理模型

client请求流量到达某node的内核空间,内核将流量转发至当前node节点用户空间的kube-proxy,kube-proxy将根据iptables规则转发至对应的Server Pod;
userspace模式需要将client的请求流量在内核空间与用户空间来回转发两次;
userspace模式中,kube-proxy每次转发流量至对应的Pod时都会检测Pod的健康性,缺点是效率低。
1.2.2 iptables&&ipvs代理模型

当前k8s版本将iptables设为默认模式;
当client 的请求流量到达某个Node的内核空间后直接根据kube-proxy生成的iptables或者ipvs规则转发至对应的Server Pod,而无须转发至用户空间的kube-proxy;
ipvs所支持的调度算法比iptables更丰富。
1.3 Endpoint
k8s集群中创建一个service,就会生成一个同名的endpoint对象,endpoint就是service关联的pod的ip地址和端口的集合;
service由一组后端pod组成,这些后端的pod通过service endpoint暴露出来,如果有一个新的pod被创建出来,且pod的标签名称(label:pod)与service里面的标签(label selector 的label)一致会自动加入到service的endpoints 里面,如果pod对象终止后,pod 会自动从edponts 中移除。
service不仅可以代理Pod还可以代理任意其他的后端服务,但是必须定义同名的service和endpoints来实现;
[root@k8s-master-01 service]# kubectl get endpointsNAME ENDPOINTS AGEkubernetes 10.1.114.220:6443 14dmyapp-svc-clusterip 10.244.1.14:80,10.244.1.15:80,10.244.3.6:80 19hmyapp-svc-headless 10.244.1.14:80,10.244.1.15:80,10.244.3.6:80 19hmyapp-svc-nodeport 10.244.1.14:80,10.244.1.15:80,10.244.3.6:80 19h
1.4 Service类型
1.4.1 ClusterIP

ClusterIP为一个虚拟IP,该模式只能用于在集群内部通信;
ClusterIP为service的默认代理类型;
[root@k8s-master-01 service]# cat myapp-svc.yamlkind: ServiceapiVersion: v1metadata:name: myapp-svc-clusteripspec:type: ClusterIPselector:app: myappports:- protocol: TCPport: 80targetPort: 80#基于客户端IP设置会话粘性sessionAffinity: ClientIPsessionAffinityConfig:clientIP:#会话保持长timeoutSeconds: 20800[root@k8s-master-01 service]# kubectl apply -f myapp-svc.yamlservice/myapp-svc-clusterip created[root@k8s-master-01 service]# kubectl get -f myapp-svc.yamlNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEmyapp-svc-clusterip ClusterIP 10.110.45.75 <none> 80/TCP 82s
1.4.2 External

用于kubernetes集群访问外部service,该外部service需要被coreDNS解析到;
[root@k8s-master-01 service]# cat myapp-svc[root@k8s-master-01 service]# cat myapp-svc-external.yamlkind: ServiceapiVersion: v1metadata:name: myapp-svc-externalspec:type: ExternalName# externalName: redis.ilinux.ioexternalName: k8s-master-02selector: {}ports:- protocol: TCPport: 6379targetPort: 6379nodePort: 0[root@k8s-master-01 service]# kubectl apply -f myapp-svc-external.yamlservice/myapp-svc-external created[root@k8s-master-01 service]# kubectl get -f myapp-svc-external.yamlNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEmyapp-svc-external ExternalName <none> k8s-master-02 6379/TCP 11s
1.4.3 NodePort

用于将service暴露在node节点的指定端口上(NodePort);
只支持端口暴露服务,不支持域名;
NodePort:3000~32767
[root@k8s-master-01 service]# cat myapp-svc-nodeport.yamlkind: ServiceapiVersion: v1metadata:name: myapp-svc-nodeportspec:type: NodePortselector:app: myappports:- protocol: TCPport: 80targetPort: 80nodePort: 32223[root@k8s-master-01 service]# kubectl apply -f myapp-svc-nodeport.yamlservice/myapp-svc-nodeport created[root@k8s-master-01 service]# kubectl get -f myapp-svc-nodeport.yamlNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEmyapp-svc-nodeport NodePort 10.106.42.85 <none> 80:32223/TCP 11s
1.4.4 Headless

配置时clusterIP需要明确指定为None;
用于将每个Pod直接暴露给外部;
[root@k8s-master-01 service]# cat myapp-svc-headless.yamlkind: ServiceapiVersion: v1metadata:name: myapp-svc-headlessspec:clusterIP: Noneselector:app: myappports:- protocol: TCPport: 80targetPort: 80name: httpport[root@k8s-master-01 service]# kubectl apply -f myapp-svc-headless.yamlservice/myapp-svc-headless created[root@k8s-master-01 service]# kubectl get -f myapp-svc-headless.yamlNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEmyapp-svc-headless ClusterIP None <none> 80/TCP 6s
二、服务发现
2.1 服务发现和ClusterDNS
kubernetes系统上用于名称解析和服务发现的clusterDNS是集群的核心组件之一;
集群中的各个Pod会自动配置ClusterDNS为其名称解析服务器;
创建service时,ClusterDNS会自动创建资源记录用于名称解析和服务注册;
每个service对象都会包含如下两个资源记录:
SVCNAME.NAMESPACE.CLUSTER_DOMAIN
SVCNAME.NAMESPACE.svc.CLUSTER_DOMAIN
kubernetes1.11版本后,CoreDNS取代kubeDNS成为默认的DNS附件;
[root@k8s-master-01 ~]# kubectl get svc -n kube-system -o wideNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTORkube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 14d k8s-app=kube-dns#创建一个临时Client Pod做测试[root@k8s-master-01 service]# kubectl run cirros --rm -it --image=cirros -- shIf you don't see a command prompt, try pressing enter./ # cat etc/resolv.confnameserver 10.96.0.10search default.svc.cluster.local svc.cluster.local cluster.local localdomain/ # ping myapp-svc-nodeport.defaultPING myapp-svc-nodeport.default (10.106.42.85): 56 data bytes
第二部分 Ingress发布Service服务
一、ingress-nginx控制器部署
ingress控制器不会随着集群部署而自动部署,需要手动创建;
ingress发布服务的方式有多种,例如nginx、keepalived等等;
1.1 在每个worker上下载需要的image
docker pull registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5docker pull registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0
1.2 下载ingress-nginx.ymal资源申明文件
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.20.0/deploy/mandatory.yaml
1.3 修改清单文件
sed -i 's#k8s.gcr.io/defaultbackend-amd64#registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64#g' mandatory.yamlsed -i 's#quay.io/kubernetes-ingress-controller/nginx-ingress-controller#registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller#g' mandatory.yamlgrep image mandatory.yaml
注意:不修改镜像源地址会报错,导致ingress-nginx部署失败。
1.4 nginx-ingress.yaml文件
[root@k8s-master-01 sucess]# cat mandatory.yamlapiVersion: v1kind: Namespacemetadata:name: ingress-nginx---apiVersion: apps/v1kind: Deploymentmetadata:name: default-http-backendlabels:app.kubernetes.io/name: default-http-backendapp.kubernetes.io/part-of: ingress-nginxnamespace: ingress-nginxspec:replicas: 1selector:matchLabels:app.kubernetes.io/name: default-http-backendapp.kubernetes.io/part-of: ingress-nginxtemplate:metadata:labels:app.kubernetes.io/name: default-http-backendapp.kubernetes.io/part-of: ingress-nginxspec:terminationGracePeriodSeconds: 60containers:- name: default-http-backend# Any image is permissible as long as:# 1. It serves a 404 page at# 2. It serves 200 on a healthz endpointimage: registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5livenessProbe:httpGet:path: /healthzport: 8080scheme: HTTPinitialDelaySeconds: 30timeoutSeconds: 5ports:- containerPort: 8080resources:limits:cpu: 10mmemory: 20Mirequests:cpu: 10mmemory: 20Mi---apiVersion: v1kind: Servicemetadata:name: default-http-backendnamespace: ingress-nginxlabels:app.kubernetes.io/name: default-http-backendapp.kubernetes.io/part-of: ingress-nginxspec:ports:- port: 80targetPort: 8080selector:app.kubernetes.io/name: default-http-backendapp.kubernetes.io/part-of: ingress-nginx---kind: ConfigMapapiVersion: v1metadata:name: nginx-configurationnamespace: ingress-nginxlabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginx---kind: ConfigMapapiVersion: v1metadata:name: tcp-servicesnamespace: ingress-nginxlabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginx---kind: ConfigMapapiVersion: v1metadata:name: udp-servicesnamespace: ingress-nginxlabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginx---apiVersion: v1kind: ServiceAccountmetadata:name: nginx-ingress-serviceaccountnamespace: ingress-nginxlabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginx---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRolemetadata:name: nginx-ingress-clusterrolelabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginxrules:- apiGroups:- ""resources:- configmaps- endpoints- nodes- pods- secretsverbs:- list- watch- apiGroups:- ""resources:- nodesverbs:- get- apiGroups:- ""resources:- servicesverbs:- get- list- watch- apiGroups:- "extensions"resources:- ingressesverbs:- get- list- watch- apiGroups:- ""resources:- eventsverbs:- create- patch- apiGroups:- "extensions"resources:- ingresses/statusverbs:- update---apiVersion: rbac.authorization.k8s.io/v1beta1kind: Rolemetadata:name: nginx-ingress-rolenamespace: ingress-nginxlabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginxrules:- apiGroups:- ""resources:- configmaps- pods- secrets- namespacesverbs:- get- apiGroups:- ""resources:- configmapsresourceNames:# Defaults to "<election-id>-<ingress-class>"# Here: "<ingress-controller-leader>-<nginx>"# This has to be adapted if you change either parameter# when launching the nginx-ingress-controller.- "ingress-controller-leader-nginx"verbs:- get- update- apiGroups:- ""resources:- configmapsverbs:- create- apiGroups:- ""resources:- endpointsverbs:- get---apiVersion: rbac.authorization.k8s.io/v1beta1kind: RoleBindingmetadata:name: nginx-ingress-role-nisa-bindingnamespace: ingress-nginxlabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginxroleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: nginx-ingress-rolesubjects:- kind: ServiceAccountname: nginx-ingress-serviceaccountnamespace: ingress-nginx---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata:name: nginx-ingress-clusterrole-nisa-bindinglabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginxroleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: nginx-ingress-clusterrolesubjects:- kind: ServiceAccountname: nginx-ingress-serviceaccountnamespace: ingress-nginx---apiVersion: apps/v1kind: Deploymentmetadata:name: nginx-ingress-controllernamespace: ingress-nginxlabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginxspec:replicas: 1selector:matchLabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginxtemplate:metadata:labels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginxannotations:prometheus.io/port: "10254"prometheus.io/scrape: "true"spec:serviceAccountName: nginx-ingress-serviceaccountcontainers:- name: nginx-ingress-controllerimage: registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0args:- /nginx-ingress-controller- --default-backend-service=$(POD_NAMESPACE)/default-http-backend- --configmap=$(POD_NAMESPACE)/nginx-configuration- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services- --udp-services-configmap=$(POD_NAMESPACE)/udp-services- --publish-service=$(POD_NAMESPACE)/ingress-nginx- --annotations-prefix=nginx.ingress.kubernetes.iosecurityContext:capabilities:drop:- ALLadd:- NET_BIND_SERVICE# www-data -> 33runAsUser: 33env:- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespaceports:- name: httpcontainerPort: 80- name: httpscontainerPort: 443livenessProbe:failureThreshold: 3httpGet:path: /healthzport: 10254scheme: HTTPinitialDelaySeconds: 10periodSeconds: 10successThreshold: 1timeoutSeconds: 1readinessProbe:failureThreshold: 3httpGet:path: /healthzport: 10254scheme: HTTPperiodSeconds: 10successThreshold: 1timeoutSeconds: 1
1.5 应用并查看创建的ingress-nginx资源
[root@k8s-master-01 sucess]# kubectl get -f mandatory.yamlNAME STATUS AGEnamespace/ingress-nginx Active 2d18hNAME READY UP-TO-DATE AVAILABLE AGEdeployment.apps/default-http-backend 1/1 1 1 2d18hNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEservice/default-http-backend ClusterIP 10.106.145.204 <none> 80/TCP 2d18hNAME DATA AGEconfigmap/nginx-configuration 0 2d18hconfigmap/tcp-services 0 2d18hconfigmap/udp-services 0 2d18hNAME SECRETS AGEserviceaccount/nginx-ingress-serviceaccount 1 2d18hNAME CREATED ATclusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole 2020-07-25T10:34:45ZNAME CREATED ATrole.rbac.authorization.k8s.io/nginx-ingress-role 2020-07-25T10:34:45ZNAME ROLE AGErolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding Role/nginx-ingress-role 2d18hNAME ROLE AGEclusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding ClusterRole/nginx-ingress-clusterrole 2d18hNAME READY UP-TO-DATE AVAILABLE AGEdeployment.apps/nginx-ingress-controller 1/1 1 1 2d18h[root@k8s-master-01 sucess]# kubectl get svc -n ingress-nginxNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEdefault-http-backend ClusterIP 10.106.145.204 <none> 80/TCP 2d18h[root@k8s-master-01 sucess]# kubectl get deployment -n ingress-nginxNAME READY UP-TO-DATE AVAILABLE AGEdefault-http-backend 1/1 1 1 2d18hnginx-ingress-controller 1/1 1 1 2d18h
二、使用service的NodePort类型对外暴露ingress-nginx
2.1 ingress-nginx暴露为NodePort
[root@k8s-master-01 sucess]# cat ingress-nginx-svc.yamlapiVersion: v1kind: Servicemetadata:name: ingress-nginxnamespace: ingress-nginxlabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginxspec:type: NodePortports:- name: httpport: 80targetPort: 80protocol: TCPnodePort: 32080 #http- name: httpsport: 443targetPort: 443protocol: TCPnodePort: 32443 #httpsselector:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginx
2.2 定义并查看ingress-nginx的svc资源
[root@k8s-master-01 sucess]# kubectl get svc -n ingress-nginxNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEingress-nginx NodePort 10.96.250.67 <none> 80:32080/TCP,443:32443/TCP 2d17h
三、tomcat测试示例
3.1 定义deployment与service
[root@k8s-master-01 ingress-tomcat]# cat svc-tomcat.yamlapiVersion: v1kind: Servicemetadata:name: tomcat-svcnamespace: tomcat-testspec:selector:app: tomcatrelease: canaryports:- name: httpport: 80targetPort: 8080- name: ajpport: 8009targetPort: 8009---apiVersion: apps/v1kind: Deploymentmetadata:name: tomcat-deploynamespace: tomcat-testspec:replicas: 3selector:matchLabels:app: tomcatrelease: canarytemplate:metadata:labels:app: tomcatrelease: canaryspec:containers:- name: tomcatimage: tomcat:7-alpineports:- name: httpdcontainerPort: 8080- name: ajpcontainerPort: 8009
3.2 定义ingress暴露规则(http)
[root@k8s-master-01 ingress-tomcat]# cat ingress-tomcat.yamlapiVersion: extensions/v1beta1kind: Ingressmetadata:name: tomcat-tomcatnamespace: tomcat-testannotations:kubernets.io/ingress.class: "nginx"spec:rules:- host: tomcat.ilinux.iohttp:paths:- path:backend:serviceName: tomcat-svcservicePort: 80
3.2.1 应用资源清单文件
kubectl apply -f svc-tomcat.yamlkubectl apply -f ingress-tomcat.yaml[root@k8s-master-01 ingress-tomcat]# kubectl get svc -n tomcat-testNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEtomcat-svc ClusterIP 10.103.136.32 <none> 80/TCP,8009/TCP 2d15h[root@k8s-master-01 ingress-tomcat]# kubectl get ingress -n tomcat-testNAME CLASS HOSTS ADDRESS PORTS AGEtomcat-tomcat <none> tomcat.ilinux.io 80 56s#svc会根据deployment的label selector选择对用的pod[root@k8s-master-01 ingress-tomcat]# kubectl get endpoints -n tomcat-test -o wideNAME ENDPOINTS AGEtomcat-svc 10.244.1.220:8080,10.244.3.29:8080,10.244.3.30:8080 + 3 more... 3d13h[root@k8s-master-01 ingress-tomcat]# kubectl get pods -n tomcat-test -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATEStomcat-deploy-f59fd664-9m2tr 1/1 Running 0 3d13h 10.244.1.220 k8s-worker-01 <none> <none>tomcat-deploy-f59fd664-jb4nk 1/1 Running 1 3d13h 10.244.3.29 k8s-worker-02 <none> <none>tomcat-deploy-f59fd664-vp9xx 1/1 Running 0 3d13h 10.244.3.30 k8s-worker-02 <none> <none>
3.2.2 测试页面
http://tomcat.ilinux.io:32080/

3.3 定义ingress的暴露规则(https)
#生成证书文件openssl genrsa -out tls.key 2048openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.magedu.comkubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key
3.3.1 定义secret资源
[root@k8s-master-01 https-ingress]# cat ingress-tomcat-tls.yamlapiVersion: extensions/v1beta1kind: Ingressmetadata:name: ingress-tomcat-tlsnamespace: defaultannotations:kubernets.io/ingress.class: "nginx"spec:tls:- hosts:- tomcat.magedu.com #与secret证书的域名需要保持一致secretName: tomcat-ingress-secret #secret证书的名称rules:- host: tomcat.magedu.comhttp:paths:- path:backend:serviceName: tomcatservicePort: 8080[root@k8s-master-01 https-ingress]# kubectl get secret -n tomcat-testNAME TYPE DATA AGEdefault-token-qkbk9 kubernetes.io/service-account-token 3 2d18h
3.3.2 测试页面
https://tomcat.magedu.com:32443/

3.4 查看ingress资源
[root@k8s-master-01 ingress-tomcat]# kubectl get ingress -n tomcat-testNAME CLASS HOSTS ADDRESS PORTS AGEingress-tomcat-tls <none> tomcat.magedu.com 80, 443 53mtomcat-tomcat <none> tomcat.ilinux.io 80 70mNAME CLASS HOSTS ADDRESS PORTS AGEingress-tomcat-tls <none> tomcat.magedu.com 80, 443 53mtomcat-tomcat <none> tomcat.ilinux.io 80 70m[root@k8s-master-01 ingress-tomcat]# kubectl describe ingress tomcat-tomcatName: tomcat-tomcatNamespace: defaultAddress:Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)Rules:Host Path Backends---- ---- --------magedu.comtomcat:8080 (10.244.1.219:8080,10.244.3.23:8080,10.244.3.24:8080)Annotations: kubernets.io/ingress.class: nginxEvents: <none>




