5 Front Proxy and TLS
本文讲述的是 front proxy 和 TLS 结合起来大体该如何使用。
5.1 Front proxy
其实 front proxy 到底是什么我在前面的文章中一直在讲 envoy 其实有两种部署类型：
一：边缘代理，我将其称为 front proxy ，通常是用来代理一整个集群的，也可能是工作在网络边缘代理整个 cluster 中所有服务路由的这个时候其实扮演的角色就是 api gateway 的角色，这里提到的 front proxy 通常是一个集群前端的统一调度器
二：以 sidecar 的方式部署在每个 service 周边，而 sidecar proxy 中主要基于 listener 所管控流量的方向，我们又有 ingress proxy、egress proxy、其实都是基于 listener 进行交互的。但是出站流量可以不用非得 配置监听器，如果说为了实现更高级的网格功能，比如
TLS
会话的自动创建发起和卸载那我们就应该对每个出站的流量加一个 egress 的 listener
在 Envoy Mesh 中，作为 Front Proxy 的 Envoy 通常是独立运行的进程，它将客户端请求代理至 Mesh 中的各 Service，而这些 Service 中的每个应用实例都会隐藏于一个 Sidecar Proxy 模式的 envoy 实例背后
5.2 TLS
正常情况下 Envoy Mesh (这里我之所以称为 envoy 网格就表示他没有控制平面的纯粹的 envoy 数据平面)中的 TLS 模式大体上有以下几种场景：
场景一：Front Proxy 面向下游客户端提供 https 服务，但 Front Proxy、Mesh 内部的各服务间依然使用 http 协议
https → http：也就是说我们的 envoy 在面向客户端的时候使用的就是 HTTPS ，这里我们假设 envoy Mesh 内部通信是安全的所以在服务网格内部就将 https 卸载了而转为了 http ，这个时候 envoy 扮演的就是 TLS 的卸载器
场景二：Front Proxy 面向下游客户端提供https服务，而且 Front Proxy、Mesh 内部的各服务间也使用 https 协议
仅客户端验证服务端证书
客户端与服务端之间互相验证彼此的证书（mTLS） ，在我们需要极高安全认证的时候才会需要这种功能
https → https：也就是说不管 envoy 面向客户端还是自己 Mesh 网格内部通信都是 https
但是内部各 Service 间的通信也有如下两种情形
注意：对于容器化的动态环境来说，证书预配和管理将成为显著难题
场景三：Front Proxy 直接以 TCP Proxy 的代理模式，在下游客户端与上游服务端之间透传 tls 协议而不在通过 envoy 中间中转；
https-passthrough
集群内部的东西向流量同样工作于 https 协议模型
5.2.1 https → http 示例
我们在看这个演示之前需要看一下伪代码，这样才能更好的理解我们在配置文件中的一些操作
# 我说过在 envoy.yaml 配置文件中，Listener 字段是用来配置面向客户端下游服务
# 而 Cluster 字段则是配置面向提供服务端上游服务
# 由此可以看出 envoy 在面向下游服务时充当的角色就是一个 server，而面向上游服务的时候充当的角色就是一个 client
# 所以 envoy 在面向下游客户端的时候就需要提供证书给客户端，所以无论如何 envoy 都需要有个证书，甚至是提供私钥
# 上面我们说过如果在 envoy Mesh 内部通信如果是走的 http 协议就不用配置证书
# 下面为 envoy.yaml 配置字段解释
Listener # 配置面向下游服务
DownStream
Server,Certs # 在面向下游服务的时候是 server 端
Cluster # 配置面向上游服务
UpStream
Client # 在面向上游服务的时候是 client 端
CA Certs # 只需要指定 CA 证书能够验证上游服务器的证书就可以了
仅需要配置 Listener 面向下游客户端提供 TLS通信，所以只需要在 Listener 字段中配置自己的 server 证书即可，而面向上游网格内服务则直接将 https 卸载不需要 TLS 通信。
下面是Front Proxy Envoy的配置示例
static_resources:
listeners:
- name: listener_http
address:
socket_address: { address: 0.0.0.0, port_value: 8443 } # 端口改为了一眼就能识别的 TLS 通信端口
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
codec_type: AUTO
route_config:
name: local_route
virtual_hosts:
- name: web_service_01
domains: ["*"]
routes:
- match: { prefix: "/" }
route: { cluster: web_cluster_01 }
http_filters:
- name: envoy.filters.http.router
# 面向下游 https 通信我们要配置 transport_socket 字段，这就表示要给下游服务器提供 TLS 通信
transport_socket:
# 这里的 name 是一个固定的字符串，表示我们要调用一个过滤器来扩展，而这个扩展有自己专用的配置
name: envoy.transport_sockets.tls
typed_config:
# 固定字符串写法 DownstreamTlsContext 面向下游专用
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context: # 通用 TLS 配置
tls_certificates: # 配置证书
# The following self-signed certificate pair is generated using:
# $ openssl req -x509 -newkey rsa:2048 -keyout front-proxy.key -out front-proxy.crt -days 3650 -nodes -subj '/CN=www.zgy.com'
- certificate_chain: # 指定证书名字
filename: "/etc/envoy/certs/front-proxy.crt" # 证书路径
private_key: # 私钥文件
filename: "/etc/envoy/certs/front-proxy.key“ # 私钥文件路径
但是上面是纯静态提供的 TLS 服务，有个坏处就是将来证书到期了想换证书该怎么办，还需要重启，所以我们最好能够动态获取证书，可以基于所谓的 xDS 协议发现服务
5.2.2 https → https 示例
除了 Listener 中面向下游提供 tls 通信，Front Proxy 还要以 tls 协议与 Envoy Mesh 中的各 Service 建立 tls 连接；
下面是 Envoy Mesh 中的某 Service 的 Sidecar Proxy Envoy 的配置示例：
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 443 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
……
http_filters:
- name: envoy.filters.http.router
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
- certificate_chain:
filename: "/etc/envoy/certs/webserver.crt"
private_key:
filename: "/etc/envoy/certs/webserver.key"
下面是 Front Proxy Envoy 中的 Cluster 面向上游通信的配置示例：
clusters:
- name: web_cluster_01
connect_timeout: 0.25s
type: STATIC
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: web_cluster_01
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address: { address: 10.0.10.11, port_value: 443 }
- endpoint:
address:
socket_address: { address: 10.0.10.12, port_value: 443 }
transport_socket: # 定义 transport_socket 字段，这就表示要接收 envoy 提供的 TLS 通信
name: envoy.transport_sockets.tls # 定义 envoy.transport_sockets.tls 开启 TLS 功能
typed_config:
# 固定字符串写法 UpstreamTlsContext 面向上游专用
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
5.2.3 https-passthrough 示例
直接透传模式示例，在下游客户端与上游服务端之间透传 tls 协议而不在通过 envoy 中间中转；
TLS passthrough 模式的 Front Proxy 需要使用 TCP Proxy 类型的 Listener，Cluster 的相关配置中也无需再指定 transport_socket 相关的配置；
但 Envoy Mesh 中各 Service 需要基于 tls 提供服务
static_resources:
listeners: # 直接透传的话在 listener 中就以为这 TCP 代理
- name: listener_http
address:
socket_address: { address: 0.0.0.0, port_value: 8443 }
filter_chains:
- filters:
- name: envoy.filters.network.tcp_proxy # 由于是直接透传因此这里的代理就使用的 tcp proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
cluster: web_cluster_01
stat_prefix: https_passthrough
# 下面就为 cluster 不在配置 TLS
clusters:
- name: web_cluster_01
connect_timeout: 0.25s
type: STATIC
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: web_cluster_01
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address: { address: 10.0.10.11, port_value: 443 }
- endpoint:
address:
socket_address: { address: 10.0.10.12, port_value: 443 }
因此 TLS 会话实在客户端和 server 端进行的，这样做有个坏处，大家都知道如果我们不在前端 envoy 中卸载 TLS 会话的话，一个最大的问题在于每次都是客户端与 server 端建立连接，而后端 server 如果有多个的话客户端的每一次请求都可能会被调度到每台不同的 server 上，
5.3 静态配置的Envoy TLS 演示
在这里的演示中将会对上面 TLS 三种示例通过一个 envoy.yaml 的测试文件进行演示
环境说明
Envoy Mesh使用的网络: 10.0.10.0/24
6个Service:
front-envoy：Front Proxy,地址为 10.0.10.10
3 个 http 后端服务没有证书的明文集群：可统一由 myservice 名称解析到；front-envoy 会通过 http（会自动跳转至https）和 https 侦听器接收对这些服务的访问请求，并将其转为 http 请求后转至后端服务上；（https-http）
service-blue
service-red
service-green
下面为三个后端服务：
2 个 https 后端服务通过 TLS 通信的证书集群
service-gray：同时提供 http 和 https 侦听器，front-envoy 在其 cluster 配置中，会向该服务发起 https 请求所以 service-gray 会提供一个 server 端证书并会验证其数字证书；（http-https, https-https）
service-purple：同时提供 http 和 https 侦听器，通过 http 接收的请求会自动重定向至 https 所以需要提供能一个 server 端证书，并且 https 侦听器强制要求验证客户端证书；front-envoy在其cluster配置中，会向该服务发起https请求，向其提供自身的客户端证书后，并会验证其数字证书；
因而我们这里演示需要用到 4 个证书：
front-envoy作为服务端：
server.crt/server.keyfront-envoy作为客户端：
client.crt/client.key
service-gray 作为服务端：
server.crt/server.keyservice-purple 作为服务端：
server.crt/server.key
他们都属于同一个 CA ，所以我们需要先创建一个私有 CA ，再由这个私有 CA 像这 4 个证书发证。
预备操作:
1.生成测试使用的数字证书
脚本 gencerts.sh 运行时，需指定证书的 Subject 名称（也将做为证书和私钥等文件存放的目录的目录名），以及 OpenSSL 配置文件中定义的证书扩展类型，这里支持使用两种类型：
envoy_server_cert：为 Envoy 创建服务器证书，用于同下游客户端建立 TLS 连接，证书和私钥文件默认名称分别为 server.crt 和 server.key；
envoy_client_cert：为 Envoy 创建客户端证书，用于同上游服务端建立 TLS 连接，证书和私钥文件默认名称分别为 client.crt 和 client.key；
脚本使用示例：
./gencerts.sh
Generating RSA private key, 4096 bit long modulus (2 primes)
.............................++++
.............++++
e is 65537 (0x010001)
Certificate Name and Certificate Extenstions(envoy_server_cert/envoy_client_cert): front-envoy envoy_server_cert
# 上面的键入的证书主体名称为front-envoy，而选择的扩展类型为envoy_server_cert
Generating RSA private key, 2048 bit long modulus (2 primes)
.................................................................+++++
...............................................................................+++++
e is 65537 (0x010001)
Generating certs/front-envoy/server.csr
Generating certs/front-envoy/server.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4096 (0x1000)
Validity
Not Before: Nov 6 08:34:05 2021 GMT
Not After : Nov 4 08:34:05 2031 GMT
Subject:
commonName = front-envoy
X509v3 extensions:
……
……
# 可在出现命令提示符时，分别为front-envoy、service-gray和service-purple创建所需的证书和私钥；
注意：各目录中的key文件，可以需要做权限调整，以便envoy用户（UID:100, GID:101）能够读取并加载。
5.3.1 操作流程
5.3.1.1 编写生成证书脚本生成证书
1.编写 openssl 文件
root@envoy:~/servicemesh_in_practise-develop/Security/tls-static# cat openssl.conf
# environment variable values
BASE_DOMAIN=ilinux.io
CERT_DIR=
[ ca ]
# `man ca`
default_ca = CA_default
[ CA_default ]
# Directory and file locations.
dir = ${ENV::CERT_DIR}
certs = $dir
crl_dir = $dir/crl
new_certs_dir = $dir
database = $dir/index.txt
serial = $dir/serial
# certificate revocation lists.
crlnumber = $dir/crlnumber
crl = $dir/crl/intermediate-ca.crl
crl_extensions = crl_ext
default_crl_days = 30
default_md = sha256
name_opt = ca_default
cert_opt = ca_default
default_days = 375
preserve = no
policy = policy_loose
[ policy_loose ]
# Allow the CA to sign a range of certificates.
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[ req ]
# `man req`
default_bits = 4096
distinguished_name = req_distinguished_name
string_mask = utf8only
default_md = sha256
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
stateOrProvinceName = State or Province Name
localityName = Locality Name
0.organizationName = Organization Name
organizationalUnitName = Organizational Unit Name
commonName = Common Name
# Certificate extensions (`man x509v3_config`)
[ v3_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
[ client_cert ]
basicConstraints = CA:FALSE
nsCertType = client
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
[ server_cert ]
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
[ envoy_server_cert ]
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
#subjectAltName = DNS.1:*.${ENV::BASE_DOMAIN}
[ peer_cert ]
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS.1:*.${ENV::BASE_DOMAIN}
[ envoy_client_cert ]
basicConstraints = CA:FALSE
nsCertType = client
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
2.编写脚本
root@envoy:~/servicemesh_in_practise-develop/Security/tls-static# vim gencerts.sh
#!/bin/bash -e
function usage() {
>&2 cat << EOF
Usage: ./envoy-certs-gen.sh
Set the following environment variables to run this script:
BASE_DOMAIN Base domain name of the cluster. For example if your API
server is running on "my-cluster-k8s.example.com", the
base domain is "example.com"
CA_CERT(optional) Path to the pem encoded CA certificate of your cluster.
CA_KEY(optional) Path to the pem encoded CA key of your cluster.
EOF
exit 1
}
BASE_DOMAIN=ilinux.io
if [ -z $BASE_DOMAIN ]; then
usage
fi
export DIR="certs"
if [ $# -eq 1 ]; then
DIR="$1"
fi
export CERT_DIR=$DIR
[ ! -e $CERT_DIR ] && mkdir -p $CERT_DIR
CA_CERT="$CERT_DIR/CA/ca.crt"
CA_KEY="$CERT_DIR/CA/ca.key"
# Configure expected OpenSSL CA configs.
touch $CERT_DIR/index
touch $CERT_DIR/index.txt
touch $CERT_DIR/index.txt.attr
echo 1000 > $CERT_DIR/serial
# Sign multiple certs for the same CN
echo "unique_subject = no" > $CERT_DIR/index.txt.attr
function openssl_req() {
openssl genrsa -out ${1}/${2}.key 2048
echo "Generating ${1}/${2}.csr"
openssl req -config openssl.conf -new -sha256 \
-key ${1}/${2}.key -out ${1}/${2}.csr -subj "$3"
}
function openssl_sign() {
echo "Generating ${3}/${4}.crt"
openssl ca -batch -config openssl.conf -extensions ${5} -days 3650 -notext \
-md sha256 -in ${3}/${4}.csr -out ${3}/${4}.crt \
-cert ${1} -keyfile ${2}
}
if [ ! -e "$CA_KEY" -o ! -e "$CA_CERT" ]; then
mkdir $CERT_DIR/CA
openssl genrsa -out $CERT_DIR/CA/ca.key 4096
openssl req -config openssl.conf \
-new -x509 -days 3650 -sha256 \
-key $CERT_DIR/CA/ca.key -extensions v3_ca \
-out $CERT_DIR/CA/ca.crt -subj "/CN=envoy-ca"
export CA_KEY="$CERT_DIR/CA/ca.key"
export CA_CERT="$CERT_DIR/CA/ca.crt"
fi
read -p "Certificate Name and Certificate Extenstions(envoy_server_cert/envoy_client_cert): " CERT EXT
while [ -n "$CERT" -a -n "$EXT" ]; do
[ ! -e $CERT_DIR/$CERT ] && mkdir $CERT_DIR/$CERT
if [ "$EXT" == "envoy_server_cert" ]; then
openssl_req $CERT_DIR/$CERT server "/CN=$CERT"
openssl_sign $CERT_DIR/CA/ca.crt $CERT_DIR/CA/ca.key $CERT_DIR/$CERT server $EXT
else
openssl_req $CERT_DIR/$CERT client "/CN=$CERT"
openssl_sign $CERT_DIR/CA/ca.crt $CERT_DIR/CA/ca.key $CERT_DIR/$CERT client $EXT
fi
read -p "Certificate Name and Certificate Extenstions(envoy_server_cert/envoy_client_cert): " CERT EXT
done
# Add debug information to directories
#for CERT in $CERT_DIR/*; do
# [ -d $CERT ] && openssl x509 -in $CERT/*.crt -noout -text > "${CERT%.crt}.txt"
#done
# Clean up openssl config
rm $CERT_DIR/index*
rm $CERT_DIR/100*
rm $CERT_DIR/serial*
for CERT in $CERT_DIR/*; do
[ -d $CERT ] && rm -f $CERT/*.csr
done
2.执行脚本生成证书文件
root@envoy:~/servicemesh_in_practise-develop/Security/tls-static# ./gencerts.sh
# 生成 CA 证书
Generating RSA private key, 4096 bit long modulus (2 primes)
.........................................................................................................++++
..................................................++++
e is 65537 (0x010001)
# 生成 front-proxy 服务端证书
Certificate Name and Certificate Extenstions(envoy_server_cert/envoy_client_cert): front-envoy envoy_server_cert # 必须写 front-envoy envoy_server_cert 因为我在 openssl 配置文件中定义好的
Generating RSA private key, 2048 bit long modulus (2 primes)
.................+++++
....+++++
e is 65537 (0x010001)
Generating certs/front-proxy/server.csr
Generating certs/front-proxy/server.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4096 (0x1000)
Validity
Not Before: Dec 3 06:29:18 2021 GMT
Not After : Dec 1 06:29:18 2031 GMT
Subject:
commonName = front-proxy
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Server
Netscape Comment:
OpenSSL Generated Server Certificate
X509v3 Subject Key Identifier:
39:E7:C4:95:0F:F0:5A:7C:40:E6:FD:38:1C:8E:B8:FF:A9:57:F6:69
X509v3 Authority Key Identifier:
keyid:57:C0:D2:A8:08:95:5D:F2:4F:AA:1F:5C:9A:B9:13:0F:2E:FE:B6:AC
DirName:/CN=envoy-ca
serial:29:EE:C5:41:6B:E7:3C:68:AA:F7:D8:E0:7A:D1:2E:42:1E:DC:56:1B
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
Certificate is to be certified until Dec 1 06:29:18 2031 GMT (3650 days)
Write out database with 1 new entries
Data Base Updated
# 生成 front-proxy 客户端证书
Certificate Name and Certificate Extenstions(envoy_server_cert/envoy_client_cert): front-envoy envoy_client_cert # 生成 client 证书
# ...... 省略 .......
Data Base Update
# 生成 service-gray 服务端证书
Certificate Name and Certificate Extenstions(envoy_server_cert/envoy_client_cert): service-gray envoy_server_cert
Generating RSA private key, 2048 bit long modulus (2 primes)
# ...... 省略 .......
Data Base Updated
# 生成 service-purple 服务端证书
Certificate Name and Certificate Extenstions(envoy_server_cert/envoy_client_cert): service-purple envoy_server_cert
Generating RSA private key, 2048 bit long modulus (2 primes)
# ...... 省略 ....... 最后回车即可
以上我们就生成了 4 个不同的证书，并且会自动创建一个 certs 目录
root@envoy:~/servicemesh_in_practise-develop/Security/tls-static# tree certs/
certs/
├── CA
│ ├── ca.crt
│ └── ca.key
├── front-envoy
│ ├── client.crt
│ ├── client.key
│ ├── server.crt
│ └── server.key
├── service-gray
│ ├── server.crt
│ └── server.key
└── service-purple
├── server.crt
└── server.key
# 随后我面的 envoy 配置文件就会通过这个 certs 目录来加载这些证书文件
3.将 certs 目录下的所有文件属组改为 envoy，因为我在 docker-compose 文件中通过卷挂载的方式到 envoy 容器中，所以很显然 envoy 的 work 进程是以 envoy 用户运行的。envoy 用户（UID:100, GID:101）能够读取并加载
# 添加属组
root@envoy:~/servicemesh_in_practise-develop/Security/tls-static# chown -R 100.101 certs/*
5.3.1.2 编写 docker-compose
1.编写 docker-compose 文件
root@envoy:~/servicemesh_in_practise-develop/Security/tls-static# cat docker-compose.yaml
version: '3.3'
services:
front-envoy:
image: envoyproxy/envoy-alpine:v1.20.0
volumes:
- ./front-envoy.yaml:/etc/envoy/envoy.yaml # 挂在 front-envoy.yaml 配置文件
- ./certs/front-envoy/:/etc/envoy/certs/ # front-envoy 提供边缘代理所以将 ca 和 front 目录下 server、client 证书挂载至 envoy 容器中
- ./certs/CA/:/etc/envoy/ca/
networks:
envoymesh:
ipv4_address: 10.0.10.10
aliases:
- front-envoy
expose:
# Expose ports 80 (for general traffic) and 9901 (for the admin server)
- "80"
- "443"
- "9901"
ports:
- "8080:80"
- "8443:443"
- "9901:9901"
# blue 后端 http 服务容器
blue:
image: ikubernetes/servicemesh-app:latest
networks:
envoymesh:
aliases:
- myservice # 定义的 DNS 为 myservice 因为在 docker-compose 内部使用的是docker 自带的 DNS 解析
- service-blue
- blue
environment:
- SERVICE_NAME=blue
expose:
- "80"
# green 后端 http 服务容器
green:
image: ikubernetes/servicemesh-app:latest
networks:
envoymesh:
aliases:
- myservice # 定义的 DNS 为 myservice 因为在 docker-compose 内部使用的是docker 自带的 DNS 解析
- service-green
- green
environment:
- SERVICE_NAME=green
expose:
- "80"
# red 后端 http 服务容器
red:
image: ikubernetes/servicemesh-app:latest
networks:
envoymesh:
aliases:
- myservice # 定义的 DNS 为 myservice 因为在 docker-compose 内部使用的是docker 自带的 DNS 解析
- service-red
- red
environment:
- SERVICE_NAME=red
expose:
- "80"
# gray 容器为后端 https 服务，提供单向认证，并且 envoy 通过 sidecar 方式部署
gray:
image: ikubernetes/servicemesh-app:latest
volumes:
- ./service-gray.yaml:/etc/envoy/envoy.yaml # 挂在配置 service-gray.yaml 配置文件来代理 web 服务
- ./certs/service-gray/:/etc/envoy/certs/ # 挂在证书至容器中
networks:
envoymesh:
ipv4_address: 10.0.10.15
aliases:
- gray
- service-gray
environment:
- SERVICE_NAME=gray
expose:
- "80"
- "443"
# purple 容器为后端 https 服务，提供双向认证所以需要 CA 证书，并且 envoy 通过 sidecar 方式部署
purple:
image: ikubernetes/servicemesh-app:latest
volumes:
- ./service-purple.yaml:/etc/envoy/envoy.yaml # service-purple.yaml 配置文件来代理 web 服务
- ./certs/service-purple/:/etc/envoy/certs/
- ./certs/CA/:/etc/envoy/ca/
networks:
envoymesh:
ipv4_address: 10.0.10.16
aliases:
- purple
- service-purple
environment:
- SERVICE_NAME=purple
expose:
- "80"
- "443"
networks:
envoymesh:
driver: bridge
ipam:
config:
- subnet: 10.0.10.0/24
5.3.1.3 编写 envoy.yaml 文件
在上面的 docker-compose 文件中，我们看到了 3 个不同的 envoy 容器分别是：service-purple
，service-gray
, front-envoy
。所以需要挂载对应的 3 个不同 envoy.yaml 配置文件
1.编写 front-envoy 容器需要的配置文件
root@envoy:~/servicemesh_in_practise-develop/Security/tls-static# cat front-envoy.yaml
node:
id: front-envoy
cluster: front-envoy
# 开启 admin 接口
admin:
profile_path: /tmp/envoy.prof
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 9901
# 开启 runtime 为管理控制台层级
layered_runtime:
layers:
- name: admin
admin_layer: {}
# 静态资源配置
static_resources:
# 定义 secrets 服务证书
secrets:
# 服务端证书和证书、私钥路径
- name: server_cert
tls_certificate:
certificate_chain:
filename: "/etc/envoy/certs/server.crt"
private_key:
filename: "/etc/envoy/certs/server.key"
# 自己为 client 的时候，客户端证书和客户端私钥文件路径
- name: client_cert
tls_certificate:
certificate_chain:
filename: "/etc/envoy/certs/client.crt"
private_key:
filename: "/etc/envoy/certs/client.key"
# 以及 CA 证书
- name: validation_context
validation_context:
trusted_ca:
filename: "/etc/envoy/ca/ca.crt"
# 定义监听器，监听 http 的请求
listeners:
- name: listener_http
address:
socket_address:
address: 0.0.0.0
port_value: 80
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
route_config:
name: local_route
virtual_hosts:
- name: backend
# 有个路由配置，无论访问的是那个虚拟主机
domains: ["*"]
routes:
- match:
prefix: "/"
# 都对所有请求做重定向
redirect:
# 意思是将所有请求重定向至 https 的 listener
https_redirect: true
port_redirect: 443
http_filters:
# 开启 7 层过滤器路由功能
- name: envoy.filters.http.router
typed_config: {}
# 定义了 https listener
- name: listener_https
address:
# 监听 443 端口
socket_address: { address: 0.0.0.0, port_value: 443 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_https
codec_type: AUTO
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
route_config:
name: https_route
virtual_hosts:
- name: https_route
domains: ["*"]
# 路由匹配
routes:
# 如果是访问的 /service/gray 前缀的 URI 就调度到后端 service-gray cluster 服务上
- match:
prefix: "/service/gray"
route:
cluster: service-gray
# 如果是访问的 /service/purple 前缀的 URI 就调度到后端 service-purple cluster 服务上
- match:
prefix: "/service/purple"
route:
cluster: service-purple
# 如果是访问了 / 前缀的 URI 就调度到后端 mycluster cluster 服务上
- match:
prefix: "/"
route:
cluster: mycluster
http_filters:
- name: envoy.filters.http.router
typed_config: {}
# 做下游客户端的 TLS 通信
transport_socket: # DownstreamTlsContext
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
# 指明证书通过那加载
common_tls_context:
# 使用 TLS 配置静态加载，也就是上面我定义的 secrets 字段
tls_certificate_sds_secret_configs:
# 加载 server_cert 证书
- name: server_cert
# 定义 cluster
clusters:
# 定义 Front Proxy 模式下的 cluster，也就是直接通过明文 http 模式
- name: mycluster
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: mycluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: myservice # 因为明文集群通过 DNS 解析访问
port_value: 80
# 定义 service-gray 集群，service-gray：同时提供 http 和 https 侦听器，front-envoy 在其 cluster 配置中，会向该服务发起 https 请求，并会验证其数字证书；（单向 TLS 通信 http -> https, https -> https）
- name: service-gray
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service-gray
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: service-gray
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
validation_context_sds_secret_config:
# 只需要校验对方
name: validation_context
# 定义 service-purple 集群，service-purple ：同时提供 http 和 https 侦听器，通过 http 接收的请求会自动重定向至 https，并且 https 侦听器强制要求验证客户端证书；front-envoy在其 cluster 配置中，会向该服务发起 https 请求，向其提供自身的客户端证书后，并会验证其数字证书；(双向 TLS 通信)
- name: service-purple
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service-purple
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: service-purple
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_certificate_sds_secret_configs:
# 以上有服务通信的时候提供 client 证书
- name: client_cert
validation_context_sds_secret_config:
# 提供 CA 证书以验证服务端证书
name: validation_context
2.编写 service-gray.yaml envoy 配置文件
root@envoy:~/servicemesh_in_practise-develop/Security/tls-static# cat service-gray.yaml
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_http
address:
socket_address:
address: 0.0.0.0
port_value: 80
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
route_config:
name: local_route
virtual_hosts:
- name: service
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: local_service
http_filters:
- name: envoy.filters.http.router
typed_config: {}
- name: listener_https
address:
socket_address: { address: 0.0.0.0, port_value: 443 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_https
codec_type: AUTO
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
route_config:
name: https_route
virtual_hosts:
- name: https_route
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: local_service
http_filters:
- name: envoy.filters.http.router
typed_config: {}
transport_socket: # DownstreamTlsContext
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates: # 基于DataSource，直接给出证书和私钥文件
certificate_chain:
filename: "/etc/envoy/certs/server.crt"
private_key:
filename: "/etc/envoy/certs/server.key"
clusters:
- name: local_service
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
load_assignment:
cluster_name: local_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
3.编写 service-purple.yaml envoy 配置文件
root@envoy:~/servicemesh_in_practise-develop/Security/tls-static# cat service-purple.yaml
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_http
address:
socket_address:
address: 0.0.0.0
port_value: 80
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
route_config:
name: local_route
virtual_hosts:
- name: service
domains:
- "*"
routes:
- match:
prefix: "/"
redirect:
https_redirect: true
port_redirect: 443
http_filters:
- name: envoy.filters.http.router
typed_config: {}
- name: listener_https
address:
socket_address: { address: 0.0.0.0, port_value: 443 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_https
codec_type: AUTO
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
route_config:
name: https_route
virtual_hosts:
- name: https_route
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: local_service
http_filters:
- name: envoy.filters.http.router
typed_config: {}
transport_socket: # DownstreamTlsContext
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates: # 基于DataSource，直接给出证书和私钥文件
certificate_chain:
filename: "/etc/envoy/certs/server.crt"
private_key:
filename: "/etc/envoy/certs/server.key"
require_client_certificate: true # 强制验证客户端证书
clusters:
- name: local_service
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
load_assignment:
cluster_name: local_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
由此我们的准备工作就做完了，接下来就是通过 docker-compose 启动
5.3.1.4 启动测试
1.启动
root@envoy:~/servicemesh_in_practise-develop/Security/tls-static# docker-compose up
2.测试，查看证书和监听的https套接字
root@envoy:~# curl 10.0.10.10:9901/listeners
listener_http::0.0.0.0:80
listener_https::0.0.0.0:443
# 上面的命令结果显示出，front-envoy上同时监听有http和https相关的套接字
3.查看front-envoy加载的证书
root@envoy:~# curl 10.0.10.10:9901/listeners
#下面的结果显示出，front-envoy已然加载了相关的数字证书
{
"certificates": [
{
"ca_cert": [
{
"path": "/etc/envoy/ca/ca.crt",
"serial_number": "778e3555311639c123fc6e55bf91d956dbb95999",
"subject_alt_names": [],
"days_until_expiration": "3649",
"valid_from": "2021-11-06T08:33:54Z",
"expiration_time": "2031-11-04T08:33:54Z"
}
],
"cert_chain": []
},
……
}
]
}
4.测试访问服务
# 直接向 front-envoy 发起的 http 请求，将会被自动跳转至https服务上。
root@envoy:~# curl -I 10.0.10.10
HTTP/1.1 301 Moved Permanently
location: https://10.0.10.10:443/
date: Fri, 03 Dec 2021 09:24:31 GMT
server: envoy
transfer-encoding: chunked
如下命令结果显示，tls传话已然能正常建立，但curl命令无法任何服务端证书的CA，除非我们给命令指定相应
root@envoy:~# openssl s_client -connect 10.0.10.10:443
CONNECTED(00000005)
depth=0 CN = front-envoy
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = front-envoy
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:CN = front-envoy
i:CN = envoy-ca
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEkDCCAnigAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwEzERMA8GA1UEAwwIZW52
b3ktY2EwHhcNMjExMjAzMDkxNzI1WhcNMzExMjAxMDkxNzI1WjAWMRQwEgYDVQQD
DAtmcm9udC1lbnZveTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALRR
BRSytbGzSZIq6k/9PzNOnDO7f1rY2iAnqbR8mr6F2raaEB4U3q2qtvCs24Z4YXzA
T5wlHsmKCVZ6uF5/tl6ihLCjsoZwfgNcwNX71rRVjEZZo+/9wY0qwh2Oj80+81YA
p0KW5X09yIHdZiWrZh5k8GY2xx4E5Hme+siafPAMi2phZ3sL+U+OoohjfUoQbgDw
HTAWxCYCjYg/8lcxr+FftxmK0OajW/u2sPr6hpB4G8xV7/9qMBOExWxTZnDIIo9E
ds8ibqMu0hJCrW/nYk4ikmpogfKJWaaqG61q2wZx52kDERVhdj/6duuClvRW9Zrk
C3TzB5J+3BNw1zpCAicCAwEAAaOB6jCB5zAJBgNVHRMEAjAAMBEGCWCGSAGG+EIB
AQQEAwIGQDAzBglghkgBhvhCAQ0EJhYkT3BlblNTTCBHZW5lcmF0ZWQgU2VydmVy
IENlcnRpZmljYXRlMB0GA1UdDgQWBBQR/NxtDxDsf8qZc8zeOGp6/lyFxDBOBgNV
HSMERzBFgBQykefWWuna/QoOWMy6lS8ddbOQnKEXpBUwEzERMA8GA1UEAwwIZW52
b3ktY2GCFBP0vx191YYtuMd0m1H9UVFLOaNnMA4GA1UdDwEB/wQEAwIFoDATBgNV
HSUEDDAKBggrBgEFBQcDATANBgkqhkiG9w0BAQsFAAOCAgEAIyS8cLUULVwuDMGK
Y5pcLgJNgQy4mtY63trBKMpMSBhMgtKaiLJuTK3YpJEKB28iGK6qt9MuYZdvyD8C
db3FSVr5E44kaOruYJ+MmJ2UoQoXusfG172bUr7GghJBzhgMnjKvD8I2OzM6ZlA5
u5Vc+yhIxeU1lSQaOrQvIzedyxBIAl8HQnkpKRRSrzsRvNPqeGnHAvfK9knx2f1H
4Ex7II72URa2AZajJf8t6hvU6xG6FSaL96CEcDwAnNtTeo8AXLyD9D4djcBxzBFh
0vCZW0dxcY0Q5G2CXtm3pL+FPNgz1SWAl5O04/QFxm0gp2vq5lYRJY9qdR3EX+hE
Ce8d+sqgGQikayYUvboAIcytEGtkGlxjwx4jlL6dTKGiS3CzR7OVLpDub0ghnDNC
A7b0/1EK5UcFv+bXyMt8Tix6LmgOaIRwboCTcHBCzwpw5+Q81Nxqh9/OvS/bpjYL
sV7P2ZMuhUzisFN2rAds6KT+yDE3M9PSOsaLGbWdpMaKWoEbz89srT0DKHhteSCp
bHWQ/FjR4uh8tC9bps4rHlVliFYysXJmr8u9xH+9DBsuL4EzkHcABLW1irwKZMMZ
owQq4Kd/TQNsPkNxUwHO+6Zv+c5USt+EXJhwlTSKIz47DKNxoYMwoyZPyNMgiqrV
XQdakNsPI+puadV5qvo0t8Q9FvY=
-----END CERTIFICATE-----
subject=CN = front-envoy
issuer=CN = envoy-ca
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1662 bytes and written 392 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
6.停止后清理
# docker-compose down文章转载自运维老狗，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
