Kubernetes环境Traefik部署与应用

发布于2021-09-11，最后编辑于2024-04-17，全文约3024字，阅读时间约7分钟。

Kubernetes Traefik Ingress Helm

概述¶

本文用于整理基于Kubernetes环境的Traefik部署与应用，实现Ingress Controller、七层/四层反向代理等功能。

随着各相关组件版本的更新，笔者将在验证通过后对本文进行补充和更新，请参考更新记录。

本次演练环境为Kubernetes集群环境，环境配置可参考笔者另一篇笔记《Kubernetes集群部署笔记》。

组件版本¶

• Traefik v2.11.0

• Traefik Helm Chart v26.1.0

配置过程¶

安装Traefik¶

• 添加Helm仓库

  1helm repo add traefik https://helm.traefik.io/traefik
  2helm repo update
  

• 安装Traefik

  本次演练中将traefik安装至kube-system命名空间，可根据需要替换。

   1# image.tag=v2.11.0                                                          设置Traefik容器镜像版本
   2# deployment.replicas=3                                                      设置Traefik部署副本数量
   3# pilot.dashboard=false                                                      禁用Dashboard中Pilot链接
   4# ingressRoute.dashboard.enabled=false                                       禁用默认Dashboard入口规则（将在后续步骤中手动创建）
   5# ingressClass.enabled=true                                                  创建IngressClass
   6# ingressClass.isDefaultClass=true                                           设置为默认IngressClass
   7# service.single=false                                                       为TCP和UDP端口分别创建负载均衡服务
   8# service.spec.externalTrafficPolicy=Local                                   启用保留客户端IP地址。注意，这可能仅适合笔者演练环境。参见：
   9#   https://metallb.universe.tf/usage/#traffic-policies
  10#   https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
  11# service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201  设置TCP服务负载均衡地址
  12# entryPoints.*.proxyProtocol.trustedIPs=10.0.0.0/24                         为信任的IP地址启用代理协议
  13helm upgrade --install --namespace kube-system \
  14  --set image.tag=v2.11.0 \
  15  --set deployment.replicas=3 \
  16  --set pilot.dashboard=false \
  17  --set ingressRoute.dashboard.enabled=false \
  18  --set ingressClass.enabled=true \
  19  --set ingressClass.isDefaultClass=true \
  20  --set service.single=false \
  21  --set service.spec.externalTrafficPolicy=Local \
  22  --set service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201 \
  23  --set additionalArguments[0]=--entryPoints.web.proxyProtocol.trustedIPs=10.0.0.0/24 \
  24  --set additionalArguments[1]=--entryPoints.websecure.proxyProtocol.trustedIPs=10.0.0.0/24 \
  25  traefik traefik/traefik --version v26.1.0
  

• 其他准备工作

  获取traefik服务的负载均衡器地址。执行该命令，记录返回的EXTERNAL-IP地址备用。本次演练环境中，已将local.choral.io和*.local.choral.io指向该地址。

  1kubectl get svc traefik -n kube-system
  

  创建一个用于部署演练用对象的命名空间。本次演练中使用apps-choral命名空间，可根据需要替换。

  1kubectl create namespace apps-choral
  

部署Dashboard¶

• 创建IngressRoute

  创建一个IngressRoute，用于配置api和dashboard的入口规则。

  本次演练中，使用traefik.local.choral.io域名访问Dashboard，可根据需要替换。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: traefik.io/v1alpha1
   3kind: IngressRoute
   4metadata:
   5  name: traefik-dashboard
   6  namespace: apps-choral
   7spec:
   8  entryPoints:
   9    - web
  10  routes:
  11    - match: Host(\`traefik.local.choral.io\`) && (PathPrefix(\`/dashboard\`) || PathPrefix(\`/api\`))
  12      kind: Rule
  13      services:
  14        - name: api@internal
  15          kind: TraefikService
  16EOF
  

• 启用BasicAuth认证

  首先，创建一个用于保存用户名和密码的Secret，其中的users字段内容可使用htpassword工具生成。本次演练中，认证username和password都是admin。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: v1
   3kind: Secret
   4metadata:
   5  name: traefik-basicauth-secret
   6  namespace: apps-choral
   7data:
   8  users: |2 # htpasswd -nb admin admin | openssl base64
   9    YWRtaW46e1NIQX0wRFBpS3VOSXJyVm1EOElVQ3V3MWhReE5xWmM9Cg==
  10EOF
  

  创建一个Traefik中间件，用于对请求启用BasicAuth认证。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: traefik.io/v1alpha1
   3kind: Middleware
   4metadata:
   5  name: traefik-basicauth
   6  namespace: apps-choral
   7spec:
   8  basicAuth:
   9    realm: traefik.local.choral.io
  10    secret: traefik-basicauth-secret
  11EOF
  

  更新Dashboard的IngressRoute，启用BasicAuth中间件。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: traefik.io/v1alpha1
   3kind: IngressRoute
   4metadata:
   5  name: traefik-dashboard
   6  namespace: apps-choral
   7spec:
   8  entryPoints:
   9    - web
  10  routes:
  11    - match: Host(\`traefik.local.choral.io\`) && (PathPrefix(\`/dashboard\`) || PathPrefix(\`/api\`))
  12      kind: Rule
  13      services:
  14        - name: api@internal
  15          kind: TraefikService
  16      middlewares:
  17        - name: traefik-basicauth
  18EOF
  

七层反向代理¶

HTTP应用示例¶

• 部署whoami应用

  创建Deployment，部署whoami应用。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: apps/v1
   3kind: Deployment
   4metadata:
   5  name: whoami
   6  namespace: apps-choral
   7spec:
   8  replicas: 3
   9  selector:
  10    matchLabels:
  11      app: whoami
  12  template:
  13    metadata:
  14      labels:
  15        app: whoami
  16    spec:
  17      containers:
  18        - name: whoami
  19          image: traefik/whoami:latest
  20          imagePullPolicy: IfNotPresent
  21          ports:
  22            - containerPort: 80
  23EOF
  

  创建一个用于访问whoami应用的服务。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: v1
   3kind: Service
   4metadata:
   5  name: whoami
   6  namespace: apps-choral
   7spec:
   8  type: ClusterIP
   9  ports:
  10    - protocol: TCP
  11      port: 80
  12  selector:
  13    app: whoami
  14EOF
  

  创建一个Ingress，用于配置whoami应用的入口规则。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: networking.k8s.io/v1
   3kind: Ingress
   4metadata:
   5  name: whoami
   6  namespace: apps-choral
   7  annotations:
   8    traefik.ingress.kubernetes.io/router.entrypoints: web
   9spec:
  10  rules:
  11    - host: local.choral.io
  12      http:
  13        paths:
  14          - path: /
  15            pathType: Prefix
  16            backend:
  17              service:
  18                name: whoami
  19                port:
  20                  number: 80
  21EOF
  

启用TLS（HTTPS）¶

本次演练使用静态证书配置TLS，该证书被手动创建，应用于local.choral.io和*.local.choral.io域名。

有关自动证书管理功能的实现，可参考笔者另一篇笔记《Kubernetes环境cert-manager部署与应用》。

• 更新Traefik运行参数

   1# additionalArguments[2]=--entrypoints.websecure.http.tls  为websecure默认启用TLS
   2# ports.web.redirectTo.port=websecure                      启用web跳转至websecure
   3helm upgrade --install --namespace kube-system \
   4  --set image.tag=v2.11.0 \
   5  --set deployment.replicas=3 \
   6  --set pilot.dashboard=false \
   7  --set ingressRoute.dashboard.enabled=false \
   8  --set ingressClass.enabled=true \
   9  --set ingressClass.isDefaultClass=true \
  10  --set service.single=false \
  11  --set service.spec.externalTrafficPolicy=Local \
  12  --set service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201 \
  13  --set additionalArguments[0]=--entryPoints.web.proxyProtocol.trustedIPs=10.0.0.0/24 \
  14  --set additionalArguments[1]=--entryPoints.websecure.proxyProtocol.trustedIPs=10.0.0.0/24 \
  15  --set additionalArguments[2]=--entrypoints.websecure.http.tls=true \
  16  --set ports.web.redirectTo.port=websecure \
  17  traefik traefik/traefik --version v26.1.0
  

• 创建TLS证书Secret

  从已准备好的证书key文件和crt文件创建Secret。

  1kubectl create secret tls local-choral-io-tls -n apps-choral \
  2  --key=local.choral.io.key --cert=local.choral.io.crt
  

• 更新Dashboard的IngressRoute

  更新Dashboard的IngressRoute，启用TLS配置。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: traefik.io/v1alpha1
   3kind: IngressRoute
   4metadata:
   5  name: traefik-dashboard
   6  namespace: apps-choral
   7spec:
   8  entryPoints:
   9    - websecure
  10  routes:
  11    - match: Host(\`traefik.local.choral.io\`) && (PathPrefix(\`/dashboard\`) || PathPrefix(\`/api\`))
  12      kind: Rule
  13      services:
  14        - name: api@internal
  15          kind: TraefikService
  16      middlewares:
  17        - name: traefik-basicauth
  18  tls:
  19    secretName: local-choral-io-tls
  20EOF
  

• 更新whoami的Ingress

  更新whoami的Ingress，启用TLS配置。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: networking.k8s.io/v1
   3kind: Ingress
   4metadata:
   5  name: whoami
   6  namespace: apps-choral
   7  annotations:
   8    traefik.ingress.kubernetes.io/router.entrypoints: websecure
   9spec:
  10  tls:
  11    - secretName: local-choral-io-tls
  12  rules:
  13    - host: local.choral.io
  14      http:
  15        paths:
  16          - path: /
  17            pathType: Prefix
  18            backend:
  19              service:
  20                name: whoami
  21                port:
  22                  number: 80
  23EOF
  

四层反向代理¶

TCP应用示例¶

• 更新Traefik运行参数

  更新Traefik运行参数，创建新的EntryPoint。

   1# ports.whoamitcp.protocol=TCP      网络协议
   2# ports.whoamitcp.port=8081         监听端口
   3# ports.whoamitcp.exposedPort=8081  服务公开端口
   4# ports.whoamitcp.expose=true       是否暴露端口
   5helm upgrade --install --namespace kube-system \
   6  --set image.tag=v2.11.0 \
   7  --set deployment.replicas=3 \
   8  --set pilot.dashboard=false \
   9  --set ingressRoute.dashboard.enabled=false \
  10  --set ingressClass.enabled=true \
  11  --set ingressClass.isDefaultClass=true \
  12  --set service.single=false \
  13  --set service.spec.externalTrafficPolicy=Local \
  14  --set service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201 \
  15  --set additionalArguments[0]=--entryPoints.web.proxyProtocol.trustedIPs=10.0.0.0/24 \
  16  --set additionalArguments[1]=--entryPoints.websecure.proxyProtocol.trustedIPs=10.0.0.0/24 \
  17  --set additionalArguments[2]=--entrypoints.websecure.http.tls=true \
  18  --set ports.web.redirectTo.port=websecure \
  19  --set ports.whoamitcp.protocol=TCP \
  20  --set ports.whoamitcp.port=8081 \
  21  --set ports.whoamitcp.exposedPort=8081 \
  22  --set ports.whoamitcp.expose=true \
  23  traefik traefik/traefik --version v26.1.0
  

• 部署whoamitcp应用

  创建Deployment，部署whoamitcp应用。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: apps/v1
   3kind: Deployment
   4metadata:
   5  name: whoamitcp
   6  namespace: apps-choral
   7spec:
   8  replicas: 3
   9  selector:
  10    matchLabels:
  11      app: whoamitcp
  12  template:
  13    metadata:
  14      labels:
  15        app: whoamitcp
  16    spec:
  17      containers:
  18        - name: whoamitcp
  19          image: traefik/whoamitcp:latest
  20          imagePullPolicy: IfNotPresent
  21          ports:
  22            - protocol: TCP
  23              containerPort: 8080
  24EOF
  

  创建一个用于访问whoamitcp应用的服务。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: v1
   3kind: Service
   4metadata:
   5  name: whoamitcp
   6  namespace: apps-choral
   7spec:
   8  type: ClusterIP
   9  ports:
  10    - protocol: TCP
  11      port: 8080
  12  selector:
  13    app: whoamitcp
  14EOF
  

  创建一个IngressRouteTCP，用于配置whoamitcp应用的入口规则。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: traefik.io/v1alpha1
   3kind: IngressRouteTCP
   4metadata:
   5  name: whoamitcp
   6  namespace: apps-choral
   7spec:
   8  entryPoints:
   9    - whoamitcp
  10  routes:
  11    - match: HostSNI(\`*\`)
  12      services:
  13        - name: whoamitcp
  14          port: 8080
  15EOF
  

  验证反向代理和服务运行状态。

  1# `10.0.0.201`是`traefik`服务的负载均衡器地址（kubectl get svc traefik -n kube-system）
  2echo "Hello" | socat - tcp4:10.0.0.201:8081
  3# 终端回显如下内容
  4Received: Hello
  

UDP应用示例¶

• 更新Traefik运行参数

  更新Traefik运行参数，创建新的EntryPoint。

   1# ports.whoamiudp.protocol=UDP                                               网络协议
   2# ports.whoamiudp.port=8082                                                  监听端口
   3# ports.whoamiudp.exposedPort=8082                                           服务公开端口
   4# ports.whoamiudp.expose=true                                                是否暴露端口
   5# service.annotationsUDP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.202  设置UDP服务负载均衡地址
   6helm upgrade --install --namespace kube-system \
   7  --set image.tag=v2.11.0 \
   8  --set deployment.replicas=3 \
   9  --set pilot.dashboard=false \
  10  --set ingressRoute.dashboard.enabled=false \
  11  --set ingressClass.enabled=true \
  12  --set ingressClass.isDefaultClass=true \
  13  --set service.single=false \
  14  --set service.spec.externalTrafficPolicy=Local \
  15  --set service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201 \
  16  --set additionalArguments[0]=--entryPoints.web.proxyProtocol.trustedIPs=10.0.0.0/24 \
  17  --set additionalArguments[1]=--entryPoints.websecure.proxyProtocol.trustedIPs=10.0.0.0/24 \
  18  --set additionalArguments[2]=--entrypoints.websecure.http.tls=true \
  19  --set ports.web.redirectTo.port=websecure \
  20  --set ports.whoamitcp.protocol=TCP \
  21  --set ports.whoamitcp.port=8081 \
  22  --set ports.whoamitcp.exposedPort=8081 \
  23  --set ports.whoamitcp.expose=true \
  24  --set ports.whoamiudp.protocol=UDP \
  25  --set ports.whoamiudp.port=8082 \
  26  --set ports.whoamiudp.exposedPort=8082 \
  27  --set ports.whoamiudp.expose=true \
  28  --set service.annotationsUDP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.202 \
  29  traefik traefik/traefik --version v26.1.0
  

• 部署whoamiudp应用

  创建Deployment，部署whoamiudp应用。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: apps/v1
   3kind: Deployment
   4metadata:
   5  name: whoamiudp
   6  namespace: apps-choral
   7spec:
   8  replicas: 3
   9  selector:
  10    matchLabels:
  11      app: whoamiudp
  12  template:
  13    metadata:
  14      labels:
  15        app: whoamiudp
  16    spec:
  17      containers:
  18        - name: whoamiudp
  19          image: traefik/whoamiudp:latest
  20          imagePullPolicy: IfNotPresent
  21          ports:
  22            - protocol: UDP
  23              containerPort: 8080
  24EOF
  

  创建一个用于访问whoamiudp应用的服务。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: v1
   3kind: Service
   4metadata:
   5  name: whoamiudp
   6  namespace: apps-choral
   7spec:
   8  type: ClusterIP
   9  ports:
  10    - protocol: UDP
  11      port: 8080
  12  selector:
  13    app: whoamiudp
  14EOF
  

  创建一个IngressRouteUDP，用于配置whoamiudp应用的入口规则。

   1cat <<EOF | kubectl apply -f - > /dev/null
   2apiVersion: traefik.io/v1alpha1
   3kind: IngressRouteUDP
   4metadata:
   5  name: whoamiudp
   6  namespace: apps-choral
   7spec:
   8  entryPoints:
   9    - whoamiudp
  10  routes:
  11    - services:
  12        - name: whoamiudp
  13          port: 8080
  14EOF
  

  验证反向代理和服务运行状态。

  1# `10.0.0.202`是`traefik-udp`服务的负载均衡器地址（kubectl get svc traefik-udp -n kube-system）
  2echo "Hello" | socat - udp4:10.0.0.202:8082
  3# 终端回显如下内容
  4Received: Hello
  

参考资料¶

• https://github.com/traefik/traefik-helm-chart

• https://doc.traefik.io/traefik/providers/kubernetes-crd/

• https://doc.traefik.io/traefik/providers/kubernetes-ingress/

• https://doc.traefik.io/traefik/middlewares/http/basicauth/