Kubernetes环境Traefik部署与应用

发布于2021-09-11,最后编辑于2024-04-20,全文约3024字,阅读时间约7分钟。

Kubernetes Traefik Ingress Helm

更新记录
  • 2021-09-17

    • 部署Traefik时禁用默认Dashboard入口规则;
  • 2021-09-30

    • Traefik版本由v2.5.1更新至v2.5.3
    • Traefik Helm Chart版本由v10.3.2更新至v10.3.6
    • 修复一些配置文件中的字符转义错误;
  • 2021-10-30

    • Traefik Helm Chart版本由v10.3.2更新至v10.6.0
    • 修复创建TLS证书命名空间配置错误;
  • 2021-11-20

    • Traefik版本由v2.5.3更新至v2.5.4
    • Traefik Helm Chart版本由v10.3.2更新至v10.6.2
  • 2022-04-30

    • Traefik版本由v2.5.4更新至v2.6.3
    • Traefik Helm Chart版本由v10.6.2更新至v10.19.4
  • 2022-05-20

    • Traefik版本由v2.6.3更新至v2.6.6
    • Traefik Helm Chart版本由v10.19.4更新至v10.19.5
  • 2022-05-29

    • Traefik版本由v2.6.6更新至v2.7.0
    • Traefik Helm Chart版本由v10.19.5更新至v10.20.0
  • 2022-06-23

    • Traefik版本由v2.7.0更新至v2.7.1
    • Traefik Helm Chart版本由v10.20.0更新至v10.22.0
  • 2022-07-09

    • Traefik版本由v2.7.1更新至v2.8.0
    • Traefik Helm Chart版本由v10.22.0更新至v10.24.0
  • 2022-08-14

    • 安装Traefik时创建并设置默认IngressClass
  • 2022-09-06

    • Traefik版本由v2.8.0更新至v2.8.4
    • Traefik Helm Chart版本由v10.24.0更新至v10.24.2
    • 启用保留客户端IP地址
  • 2022-09-17

    • Traefik版本由v2.8.5更新至v2.8.7
    • Traefik Helm Chart版本由v10.24.3更新至v10.30.1
  • 2022-10-14

    • Traefik版本由v2.8.7更新至v2.9.1
    • Traefik Helm Chart版本由v10.30.1更新至v15.0.0
  • 2022-10-20

    • Traefik Helm Chart版本由v15.0.0更新至v16.1.0
  • 2022-11-08

    • Traefik版本由v2.9.1更新至v2.9.4
    • Traefik Helm Chart版本由v16.1.0更新至v19.0.3
  • 2022-11-12

    • Traefik Helm Chart版本由v19.0.3更新至v20.1.1
  • 2022-11-18

    • Traefik Helm Chart版本由v20.1.1更新至v20.3.0
  • 2022-12-08

    • Traefik版本由v2.9.5更新至v2.9.6
    • Traefik Helm Chart版本由v20.6.0更新至v20.8.0
  • 2023-02-11

    • 添加cert-manager证书管理方案文章引用;
    • Traefik Helm Chart版本由v20.8.0更新至v21.0.0
  • 2023-02-18

    • Traefik版本由v2.9.6更新至v2.9.7
    • Traefik Helm Chart版本由v21.0.0更新至v21.1.0
  • 2023-03-03

    • Traefik版本由v2.9.7更新至v2.9.8
  • 2023-03-22

    • Traefik Helm Chart版本由v21.1.0更新至v21.2.0
  • 2023-03-27

    • Traefik版本由v2.9.8更新至v2.9.9
  • 2023-04-14

    • Traefik版本由v2.9.9更新至v2.9.10
    • Traefik Helm Chart版本由v21.2.0更新至v22.1.0
  • 2023-05-03

    • Traefik版本由v2.9.10更新至v2.10.1
    • Traefik Helm Chart版本由v22.1.0更新至v23.0.1
  • 2023-06-24

    • Traefik版本由v2.10.1更新至v2.10.3
    • Traefik Helm Chart版本由v23.0.1更新至v23.1.0
  • 2023-08-08

    • Traefik版本由v2.10.3更新至v2.10.4
    • Traefik Helm Chart版本由v23.1.0更新至v23.2.0
  • 2023-09-01

    • Traefik Helm Chart版本由v23.2.0更新至v24.0.0
  • 2023-11-18

    • Traefik版本由v2.10.4更新至v2.10.5
    • Traefik Helm Chart版本由v24.0.0更新至v25.0.0
  • 2024-01-01

    • Traefik版本由v2.10.5更新至v2.10.7
    • Traefik Helm Chart版本由v25.0.0更新至v26.0.0
  • 2024-03-31

    • Traefik版本由v2.10.7更新至v2.11.0
    • Traefik Helm Chart版本由v26.0.0更新至v26.1.0

概述

本文用于整理基于Kubernetes环境的Traefik部署与应用,实现Ingress Controller、七层/四层反向代理等功能。

随着各相关组件版本的更新,笔者将在验证通过后对本文进行补充和更新,请参考更新记录

本次演练环境为Kubernetes集群环境,环境配置可参考笔者另一篇笔记《Kubernetes集群部署笔记》。

组件版本

配置过程

安装Traefik

  • 添加Helm仓库

    1helm repo add traefik https://helm.traefik.io/traefik
    2helm repo update
    
  • 安装Traefik

    本次演练中将traefik安装至kube-system命名空间,可根据需要替换。

     1# image.tag=v2.11.0                                                          设置Traefik容器镜像版本
     2# deployment.replicas=3                                                      设置Traefik部署副本数量
     3# pilot.dashboard=false                                                      禁用Dashboard中Pilot链接
     4# ingressRoute.dashboard.enabled=false                                       禁用默认Dashboard入口规则(将在后续步骤中手动创建)
     5# ingressClass.enabled=true                                                  创建IngressClass
     6# ingressClass.isDefaultClass=true                                           设置为默认IngressClass
     7# service.single=false                                                       为TCP和UDP端口分别创建负载均衡服务
     8# service.spec.externalTrafficPolicy=Local                                   启用保留客户端IP地址。注意,这可能仅适合笔者演练环境。参见:
     9#   https://metallb.universe.tf/usage/#traffic-policies
    10#   https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
    11# service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201  设置TCP服务负载均衡地址
    12# entryPoints.*.proxyProtocol.trustedIPs=10.0.0.0/24                         为信任的IP地址启用代理协议
    13helm upgrade --install --namespace kube-system \
    14  --set image.tag=v2.11.0 \
    15  --set deployment.replicas=3 \
    16  --set pilot.dashboard=false \
    17  --set ingressRoute.dashboard.enabled=false \
    18  --set ingressClass.enabled=true \
    19  --set ingressClass.isDefaultClass=true \
    20  --set service.single=false \
    21  --set service.spec.externalTrafficPolicy=Local \
    22  --set service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201 \
    23  --set additionalArguments[0]=--entryPoints.web.proxyProtocol.trustedIPs=10.0.0.0/24 \
    24  --set additionalArguments[1]=--entryPoints.websecure.proxyProtocol.trustedIPs=10.0.0.0/24 \
    25  traefik traefik/traefik --version v26.1.0
    
  • 其他准备工作

    获取traefik服务的负载均衡器地址。执行该命令,记录返回的EXTERNAL-IP地址备用。本次演练环境中,已将local.choral.io*.local.choral.io指向该地址。

    1kubectl get svc traefik -n kube-system
    

    创建一个用于部署演练用对象的命名空间。本次演练中使用apps-choral命名空间,可根据需要替换。

    1kubectl create namespace apps-choral
    

部署Dashboard

  • 创建IngressRoute

    创建一个IngressRoute,用于配置apidashboard的入口规则。

    本次演练中,使用traefik.local.choral.io域名访问Dashboard,可根据需要替换。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: traefik.io/v1alpha1
     3kind: IngressRoute
     4metadata:
     5  name: traefik-dashboard
     6  namespace: apps-choral
     7spec:
     8  entryPoints:
     9    - web
    10  routes:
    11    - match: Host(\`traefik.local.choral.io\`) && (PathPrefix(\`/dashboard\`) || PathPrefix(\`/api\`))
    12      kind: Rule
    13      services:
    14        - name: api@internal
    15          kind: TraefikService
    16EOF
    
  • 启用BasicAuth认证

    首先,创建一个用于保存用户名和密码的Secret,其中的users字段内容可使用htpassword工具生成。本次演练中,认证usernamepassword都是admin

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: v1
     3kind: Secret
     4metadata:
     5  name: traefik-basicauth-secret
     6  namespace: apps-choral
     7data:
     8  users: |2 # htpasswd -nb admin admin | openssl base64
     9    YWRtaW46e1NIQX0wRFBpS3VOSXJyVm1EOElVQ3V3MWhReE5xWmM9Cg==
    10EOF
    

    创建一个Traefik中间件,用于对请求启用BasicAuth认证。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: traefik.io/v1alpha1
     3kind: Middleware
     4metadata:
     5  name: traefik-basicauth
     6  namespace: apps-choral
     7spec:
     8  basicAuth:
     9    realm: traefik.local.choral.io
    10    secret: traefik-basicauth-secret
    11EOF
    

    更新DashboardIngressRoute,启用BasicAuth中间件。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: traefik.io/v1alpha1
     3kind: IngressRoute
     4metadata:
     5  name: traefik-dashboard
     6  namespace: apps-choral
     7spec:
     8  entryPoints:
     9    - web
    10  routes:
    11    - match: Host(\`traefik.local.choral.io\`) && (PathPrefix(\`/dashboard\`) || PathPrefix(\`/api\`))
    12      kind: Rule
    13      services:
    14        - name: api@internal
    15          kind: TraefikService
    16      middlewares:
    17        - name: traefik-basicauth
    18EOF
    

七层反向代理

HTTP应用示例

  • 部署whoami应用

    创建Deployment,部署whoami应用。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: apps/v1
     3kind: Deployment
     4metadata:
     5  name: whoami
     6  namespace: apps-choral
     7spec:
     8  replicas: 3
     9  selector:
    10    matchLabels:
    11      app: whoami
    12  template:
    13    metadata:
    14      labels:
    15        app: whoami
    16    spec:
    17      containers:
    18        - name: whoami
    19          image: traefik/whoami:latest
    20          imagePullPolicy: IfNotPresent
    21          ports:
    22            - containerPort: 80
    23EOF
    

    创建一个用于访问whoami应用的服务。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: v1
     3kind: Service
     4metadata:
     5  name: whoami
     6  namespace: apps-choral
     7spec:
     8  type: ClusterIP
     9  ports:
    10    - protocol: TCP
    11      port: 80
    12  selector:
    13    app: whoami
    14EOF
    

    创建一个Ingress,用于配置whoami应用的入口规则。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: networking.k8s.io/v1
     3kind: Ingress
     4metadata:
     5  name: whoami
     6  namespace: apps-choral
     7  annotations:
     8    traefik.ingress.kubernetes.io/router.entrypoints: web
     9spec:
    10  rules:
    11    - host: local.choral.io
    12      http:
    13        paths:
    14          - path: /
    15            pathType: Prefix
    16            backend:
    17              service:
    18                name: whoami
    19                port:
    20                  number: 80
    21EOF
    

启用TLS(HTTPS)

本次演练使用静态证书配置TLS,该证书被手动创建,应用于local.choral.io*.local.choral.io域名。

有关自动证书管理功能的实现,可参考笔者另一篇笔记《Kubernetes环境cert-manager部署与应用》。

  • 更新Traefik运行参数

     1# additionalArguments[2]=--entrypoints.websecure.http.tls  为websecure默认启用TLS
     2# ports.web.redirectTo.port=websecure                      启用web跳转至websecure
     3helm upgrade --install --namespace kube-system \
     4  --set image.tag=v2.11.0 \
     5  --set deployment.replicas=3 \
     6  --set pilot.dashboard=false \
     7  --set ingressRoute.dashboard.enabled=false \
     8  --set ingressClass.enabled=true \
     9  --set ingressClass.isDefaultClass=true \
    10  --set service.single=false \
    11  --set service.spec.externalTrafficPolicy=Local \
    12  --set service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201 \
    13  --set additionalArguments[0]=--entryPoints.web.proxyProtocol.trustedIPs=10.0.0.0/24 \
    14  --set additionalArguments[1]=--entryPoints.websecure.proxyProtocol.trustedIPs=10.0.0.0/24 \
    15  --set additionalArguments[2]=--entrypoints.websecure.http.tls=true \
    16  --set ports.web.redirectTo.port=websecure \
    17  traefik traefik/traefik --version v26.1.0
    
  • 创建TLS证书Secret

    从已准备好的证书key文件和crt文件创建Secret

    1kubectl create secret tls local-choral-io-tls -n apps-choral \
    2  --key=local.choral.io.key --cert=local.choral.io.crt
    
  • 更新DashboardIngressRoute

    更新DashboardIngressRoute,启用TLS配置。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: traefik.io/v1alpha1
     3kind: IngressRoute
     4metadata:
     5  name: traefik-dashboard
     6  namespace: apps-choral
     7spec:
     8  entryPoints:
     9    - websecure
    10  routes:
    11    - match: Host(\`traefik.local.choral.io\`) && (PathPrefix(\`/dashboard\`) || PathPrefix(\`/api\`))
    12      kind: Rule
    13      services:
    14        - name: api@internal
    15          kind: TraefikService
    16      middlewares:
    17        - name: traefik-basicauth
    18  tls:
    19    secretName: local-choral-io-tls
    20EOF
    
  • 更新whoamiIngress

    更新whoamiIngress,启用TLS配置。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: networking.k8s.io/v1
     3kind: Ingress
     4metadata:
     5  name: whoami
     6  namespace: apps-choral
     7  annotations:
     8    traefik.ingress.kubernetes.io/router.entrypoints: websecure
     9spec:
    10  tls:
    11    - secretName: local-choral-io-tls
    12  rules:
    13    - host: local.choral.io
    14      http:
    15        paths:
    16          - path: /
    17            pathType: Prefix
    18            backend:
    19              service:
    20                name: whoami
    21                port:
    22                  number: 80
    23EOF
    

四层反向代理

TCP应用示例

  • 更新Traefik运行参数

    更新Traefik运行参数,创建新的EntryPoint

     1# ports.whoamitcp.protocol=TCP      网络协议
     2# ports.whoamitcp.port=8081         监听端口
     3# ports.whoamitcp.exposedPort=8081  服务公开端口
     4# ports.whoamitcp.expose=true       是否暴露端口
     5helm upgrade --install --namespace kube-system \
     6  --set image.tag=v2.11.0 \
     7  --set deployment.replicas=3 \
     8  --set pilot.dashboard=false \
     9  --set ingressRoute.dashboard.enabled=false \
    10  --set ingressClass.enabled=true \
    11  --set ingressClass.isDefaultClass=true \
    12  --set service.single=false \
    13  --set service.spec.externalTrafficPolicy=Local \
    14  --set service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201 \
    15  --set additionalArguments[0]=--entryPoints.web.proxyProtocol.trustedIPs=10.0.0.0/24 \
    16  --set additionalArguments[1]=--entryPoints.websecure.proxyProtocol.trustedIPs=10.0.0.0/24 \
    17  --set additionalArguments[2]=--entrypoints.websecure.http.tls=true \
    18  --set ports.web.redirectTo.port=websecure \
    19  --set ports.whoamitcp.protocol=TCP \
    20  --set ports.whoamitcp.port=8081 \
    21  --set ports.whoamitcp.exposedPort=8081 \
    22  --set ports.whoamitcp.expose=true \
    23  traefik traefik/traefik --version v26.1.0
    
  • 部署whoamitcp应用

    创建Deployment,部署whoamitcp应用。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: apps/v1
     3kind: Deployment
     4metadata:
     5  name: whoamitcp
     6  namespace: apps-choral
     7spec:
     8  replicas: 3
     9  selector:
    10    matchLabels:
    11      app: whoamitcp
    12  template:
    13    metadata:
    14      labels:
    15        app: whoamitcp
    16    spec:
    17      containers:
    18        - name: whoamitcp
    19          image: traefik/whoamitcp:latest
    20          imagePullPolicy: IfNotPresent
    21          ports:
    22            - protocol: TCP
    23              containerPort: 8080
    24EOF
    

    创建一个用于访问whoamitcp应用的服务。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: v1
     3kind: Service
     4metadata:
     5  name: whoamitcp
     6  namespace: apps-choral
     7spec:
     8  type: ClusterIP
     9  ports:
    10    - protocol: TCP
    11      port: 8080
    12  selector:
    13    app: whoamitcp
    14EOF
    

    创建一个IngressRouteTCP,用于配置whoamitcp应用的入口规则。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: traefik.io/v1alpha1
     3kind: IngressRouteTCP
     4metadata:
     5  name: whoamitcp
     6  namespace: apps-choral
     7spec:
     8  entryPoints:
     9    - whoamitcp
    10  routes:
    11    - match: HostSNI(\`*\`)
    12      services:
    13        - name: whoamitcp
    14          port: 8080
    15EOF
    

    验证反向代理和服务运行状态。

    1# `10.0.0.201`是`traefik`服务的负载均衡器地址(kubectl get svc traefik -n kube-system)
    2echo "Hello" | socat - tcp4:10.0.0.201:8081
    3# 终端回显如下内容
    4Received: Hello
    

UDP应用示例

  • 更新Traefik运行参数

    更新Traefik运行参数,创建新的EntryPoint

     1# ports.whoamiudp.protocol=UDP                                               网络协议
     2# ports.whoamiudp.port=8082                                                  监听端口
     3# ports.whoamiudp.exposedPort=8082                                           服务公开端口
     4# ports.whoamiudp.expose=true                                                是否暴露端口
     5# service.annotationsUDP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.202  设置UDP服务负载均衡地址
     6helm upgrade --install --namespace kube-system \
     7  --set image.tag=v2.11.0 \
     8  --set deployment.replicas=3 \
     9  --set pilot.dashboard=false \
    10  --set ingressRoute.dashboard.enabled=false \
    11  --set ingressClass.enabled=true \
    12  --set ingressClass.isDefaultClass=true \
    13  --set service.single=false \
    14  --set service.spec.externalTrafficPolicy=Local \
    15  --set service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201 \
    16  --set additionalArguments[0]=--entryPoints.web.proxyProtocol.trustedIPs=10.0.0.0/24 \
    17  --set additionalArguments[1]=--entryPoints.websecure.proxyProtocol.trustedIPs=10.0.0.0/24 \
    18  --set additionalArguments[2]=--entrypoints.websecure.http.tls=true \
    19  --set ports.web.redirectTo.port=websecure \
    20  --set ports.whoamitcp.protocol=TCP \
    21  --set ports.whoamitcp.port=8081 \
    22  --set ports.whoamitcp.exposedPort=8081 \
    23  --set ports.whoamitcp.expose=true \
    24  --set ports.whoamiudp.protocol=UDP \
    25  --set ports.whoamiudp.port=8082 \
    26  --set ports.whoamiudp.exposedPort=8082 \
    27  --set ports.whoamiudp.expose=true \
    28  --set service.annotationsUDP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.202 \
    29  traefik traefik/traefik --version v26.1.0
    
  • 部署whoamiudp应用

    创建Deployment,部署whoamiudp应用。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: apps/v1
     3kind: Deployment
     4metadata:
     5  name: whoamiudp
     6  namespace: apps-choral
     7spec:
     8  replicas: 3
     9  selector:
    10    matchLabels:
    11      app: whoamiudp
    12  template:
    13    metadata:
    14      labels:
    15        app: whoamiudp
    16    spec:
    17      containers:
    18        - name: whoamiudp
    19          image: traefik/whoamiudp:latest
    20          imagePullPolicy: IfNotPresent
    21          ports:
    22            - protocol: UDP
    23              containerPort: 8080
    24EOF
    

    创建一个用于访问whoamiudp应用的服务。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: v1
     3kind: Service
     4metadata:
     5  name: whoamiudp
     6  namespace: apps-choral
     7spec:
     8  type: ClusterIP
     9  ports:
    10    - protocol: UDP
    11      port: 8080
    12  selector:
    13    app: whoamiudp
    14EOF
    

    创建一个IngressRouteUDP,用于配置whoamiudp应用的入口规则。

     1cat <<EOF | kubectl apply -f - > /dev/null
     2apiVersion: traefik.io/v1alpha1
     3kind: IngressRouteUDP
     4metadata:
     5  name: whoamiudp
     6  namespace: apps-choral
     7spec:
     8  entryPoints:
     9    - whoamiudp
    10  routes:
    11    - services:
    12        - name: whoamiudp
    13          port: 8080
    14EOF
    

    验证反向代理和服务运行状态。

    1# `10.0.0.202`是`traefik-udp`服务的负载均衡器地址(kubectl get svc traefik-udp -n kube-system)
    2echo "Hello" | socat - udp4:10.0.0.202:8082
    3# 终端回显如下内容
    4Received: Hello
    

参考资料