Kubernetes环境Traefik部署与应用

概述

本文用于整理基于Kubernetes环境的Traefik部署与应用，实现Ingress Controller、七层/四层反向代理等功能。

随着各相关组件版本的更新，笔者将在验证通过后对本文进行补充和更新，请参考更新记录。

本次演练环境为Kubernetes集群环境，环境配置可参考笔者另一篇笔记《Kubernetes集群部署笔记》。

组件版本

• Traefik v3.6.12

• Traefik Helm Chart v39.0.7

配置过程

安装Traefik

• 添加Helm仓库

  helm repo add traefik https://helm.traefik.io/traefik
  helm repo update traefik

• 安装Traefik

  本次演练中将traefik安装至kube-system命名空间，可根据需要替换。

  # image.tag=v3.6.12                                                          设置Traefik容器镜像版本
  # image.registry=quay.io                                                     设置Traefik容器镜像服务
  # image.repository=choral-k8s/traefik                                        设置Traefik容器镜像仓库
  # deployment.replicas=3                                                      设置Traefik部署副本数量
  # ingressRoute.dashboard.enabled=false                                       禁用默认Dashboard入口规则（将在后续步骤中手动创建）
  # ingressClass.enabled=true                                                  创建IngressClass
  # ingressClass.isDefaultClass=true                                           设置为默认IngressClass
  # service.single=false                                                       为TCP和UDP端口分别创建负载均衡服务
  # service.spec.externalTrafficPolicy=Local                                   启用保留客户端IP地址。注意，这可能仅适合笔者演练环境。参见：
  #   https://metallb.universe.tf/usage/#traffic-policies
  #   https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
  # service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201  设置TCP服务负载均衡地址
  # ports.*.proxyProtocol.trustedIPs[0]=10.0.0.0/24                            为信任的IP地址启用代理协议
  helm upgrade --install --namespace kube-system \
    --set image.tag=v3.6.12 \
    --set image.registry=quay.io \
    --set image.repository=choral-k8s/traefik \
    --set deployment.replicas=3 \
    --set ingressRoute.dashboard.enabled=false \
    --set ingressClass.enabled=true \
    --set ingressClass.isDefaultClass=true \
    --set service.single=false \
    --set service.spec.externalTrafficPolicy=Local \
    --set service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201 \
    --set ports.websecure.proxyProtocol.trustedIPs[0]=10.0.0.0/24 \
    --set ports.web.proxyProtocol.trustedIPs[0]=10.0.0.0/24 \
    traefik traefik/traefik --version 39.0.7

• 其他准备工作

  获取traefik服务的负载均衡器地址。执行该命令，记录返回的EXTERNAL-IP地址备用。本次演练环境中，已将local.choral.io和*.local.choral.io指向该地址。

  kubectl get svc traefik -n kube-system

  创建一个用于部署演练用对象的命名空间。本次演练中使用apps-choral命名空间，可根据需要替换。

  kubectl create namespace apps-choral

部署Dashboard

• 创建IngressRoute

  创建一个IngressRoute，用于配置api和dashboard的入口规则。

  本次演练中，使用traefik.local.choral.io域名访问Dashboard，可根据需要替换。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: traefik.io/v1alpha1
  kind: IngressRoute
  metadata:
    name: traefik-dashboard
    namespace: apps-choral
  spec:
    entryPoints:
      - web
    routes:
      - match: Host(\`traefik.local.choral.io\`) && (PathPrefix(\`/dashboard\`) || PathPrefix(\`/api\`))
        kind: Rule
        services:
          - name: api@internal
            kind: TraefikService
  EOF

• 启用BasicAuth认证

  首先，创建一个用于保存用户名和密码的Secret，其中的users字段内容可使用htpasswd工具生成。本次演练中，认证username和password都是admin。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: v1
  kind: Secret
  metadata:
    name: traefik-basicauth-secret
    namespace: apps-choral
  data:
    users: |2 # htpasswd -nb admin admin | openssl base64
      YWRtaW46e1NIQX0wRFBpS3VOSXJyVm1EOElVQ3V3MWhReE5xWmM9Cg==
  EOF

  创建一个Traefik中间件，用于对请求启用BasicAuth认证。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: traefik.io/v1alpha1
  kind: Middleware
  metadata:
    name: traefik-basicauth
    namespace: apps-choral
  spec:
    basicAuth:
      realm: traefik.local.choral.io
      secret: traefik-basicauth-secret
  EOF

  更新Dashboard的IngressRoute，启用BasicAuth中间件。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: traefik.io/v1alpha1
  kind: IngressRoute
  metadata:
    name: traefik-dashboard
    namespace: apps-choral
  spec:
    entryPoints:
      - web
    routes:
      - match: Host(\`traefik.local.choral.io\`) && (PathPrefix(\`/dashboard\`) || PathPrefix(\`/api\`))
        kind: Rule
        services:
          - name: api@internal
            kind: TraefikService
        middlewares:
          - name: traefik-basicauth
  EOF

七层反向代理

HTTP应用示例

• 部署whoami应用

  创建Deployment，部署whoami应用。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: whoami
    namespace: apps-choral
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: whoami
    template:
      metadata:
        labels:
          app: whoami
      spec:
        containers:
          - name: whoami
            image: traefik/whoami:latest
            imagePullPolicy: IfNotPresent
            ports:
              - containerPort: 80
  EOF

  创建一个用于访问whoami应用的服务。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: v1
  kind: Service
  metadata:
    name: whoami
    namespace: apps-choral
  spec:
    type: ClusterIP
    ports:
      - protocol: TCP
        port: 80
    selector:
      app: whoami
  EOF

  创建一个Ingress，用于配置whoami应用的入口规则。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: whoami
    namespace: apps-choral
    annotations:
      traefik.ingress.kubernetes.io/router.entrypoints: web
  spec:
    rules:
      - host: local.choral.io
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: whoami
                  port:
                    number: 80
  EOF

启用TLS（HTTPS）

本次演练使用静态证书配置TLS，该证书被手动创建，应用于local.choral.io和*.local.choral.io域名。

有关自动证书管理功能的实现，可参考笔者另一篇笔记《Kubernetes环境cert-manager部署与应用》。

• 更新Traefik运行参数

  # additionalArguments[2]=--entrypoints.websecure.http.tls  为websecure默认启用TLS
  # ports.web.http.redirections.entryPoint.to=websecure      启用web跳转至websecure
  # ports.web.http.redirections.entryPoint.scheme=https      启用web跳转至websecure
  # ports.web.http.redirections.entryPoint.permanent=true    启用web跳转至websecure
  helm upgrade --install --namespace kube-system \
    --set image.tag=v3.6.12 \
    --set image.registry=quay.io \
    --set image.repository=choral-k8s/traefik \
    --set deployment.replicas=3 \
    --set ingressRoute.dashboard.enabled=false \
    --set ingressClass.enabled=true \
    --set ingressClass.isDefaultClass=true \
    --set service.single=false \
    --set service.spec.externalTrafficPolicy=Local \
    --set service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201 \
    --set ports.websecure.proxyProtocol.trustedIPs[0]=10.0.0.0/24 \
    --set ports.web.proxyProtocol.trustedIPs[0]=10.0.0.0/24 \
    --set ports.web.http.redirections.entryPoint.to=websecure \
    --set ports.web.http.redirections.entryPoint.scheme=https \
    --set ports.web.http.redirections.entryPoint.permanent=true \
    traefik traefik/traefik --version 39.0.7

• 创建TLS证书Secret

  从已准备好的证书key文件和crt文件创建Secret。

  kubectl create secret tls local-choral-io-tls -n apps-choral \
    --key=local.choral.io.key --cert=local.choral.io.crt

• 更新Dashboard的IngressRoute

  更新Dashboard的IngressRoute，启用TLS配置。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: traefik.io/v1alpha1
  kind: IngressRoute
  metadata:
    name: traefik-dashboard
    namespace: apps-choral
  spec:
    entryPoints:
      - websecure
    routes:
      - match: Host(\`traefik.local.choral.io\`) && (PathPrefix(\`/dashboard\`) || PathPrefix(\`/api\`))
        kind: Rule
        services:
          - name: api@internal
            kind: TraefikService
        middlewares:
          - name: traefik-basicauth
    tls:
      secretName: local-choral-io-tls
  EOF

• 更新whoami的Ingress

  更新whoami的Ingress，启用TLS配置。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: whoami
    namespace: apps-choral
    annotations:
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
  spec:
    tls:
      - secretName: local-choral-io-tls
    rules:
      - host: local.choral.io
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: whoami
                  port:
                    number: 80
  EOF

四层反向代理

TCP应用示例

• 更新Traefik运行参数

  更新Traefik运行参数，创建新的EntryPoint。

  # ports.whoamitcp.protocol=TCP         网络协议
  # ports.whoamitcp.port=8081            监听端口
  # ports.whoamitcp.exposedPort=8081     服务公开端口
  # ports.whoamitcp.expose.default=true  是否暴露端口
  helm upgrade --install --namespace kube-system \
    --set image.tag=v3.6.12 \
    --set image.registry=quay.io \
    --set image.repository=choral-k8s/traefik \
    --set deployment.replicas=3 \
    --set ingressRoute.dashboard.enabled=false \
    --set ingressClass.enabled=true \
    --set ingressClass.isDefaultClass=true \
    --set service.single=false \
    --set service.spec.externalTrafficPolicy=Local \
    --set service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201 \
    --set ports.websecure.proxyProtocol.trustedIPs[0]=10.0.0.0/24 \
    --set ports.web.proxyProtocol.trustedIPs[0]=10.0.0.0/24 \
    --set ports.web.http.redirections.entryPoint.to=websecure \
    --set ports.web.http.redirections.entryPoint.scheme=https \
    --set ports.web.http.redirections.entryPoint.permanent=true \
    --set ports.whoamitcp.protocol=TCP \
    --set ports.whoamitcp.port=8081 \
    --set ports.whoamitcp.exposedPort=8081 \
    --set ports.whoamitcp.expose.default=true \
    traefik traefik/traefik --version 39.0.7

• 部署whoamitcp应用

  创建Deployment，部署whoamitcp应用。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: whoamitcp
    namespace: apps-choral
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: whoamitcp
    template:
      metadata:
        labels:
          app: whoamitcp
      spec:
        containers:
          - name: whoamitcp
            image: traefik/whoamitcp:latest
            imagePullPolicy: IfNotPresent
            ports:
              - protocol: TCP
                containerPort: 8080
  EOF

  创建一个用于访问whoamitcp应用的服务。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: v1
  kind: Service
  metadata:
    name: whoamitcp
    namespace: apps-choral
  spec:
    type: ClusterIP
    ports:
      - protocol: TCP
        port: 8080
    selector:
      app: whoamitcp
  EOF

  创建一个IngressRouteTCP，用于配置whoamitcp应用的入口规则。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: traefik.io/v1alpha1
  kind: IngressRouteTCP
  metadata:
    name: whoamitcp
    namespace: apps-choral
  spec:
    entryPoints:
      - whoamitcp
    routes:
      - match: HostSNI(\`*\`)
        services:
          - name: whoamitcp
            port: 8080
  EOF

  验证反向代理和服务运行状态。

  # `10.0.0.201`是`traefik`服务的负载均衡器地址（kubectl get svc traefik -n kube-system）
  echo "Hello" | socat - tcp4:10.0.0.201:8081
  # 终端回显如下内容
  Received: Hello

UDP应用示例

• 更新Traefik运行参数

  更新Traefik运行参数，创建新的EntryPoint。

  # ports.whoamiudp.protocol=UDP                                               网络协议
  # ports.whoamiudp.port=8082                                                  监听端口
  # ports.whoamiudp.exposedPort=8082                                           服务公开端口
  # ports.whoamiudp.expose.default=true                                        是否暴露端口
  # service.annotationsUDP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.202  设置UDP服务负载均衡地址
  helm upgrade --install --namespace kube-system \
    --set image.tag=v3.6.12 \
    --set image.registry=quay.io \
    --set image.repository=choral-k8s/traefik \
    --set deployment.replicas=3 \
    --set ingressRoute.dashboard.enabled=false \
    --set ingressClass.enabled=true \
    --set ingressClass.isDefaultClass=true \
    --set service.single=false \
    --set service.spec.externalTrafficPolicy=Local \
    --set service.annotationsTCP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.201 \
    --set ports.websecure.proxyProtocol.trustedIPs[0]=10.0.0.0/24 \
    --set ports.web.proxyProtocol.trustedIPs[0]=10.0.0.0/24 \
    --set ports.web.http.redirections.entryPoint.to=websecure \
    --set ports.web.http.redirections.entryPoint.scheme=https \
    --set ports.web.http.redirections.entryPoint.permanent=true \
    --set ports.whoamitcp.protocol=TCP \
    --set ports.whoamitcp.port=8081 \
    --set ports.whoamitcp.exposedPort=8081 \
    --set ports.whoamitcp.expose.default=true \
    --set ports.whoamiudp.protocol=UDP \
    --set ports.whoamiudp.port=8082 \
    --set ports.whoamiudp.exposedPort=8082 \
    --set ports.whoamiudp.expose.default=true \
    --set service.annotationsUDP."metallb\.universe\.tf/loadBalancerIPs"=10.0.0.202 \
    traefik traefik/traefik --version 39.0.7

• 部署whoamiudp应用

  创建Deployment，部署whoamiudp应用。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: whoamiudp
    namespace: apps-choral
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: whoamiudp
    template:
      metadata:
        labels:
          app: whoamiudp
      spec:
        containers:
          - name: whoamiudp
            image: traefik/whoamiudp:latest
            imagePullPolicy: IfNotPresent
            ports:
              - protocol: UDP
                containerPort: 8080
  EOF

  创建一个用于访问whoamiudp应用的服务。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: v1
  kind: Service
  metadata:
    name: whoamiudp
    namespace: apps-choral
  spec:
    type: ClusterIP
    ports:
      - protocol: UDP
        port: 8080
    selector:
      app: whoamiudp
  EOF

  创建一个IngressRouteUDP，用于配置whoamiudp应用的入口规则。

  cat <<EOF | kubectl apply -f - > /dev/null
  apiVersion: traefik.io/v1alpha1
  kind: IngressRouteUDP
  metadata:
    name: whoamiudp
    namespace: apps-choral
  spec:
    entryPoints:
      - whoamiudp
    routes:
      - services:
          - name: whoamiudp
            port: 8080
  EOF

  验证反向代理和服务运行状态。

  # `10.0.0.202`是`traefik-udp`服务的负载均衡器地址（kubectl get svc traefik-udp -n kube-system）
  echo "Hello" | socat - udp4:10.0.0.202:8082
  # 终端回显如下内容
  Received: Hello

参考资料

• https://github.com/traefik/traefik-helm-chart

• https://doc.traefik.io/traefik/providers/kubernetes-crd/

• https://doc.traefik.io/traefik/providers/kubernetes-ingress/

• https://doc.traefik.io/traefik/middlewares/http/basicauth/