feat: CoreDNS IPv4 上游、03-03 Tomcat 修复、HAProxy 与验证脚本

- Ansible: 部署时自动配置 CoreDNS forward 为 IPv4，避免 ACME 解析失败 - 01-01/01-07: 文档增加 CoreDNS 设置说明 - 03-03: Tomcat webapps.dist 复制、HTTP/HTTPS 双 Ingress、显式 Dashboard IngressRoute - traefik-dashboard-acme: tomcat-acme.yaml、404 排查说明 - HAProxy: 健康检查与 PROXY 配置拆分，18080/18443 部署与验证脚本 Made-with: Cursor
2026-03-22 19:02:46 +08:00
parent de1be1dbe5
commit 8a54cac61f
25 changed files with 924 additions and 113 deletions
--- a/ansible/files/01-08-haproxy/README.md
+++ b/ansible/files/01-08-haproxy/README.md
@@ -1,11 +1,38 @@
 # 01-08 HAProxy 配置

-用于 `docs/01-08-openwrt-haproxy.md`，可与 Ansible 共用（复制到 OpenWrt 或通过 playbook 下发）。
+## 核心目标

-| 文件 | 说明 |
-|------|------|
-| haproxy.cfg | 基础配置，TCP 健康检查 |
-| haproxy-proxy.cfg | 启用 send-proxy-v2（Traefik 真实 IP） |
-| haproxy-proxy-http-tls.cfg | HTTP 检查 + TLS 检查 + PROXY 组合 |
+本目录下的 **所有 `*.cfg` 必须可被 HAProxy 正确解析并符合文档意图**。验证分两层：

-按实际节点 IP 修改 `192.168.2.61`～`192.168.2.64`。80/443 被封时可将 `bind *:80` / `bind *:443` 改为 `*:18080` / `*:18443`。
+| 层次 | 含义 | 如何验证 |
+|------|------|----------|
+| **① 语法正确** | `haproxy -c -f <cfg>` 无致命错误 | 见下文「仅校验 cfg」或主验证脚本第 2 步 |
+| **② 运行与后端** | 在 OpenWrt 上实际监听 18080/18443 时，经第三方主机 curl 可达 K3s/Traefik 后端 | `./scripts/01-08-verify-haproxy.sh`（完整流程，含 curl） |
+
+仓库内 **frontend 已统一为 `18080` / `18443`**（与 LuCI 的 80/443 分离）；backend 仍指向各节点 **80/443**（Traefik 入口）。按环境修改 `192.168.2.61`～`192.168.2.64`。
+
+## 仅校验本目录 cfg（不跑 curl）
+
+仅需确认 **① 语法**，在仓库根目录执行：
+
+```bash
+./scripts/01-08-verify-haproxy.sh --cfg-only
+```
+
+会将本目录全部 `*.cfg` 拷到 OpenWrt 的 `/tmp/haproxy-verify/`，对每台文件执行 `haproxy -c`（与 OpenWrt 上安装的 HAProxy 版本一致）。
+
+**说明**：`haproxy-https.cfg` 含 `ssl crt /etc/ssl/haproxy.pem`；若路由器上**没有**该 pem，语法检查可能失败，脚本会标为 `[SKIP]`。在 OpenWrt 放置有效 pem 后应能通过 `haproxy -c`。
+
+## 文件一览
+
+| 文件 | 说明（对应 `docs/01-08-openwrt-haproxy.md`） |
+|------|-----------------------------------------------|
+| `haproxy-no-check.cfg` | §2 最简；§3.1 在其 `server` 行加 `check` |
+| `haproxy-http.cfg` | §3.2 HTTP 健康检查（明文 80 后端） |
+| `haproxy-tls.cfg` | §3.3 TLS 握手检查（443 后端，`mode tcp`） |
+| `haproxy-https.cfg` | §3.4 HTTPS 应用层检查（需 HAProxy 终结 TLS，由 HAProxy 提供证书） |
+| `haproxy-proxy-http-tls.cfg` | §5 PROXY + HTTP/TLS 检查 |
+
+## 与 Ansible / OpenWrt
+
+可与 Ansible 共用（复制到 OpenWrt 或通过 playbook 下发）。一键把 **uhttpd 80/443 + HAProxy 18080/18443** 落到路由器见 `scripts/01-08-deploy-openwrt-haproxy.sh`。
--- a/ansible/files/01-08-haproxy/haproxy-http.cfg
+++ b/ansible/files/01-08-haproxy/haproxy-http.cfg
@@ -1,10 +1,9 @@
-# 01-08 OpenWrt HAProxy 负载均衡 - 基础配置
-# 文档：docs/01-08-openwrt-haproxy.md
-# 将 192.168.2.61～64 按实际 K3s 节点 IP 修改
+# 01-08 HAProxy - 3.2 HTTP 健康检查（80 明文）
+# backend k3s_http 增加 option httpchk GET /
+# 文档：docs/01-08-openwrt-haproxy.md 第 3.2 节
 global
    log /dev/log local0
    maxconn 4096
-    # 部分 OpenWrt 需 daemon / pidfile，按发行版调整；若无 /dev/log 可改 log 127.0.0.1 local0

 defaults
    mode http
@@ -14,15 +13,16 @@ defaults
    timeout server 30s

 frontend http_in
-    bind *:80
+    bind *:18080
    default_backend k3s_http

 frontend https_in
-    bind *:443
+    bind *:18443
    mode tcp
    default_backend k3s_https

 backend k3s_http
+    option httpchk GET /
    balance roundrobin
    server ylc61 192.168.2.61:80 check
    server ylc62 192.168.2.62:80 check
--- a/ansible/files/01-08-haproxy/haproxy-https.cfg
+++ b/ansible/files/01-08-haproxy/haproxy-https.cfg
@@ -0,0 +1,41 @@
+# 01-08 HAProxy - 3.4 HTTPS 健康检查（443 应用层，HAProxy 终结 TLS，由 HAProxy 提供证书）
+# frontend 需 bind *:443 ssl，backend mode http 连 K3s:443 做 HTTP over TLS 检查
+# 将 your-ingress.example.com 改为实际 Host；将 /etc/ssl/haproxy.pem 改为实际证书路径
+# 自签/内网 CA 用 verify none，生产建议 ca-file
+# 文档：docs/01-08-openwrt-haproxy.md 第 3.4 节
+global
+    log /dev/log local0
+    maxconn 4096
+
+defaults
+    mode http
+    option httplog
+    timeout connect 5s
+    timeout client 30s
+    timeout server 30s
+
+frontend http_in
+    bind *:18080
+    default_backend k3s_http
+
+frontend https_in
+    bind *:18443 ssl crt /etc/ssl/haproxy.pem
+    mode http
+    default_backend k3s_https
+
+backend k3s_http
+    balance roundrobin
+    server ylc61 192.168.2.61:80 check
+    server ylc62 192.168.2.62:80 check
+    server ylc63 192.168.2.63:80 check
+    server ylc64 192.168.2.64:80 check
+
+backend k3s_https
+    mode http
+    option httpchk GET / HTTP/1.1\r\nHost:\ your-ingress.example.com
+    default-server ssl verify none
+    balance roundrobin
+    server ylc61 192.168.2.61:443 check
+    server ylc62 192.168.2.62:443 check
+    server ylc63 192.168.2.63:443 check
+    server ylc64 192.168.2.64:443 check
--- a/ansible/files/01-08-haproxy/haproxy-no-check.cfg
+++ b/ansible/files/01-08-haproxy/haproxy-no-check.cfg
@@ -0,0 +1,38 @@
+# 01-08 OpenWrt HAProxy 负载均衡 - 原生最简（无健康检查）
+# 文档：docs/01-08-openwrt-haproxy.md 第 2 节
+# 将 192.168.2.61～64 按实际 K3s 节点 IP 修改
+# 如需健康检查，见第 3 节对应 cfg
+global
+    log /dev/log local0
+    maxconn 4096
+
+defaults
+    mode http
+    option httplog
+    timeout connect 5s
+    timeout client 30s
+    timeout server 30s
+
+frontend http_in
+    bind *:18080
+    default_backend k3s_http
+
+frontend https_in
+    bind *:18443
+    mode tcp
+    default_backend k3s_https
+
+backend k3s_http
+    balance roundrobin
+    server ylc61 192.168.2.61:80
+    server ylc62 192.168.2.62:80
+    server ylc63 192.168.2.63:80
+    server ylc64 192.168.2.64:80
+
+backend k3s_https
+    mode tcp
+    balance roundrobin
+    server ylc61 192.168.2.61:443
+    server ylc62 192.168.2.62:443
+    server ylc63 192.168.2.63:443
+    server ylc64 192.168.2.64:443
--- a/ansible/files/01-08-haproxy/haproxy-proxy-http-tls.cfg
+++ b/ansible/files/01-08-haproxy/haproxy-proxy-http-tls.cfg
@@ -13,11 +13,11 @@ defaults
    timeout server 30s

 frontend http_in
-    bind *:80
+    bind *:18080
    default_backend k3s_http

 frontend https_in
-    bind *:443
+    bind *:18443
    mode tcp
    default_backend k3s_https

--- a/ansible/files/01-08-haproxy/haproxy-proxy.cfg
+++ b/ansible/files/01-08-haproxy/haproxy-proxy.cfg
@@ -1,37 +0,0 @@
-# 01-08 HAProxy - 启用 PROXY Protocol（send-proxy-v2）
-# 用于 Traefik 获取真实客户端 IP，需配合 Traefik trustedIPs
-# 文档：docs/01-08-openwrt-haproxy.md 第 5 节
-global
-    log /dev/log local0
-    maxconn 4096
-
-defaults
-    mode http
-    option httplog
-    timeout connect 5s
-    timeout client 30s
-    timeout server 30s
-
-frontend http_in
-    bind *:80
-    default_backend k3s_http
-
-frontend https_in
-    bind *:443
-    mode tcp
-    default_backend k3s_https
-
-backend k3s_http
-    balance roundrobin
-    server ylc61 192.168.2.61:80 check send-proxy-v2
-    server ylc62 192.168.2.62:80 check send-proxy-v2
-    server ylc63 192.168.2.63:80 check send-proxy-v2
-    server ylc64 192.168.2.64:80 check send-proxy-v2
-
-backend k3s_https
-    mode tcp
-    balance roundrobin
-    server ylc61 192.168.2.61:443 check send-proxy-v2
-    server ylc62 192.168.2.62:443 check send-proxy-v2
-    server ylc63 192.168.2.63:443 check send-proxy-v2
-    server ylc64 192.168.2.64:443 check send-proxy-v2
--- a/ansible/files/01-08-haproxy/haproxy-tls.cfg
+++ b/ansible/files/01-08-haproxy/haproxy-tls.cfg
@@ -0,0 +1,38 @@
+# 01-08 HAProxy - 3.3 TLS 健康检查（443 握手，mode tcp）
+# backend k3s_https 增加 option ssl-hello-chk
+# 文档：docs/01-08-openwrt-haproxy.md 第 3.3 节
+global
+    log /dev/log local0
+    maxconn 4096
+
+defaults
+    mode http
+    option httplog
+    timeout connect 5s
+    timeout client 30s
+    timeout server 30s
+
+frontend http_in
+    bind *:18080
+    default_backend k3s_http
+
+frontend https_in
+    bind *:18443
+    mode tcp
+    default_backend k3s_https
+
+backend k3s_http
+    balance roundrobin
+    server ylc61 192.168.2.61:80 check
+    server ylc62 192.168.2.62:80 check
+    server ylc63 192.168.2.63:80 check
+    server ylc64 192.168.2.64:80 check
+
+backend k3s_https
+    mode tcp
+    option ssl-hello-chk
+    balance roundrobin
+    server ylc61 192.168.2.61:443 check
+    server ylc62 192.168.2.62:443 check
+    server ylc63 192.168.2.63:443 check
+    server ylc64 192.168.2.64:443 check
--- a/ansible/files/traefik-dashboard-acme/tomcat-acme-test05.yaml
+++ b/ansible/files/traefik-dashboard-acme/tomcat-acme-test05.yaml
@@ -20,6 +20,16 @@ spec:
      containers:
        - name: tomcat
          image: tomcat:9.0
+          # 官方镜像默认 webapps 在 webapps.dist；整目录复制到 webapps（与 Docker Compose cp -a webapps.dist/* webapps 等价）
+          command:
+            - sh
+            - -c
+            - |
+              set -e
+              CATALINA_HOME=/usr/local/tomcat
+              mkdir -p "${CATALINA_HOME}/webapps"
+              cp -a "${CATALINA_HOME}/webapps.dist/." "${CATALINA_HOME}/webapps/"
+              exec "${CATALINA_HOME}/bin/catalina.sh" run
          ports:
            - containerPort: 8080
 ---
@@ -35,6 +45,7 @@ spec:
    - port: 8080
      targetPort: 8080
 ---
+# HTTPS（websecure）
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
@@ -44,6 +55,7 @@ metadata:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls.certresolver: cloudflare
 spec:
+  ingressClassName: traefik
  tls:
    - hosts:
        - test05.jackadam.top
@@ -58,3 +70,25 @@ spec:
                name: tomcat-test05
                port:
                  number: 8080
+---
+# HTTP（web，与 03-02 nginx-matrix-tls 一致：拆成两个 Ingress）
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: tomcat-test05-http
+  namespace: default
+  annotations:
+    traefik.ingress.kubernetes.io/router.entrypoints: web
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: test05.jackadam.top
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: tomcat-test05
+                port:
+                  number: 8080
--- a/ansible/files/traefik-dashboard-acme/traefik-dashboard-acme.yaml
+++ b/ansible/files/traefik-dashboard-acme/traefik-dashboard-acme.yaml
@@ -44,6 +44,19 @@ spec:
    nodeSelector:
      kubernetes.io/hostname: ylc61

-    ingressRoute:
-      dashboard:
-        enabled: true
+---
+# 显式 IngressRoute（与 03-01 一致，确保 /dashboard 可达； Helm ingressRoute.dashboard 在 K3s chart 中未必生效）
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: traefik-dashboard
+  namespace: kube-system
+spec:
+  entryPoints:
+    - web
+  routes:
+    - match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
+      kind: Rule
+      services:
+        - name: api@internal
+          kind: TraefikService
--- a/ansible/group_vars/all.yml
+++ b/ansible/group_vars/all.yml
@@ -20,3 +20,9 @@ k3s_manage_firewalld: true
 # 可开启此开关；默认 true 表示自动按 inventory 中的 k3s_server / k3s_worker 分组打标。
 # 如需完全手动管理角色标签，可改为 false，并参考 `01-02-k3s-工作节点.md` 中的 kubectl 示例。
 k3s_manage_role_labels: true
+
+## CoreDNS 上游 DNS（ACME 需集群内解析 Let's Encrypt）
+# 宿主机若使用 IPv6 DNS（/etc/resolv.conf），K3s Pod 网络仅 IPv4 时无法访问，导致 ACME 申请失败。
+# 将 CoreDNS forward 改为明确 IPv4 地址可规避。见 docs/03-02 常见问题。
+k3s_manage_coredns: true
+coredns_forward_servers: "223.5.5.5 8.8.8.8"
--- a/ansible/playbooks/k3s-init-and-install.yml
+++ b/ansible/playbooks/k3s-init-and-install.yml
@@ -136,6 +136,46 @@
          when: firewalld_check.stdout == 'running'
      when: k3s_manage_firewalld | default(true) | bool

+- name: Configure CoreDNS (IPv4 upstream for ACME)
+  hosts: k3s_server
+  become: true
+  run_once: true
+  vars:
+    k3s_kubeconfig: /etc/rancher/k3s/k3s.yaml
+  tasks:
+    - name: Wait for CoreDNS deployment to be ready
+      ansible.builtin.shell: |
+        KUBECONFIG={{ k3s_kubeconfig }} kubectl rollout status deployment/coredns -n kube-system --timeout=120s
+      when: k3s_manage_coredns | default(true) | bool
+
+    - name: Extract CoreDNS Corefile from ConfigMap
+      ansible.builtin.shell: |
+        KUBECONFIG={{ k3s_kubeconfig }} kubectl get configmap coredns -n kube-system -o jsonpath='{.data.Corefile}' > /tmp/coredns-corefile.txt
+      when: k3s_manage_coredns | default(true) | bool
+
+    - name: Patch Corefile forward to IPv4 (avoid IPv6 upstream in Pod network)
+      ansible.builtin.replace:
+        path: /tmp/coredns-corefile.txt
+        regexp: 'forward \. /etc/resolv\.conf'
+        replace: 'forward . {{ coredns_forward_servers }}'
+      register: coredns_patched
+      when: k3s_manage_coredns | default(true) | bool
+
+    - name: Apply patched CoreDNS ConfigMap and restart
+      ansible.builtin.shell: |
+        KUBECONFIG={{ k3s_kubeconfig }} kubectl create configmap coredns --from-file=Corefile=/tmp/coredns-corefile.txt -n kube-system --dry-run=client -o yaml | KUBECONFIG={{ k3s_kubeconfig }} kubectl apply -f -
+        KUBECONFIG={{ k3s_kubeconfig }} kubectl rollout restart deployment/coredns -n kube-system
+        KUBECONFIG={{ k3s_kubeconfig }} kubectl rollout status deployment/coredns -n kube-system --timeout=60s
+      when:
+        - k3s_manage_coredns | default(true) | bool
+        - coredns_patched is changed
+
+    - name: Remove temp Corefile
+      ansible.builtin.file:
+        path: /tmp/coredns-corefile.txt
+        state: absent
+      when: k3s_manage_coredns | default(true) | bool
+
 - name: 安装后验证 - traefik / nodes / curl
  hosts: k3s_server
  become: true