Botim Lighthouse · Incident report

Lighthouse restarting every 4 minutes drove ~300 Fabric queries/hour

Date 2026-10-22 Window since ~01:00 UTC Severity high Owner Zhifeng Guo · Core CoE Briefing Lighthouse briefings post

0 · What is happening

Fabric's monitoring attributes 280–320 queries/hour, ~100 CPU-hours per hour, to the botim-lighthouse-fabric app registration since ~01:00 UTC on 2026-10-22 — several times the next-largest consumer, a Power BI refresh.

Root cause, in one line: the production Lighthouse process is being restarted every ~4 minutes, and every boot replays ~45–50 Fabric "warm-up" queries. ~14 boots/hour × ~45–50 statements ≈ the exact rate Fabric reports.

280–320 / hFabric queries attributed to the Lighthouse app registration

~4 minbetween consecutive process starts, measured live

~45–50Fabric statements fired by every process start

Evidence 1 — the restart loop, measured live

Each ETL records the run it performs at process start, so start timestamps in /api/etl/status reveal process boots. Four consecutive samples:

06:53:46Z  →  06:58:03Z (+4m17s)  →  07:02:12Z (+4m09s)  →  07:06:16Z (+4m04s)

crashlytics is a 6-hourly ETL and app-errors a 12-hourly one; both showed "ran 1 minute ago, run count = 1" on every sample. That is only possible if the process keeps dying and starting over. The loop was still running at the last sample (process age 0.4 min).

Evidence 2 — what every boot fires at Fabric

server/index.ts registers three boot-time warm-ups that run unconditionally 45–75 s after start. Their results live in an in-memory cache that dies with the process, so each restart starts from zero and pays the full cost again. That is the amplifier.

Boot warm-up	Fabric statements / boot	Module	Added by
`warmOnboardingFunnelDefaults`	~40 (18 jobs × 1–4) + 2 frontier lookups	`queries/onboarding/funnel-routes.ts`	PR #8 · CE-1896 · 2026-09-24
`warmAppLanguageDefaults`	~3–4	`queries/cx-app-language/routes.ts`	PR #43 · 2026-10-02
`warmOsVersionDefaults`	~1–2	`queries/cx-os-version/routes.ts`	PR #105 · CE-2136 · 2026-10-21

The repeated statements Fabric should show: the onboarding WITH attempts … THRESHOLD_CTE … family, the SELECT CONVERT(char(8),[date],112) … MAX(etl_updated_at) frontier lookups, and the app-language / os-version … WHERE dt IN (…) statements. Please confirm against the Fabric monitoring hub.

Why it started at 01:00 UTC

PR #105 (CE-2136, OS version dashboard) merged on 10-21 and added the third boot warm-up, on top of #8 (CE-1896, onboarding, 09-24) and #43 (app language, 10-02). The deploy that followed is when the loop began. The boot-warm pattern predates these — warmServerTraceDefaults (VPQ-375, July) reads StarRocks with one small query and is harmless. Moving the pattern onto Fabric at ~40 statements per boot and stacking three of them is what changed its nature.

What this is not — ruled out, so nobody chases it

Candidate	What was read	Result
22 scheduled ETLs	`/api/etl/status` on production	All at ≥10-minute intervals; only `fabric-bad-app` reads Fabric (6-hourly, healthy)
Alert engine	`server/alerts/*`	Kibana-only source; daily / weekly / monthly cadence
Anomaly detector	`server/anomaly/*`	No Fabric reads
Fabric client retries	`server/fabric-sql-client.ts`	At most one retry, on connection errors only
Frontend pollers	`web/src` `setInterval` sites	All poll ES / StarRocks endpoints
Developer laptops	Local `.env` of running servers	No app-registration credentials; local servers run as the developer, not the SP
PR #102 Payment Core	`queries/money/payment-core.ts`	Uncached, but request-path only; nothing calls it on a timer

Still to confirm on the VM

The repo has no code path that exits the process (unhandled errors are logged, not fatal; process.exit runs only inside the SIGTERM handler). So the process is being killed externally — most likely pm2 max_memory_restart or a kernel OOM during the heavy boot window (76-hour ES backfill + Crashlytics pull + three Fabric warm-ups all start within 75 s). Production runs under pm2 (deploy/pull_and_deploy.sh); the systemd unit in the repo is legacy.

pm2 describe voip-lighthouse          # restarts / unstable restarts / max memory restart
pm2 logs voip-lighthouse --err --lines 100
dmesg -T | grep -iE "killed process|out of memory"

1 · Stop it now

Immediately: stop the restart loop — pm2 stop voip-lighthouse (accepting downtime), or if the cause is the memory cap, pm2 restart voip-lighthouse --max-memory-restart 0 so the process stops being killed mid-boot. Either stops the Fabric bleed within one cycle.
Today — hotfix PR (Core CoE): make boot warm-ups restart-aware: persist the boot time; if the previous boot was < 30 minutes ago, skip every Fabric warm-up. A crash loop then costs zero Fabric queries. Run the warm-ups serially and staggered (10 minutes after boot, not 45 s) so they stop competing with the boot backfills for memory.
This week — CX module owner + Core CoE: persist warm-up results in MySQL (already the warehouse) instead of process memory, so a restart no longer discards them; review which of the 18 onboarding jobs need pre-warming versus on-demand.
Hygiene: the app registration's client secret was found in a developer worktree .env on a laptop. Rotate it; SP credentials belong only in the VM's /etc/botim-lighthouse.env.

2 · Prevent it at merge time

The gap: nothing today reviews what a PR adds to the boot path or how many Fabric statements it fires per process start. Proposed gate, enforced on every PR to main:

2a · One sanctioned way to register boot work

Introduce bootWarm(name, fn, { fabricBudget }) in server/boot-warm.ts as the only API for start-up work. It enforces restart-aware skip, serial + staggered execution, and a per-boot Fabric statement budget (hard cap, e.g. 10 per warm, 20 total). Raw setTimeout(() => void warm…) or top-level void run…() in server/index.ts becomes a lint failure.

2b · CI check that blocks merge

Extend the existing .github/workflows/pr-review.yml with a boot-cost job that fails the PR when:

server/index.ts gains a new top-level timer or call that reaches fabricQuery outside bootWarm;
any new bootWarm declares a budget above the cap;
a new module imports fabric-sql-client and registers request-path routes without a createCache namespace — the same gap that let #102 ship uncached.

2c · Automated PR review lens

Add a "Boot & Fabric cost" lens to .claude/skills/pr-review/lenses/ beside the existing ETL lens: every PR must state statements per boot and statements per request for each new Fabric route, and flag anything that runs at boot or lacks a cache.

2d · Ownership

CODEOWNERS: server/index.ts, server/fabric-sql-client.ts and server/boot-warm.ts require a Core CoE review. Changes to the boot path stop landing as a side effect of a dashboard PR.

2e · Runtime guardrail (defence in depth)

A per-process Fabric statement counter exposed on /api/health; an alert when a single process exceeds N statements in its first 10 minutes, or when boots per hour exceed 3. This loop ran ~6 hours before anyone noticed because nothing watched either number.

Asks. (1) Someone with VM access to run the three commands above and confirm the kill reason. (2) A Fabric admin to confirm the repeated query texts. (3) CX to co-own the warm-up redesign. Core CoE opens the hotfix PR (§1.2) and the gate PR (§2).

0 · 线上发生了什么

Fabric 监控显示,自 2026-10-22 约 01:00 UTC 起,botim-lighthouse-fabric 这个应用注册每小时发出 280–320 条查询、消耗约 100 CPU-hours——是第二大消费者(一个 Power BI 刷新任务)的数倍。

一句话根因:线上 Lighthouse 进程每约 4 分钟被重启一次,而每次启动都会重放 45–50 条 Fabric "预热"查询。每小时约 14 次启动 × 每次 45–50 条 ≈ Fabric 报告的那个数。

280–320 / 小时归因到 Lighthouse 应用注册的 Fabric 查询

~4 分钟相邻两次进程启动的间隔(实测)

~45–50 条每次进程启动发出的 Fabric 语句

证据一:重启循环(实测)

每个 ETL 都会记录它在进程启动时跑的那一次(运行计数归 1、开始时间刷新),所以 /api/etl/status 里的开始时间能反推进程何时启动。连续四次采样:

06:53:46Z  →  06:58:03Z (+4分17秒)  →  07:02:12Z (+4分09秒)  →  07:06:16Z (+4分04秒)

crashlytics 是 6 小时一次的 ETL,app-errors 是 12 小时一次;每次采样它们都显示"1 分钟前跑过、运行计数 = 1"。只有进程不断死掉再重来才会这样。最后一次采样时循环仍在进行(进程存活 0.4 分钟)。

证据二:每次启动都向 Fabric 发什么

server/index.ts 注册了三个启动预热,启动后 45–75 秒无条件执行。它们的结果存在随进程消亡的内存缓存里,所以每次重启都从零开始、重付全部成本——这就是放大器。

启动预热	每次启动的 Fabric 语句数	模块	引入
`warmOnboardingFunnelDefaults`	约 40(18 个 job × 1–4 条)+ 2 条 frontier 查询	`queries/onboarding/funnel-routes.ts`	PR #8 · CE-1896 · 2026-09-24
`warmAppLanguageDefaults`	约 3–4	`queries/cx-app-language/routes.ts`	PR #43 · 2026-10-02
`warmOsVersionDefaults`	约 1–2	`queries/cx-os-version/routes.ts`	PR #105 · CE-2136 · 2026-10-21

Fabric 上应能看到反复出现的语句:onboarding 的 WITH attempts … THRESHOLD_CTE … 这一族、SELECT CONVERT(char(8),[date],112) … MAX(etl_updated_at) 的 frontier 查询,以及 app-language / os-version 的 … WHERE dt IN (…) 大语句。请在 Fabric 监控中心核对确认。

为什么是 01:00 UTC 开始

PR #105(CE-2136,OS 版本看板)于 10-21 合并,在 #8(CE-1896,onboarding,09-24)和 #43(app language,10-02)之上加入了第三个启动预热。随后的部署就是循环开始的时间。"启动预热"这个模式更早就有——warmServerTraceDefaults(VPQ-375,7 月)读的是 StarRocks、只有一条小查询,无害。把这个模式搬到 Fabric 上、一次约 40 条、再叠三组,性质就变了。

排除项——这些不是原因,不用再查

候选	读了什么	结论
22 个定时 ETL	线上 `/api/etl/status`	全部 ≥10 分钟一次;只有 `fabric-bad-app` 读 Fabric(6 小时一次,健康)
告警引擎	`server/alerts/*`	数据源仅 Kibana;周期为日/周/月
异常检测	`server/anomaly/*`	不读 Fabric
Fabric 客户端重试	`server/fabric-sql-client.ts`	最多重试一次,且仅对连接错误
前端轮询	`web/src` 中的 `setInterval`	全部打 ES / StarRocks 接口
开发者本机	正在运行的本地 server 的 `.env`	没有应用注册凭据;本地 server 以开发者本人身份认证,不计入 SP
PR #102 Payment Core	`queries/money/payment-core.ts`	无缓存,但只在请求路径上;没有任何定时器调用它

仍需在 VM 上确认

仓库里没有任何会让进程退出的代码路径(未捕获错误只记日志、不致命;process.exit 只在 SIGTERM 处理里)。所以进程是被外部终止的——最可能是 pm2 的 max_memory_restart 内存上限,或启动高峰期(76 小时 ES 回填 + Crashlytics 拉取 + 三组 Fabric 预热在 75 秒内同时开跑)触发的内核 OOM。线上实际由 pm2 托管(见 deploy/pull_and_deploy.sh);仓库里的 systemd unit 是旧文档。

pm2 describe voip-lighthouse          # 重启次数 / 不稳定重启 / 内存重启上限
pm2 logs voip-lighthouse --err --lines 100
dmesg -T | grep -iE "killed process|out of memory"

1 · 立即止血

马上:终止重启循环——pm2 stop voip-lighthouse(接受短暂不可用),或者如果原因是内存上限,执行 pm2 restart voip-lighthouse --max-memory-restart 0,让进程不再在启动中途被杀。两种做法都能在一个周期内停止 Fabric 的消耗。
今天——hotfix PR(Core CoE):让启动预热具备重启感知:把启动时间落盘;若上一次启动在 30 分钟以内,跳过全部 Fabric 预热。这样崩溃循环的 Fabric 成本为零。预热改为串行、错峰(启动后 10 分钟再跑,而不是 45 秒),不再与启动期回填争抢内存。
本周——CX 模块负责人 + Core CoE:把预热结果落到 MySQL(本来就是数仓),而不是进程内存,重启不再丢;并复核 onboarding 的 18 个 job 里哪些真需要预热、哪些可以按需。
卫生问题:在一台开发机的 worktree .env 里发现了应用注册的 client secret。请轮换该密钥;SP 凭据只应存在于 VM 的 /etc/botim-lighthouse.env。

2 · 在合并到 main 时强制拦截

缺口在于:今天没有任何环节审查一个 PR 往启动路径里加了什么、以及每次进程启动会发多少条 Fabric 语句。建议对每个合入 main 的 PR 强制执行以下门禁:

2a · 启动期工作只有一个合法入口

在 server/boot-warm.ts 引入 bootWarm(name, fn, { fabricBudget }),作为启动期工作唯一允许的 API。它强制:重启感知跳过、串行+错峰执行、以及每次启动的 Fabric 语句预算(硬上限,例如单个预热 10 条、总计 20 条)。在 server/index.ts 里裸写 setTimeout(() => void warm…) 或顶层 void run…() 直接判为 lint 失败。

2b · 阻止合并的 CI 检查

在现有 .github/workflows/pr-review.yml 中增加一个 boot-cost 作业,出现以下任一情况即让 PR 失败:

server/index.ts 新增了会触达 fabricQuery 且不经由 bootWarm 的顶层定时器或调用;
任何新的 bootWarm 声明的预算超过上限;
新模块引入 fabric-sql-client 并注册请求路径路由却没有 createCache 命名空间——这正是让 #102 无缓存上线的那个缺口。

2c · 自动 PR 评审视角

在 .claude/skills/pr-review/lenses/ 里、现有 ETL 视角旁边,增加"启动与 Fabric 成本"视角:每个 PR 必须写明每条新 Fabric 路由的每次启动语句数和每次请求语句数,并标出任何在启动期运行或没有缓存的内容。

2d · 归属

CODEOWNERS:server/index.ts、server/fabric-sql-client.ts、server/boot-warm.ts 需 Core CoE 评审。启动路径的改动不再作为某个看板 PR 的副作用悄悄落地。

2e · 运行时护栏(纵深防御)

在 /api/health 暴露每进程的 Fabric 语句计数;当单个进程在启动后 10 分钟内超过 N 条、或每小时启动次数超过 3 次时告警。这次循环跑了约 6 小时才被发现,因为没有任何东西在盯这两个数。

需要的支持。(1)有 VM 权限的人执行上面三条命令,确认进程被杀的原因;(2)Fabric 管理员核对重复出现的查询文本;(3)CX 共同负责预热方案的重新设计。Core CoE 负责开 hotfix PR(§1.2)和门禁 PR(§2)。

Core CoE · Lighthouse · 2026-10-22. Figures measured against the production instance and origin/main; VM-side kill reason not yet read at time of writing.