--- the fixture server refuses other paths, databases and accounts, replays SQL as GreptimeDB does, and detects two open requests ok: fixture server refusals, the SQL answer shape and the concurrency guard --- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears --- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears --- a failed state for every notified unit posts no SystemdUnitFailed, and nginx, not notified, posts one --- a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes --- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once --- HostOOMKill last seen before 7 runs of store error resolves 5 min after the store answers again, not on its first answer --- HostOOMKill on stibnite last seen before 7 silent runs resolves 5 min after stibnite reports again, not on its first report --- node_vmstat_oom_kill empty for 30 min while magnetite reports raises one RuleSignalMissing --- magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again --- WeeklyPipelineTest posts once at 09:02 New York time on both sides of the daylight-saving change, never at 08:59, 09:06 or 09:02 EDT clock time in EST --- 11 instances within 24 h post all 11 FIRINGs and exactly one AlertBudgetExceeded --- 4 instances on stibnite within 24 h post exactly one AlertBudgetExceeded --- 10 FAILED lines of the last 24 h and one FIRING exceed the budget --- FAILED lines older than 24 h are not counted --- an existing failed.log that cannot be read fails the run before any post --- a failed heartbeat fails the run, and the FIRING it left pending is posted once on the next run --- PullRequestQueueWaitBurn answered 0.15 on 65 runs fires once on the 61st, posts nothing on the 62nd to 65th, and resolves once 5 min after its row disappears --- a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual --- one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL === mutant allDatabasesFreeze --- a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual --- /nix/store/3ksgnvzycis2fgfdfci3lnshpnwcfzqx-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.M6GhLqCsjZ/actual 2026-10-23 23:15:25.280984217 +0000 @@ -1,4 +1,3 @@ 0 [FIRING] HostOOMKill magnetite high 4 [FIRING] StoreQueryFailing devloop high -5 [RESOLVED] HostOOMKill magnetite high 11 [RESOLVED] StoreQueryFailing devloop high MISS a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual: the alert posts differ from the expected list mutant allDatabasesFreeze rejected: a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual: the alert posts differ from the expected list === mutant everyQueryToFleet --- one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL MISS one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL: the server did not record exactly one: REQUEST 0 /v1/sql devloop SELECT CAST(sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) AS DOUBLE) / count(*) AS value FROM ci_build c JOIN (SELECT build_id, min(finished_at) AS first_finished FROM ci_build GROUP BY build_id) f ON c.build_id = f.build_id AND c.finished_at = f.first_finished WHERE c.pr_number IS NOT NULL AND c.started_at IS NOT NULL AND c.finished_at >= now() - INTERVAL '24 hours' HAVING count(*) >= 10 AND sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) * 10 > count(*) mutant everyQueryToFleet rejected: one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL: the server did not record exactly one: REQUEST 0 /v1/sql devloop SELECT CAST(sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) AS DOUBLE) / count(*) AS value FROM ci_build c JOIN (SELECT build_id, min(finished_at) AS first_finished FROM ci_build GROUP BY build_id) f ON c.build_id = f.build_id AND c.finished_at = f.first_finished WHERE c.pr_number IS NOT NULL AND c.started_at IS NOT NULL AND c.finished_at >= now() - INTERVAL '24 hours' HAVING count(*) >= 10 AND sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) * 10 > count(*) === mutant forZero --- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears --- /nix/store/2ngrim690axpydj1bk1ghjafymwc0v6c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.GGU87KFo7F/actual 2026-10-23 23:15:58.882052151 +0000 @@ -1,2 +1,2 @@ -5 [FIRING] HostMemoryPressure magnetite default +0 [FIRING] HostMemoryPressure magnetite default 11 [RESOLVED] HostMemoryPressure magnetite default MISS HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list mutant forZero rejected: HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list === mutant noAbsenceFreeze --- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once --- /nix/store/2w5szkyqg1flnpplqw9sd3fy1894ks1c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.ZcG0X8Lnhv/actual 2026-10-23 23:16:41.895140584 +0000 @@ -1,2 +1,4 @@ 0 [FIRING] HostOOMKill stibnite high +5 [RESOLVED] HostOOMKill stibnite high +11 [FIRING] HostOOMKill stibnite high 16 [RESOLVED] HostOOMKill stibnite high MISS stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list mutant noAbsenceFreeze rejected: stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list === mutant noHostsUpdate --- magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again --- /nix/store/zvnynpdbgmllcm71apyvlc53kfxvs8m4-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.bThSO3fZHa/actual 2026-10-23 23:17:03.605185702 +0000 @@ -1,2 +1,2 @@ -3 [FIRING] RuleSignalMissing magnetite default +2 [FIRING] RuleSignalMissing magnetite default 10 [RESOLVED] RuleSignalMissing magnetite default MISS magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again: the alert posts differ from the expected list mutant noHostsUpdate rejected: magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again: the alert posts differ from the expected list === mutant noStateWrite --- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.J21z8Sulds/state.json: No such file or directory MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 1 exited 2, expected 0: jq: error: Could not open file /build/tmp.J21z8Sulds/state.json: No such file or directory --- /nix/store/qfv7fkygc10w316wiynys329fsmlx32w-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.J21z8Sulds/actual 2026-10-23 23:17:08.559196036 +0000 @@ -1,2 +1,2 @@ 0 [FIRING] HostOOMKill magnetite high -6 [RESOLVED] HostOOMKill magnetite high +1 [FIRING] HostOOMKill magnetite high MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: the alert posts differ from the expected list mutant noStateWrite rejected: HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.J21z8Sulds/state.json: No such file or directory === mutant noStoreFreeze --- a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes --- /nix/store/yc63wbqnip77jf3vlvryjari8p7bs33s-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.M0jTzinUWG/actual 2026-10-23 23:17:32.912247011 +0000 @@ -1,6 +1,6 @@ 0 [FIRING] HostOOMKill magnetite high 4 [FIRING] StoreQueryFailing devloop high 4 [FIRING] StoreQueryFailing fleet high +5 [RESOLVED] HostOOMKill magnetite high 13 [RESOLVED] StoreQueryFailing devloop high 13 [RESOLVED] StoreQueryFailing fleet high -14 [RESOLVED] HostOOMKill magnetite high MISS a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes: the alert posts differ from the expected list mutant noStoreFreeze rejected: a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes: the alert posts differ from the expected list ok: every leg passed