--- the fixture server refuses other paths, databases and accounts, replays SQL as GreptimeDB does, and detects two open requests ok: fixture server refusals, the SQL answer shape and the concurrency guard --- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears --- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears --- a failed state for every notified unit posts no SystemdUnitFailed, and nginx, not notified, posts one --- a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes --- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once --- HostOOMKill last seen before 7 runs of store error resolves 5 min after the store answers again, not on its first answer --- HostOOMKill on stibnite last seen before 7 silent runs resolves 5 min after stibnite reports again, not on its first report --- node_vmstat_oom_kill empty for 30 min while magnetite reports raises one RuleSignalMissing --- magnetite and cinnabar silent for 40 min while the store answers raise RuleSignalMissing for each 10 min after they last reported, none for node_vmstat_oom_kill, and resolve when they report again --- cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again --- cinnabar's / above 85 % for 15 min fires FilesystemFillingWarn at default, then above 95 % for 5 min fires FilesystemFull at high --- WeeklyPipelineTest posts once at 09:02 New York time on both sides of the daylight-saving change, never at 08:59, 09:06 or 09:02 EDT clock time in EST --- 11 instances within 24 h post all 11 FIRINGs and exactly one AlertBudgetExceeded --- 4 instances on stibnite within 24 h post exactly one AlertBudgetExceeded --- 10 FAILED lines of the last 24 h and one FIRING exceed the budget --- FAILED lines older than 24 h are not counted --- an existing failed.log that cannot be read fails the run before any post --- a failed heartbeat fails the run, and the FIRING it left pending is posted once on the next run --- PullRequestQueueWaitBurn answered 0.15 on 65 runs fires once on the 61st, posts nothing on the 62nd to 65th, and resolves once 5 min after its row disappears --- a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual --- one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL --- a unit notified on one host posts no SystemdUnitFailed on another, and nginx, notified nowhere, posts one --- pyrite, with HostOOMKill firing, and electrum silent for 40 min post nothing after the FIRING: no silence, no resolution, no repeat --- no SMART series for 30 min posts nothing while pyrite is silent, then one RuleSignalMissing for each NVMe rule 30 min after pyrite reports --- pyrite's NVMe critical warning on one run fires NvmeCriticalWarning pyrite at once at high --- pyrite, not collecting, reporting without SMART series for 35 min and then silent for 40 min posts nothing === mutant allDatabasesFreeze --- a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual --- /nix/store/3ksgnvzycis2fgfdfci3lnshpnwcfzqx-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.ItMxvPOg3t/actual 2026-11-01 05:48:49.889974485 +0000 @@ -1,4 +1,3 @@ 0 [FIRING] HostOOMKill magnetite high 4 [FIRING] StoreQueryFailing devloop high -5 [RESOLVED] HostOOMKill magnetite high 11 [RESOLVED] StoreQueryFailing devloop high MISS a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual: the alert posts differ from the expected list mutant allDatabasesFreeze rejected: a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual: the alert posts differ from the expected list === mutant alwaysOnMagnetiteOnly --- cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again --- /nix/store/aarcpx64r6vljqa73iglqd5hxmfbycl5-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.xLkVIKe2Sm/actual 2026-11-01 05:48:49.912974529 +0000 @@ -1,2 +0,0 @@ -3 [FIRING] RuleSignalMissing cinnabar default -10 [RESOLVED] RuleSignalMissing cinnabar default MISS cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again: the alert posts differ from the expected list MISS cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again: no posted message carries: selector=system_memory_usage_bytes mutant alwaysOnMagnetiteOnly rejected: cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again: the alert posts differ from the expected list === mutant everyQueryToFleet --- one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL MISS one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL: the server did not record exactly one: REQUEST 0 /v1/sql devloop SELECT CAST(sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) AS DOUBLE) / count(*) AS value FROM ci_build c JOIN (SELECT build_id, min(finished_at) AS first_finished FROM ci_build GROUP BY build_id) f ON c.build_id = f.build_id AND c.finished_at = f.first_finished WHERE c.pr_number IS NOT NULL AND c.started_at IS NOT NULL AND c.finished_at >= now() - INTERVAL '24 hours' HAVING count(*) >= 10 AND sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) * 10 > count(*) mutant everyQueryToFleet rejected: one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL: the server did not record exactly one: REQUEST 0 /v1/sql devloop SELECT CAST(sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) AS DOUBLE) / count(*) AS value FROM ci_build c JOIN (SELECT build_id, min(finished_at) AS first_finished FROM ci_build GROUP BY build_id) f ON c.build_id = f.build_id AND c.finished_at = f.first_finished WHERE c.pr_number IS NOT NULL AND c.started_at IS NOT NULL AND c.finished_at >= now() - INTERVAL '24 hours' HAVING count(*) >= 10 AND sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) * 10 > count(*) === mutant forZero --- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears --- /nix/store/2ngrim690axpydj1bk1ghjafymwc0v6c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.G2xMANtdtP/actual 2026-11-01 05:49:21.571035554 +0000 @@ -1,2 +1,2 @@ -5 [FIRING] HostMemoryPressure magnetite default +0 [FIRING] HostMemoryPressure magnetite default 11 [RESOLVED] HostMemoryPressure magnetite default MISS HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list mutant forZero rejected: HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list === mutant noAbsenceFreeze --- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once --- /nix/store/2w5szkyqg1flnpplqw9sd3fy1894ks1c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.e6zWwGXtPW/actual 2026-11-01 05:49:51.990093955 +0000 @@ -1,2 +1,4 @@ 0 [FIRING] HostOOMKill stibnite high +5 [RESOLVED] HostOOMKill stibnite high +11 [FIRING] HostOOMKill stibnite high 16 [RESOLVED] HostOOMKill stibnite high MISS stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list mutant noAbsenceFreeze rejected: stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list === mutant noHostsUpdate --- magnetite and cinnabar silent for 40 min while the store answers raise RuleSignalMissing for each 10 min after they last reported, none for node_vmstat_oom_kill, and resolve when they report again --- /nix/store/kbmhj18ibd15nbjl6jgr9dkrqlclx8a2-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.tk62OQlomD/actual 2026-11-01 05:50:09.462127400 +0000 @@ -1,4 +1,4 @@ -3 [FIRING] RuleSignalMissing cinnabar default -3 [FIRING] RuleSignalMissing magnetite default +2 [FIRING] RuleSignalMissing cinnabar default +2 [FIRING] RuleSignalMissing magnetite default 10 [RESOLVED] RuleSignalMissing cinnabar default 10 [RESOLVED] RuleSignalMissing magnetite default MISS magnetite and cinnabar silent for 40 min while the store answers raise RuleSignalMissing for each 10 min after they last reported, none for node_vmstat_oom_kill, and resolve when they report again: the alert posts differ from the expected list mutant noHostsUpdate rejected: magnetite and cinnabar silent for 40 min while the store answers raise RuleSignalMissing for each 10 min after they last reported, none for node_vmstat_oom_kill, and resolve when they report again: the alert posts differ from the expected list === mutant noStateWrite --- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.zBfLp9OWxo/state.json: No such file or directory MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 1 exited 2, expected 0: jq: error: Could not open file /build/tmp.zBfLp9OWxo/state.json: No such file or directory --- /nix/store/qfv7fkygc10w316wiynys329fsmlx32w-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.zBfLp9OWxo/actual 2026-11-01 05:50:13.364134860 +0000 @@ -1,2 +1,2 @@ 0 [FIRING] HostOOMKill magnetite high -6 [RESOLVED] HostOOMKill magnetite high +1 [FIRING] HostOOMKill magnetite high MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: the alert posts differ from the expected list mutant noStateWrite rejected: HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.zBfLp9OWxo/state.json: No such file or directory === mutant noStoreFreeze --- a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes --- /nix/store/yc63wbqnip77jf3vlvryjari8p7bs33s-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.Gt6VKq3ZpG/actual 2026-11-01 05:50:36.118178293 +0000 @@ -1,6 +1,6 @@ 0 [FIRING] HostOOMKill magnetite high 4 [FIRING] StoreQueryFailing devloop high 4 [FIRING] StoreQueryFailing fleet high +5 [RESOLVED] HostOOMKill magnetite high 13 [RESOLVED] StoreQueryFailing devloop high 13 [RESOLVED] StoreQueryFailing fleet high -14 [RESOLVED] HostOOMKill magnetite high MISS a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes: the alert posts differ from the expected list mutant noStoreFreeze rejected: a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes: the alert posts differ from the expected list ok: every leg passed