--- the fixture server refuses other paths, databases and accounts, and detects two open requests ok: fixture server refusals and the concurrency guard --- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears --- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears --- a failed state for every notified unit posts no SystemdUnitFailed, and nginx, not notified, posts one --- a store error freezes a firing HostOOMKill, raises StoreQueryFailing after 3 min, heartbeats, and resumes --- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once --- HostOOMKill last seen before 7 runs of store error resolves 5 min after the store answers again, not on its first answer --- HostOOMKill on stibnite last seen before 7 silent runs resolves 5 min after stibnite reports again, not on its first report --- node_vmstat_oom_kill empty for 30 min while magnetite reports raises one RuleSignalMissing --- magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again --- WeeklyPipelineTest posts once at 09:02 New York time on both sides of the daylight-saving change, never at 08:59, 09:06 or 09:02 EDT clock time in EST --- 11 instances within 24 h post all 11 FIRINGs and exactly one AlertBudgetExceeded --- 4 instances on stibnite within 24 h post exactly one AlertBudgetExceeded --- 10 FAILED lines of the last 24 h and one FIRING exceed the budget --- FAILED lines older than 24 h are not counted --- an existing failed.log that cannot be read fails the run before any post --- a failed heartbeat fails the run, and the FIRING it left pending is posted once on the next run === mutant forZero --- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears --- /nix/store/2ngrim690axpydj1bk1ghjafymwc0v6c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.BREJzxQBmf/actual 2026-10-21 01:40:03.407793486 +0000 @@ -1,2 +1,2 @@ -5 [FIRING] HostMemoryPressure magnetite default +0 [FIRING] HostMemoryPressure magnetite default 11 [RESOLVED] HostMemoryPressure magnetite default MISS HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list mutant forZero rejected: HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list === mutant noAbsenceFreeze --- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once --- /nix/store/2w5szkyqg1flnpplqw9sd3fy1894ks1c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.L0eSIcAXmg/actual 2026-10-21 01:40:36.031854782 +0000 @@ -1,2 +1,4 @@ 0 [FIRING] HostOOMKill stibnite high +5 [RESOLVED] HostOOMKill stibnite high +11 [FIRING] HostOOMKill stibnite high 16 [RESOLVED] HostOOMKill stibnite high MISS stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list mutant noAbsenceFreeze rejected: stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list === mutant noHostsUpdate --- magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again --- /nix/store/zvnynpdbgmllcm71apyvlc53kfxvs8m4-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.8rT4KF0dK4/actual 2026-10-21 01:41:04.797908677 +0000 @@ -1,2 +1,2 @@ -3 [FIRING] RuleSignalMissing magnetite default +2 [FIRING] RuleSignalMissing magnetite default 10 [RESOLVED] RuleSignalMissing magnetite default MISS magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again: the alert posts differ from the expected list mutant noHostsUpdate rejected: magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again: the alert posts differ from the expected list === mutant noStateWrite --- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.b22P9A78l9/state.json: No such file or directory MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 1 exited 2, expected 0: jq: error: Could not open file /build/tmp.b22P9A78l9/state.json: No such file or directory --- /nix/store/qfv7fkygc10w316wiynys329fsmlx32w-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.b22P9A78l9/actual 2026-10-21 01:41:14.025925938 +0000 @@ -1,2 +1,2 @@ 0 [FIRING] HostOOMKill magnetite high -6 [RESOLVED] HostOOMKill magnetite high +1 [FIRING] HostOOMKill magnetite high MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: the alert posts differ from the expected list mutant noStateWrite rejected: HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.b22P9A78l9/state.json: No such file or directory === mutant noStoreFreeze --- a store error freezes a firing HostOOMKill, raises StoreQueryFailing after 3 min, heartbeats, and resumes --- /nix/store/pmmcnbm8d54rbpdzc14rriwxc5dc59b0-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.CIO7EK1Pij/actual 2026-10-21 01:41:46.043985721 +0000 @@ -1,4 +1,4 @@ 0 [FIRING] HostOOMKill magnetite high 4 [FIRING] StoreQueryFailing high +5 [RESOLVED] HostOOMKill magnetite high 13 [RESOLVED] StoreQueryFailing high -14 [RESOLVED] HostOOMKill magnetite high MISS a store error freezes a firing HostOOMKill, raises StoreQueryFailing after 3 min, heartbeats, and resumes: the alert posts differ from the expected list mutant noStoreFreeze rejected: a store error freezes a firing HostOOMKill, raises StoreQueryFailing after 3 min, heartbeats, and resumes: the alert posts differ from the expected list ok: every leg passed