--- the fixture server refuses other paths, databases and accounts, and detects two open requests ok: fixture server refusals and the concurrency guard --- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears --- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears --- a failed state for every notified unit posts no SystemdUnitFailed, and nginx, not notified, posts one --- a store error freezes a firing HostOOMKill, raises StoreQueryFailing after 3 min, heartbeats, and resumes --- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once --- HostOOMKill last seen before 7 runs of store error resolves 5 min after the store answers again, not on its first answer --- HostOOMKill on stibnite last seen before 7 silent runs resolves 5 min after stibnite reports again, not on its first report --- node_vmstat_oom_kill empty for 30 min while magnetite reports raises one RuleSignalMissing --- magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again --- WeeklyPipelineTest posts once at 09:02 New York time on both sides of the daylight-saving change, never at 08:59, 09:06 or 09:02 EDT clock time in EST --- 11 instances within 24 h post all 11 FIRINGs and exactly one AlertBudgetExceeded --- 4 instances on stibnite within 24 h post exactly one AlertBudgetExceeded --- 10 FAILED lines of the last 24 h and one FIRING exceed the budget --- FAILED lines older than 24 h are not counted --- an existing failed.log that cannot be read fails the run before any post --- a failed heartbeat fails the run, and the FIRING it left pending is posted once on the next run === mutant forZero --- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears --- /nix/store/6y66jscp5h48z98aiwfi48jrqx1x1a4x-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.bIpDcUkE7j/actual 2026-10-22 01:05:49.052437165 +0000 @@ -1,2 +1,2 @@ -5 [FIRING] HostMemoryPressure magnetite default +0 [FIRING] HostMemoryPressure magnetite default 11 [RESOLVED] HostMemoryPressure magnetite default MISS HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list mutant forZero rejected: HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list === mutant noAbsenceFreeze --- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once --- /nix/store/wc7jnmxya82m0in69d63pn82jb07r6b5-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.sgLVkgpbar/actual 2026-10-22 01:06:02.000475079 +0000 @@ -1,2 +1,4 @@ 0 [FIRING] HostOOMKill stibnite high +5 [RESOLVED] HostOOMKill stibnite high +11 [FIRING] HostOOMKill stibnite high 16 [RESOLVED] HostOOMKill stibnite high MISS stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list mutant noAbsenceFreeze rejected: stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list === mutant noHostsUpdate --- magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again --- /nix/store/j8qp89cn45j5s16cl1v27ck5pdpwf368-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.KiFppcPt4k/actual 2026-10-22 01:06:11.108501595 +0000 @@ -1,2 +1,2 @@ -3 [FIRING] RuleSignalMissing magnetite default +2 [FIRING] RuleSignalMissing magnetite default 10 [RESOLVED] RuleSignalMissing magnetite default MISS magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again: the alert posts differ from the expected list mutant noHostsUpdate rejected: magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again: the alert posts differ from the expected list === mutant noStateWrite --- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.Ocv26IgyFO/state.json: No such file or directory MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 1 exited 2, expected 0: jq: error: Could not open file /build/tmp.Ocv26IgyFO/state.json: No such file or directory --- /nix/store/6grbrjs6fqnl7rfkpg165dxa6zlnjdha-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.Ocv26IgyFO/actual 2026-10-22 01:06:14.113510316 +0000 @@ -1,2 +1,2 @@ 0 [FIRING] HostOOMKill magnetite high -6 [RESOLVED] HostOOMKill magnetite high +1 [FIRING] HostOOMKill magnetite high MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: the alert posts differ from the expected list mutant noStateWrite rejected: HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.Ocv26IgyFO/state.json: No such file or directory === mutant noStoreFreeze --- a store error freezes a firing HostOOMKill, raises StoreQueryFailing after 3 min, heartbeats, and resumes --- /nix/store/n6m027z9nnbx170kn9hmzcy6793i91nc-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 +++ /build/tmp.cug8nhuaz5/actual 2026-10-22 01:06:25.901544397 +0000 @@ -1,4 +1,4 @@ 0 [FIRING] HostOOMKill magnetite high 4 [FIRING] StoreQueryFailing high +5 [RESOLVED] HostOOMKill magnetite high 13 [RESOLVED] StoreQueryFailing high -14 [RESOLVED] HostOOMKill magnetite high MISS a store error freezes a firing HostOOMKill, raises StoreQueryFailing after 3 min, heartbeats, and resumes: the alert posts differ from the expected list mutant noStoreFreeze rejected: a store error freezes a firing HostOOMKill, raises StoreQueryFailing after 3 min, heartbeats, and resumes: the alert posts differ from the expected list ok: every leg passed