fleet-alert-fixtures
checks.x86_64-linux.fleet-alert
· build #1761
· raw
1--- the fixture server refuses other paths, databases and accounts, and detects two open requests2ok: fixture server refusals and the concurrency guard3--- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears4--- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears5--- a failed state for every notified unit posts no SystemdUnitFailed, and nginx, not notified, posts one6--- a store error freezes a firing HostOOMKill, raises StoreQueryFailing after 3 min, heartbeats, and resumes7--- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once8--- HostOOMKill last seen before 7 runs of store error resolves 5 min after the store answers again, not on its first answer9--- HostOOMKill on stibnite last seen before 7 silent runs resolves 5 min after stibnite reports again, not on its first report10--- node_vmstat_oom_kill empty for 30 min while magnetite reports raises one RuleSignalMissing11--- magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again12--- WeeklyPipelineTest posts once at 09:02 New York time on both sides of the daylight-saving change, never at 08:59, 09:06 or 09:02 EDT clock time in EST13--- 11 instances within 24 h post all 11 FIRINGs and exactly one AlertBudgetExceeded14--- 4 instances on stibnite within 24 h post exactly one AlertBudgetExceeded15--- 10 FAILED lines of the last 24 h and one FIRING exceed the budget16--- FAILED lines older than 24 h are not counted17--- an existing failed.log that cannot be read fails the run before any post18--- a failed heartbeat fails the run, and the FIRING it left pending is posted once on the next run19=== mutant forZero20--- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears21--- /nix/store/6y66jscp5h48z98aiwfi48jrqx1x1a4x-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000022+++ /build/tmp.bIpDcUkE7j/actual 2026-10-22 01:05:49.052437165 +000023@@ -1,2 +1,2 @@24-5 [FIRING] HostMemoryPressure magnetite default25+0 [FIRING] HostMemoryPressure magnetite default26 11 [RESOLVED] HostMemoryPressure magnetite default27MISS HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list28mutant forZero rejected: HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list29=== mutant noAbsenceFreeze30--- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once31--- /nix/store/wc7jnmxya82m0in69d63pn82jb07r6b5-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000032+++ /build/tmp.sgLVkgpbar/actual 2026-10-22 01:06:02.000475079 +000033@@ -1,2 +1,4 @@34 0 [FIRING] HostOOMKill stibnite high35+5 [RESOLVED] HostOOMKill stibnite high36+11 [FIRING] HostOOMKill stibnite high37 16 [RESOLVED] HostOOMKill stibnite high38MISS stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list39mutant noAbsenceFreeze rejected: stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list40=== mutant noHostsUpdate41--- magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again42--- /nix/store/j8qp89cn45j5s16cl1v27ck5pdpwf368-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000043+++ /build/tmp.KiFppcPt4k/actual 2026-10-22 01:06:11.108501595 +000044@@ -1,2 +1,2 @@45-3 [FIRING] RuleSignalMissing magnetite default46+2 [FIRING] RuleSignalMissing magnetite default47 10 [RESOLVED] RuleSignalMissing magnetite default48MISS magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again: the alert posts differ from the expected list49mutant noHostsUpdate rejected: magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again: the alert posts differ from the expected list50=== mutant noStateWrite51--- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears52MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.Ocv26IgyFO/state.json: No such file or directory 53MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 1 exited 2, expected 0: jq: error: Could not open file /build/tmp.Ocv26IgyFO/state.json: No such file or directory 54--- /nix/store/6grbrjs6fqnl7rfkpg165dxa6zlnjdha-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000055+++ /build/tmp.Ocv26IgyFO/actual 2026-10-22 01:06:14.113510316 +000056@@ -1,2 +1,2 @@57 0 [FIRING] HostOOMKill magnetite high58-6 [RESOLVED] HostOOMKill magnetite high59+1 [FIRING] HostOOMKill magnetite high60MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: the alert posts differ from the expected list61mutant noStateWrite rejected: HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.Ocv26IgyFO/state.json: No such file or directory 62=== mutant noStoreFreeze63--- a store error freezes a firing HostOOMKill, raises StoreQueryFailing after 3 min, heartbeats, and resumes64--- /nix/store/n6m027z9nnbx170kn9hmzcy6793i91nc-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000065+++ /build/tmp.cug8nhuaz5/actual 2026-10-22 01:06:25.901544397 +000066@@ -1,4 +1,4 @@67 0 [FIRING] HostOOMKill magnetite high68 4 [FIRING] StoreQueryFailing high69+5 [RESOLVED] HostOOMKill magnetite high70 13 [RESOLVED] StoreQueryFailing high71-14 [RESOLVED] HostOOMKill magnetite high72MISS a store error freezes a firing HostOOMKill, raises StoreQueryFailing after 3 min, heartbeats, and resumes: the alert posts differ from the expected list73mutant noStoreFreeze rejected: a store error freezes a firing HostOOMKill, raises StoreQueryFailing after 3 min, heartbeats, and resumes: the alert posts differ from the expected list74ok: every leg passed