fleet-alert-fixtures
checks.x86_64-linux.fleet-alert
· build #1791
· raw
1--- the fixture server refuses other paths, databases and accounts, replays SQL as GreptimeDB does, and detects two open requests2ok: fixture server refusals, the SQL answer shape and the concurrency guard3--- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears4--- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears5--- a failed state for every notified unit posts no SystemdUnitFailed, and nginx, not notified, posts one6--- a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes7--- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once8--- HostOOMKill last seen before 7 runs of store error resolves 5 min after the store answers again, not on its first answer9--- HostOOMKill on stibnite last seen before 7 silent runs resolves 5 min after stibnite reports again, not on its first report10--- node_vmstat_oom_kill empty for 30 min while magnetite reports raises one RuleSignalMissing11--- magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again12--- WeeklyPipelineTest posts once at 09:02 New York time on both sides of the daylight-saving change, never at 08:59, 09:06 or 09:02 EDT clock time in EST13--- 11 instances within 24 h post all 11 FIRINGs and exactly one AlertBudgetExceeded14--- 4 instances on stibnite within 24 h post exactly one AlertBudgetExceeded15--- 10 FAILED lines of the last 24 h and one FIRING exceed the budget16--- FAILED lines older than 24 h are not counted17--- an existing failed.log that cannot be read fails the run before any post18--- a failed heartbeat fails the run, and the FIRING it left pending is posted once on the next run19--- PullRequestQueueWaitBurn answered 0.15 on 65 runs fires once on the 61st, posts nothing on the 62nd to 65th, and resolves once 5 min after its row disappears20--- a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual21--- one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL22=== mutant allDatabasesFreeze23--- a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual24--- /nix/store/3ksgnvzycis2fgfdfci3lnshpnwcfzqx-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000025+++ /build/tmp.M6GhLqCsjZ/actual 2026-10-23 23:15:25.280984217 +000026@@ -1,4 +1,3 @@27 0 [FIRING] HostOOMKill magnetite high28 4 [FIRING] StoreQueryFailing devloop high29-5 [RESOLVED] HostOOMKill magnetite high30 11 [RESOLVED] StoreQueryFailing devloop high31MISS a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual: the alert posts differ from the expected list32mutant allDatabasesFreeze rejected: a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual: the alert posts differ from the expected list33=== mutant everyQueryToFleet34--- one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL35MISS one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL: the server did not record exactly one: REQUEST 0 /v1/sql devloop SELECT CAST(sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) AS DOUBLE) / count(*) AS value FROM ci_build c JOIN (SELECT build_id, min(finished_at) AS first_finished FROM ci_build GROUP BY build_id) f ON c.build_id = f.build_id AND c.finished_at = f.first_finished WHERE c.pr_number IS NOT NULL AND c.started_at IS NOT NULL AND c.finished_at >= now() - INTERVAL '24 hours' HAVING count(*) >= 10 AND sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) * 10 > count(*)36mutant everyQueryToFleet rejected: one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL: the server did not record exactly one: REQUEST 0 /v1/sql devloop SELECT CAST(sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) AS DOUBLE) / count(*) AS value FROM ci_build c JOIN (SELECT build_id, min(finished_at) AS first_finished FROM ci_build GROUP BY build_id) f ON c.build_id = f.build_id AND c.finished_at = f.first_finished WHERE c.pr_number IS NOT NULL AND c.started_at IS NOT NULL AND c.finished_at >= now() - INTERVAL '24 hours' HAVING count(*) >= 10 AND sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) * 10 > count(*)37=== mutant forZero38--- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears39--- /nix/store/2ngrim690axpydj1bk1ghjafymwc0v6c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000040+++ /build/tmp.GGU87KFo7F/actual 2026-10-23 23:15:58.882052151 +000041@@ -1,2 +1,2 @@42-5 [FIRING] HostMemoryPressure magnetite default43+0 [FIRING] HostMemoryPressure magnetite default44 11 [RESOLVED] HostMemoryPressure magnetite default45MISS HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list46mutant forZero rejected: HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list47=== mutant noAbsenceFreeze48--- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once49--- /nix/store/2w5szkyqg1flnpplqw9sd3fy1894ks1c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000050+++ /build/tmp.ZcG0X8Lnhv/actual 2026-10-23 23:16:41.895140584 +000051@@ -1,2 +1,4 @@52 0 [FIRING] HostOOMKill stibnite high53+5 [RESOLVED] HostOOMKill stibnite high54+11 [FIRING] HostOOMKill stibnite high55 16 [RESOLVED] HostOOMKill stibnite high56MISS stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list57mutant noAbsenceFreeze rejected: stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list58=== mutant noHostsUpdate59--- magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again60--- /nix/store/zvnynpdbgmllcm71apyvlc53kfxvs8m4-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000061+++ /build/tmp.bThSO3fZHa/actual 2026-10-23 23:17:03.605185702 +000062@@ -1,2 +1,2 @@63-3 [FIRING] RuleSignalMissing magnetite default64+2 [FIRING] RuleSignalMissing magnetite default65 10 [RESOLVED] RuleSignalMissing magnetite default66MISS magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again: the alert posts differ from the expected list67mutant noHostsUpdate rejected: magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again: the alert posts differ from the expected list68=== mutant noStateWrite69--- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears70MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.J21z8Sulds/state.json: No such file or directory 71MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 1 exited 2, expected 0: jq: error: Could not open file /build/tmp.J21z8Sulds/state.json: No such file or directory 72--- /nix/store/qfv7fkygc10w316wiynys329fsmlx32w-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000073+++ /build/tmp.J21z8Sulds/actual 2026-10-23 23:17:08.559196036 +000074@@ -1,2 +1,2 @@75 0 [FIRING] HostOOMKill magnetite high76-6 [RESOLVED] HostOOMKill magnetite high77+1 [FIRING] HostOOMKill magnetite high78MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: the alert posts differ from the expected list79mutant noStateWrite rejected: HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.J21z8Sulds/state.json: No such file or directory 80=== mutant noStoreFreeze81--- a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes82--- /nix/store/yc63wbqnip77jf3vlvryjari8p7bs33s-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000083+++ /build/tmp.M0jTzinUWG/actual 2026-10-23 23:17:32.912247011 +000084@@ -1,6 +1,6 @@85 0 [FIRING] HostOOMKill magnetite high86 4 [FIRING] StoreQueryFailing devloop high87 4 [FIRING] StoreQueryFailing fleet high88+5 [RESOLVED] HostOOMKill magnetite high89 13 [RESOLVED] StoreQueryFailing devloop high90 13 [RESOLVED] StoreQueryFailing fleet high91-14 [RESOLVED] HostOOMKill magnetite high92MISS a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes: the alert posts differ from the expected list93mutant noStoreFreeze rejected: a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes: the alert posts differ from the expected list94ok: every leg passed