fleet-alert-fixtures
checks.x86_64-linux.fleet-alert
· build #1991
· raw
1--- the fixture server refuses other paths, databases and accounts, replays SQL as GreptimeDB does, and detects two open requests2ok: fixture server refusals, the SQL answer shape and the concurrency guard3--- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears4--- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears5--- a failed state for every notified unit posts no SystemdUnitFailed, and nginx, not notified, posts one6--- a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes7--- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once8--- HostOOMKill last seen before 7 runs of store error resolves 5 min after the store answers again, not on its first answer9--- HostOOMKill on stibnite last seen before 7 silent runs resolves 5 min after stibnite reports again, not on its first report10--- node_vmstat_oom_kill empty for 30 min while magnetite reports raises one RuleSignalMissing11--- magnetite and cinnabar silent for 40 min while the store answers raise RuleSignalMissing for each 10 min after they last reported, none for node_vmstat_oom_kill, and resolve when they report again12--- cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again13--- cinnabar's / above 85 % for 15 min fires FilesystemFillingWarn at default, then above 95 % for 5 min fires FilesystemFull at high14--- WeeklyPipelineTest posts once at 09:02 New York time on both sides of the daylight-saving change, never at 08:59, 09:06 or 09:02 EDT clock time in EST15--- 11 instances within 24 h post all 11 FIRINGs and exactly one AlertBudgetExceeded16--- 4 instances on stibnite within 24 h post exactly one AlertBudgetExceeded17--- 10 FAILED lines of the last 24 h and one FIRING exceed the budget18--- FAILED lines older than 24 h are not counted19--- an existing failed.log that cannot be read fails the run before any post20--- a failed heartbeat fails the run, and the FIRING it left pending is posted once on the next run21--- PullRequestQueueWaitBurn answered 0.15 on 65 runs fires once on the 61st, posts nothing on the 62nd to 65th, and resolves once 5 min after its row disappears22--- a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual23--- one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL24--- a unit notified on one host posts no SystemdUnitFailed on another, and nginx, notified nowhere, posts one25--- pyrite, with HostOOMKill firing, and electrum silent for 40 min post nothing after the FIRING: no silence, no resolution, no repeat26--- no SMART series for 30 min posts nothing while pyrite is silent, then one RuleSignalMissing for each NVMe rule 30 min after pyrite reports27--- pyrite's NVMe critical warning on one run fires NvmeCriticalWarning pyrite at once at high28--- pyrite, not collecting, reporting without SMART series for 35 min and then silent for 40 min posts nothing29=== mutant allDatabasesFreeze30--- a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual31--- /nix/store/3ksgnvzycis2fgfdfci3lnshpnwcfzqx-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000032+++ /build/tmp.ItMxvPOg3t/actual 2026-11-01 05:48:49.889974485 +000033@@ -1,4 +1,3 @@34 0 [FIRING] HostOOMKill magnetite high35 4 [FIRING] StoreQueryFailing devloop high36-5 [RESOLVED] HostOOMKill magnetite high37 11 [RESOLVED] StoreQueryFailing devloop high38MISS a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual: the alert posts differ from the expected list39mutant allDatabasesFreeze rejected: a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual: the alert posts differ from the expected list40=== mutant alwaysOnMagnetiteOnly41--- cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again42--- /nix/store/aarcpx64r6vljqa73iglqd5hxmfbycl5-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000043+++ /build/tmp.xLkVIKe2Sm/actual 2026-11-01 05:48:49.912974529 +000044@@ -1,2 +0,0 @@45-3 [FIRING] RuleSignalMissing cinnabar default46-10 [RESOLVED] RuleSignalMissing cinnabar default47MISS cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again: the alert posts differ from the expected list48MISS cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again: no posted message carries: selector=system_memory_usage_bytes49mutant alwaysOnMagnetiteOnly rejected: cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again: the alert posts differ from the expected list50=== mutant everyQueryToFleet51--- one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL52MISS one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL: the server did not record exactly one: REQUEST 0 /v1/sql devloop SELECT CAST(sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) AS DOUBLE) / count(*) AS value FROM ci_build c JOIN (SELECT build_id, min(finished_at) AS first_finished FROM ci_build GROUP BY build_id) f ON c.build_id = f.build_id AND c.finished_at = f.first_finished WHERE c.pr_number IS NOT NULL AND c.started_at IS NOT NULL AND c.finished_at >= now() - INTERVAL '24 hours' HAVING count(*) >= 10 AND sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) * 10 > count(*)53mutant everyQueryToFleet rejected: one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL: the server did not record exactly one: REQUEST 0 /v1/sql devloop SELECT CAST(sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) AS DOUBLE) / count(*) AS value FROM ci_build c JOIN (SELECT build_id, min(finished_at) AS first_finished FROM ci_build GROUP BY build_id) f ON c.build_id = f.build_id AND c.finished_at = f.first_finished WHERE c.pr_number IS NOT NULL AND c.started_at IS NOT NULL AND c.finished_at >= now() - INTERVAL '24 hours' HAVING count(*) >= 10 AND sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) * 10 > count(*)54=== mutant forZero55--- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears56--- /nix/store/2ngrim690axpydj1bk1ghjafymwc0v6c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000057+++ /build/tmp.G2xMANtdtP/actual 2026-11-01 05:49:21.571035554 +000058@@ -1,2 +1,2 @@59-5 [FIRING] HostMemoryPressure magnetite default60+0 [FIRING] HostMemoryPressure magnetite default61 11 [RESOLVED] HostMemoryPressure magnetite default62MISS HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list63mutant forZero rejected: HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list64=== mutant noAbsenceFreeze65--- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once66--- /nix/store/2w5szkyqg1flnpplqw9sd3fy1894ks1c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000067+++ /build/tmp.e6zWwGXtPW/actual 2026-11-01 05:49:51.990093955 +000068@@ -1,2 +1,4 @@69 0 [FIRING] HostOOMKill stibnite high70+5 [RESOLVED] HostOOMKill stibnite high71+11 [FIRING] HostOOMKill stibnite high72 16 [RESOLVED] HostOOMKill stibnite high73MISS stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list74mutant noAbsenceFreeze rejected: stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list75=== mutant noHostsUpdate76--- magnetite and cinnabar silent for 40 min while the store answers raise RuleSignalMissing for each 10 min after they last reported, none for node_vmstat_oom_kill, and resolve when they report again77--- /nix/store/kbmhj18ibd15nbjl6jgr9dkrqlclx8a2-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000078+++ /build/tmp.tk62OQlomD/actual 2026-11-01 05:50:09.462127400 +000079@@ -1,4 +1,4 @@80-3 [FIRING] RuleSignalMissing cinnabar default81-3 [FIRING] RuleSignalMissing magnetite default82+2 [FIRING] RuleSignalMissing cinnabar default83+2 [FIRING] RuleSignalMissing magnetite default84 10 [RESOLVED] RuleSignalMissing cinnabar default85 10 [RESOLVED] RuleSignalMissing magnetite default86MISS magnetite and cinnabar silent for 40 min while the store answers raise RuleSignalMissing for each 10 min after they last reported, none for node_vmstat_oom_kill, and resolve when they report again: the alert posts differ from the expected list87mutant noHostsUpdate rejected: magnetite and cinnabar silent for 40 min while the store answers raise RuleSignalMissing for each 10 min after they last reported, none for node_vmstat_oom_kill, and resolve when they report again: the alert posts differ from the expected list88=== mutant noStateWrite89--- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears90MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.zBfLp9OWxo/state.json: No such file or directory 91MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 1 exited 2, expected 0: jq: error: Could not open file /build/tmp.zBfLp9OWxo/state.json: No such file or directory 92--- /nix/store/qfv7fkygc10w316wiynys329fsmlx32w-fleet-alert-expected 1970-01-01 00:00:01.000000000 +000093+++ /build/tmp.zBfLp9OWxo/actual 2026-11-01 05:50:13.364134860 +000094@@ -1,2 +1,2 @@95 0 [FIRING] HostOOMKill magnetite high96-6 [RESOLVED] HostOOMKill magnetite high97+1 [FIRING] HostOOMKill magnetite high98MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: the alert posts differ from the expected list99mutant noStateWrite rejected: HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.zBfLp9OWxo/state.json: No such file or directory 100=== mutant noStoreFreeze101--- a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes102--- /nix/store/yc63wbqnip77jf3vlvryjari8p7bs33s-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000103+++ /build/tmp.Gt6VKq3ZpG/actual 2026-11-01 05:50:36.118178293 +0000104@@ -1,6 +1,6 @@105 0 [FIRING] HostOOMKill magnetite high106 4 [FIRING] StoreQueryFailing devloop high107 4 [FIRING] StoreQueryFailing fleet high108+5 [RESOLVED] HostOOMKill magnetite high109 13 [RESOLVED] StoreQueryFailing devloop high110 13 [RESOLVED] StoreQueryFailing fleet high111-14 [RESOLVED] HostOOMKill magnetite high112MISS a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes: the alert posts differ from the expected list113mutant noStoreFreeze rejected: a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes: the alert posts differ from the expected list114ok: every leg passed