these 3 derivations will be built: /nix/store/ccnf2mrwmwxscdiyzbka0grqyy3py6bg-fleet-alert-property.drv /nix/store/zjqgkxwwkmd8yxf355vixr4vav0hj49r-fleet-alert-fixtures.drv /nix/store/kfdnm841lf4d8s5fh6sy7h3h81flvs79-fleet-alert.drv building '/nix/store/zjqgkxwwkmd8yxf355vixr4vav0hj49r-fleet-alert-fixtures.drv' building '/nix/store/ccnf2mrwmwxscdiyzbka0grqyy3py6bg-fleet-alert-property.drv' fleet-alert-fixtures> --- the fixture server refuses other paths, databases and accounts, replays SQL as GreptimeDB does, and detects two open requests fleet-alert-fixtures> ok: fixture server refusals, the SQL answer shape and the concurrency guard fleet-alert-fixtures> --- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears fleet-alert-property> ok: transition.jq equals the reference model on every generated timeline fleet-alert-property> ok: a rule without expr but with a results entry fires and resolves as its expr twin fleet-alert-property> ok: a rule without expr or results entry, and results named for internal rules, produce nothing fleet-alert-property> ok: a RESOLVED follows 5 min of absence observed while neither the store nor the host is frozen fleet-alert-fixtures> --- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears fleet-alert-property> ok: a devloop failure leaves fleet's posts as they are without it fleet-alert-fixtures> --- a failed state for every notified unit posts no SystemdUnitFailed, and nginx, not notified, posts one fleet-alert-property> ok: StoreQueryFailing is one instance per failing database, labelled and titled with it fleet-alert-fixtures> --- a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes fleet-alert-fixtures> --- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once fleet-alert-property> Traceback (most recent call last): fleet-alert-property> File "/nix/store/ww3f8n7yv1z2h6m9a3x7pg230i27xywc-fleet-alert/test_transition.py", line 404, in fleet-alert-property> test_transition_matches_model() fleet-alert-property> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ fleet-alert-property> File "/nix/store/ww3f8n7yv1z2h6m9a3x7pg230i27xywc-fleet-alert/test_transition.py", line 174, in test_transition_matches_model fleet-alert-property> derandomize=True, fleet-alert-property> ^^^ fleet-alert-property> File "/nix/store/0wq2wpqi3611jav9npnssb4c3r3mvyca-python3-3.14.8-env/lib/python3.14/site-packages/hypothesis/core.py", line 2295, in wrapped_test fleet-alert-property> raise the_error_hypothesis_found fleet-alert-property> File "/nix/store/ww3f8n7yv1z2h6m9a3x7pg230i27xywc-fleet-alert/test_transition.py", line 187, in test_transition_matches_model fleet-alert-property> assert want == got, f"run {index}: model posts {want}, transition.jq posts {got}" fleet-alert-property> ^^^^^^^^^^^ fleet-alert-property> AssertionError: run 2: model posts [{'title': '[FIRING] StoreQueryFailing devloop', 'priority': 'high', 'kind': 'repost', 'id': '["StoreQueryFailing",[{"key":"database","value":"devloop"}]]'}], transition.jq posts [] fleet-alert-property> Failing test case: test_transition_matches_model( fleet-alert-property> # The test always failed when commented parts were varied together. fleet-alert-property> steps=[{ fleet-alert-property> 'dt': 60, # or any other generated value fleet-alert-property> 'storeErrors': ['devloop'], fleet-alert-property> 'reporting': ['magnetite', fleet-alert-property> 'stibnite'], # or any other generated value fleet-alert-property> 'results': { fleet-alert-property> 'Crit': [], # or any other generated value fleet-alert-property> 'Warn': [], # or any other generated value fleet-alert-property> 'Info': [], # or any other generated value fleet-alert-property> 'Dev': [], # or any other generated value fleet-alert-property> }, fleet-alert-property> 'failAt': None, # or any other generated value fleet-alert-property> }, fleet-alert-property> { fleet-alert-property> 'dt': 300, fleet-alert-property> 'storeErrors': ['devloop'], fleet-alert-property> 'reporting': ['magnetite', fleet-alert-property> 'stibnite'], # or any other generated value fleet-alert-property> 'results': { fleet-alert-property> 'Crit': [], # or any other generated value fleet-alert-property> 'Warn': [], # or any other generated value fleet-alert-property> 'Info': [], # or any other generated value fleet-alert-property> 'Dev': [], # or any other generated value fleet-alert-property> }, fleet-alert-property> 'failAt': None, fleet-alert-property> }, fleet-alert-property> { fleet-alert-property> 'dt': 86400, fleet-alert-property> 'storeErrors': ['devloop'], fleet-alert-property> 'reporting': ['magnetite', fleet-alert-property> 'stibnite'], # or any other generated value fleet-alert-property> 'results': {'Crit': [], fleet-alert-property> 'Warn': [], fleet-alert-property> 'Info': [], fleet-alert-property> 'Dev': []}, # or any other generated value fleet-alert-property> 'failAt': None, # or any other generated value fleet-alert-property> }], fleet-alert-property> ) fleet-alert-property> Explanation: fleet-alert-property> These lines were always and only run by failing test cases: fleet-alert-property> /nix/store/0wq2wpqi3611jav9npnssb4c3r3mvyca-python3-3.14.8-env/lib/python3.14/contextlib.py:157 fleet-alert-property> /nix/store/ww3f8n7yv1z2h6m9a3x7pg230i27xywc-fleet-alert/model.py:176 error: Cannot build '/nix/store/ccnf2mrwmwxscdiyzbka0grqyy3py6bg-fleet-alert-property.drv'. Reason: builder failed with exit code 1. Output paths: /nix/store/wcjb2sz5y68iaai7m3nhk7fn6hkqphd0-fleet-alert-property Last 25 log lines: > 'stibnite'], # or any other generated value > 'results': { > 'Crit': [], # or any other generated value > 'Warn': [], # or any other generated value > 'Info': [], # or any other generated value > 'Dev': [], # or any other generated value > }, > 'failAt': None, > }, > { > 'dt': 86400, > 'storeErrors': ['devloop'], > 'reporting': ['magnetite', > 'stibnite'], # or any other generated value > 'results': {'Crit': [], > 'Warn': [], > 'Info': [], > 'Dev': []}, # or any other generated value > 'failAt': None, # or any other generated value > }], > ) > Explanation: > These lines were always and only run by failing test cases: > /nix/store/0wq2wpqi3611jav9npnssb4c3r3mvyca-python3-3.14.8-env/lib/python3.14/contextlib.py:157 > /nix/store/ww3f8n7yv1z2h6m9a3x7pg230i27xywc-fleet-alert/model.py:176 For full logs, run: nix log /nix/store/ccnf2mrwmwxscdiyzbka0grqyy3py6bg-fleet-alert-property.drv fleet-alert-fixtures> --- HostOOMKill last seen before 7 runs of store error resolves 5 min after the store answers again, not on its first answer fleet-alert-fixtures> --- HostOOMKill on stibnite last seen before 7 silent runs resolves 5 min after stibnite reports again, not on its first report fleet-alert-fixtures> --- node_vmstat_oom_kill empty for 30 min while magnetite reports raises one RuleSignalMissing fleet-alert-fixtures> --- magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again fleet-alert-fixtures> --- WeeklyPipelineTest posts once at 09:02 New York time on both sides of the daylight-saving change, never at 08:59, 09:06 or 09:02 EDT clock time in EST fleet-alert-fixtures> --- 11 instances within 24 h post all 11 FIRINGs and exactly one AlertBudgetExceeded fleet-alert-fixtures> --- 4 instances on stibnite within 24 h post exactly one AlertBudgetExceeded fleet-alert-fixtures> --- 10 FAILED lines of the last 24 h and one FIRING exceed the budget fleet-alert-fixtures> --- FAILED lines older than 24 h are not counted fleet-alert-fixtures> --- an existing failed.log that cannot be read fails the run before any post fleet-alert-fixtures> --- a failed heartbeat fails the run, and the FIRING it left pending is posted once on the next run fleet-alert-fixtures> --- PullRequestQueueWaitBurn answered 0.15 on 65 runs fires once on the 61st, posts nothing on the 62nd to 65th, and resolves once 5 min after its row disappears fleet-alert-fixtures> --- a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual fleet-alert-fixtures> --- one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL fleet-alert-fixtures> === mutant allDatabasesFreeze fleet-alert-fixtures> --- a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual fleet-alert-fixtures> --- /nix/store/3ksgnvzycis2fgfdfci3lnshpnwcfzqx-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.M6GhLqCsjZ/actual 2026-10-23 23:15:25.280984217 +0000 fleet-alert-fixtures> @@ -1,4 +1,3 @@ fleet-alert-fixtures> 0 [FIRING] HostOOMKill magnetite high fleet-alert-fixtures> 4 [FIRING] StoreQueryFailing devloop high fleet-alert-fixtures> -5 [RESOLVED] HostOOMKill magnetite high fleet-alert-fixtures> 11 [RESOLVED] StoreQueryFailing devloop high fleet-alert-fixtures> MISS a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual: the alert posts differ from the expected list fleet-alert-fixtures> mutant allDatabasesFreeze rejected: a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual: the alert posts differ from the expected list fleet-alert-fixtures> === mutant everyQueryToFleet fleet-alert-fixtures> --- one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL fleet-alert-fixtures> MISS one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL: the server did not record exactly one: REQUEST 0 /v1/sql devloop SELECT CAST(sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) AS DOUBLE) / count(*) AS value FROM ci_build c JOIN (SELECT build_id, min(finished_at) AS first_finished FROM ci_build GROUP BY build_id) f ON c.build_id = f.build_id AND c.finished_at = f.first_finished WHERE c.pr_number IS NOT NULL AND c.started_at IS NOT NULL AND c.finished_at >= now() - INTERVAL '24 hours' HAVING count(*) >= 10 AND sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) * 10 > count(*) fleet-alert-fixtures> mutant everyQueryToFleet rejected: one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL: the server did not record exactly one: REQUEST 0 /v1/sql devloop SELECT CAST(sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) AS DOUBLE) / count(*) AS value FROM ci_build c JOIN (SELECT build_id, min(finished_at) AS first_finished FROM ci_build GROUP BY build_id) f ON c.build_id = f.build_id AND c.finished_at = f.first_finished WHERE c.pr_number IS NOT NULL AND c.started_at IS NOT NULL AND c.finished_at >= now() - INTERVAL '24 hours' HAVING count(*) >= 10 AND sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) * 10 > count(*) fleet-alert-fixtures> === mutant forZero fleet-alert-fixtures> --- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears fleet-alert-fixtures> --- /nix/store/2ngrim690axpydj1bk1ghjafymwc0v6c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.GGU87KFo7F/actual 2026-10-23 23:15:58.882052151 +0000 fleet-alert-fixtures> @@ -1,2 +1,2 @@ fleet-alert-fixtures> -5 [FIRING] HostMemoryPressure magnetite default fleet-alert-fixtures> +0 [FIRING] HostMemoryPressure magnetite default fleet-alert-fixtures> 11 [RESOLVED] HostMemoryPressure magnetite default fleet-alert-fixtures> MISS HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list fleet-alert-fixtures> mutant forZero rejected: HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list fleet-alert-fixtures> === mutant noAbsenceFreeze fleet-alert-fixtures> --- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once fleet-alert-fixtures> --- /nix/store/2w5szkyqg1flnpplqw9sd3fy1894ks1c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.ZcG0X8Lnhv/actual 2026-10-23 23:16:41.895140584 +0000 fleet-alert-fixtures> @@ -1,2 +1,4 @@ fleet-alert-fixtures> 0 [FIRING] HostOOMKill stibnite high fleet-alert-fixtures> +5 [RESOLVED] HostOOMKill stibnite high fleet-alert-fixtures> +11 [FIRING] HostOOMKill stibnite high fleet-alert-fixtures> 16 [RESOLVED] HostOOMKill stibnite high fleet-alert-fixtures> MISS stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list fleet-alert-fixtures> mutant noAbsenceFreeze rejected: stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list fleet-alert-fixtures> === mutant noHostsUpdate fleet-alert-fixtures> --- magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again fleet-alert-fixtures> --- /nix/store/zvnynpdbgmllcm71apyvlc53kfxvs8m4-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.bThSO3fZHa/actual 2026-10-23 23:17:03.605185702 +0000 fleet-alert-fixtures> @@ -1,2 +1,2 @@ fleet-alert-fixtures> -3 [FIRING] RuleSignalMissing magnetite default fleet-alert-fixtures> +2 [FIRING] RuleSignalMissing magnetite default fleet-alert-fixtures> 10 [RESOLVED] RuleSignalMissing magnetite default fleet-alert-fixtures> MISS magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again: the alert posts differ from the expected list fleet-alert-fixtures> mutant noHostsUpdate rejected: magnetite silent for 40 min while the store answers raises RuleSignalMissing magnetite 10 min after it last reported, none for node_vmstat_oom_kill, and resolves when it reports again: the alert posts differ from the expected list fleet-alert-fixtures> === mutant noStateWrite fleet-alert-fixtures> --- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears fleet-alert-fixtures> MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.J21z8Sulds/state.json: No such file or directory fleet-alert-fixtures> MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 1 exited 2, expected 0: jq: error: Could not open file /build/tmp.J21z8Sulds/state.json: No such file or directory fleet-alert-fixtures> --- /nix/store/qfv7fkygc10w316wiynys329fsmlx32w-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.J21z8Sulds/actual 2026-10-23 23:17:08.559196036 +0000 fleet-alert-fixtures> @@ -1,2 +1,2 @@ fleet-alert-fixtures> 0 [FIRING] HostOOMKill magnetite high fleet-alert-fixtures> -6 [RESOLVED] HostOOMKill magnetite high fleet-alert-fixtures> +1 [FIRING] HostOOMKill magnetite high fleet-alert-fixtures> MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: the alert posts differ from the expected list fleet-alert-fixtures> mutant noStateWrite rejected: HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.J21z8Sulds/state.json: No such file or directory fleet-alert-fixtures> === mutant noStoreFreeze fleet-alert-fixtures> --- a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes fleet-alert-fixtures> --- /nix/store/yc63wbqnip77jf3vlvryjari8p7bs33s-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.M0jTzinUWG/actual 2026-10-23 23:17:32.912247011 +0000 fleet-alert-fixtures> @@ -1,6 +1,6 @@ fleet-alert-fixtures> 0 [FIRING] HostOOMKill magnetite high fleet-alert-fixtures> 4 [FIRING] StoreQueryFailing devloop high fleet-alert-fixtures> 4 [FIRING] StoreQueryFailing fleet high fleet-alert-fixtures> +5 [RESOLVED] HostOOMKill magnetite high fleet-alert-fixtures> 13 [RESOLVED] StoreQueryFailing devloop high fleet-alert-fixtures> 13 [RESOLVED] StoreQueryFailing fleet high fleet-alert-fixtures> -14 [RESOLVED] HostOOMKill magnetite high fleet-alert-fixtures> MISS a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes: the alert posts differ from the expected list fleet-alert-fixtures> mutant noStoreFreeze rejected: a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes: the alert posts differ from the expected list fleet-alert-fixtures> ok: every leg passed warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy error: Cannot build '/nix/store/kfdnm841lf4d8s5fh6sy7h3h81flvs79-fleet-alert.drv'. Reason: 1 dependency failed. Output paths: /nix/store/9gxgja3wwb5nas4a62pj5kzdnx7xflzm-fleet-alert