these 46 derivations will be built: /nix/store/1jxl43z4mc50h5qb3wg0dax9c1sl5hwn-fleet-alert-expected.drv /nix/store/2az9b18q5n0ydhnf63bnyi0kifyywa0b-fleet-alert-check-config.json.drv /nix/store/4gc05y7x6hrca7w3hqbghfgblmsrng26-fleet-alert-responses.json.drv /nix/store/58gq40c14f4lhhzpkllp0ksavn3086p3-fleet-alert-responses.json.drv /nix/store/5nx3mmfrf8nm2fcc6x99gbmgl70cgpwc-fleet-alert-responses.json.drv /nix/store/6px351ldllsyxyh86vwnjwg9lhjx4ln8-fleet-alert-check-config.json.drv /nix/store/7vz0hgg7g4mlxfrylkxyfsvfhpmzcban-fleet-alert-expected.drv /nix/store/8j5iarwvf1izx71ndv6mkz5zmzqnkdhy-fleet-alert-responses.json.drv /nix/store/8pqn2m19nacvb11dsbxk3rdmrslx6bkq-fleet-alert-responses.json.drv /nix/store/92cahj4sh26hsfgvvkmbf3mcnwiwf5z7-fleet-alert-responses.json.drv /nix/store/9jazzrzd7n86p7drzla176dh2vx57wi9-fleet-alert-responses.json.drv /nix/store/9nzfpmc1f25gjqi8jw3jb3k0imb3cwil-fleet-alert-responses.json.drv /nix/store/9xza8x44gzacqfvjqvr182v24a30219h-fleet-alert-responses.json.drv /nix/store/c5r697frbzrimzqq8a0g1p75i10ld23m-fleet-alert-responses.json.drv /nix/store/c6pzzc9jffjm59hmjnx86vjmw2png2r4-fleet-alert-responses.json.drv /nix/store/c7kdqw1flh80bc3nw1v5lyp57pnw3m8x-fleet-alert-databases.json.drv /nix/store/cmaliwdjzki2mrm5227kb7npp9rkxaji-fleet-alert-responses.json.drv /nix/store/d69cnadvyrkngaxz5ivj6kqy5b5c0j14-fleet-alert-check-config.json.drv /nix/store/fm5sglkf9wq5a0si23iql92d8jkridk7-fleet-alert-responses.json.drv /nix/store/gb5msqq1krp1clka73jw8p5mddaicza0-fleet-alert-expected.drv /nix/store/gm636mf6iikjzgyi0l6nwj90ll12a8dg-fleet-alert-expected.drv /nix/store/if3vpzc8v1wy0smcp6kv1swycscdawc9-fleet-alert-expected.drv /nix/store/il6n9hsvbxz9q0r39xszg7zfpq9brzpk-fleet-alert-responses.json.drv /nix/store/j8s29j0ngsdxip8ynw866wpjdbyhmgnw-fleet-alert-responses.json.drv /nix/store/jfqjj8qx98dlxay3bqzizzma06jnm4k1-fleet-alert-check-config.json.drv /nix/store/jjmq67fq772zhckpd19p0w89jdraydvx-fleet-alert-responses.json.drv /nix/store/lcp0akp862vxl2344b2q03b19w5dd722-fleet-alert-responses.json.drv /nix/store/m6aw8niq36qmbajbwszljzlkxmpjxyzp-fleet-alert-responses.json.drv /nix/store/m7r74adg8yg8q01kqa03vnnvbs6kf4dl-fleet-alert-databases.json.drv /nix/store/mp7p7l3pn7fvjndw4xy4iwi8kxk2hrjy-fleet-alert-responses.json.drv /nix/store/n4crhwpglizca9mzhywk2gx0z3gcpvd1-fleet-alert-check-config.json.drv /nix/store/pkl1j6n8rw9l1irh8xwmvf8l79xkgchc-fleet-alert-check-config.json.drv /nix/store/qikyiqv6c64jp456hrv27016jnmyiqdf-fleet-alert-responses.json.drv /nix/store/s13ryq6cv93m0s1874q0g8qpyr0xl1r4-fleet-alert-responses.json.drv /nix/store/sgk41d37jzna3q04lky7k9zffpxr9g5c-fleet-alert-responses.json.drv /nix/store/v05lld1znv548rw7mwq34y8g6ywb1n9q-fleet-alert-databases.json.drv /nix/store/vy6xfswvgili52kwlvxpx20z2avd8bv3-fleet-alert-expected.drv /nix/store/xh4ad2875wqmfc2ljcm9rmlzhj1j782d-fleet-alert-responses.json.drv /nix/store/zhlghbngj0l2p256inlsfqw6giw4zacs-fleet-alert-responses.json.drv /nix/store/zkaad2wvbdz9lz2qzhygyamd7f6g283k-fleet-alert-expected.drv /nix/store/2wfxqw3b07rigqxy41jb3v9l2yl1kqf6-fleet-alert-fixtures.drv /nix/store/k5gvkibalmnkkp0jmlk3cwgahc3qlpl3-fleet-alert-rules.yaml.drv /nix/store/x7lfh8kz5fb2vz12h5kcr5pqvyygzcny-fleet-alert-rules-malformed.yaml.drv /nix/store/lkjgkxwgm6kxjpmymfrd52wz8axrrn18-fleet-alert-promtool.drv /nix/store/r59gq13sl3knliy61fsdy1rblp90qkqa-fleet-alert-property.drv /nix/store/cza475sjfq5hp1d5z62gg3xn4fp7b48c-fleet-alert.drv these 44 paths will be fetched (22.8 MiB download, 89.5 MiB unpacked): /nix/store/6s6l3fbzhdh1fv1b1471mz1m3rb3p3rs-fleet-alert /nix/store/76rcgwl41qrhm86q6pividy2jfly8811-fleet-alert /nix/store/94cfdlrm1af11zpf8jz8q5lz7mygm8a4-fleet-alert /nix/store/adpwvnwhb6x8sr777gp1s9l2yz9n7jap-fleet-alert /nix/store/mwcmhsx5wn9qwgc3c3ra88k94x4cjm16-fleet-alert /nix/store/skwvmxrr4h6sqqwlsg38k8g2z4f8q184-fleet-alert /nix/store/y62rjxyypp92avjvb71ll9gnj65wi9y8-fleet-alert /nix/store/nkmc2s29zv6d3dz4d56nyxz7nsqr42lm-fleet-alert-control-databases.json /nix/store/vj5dmisgkhxm41f0f27jhwx2af09ibh8-fleet-alert-control-responses.json /nix/store/lw02ixzcpwwh1bfpcs9b6hvqfr6y01p4-fleet-alert-deadman-clause /nix/store/0z5r61g829jw2vjhqpnz35lw8hfff88p-fleet-alert-expected /nix/store/10ir5jdh63gmsv8nrafvcis9180nnl71-fleet-alert-expected /nix/store/1n5v9hcvx7hiv3z159f2hdnccyy9mwfx-fleet-alert-expected /nix/store/285li9xjcxm8mmy5mk5y416v4i4xsqb1-fleet-alert-expected /nix/store/2ngrim690axpydj1bk1ghjafymwc0v6c-fleet-alert-expected /nix/store/2w5szkyqg1flnpplqw9sd3fy1894ks1c-fleet-alert-expected /nix/store/3ksgnvzycis2fgfdfci3lnshpnwcfzqx-fleet-alert-expected /nix/store/41xddkij7m9fki1dddqgjw50bg82x759-fleet-alert-expected /nix/store/7y6kshvvgd2jvqfhv1crwh0dghhcimq8-fleet-alert-expected /nix/store/8mnv9jm4hm1pzxwqhdqfgl4ygrn2vm2w-fleet-alert-expected /nix/store/b55wb5942fnvr54579blsyx6fs1yj06r-fleet-alert-expected /nix/store/ki4fb30693420hxdb5a15grdhryjdmm1-fleet-alert-expected /nix/store/qfv7fkygc10w316wiynys329fsmlx32w-fleet-alert-expected /nix/store/qirbam0n0dch5brm1pjrgjjfbn3balkf-fleet-alert-expected /nix/store/qnq4z9hvvzhz43y2y8p03cgl1d411az8-fleet-alert-expected /nix/store/wm47qc294vwyl162xfb9xd9j4bra1glg-fleet-alert-expected /nix/store/yc63wbqnip77jf3vlvryjari8p7bs33s-fleet-alert-expected /nix/store/0213i35ddf0vlfsb9xbzd50ncbyazhah-fleet-alert-fixture-credentials /nix/store/4sxdh5kqb2vbsmp20p728lfk1ngag44g-fleet-alert-jq /nix/store/7lblph1bj318ipxg6yymcwqk00vc5mwn-fleet-alert-jq /nix/store/wsd3980pdsph8ybkhab3fbsag38ky64k-fleet-alert-jq /nix/store/x22z8s91l8kq976kaqf4g8b4711cg8wi-fleet-alert-jq /nix/store/gzwmqyybkrc5b43mqyw01mgv5xildx7a-fleet-alert-jq-noRepost /nix/store/bnqwcqn5zz9yx2qy5n8zala9aaw1lwyw-fleet-alert-jq-real /nix/store/58zjhz78d81l8ybx2b0l1zc20ig03abg-fleet-deadman-rehearsal /nix/store/vqqp72svyvrsha001jg0qg51rqs7w6p9-ntfy-send /nix/store/vccyr64d1m6lwa3f3ak22xg2rd9vdmwd-prometheus-3.15.0-cli /nix/store/0wq2wpqi3611jav9npnssb4c3r3mvyca-python3-3.14.8-env /nix/store/g46fg6ncdn8w31hpz1q8pfd7cxiiciri-python3.14-hypothesis-6.168.1 /nix/store/9yxirjvsklliz74ga1avpskhjm2n0piq-transition-allDatabasesFreeze.jq /nix/store/1djha7k27f9rv8jl971f35d1g8kj0pkp-transition-noAbsenceFreeze.jq /nix/store/rxkpjsazimz0xrrydx9n9ppw9ga91r7j-transition-noHostsUpdate.jq /nix/store/2mxlcnm3xwrl4r3l90chbf6r0b0qnaa2-transition-noRepost.jq /nix/store/gvkc8lq75pi1fwwzwa6psv8q5dxlx0xs-transition-noStoreFreeze.jq building '/nix/store/2az9b18q5n0ydhnf63bnyi0kifyywa0b-fleet-alert-check-config.json.drv' building '/nix/store/6px351ldllsyxyh86vwnjwg9lhjx4ln8-fleet-alert-check-config.json.drv' building '/nix/store/d69cnadvyrkngaxz5ivj6kqy5b5c0j14-fleet-alert-check-config.json.drv' building '/nix/store/jfqjj8qx98dlxay3bqzizzma06jnm4k1-fleet-alert-check-config.json.drv' building '/nix/store/n4crhwpglizca9mzhywk2gx0z3gcpvd1-fleet-alert-check-config.json.drv' building '/nix/store/pkl1j6n8rw9l1irh8xwmvf8l79xkgchc-fleet-alert-check-config.json.drv' building '/nix/store/c7kdqw1flh80bc3nw1v5lyp57pnw3m8x-fleet-alert-databases.json.drv' building '/nix/store/m7r74adg8yg8q01kqa03vnnvbs6kf4dl-fleet-alert-databases.json.drv' building '/nix/store/v05lld1znv548rw7mwq34y8g6ywb1n9q-fleet-alert-databases.json.drv' building '/nix/store/1jxl43z4mc50h5qb3wg0dax9c1sl5hwn-fleet-alert-expected.drv' building '/nix/store/7vz0hgg7g4mlxfrylkxyfsvfhpmzcban-fleet-alert-expected.drv' building '/nix/store/gb5msqq1krp1clka73jw8p5mddaicza0-fleet-alert-expected.drv' building '/nix/store/gm636mf6iikjzgyi0l6nwj90ll12a8dg-fleet-alert-expected.drv' building '/nix/store/if3vpzc8v1wy0smcp6kv1swycscdawc9-fleet-alert-expected.drv' building '/nix/store/vy6xfswvgili52kwlvxpx20z2avd8bv3-fleet-alert-expected.drv' building '/nix/store/zkaad2wvbdz9lz2qzhygyamd7f6g283k-fleet-alert-expected.drv' fleet-alert-check-config.json> structuredAttrs is enabled fleet-alert-check-config.json> structuredAttrs is enabled fleet-alert-check-config.json> structuredAttrs is enabled fleet-alert-check-config.json> structuredAttrs is enabled fleet-alert-check-config.json> structuredAttrs is enabled fleet-alert-check-config.json> structuredAttrs is enabled fleet-alert-databases.json> structuredAttrs is enabled fleet-alert-databases.json> structuredAttrs is enabled fleet-alert-databases.json> structuredAttrs is enabled fleet-alert-expected> structuredAttrs is enabled fleet-alert-expected> structuredAttrs is enabled fleet-alert-expected> structuredAttrs is enabled fleet-alert-expected> structuredAttrs is enabled fleet-alert-expected> structuredAttrs is enabled fleet-alert-expected> structuredAttrs is enabled fleet-alert-expected> structuredAttrs is enabled building '/nix/store/4gc05y7x6hrca7w3hqbghfgblmsrng26-fleet-alert-responses.json.drv' building '/nix/store/58gq40c14f4lhhzpkllp0ksavn3086p3-fleet-alert-responses.json.drv' building '/nix/store/5nx3mmfrf8nm2fcc6x99gbmgl70cgpwc-fleet-alert-responses.json.drv' building '/nix/store/8j5iarwvf1izx71ndv6mkz5zmzqnkdhy-fleet-alert-responses.json.drv' building '/nix/store/8pqn2m19nacvb11dsbxk3rdmrslx6bkq-fleet-alert-responses.json.drv' building '/nix/store/92cahj4sh26hsfgvvkmbf3mcnwiwf5z7-fleet-alert-responses.json.drv' building '/nix/store/9jazzrzd7n86p7drzla176dh2vx57wi9-fleet-alert-responses.json.drv' building '/nix/store/9nzfpmc1f25gjqi8jw3jb3k0imb3cwil-fleet-alert-responses.json.drv' building '/nix/store/cmaliwdjzki2mrm5227kb7npp9rkxaji-fleet-alert-responses.json.drv' building '/nix/store/fm5sglkf9wq5a0si23iql92d8jkridk7-fleet-alert-responses.json.drv' building '/nix/store/il6n9hsvbxz9q0r39xszg7zfpq9brzpk-fleet-alert-responses.json.drv' building '/nix/store/j8s29j0ngsdxip8ynw866wpjdbyhmgnw-fleet-alert-responses.json.drv' building '/nix/store/jjmq67fq772zhckpd19p0w89jdraydvx-fleet-alert-responses.json.drv' building '/nix/store/lcp0akp862vxl2344b2q03b19w5dd722-fleet-alert-responses.json.drv' fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled building '/nix/store/c5r697frbzrimzqq8a0g1p75i10ld23m-fleet-alert-responses.json.drv' building '/nix/store/c6pzzc9jffjm59hmjnx86vjmw2png2r4-fleet-alert-responses.json.drv' fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled building '/nix/store/9xza8x44gzacqfvjqvr182v24a30219h-fleet-alert-responses.json.drv' building '/nix/store/m6aw8niq36qmbajbwszljzlkxmpjxyzp-fleet-alert-responses.json.drv' building '/nix/store/mp7p7l3pn7fvjndw4xy4iwi8kxk2hrjy-fleet-alert-responses.json.drv' building '/nix/store/qikyiqv6c64jp456hrv27016jnmyiqdf-fleet-alert-responses.json.drv' building '/nix/store/s13ryq6cv93m0s1874q0g8qpyr0xl1r4-fleet-alert-responses.json.drv' building '/nix/store/sgk41d37jzna3q04lky7k9zffpxr9g5c-fleet-alert-responses.json.drv' building '/nix/store/xh4ad2875wqmfc2ljcm9rmlzhj1j782d-fleet-alert-responses.json.drv' building '/nix/store/zhlghbngj0l2p256inlsfqw6giw4zacs-fleet-alert-responses.json.drv' building '/nix/store/x7lfh8kz5fb2vz12h5kcr5pqvyygzcny-fleet-alert-rules-malformed.yaml.drv' building '/nix/store/k5gvkibalmnkkp0jmlk3cwgahc3qlpl3-fleet-alert-rules.yaml.drv' fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-responses.json> structuredAttrs is enabled fleet-alert-rules-malformed.yaml> structuredAttrs is enabled fleet-alert-rules.yaml> structuredAttrs is enabled building '/nix/store/lkjgkxwgm6kxjpmymfrd52wz8axrrn18-fleet-alert-promtool.drv' building '/nix/store/r59gq13sl3knliy61fsdy1rblp90qkqa-fleet-alert-property.drv' fleet-alert-promtool> Checking /nix/store/53r7hx2qbdf983snzx64qc5c36kbvsjq-fleet-alert-rules.yaml fleet-alert-promtool> SUCCESS: 11 rules found fleet-alert-promtool> fleet-alert-promtool> Checking /nix/store/xlsyhfyi69fanskx355hqfff8nq4zsi9-fleet-alert-rules-malformed.yaml fleet-alert-promtool> FAILED: fleet-alert-promtool> /nix/store/xlsyhfyi69fanskx355hqfff8nq4zsi9-fleet-alert-rules-malformed.yaml: 1:758: group "fleet-alert", rule 2, "HostOOMKill": could not parse expression: 1:39: parse error: unclosed left parenthesis fleet-alert-promtool> mutant malformedExpression rejected building '/nix/store/2wfxqw3b07rigqxy41jb3v9l2yl1kqf6-fleet-alert-fixtures.drv' fleet-alert-fixtures> --- the fixture server refuses other paths, databases and accounts, replays SQL as GreptimeDB does, and detects two open requests fleet-alert-fixtures> ok: fixture server refusals, the SQL answer shape and the concurrency guard fleet-alert-fixtures> --- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears fleet-alert-property> ok: transition.jq equals the reference model on every generated timeline fleet-alert-property> ok: a rule without expr but with a results entry fires and resolves as its expr twin fleet-alert-property> ok: a rule without expr or results entry, and results named for internal rules, produce nothing fleet-alert-property> ok: a RESOLVED follows 5 min of absence observed while neither the store nor the host is frozen fleet-alert-fixtures> --- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears fleet-alert-property> ok: a devloop failure leaves fleet's posts as they are without it fleet-alert-fixtures> --- a failed state for every notified unit posts no SystemdUnitFailed, and nginx, not notified, posts one fleet-alert-property> ok: StoreQueryFailing is one instance per failing database, labelled and titled with it fleet-alert-fixtures> --- a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes fleet-alert-fixtures> --- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once fleet-alert-property> Traceback (most recent call last): fleet-alert-property> File "/nix/store/vixziicimr33k31hsz07nc2q0r1dghhg-fleet-alert/test_transition.py", line 404, in fleet-alert-property> test_transition_matches_model() fleet-alert-property> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ fleet-alert-property> File "/nix/store/vixziicimr33k31hsz07nc2q0r1dghhg-fleet-alert/test_transition.py", line 174, in test_transition_matches_model fleet-alert-property> derandomize=True, fleet-alert-property> ^^^ fleet-alert-property> File "/nix/store/0wq2wpqi3611jav9npnssb4c3r3mvyca-python3-3.14.8-env/lib/python3.14/site-packages/hypothesis/core.py", line 2295, in wrapped_test fleet-alert-property> raise the_error_hypothesis_found fleet-alert-property> File "/nix/store/vixziicimr33k31hsz07nc2q0r1dghhg-fleet-alert/test_transition.py", line 187, in test_transition_matches_model fleet-alert-property> assert want == got, f"run {index}: model posts {want}, transition.jq posts {got}" fleet-alert-property> ^^^^^^^^^^^ fleet-alert-property> AssertionError: run 2: model posts [{'title': '[FIRING] StoreQueryFailing devloop', 'priority': 'high', 'kind': 'repost', 'id': '["StoreQueryFailing",[{"key":"database","value":"devloop"}]]'}], transition.jq posts [] fleet-alert-property> Failing test case: test_transition_matches_model( fleet-alert-property> # The test always failed when commented parts were varied together. fleet-alert-property> steps=[{ fleet-alert-property> 'dt': 60, # or any other generated value fleet-alert-property> 'storeErrors': ['devloop'], fleet-alert-property> 'reporting': ['magnetite', fleet-alert-property> 'stibnite'], # or any other generated value fleet-alert-property> 'results': { fleet-alert-property> 'Crit': [], # or any other generated value fleet-alert-property> 'Warn': [], # or any other generated value fleet-alert-property> 'Info': [], # or any other generated value fleet-alert-property> 'Dev': [], # or any other generated value fleet-alert-property> }, fleet-alert-property> 'failAt': None, # or any other generated value fleet-alert-property> }, fleet-alert-property> { fleet-alert-property> 'dt': 300, fleet-alert-property> 'storeErrors': ['devloop'], fleet-alert-property> 'reporting': ['magnetite', fleet-alert-property> 'stibnite'], # or any other generated value fleet-alert-property> 'results': { fleet-alert-property> 'Crit': [], # or any other generated value fleet-alert-property> 'Warn': [], # or any other generated value fleet-alert-property> 'Info': [], # or any other generated value fleet-alert-property> 'Dev': [], # or any other generated value fleet-alert-property> }, fleet-alert-property> 'failAt': None, fleet-alert-property> }, fleet-alert-property> { fleet-alert-property> 'dt': 86400, fleet-alert-property> 'storeErrors': ['devloop'], fleet-alert-property> 'reporting': ['magnetite', fleet-alert-property> 'stibnite'], # or any other generated value fleet-alert-property> 'results': {'Crit': [], fleet-alert-property> 'Warn': [], fleet-alert-property> 'Info': [], fleet-alert-property> 'Dev': []}, # or any other generated value fleet-alert-property> 'failAt': None, # or any other generated value fleet-alert-property> }], fleet-alert-property> ) fleet-alert-property> Explanation: fleet-alert-property> These lines were always and only run by failing test cases: fleet-alert-property> /nix/store/0wq2wpqi3611jav9npnssb4c3r3mvyca-python3-3.14.8-env/lib/python3.14/contextlib.py:157 fleet-alert-property> /nix/store/vixziicimr33k31hsz07nc2q0r1dghhg-fleet-alert/model.py:176 fleet-alert-property> mutant noRepost rejected fleet-alert-fixtures> --- HostOOMKill last seen before 7 runs of store error resolves 5 min after the store answers again, not on its first answer fleet-alert-fixtures> --- HostOOMKill on stibnite last seen before 7 silent runs resolves 5 min after stibnite reports again, not on its first report fleet-alert-fixtures> --- node_vmstat_oom_kill empty for 30 min while magnetite reports raises one RuleSignalMissing fleet-alert-fixtures> --- magnetite and cinnabar silent for 40 min while the store answers raise RuleSignalMissing for each 10 min after they last reported, none for node_vmstat_oom_kill, and resolve when they report again fleet-alert-fixtures> --- cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again fleet-alert-fixtures> --- cinnabar's / above 85 % for 15 min fires FilesystemFillingWarn at default, then above 95 % for 5 min fires FilesystemFull at high fleet-alert-fixtures> --- WeeklyPipelineTest posts once at 09:02 New York time on both sides of the daylight-saving change, never at 08:59, 09:06 or 09:02 EDT clock time in EST fleet-alert-fixtures> --- 11 instances within 24 h post all 11 FIRINGs and exactly one AlertBudgetExceeded fleet-alert-fixtures> --- 4 instances on stibnite within 24 h post exactly one AlertBudgetExceeded fleet-alert-fixtures> --- 10 FAILED lines of the last 24 h and one FIRING exceed the budget fleet-alert-fixtures> --- FAILED lines older than 24 h are not counted fleet-alert-fixtures> --- an existing failed.log that cannot be read fails the run before any post fleet-alert-fixtures> --- a failed heartbeat fails the run, and the FIRING it left pending is posted once on the next run fleet-alert-fixtures> --- PullRequestQueueWaitBurn answered 0.15 on 65 runs fires once on the 61st, posts nothing on the 62nd to 65th, and resolves once 5 min after its row disappears fleet-alert-fixtures> --- a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual fleet-alert-fixtures> --- one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL fleet-alert-fixtures> --- a unit notified on one host posts no SystemdUnitFailed on another, and nginx, notified nowhere, posts one fleet-alert-fixtures> --- pyrite, with HostOOMKill firing, and electrum silent for 40 min post nothing after the FIRING: no silence, no resolution, no repeat fleet-alert-fixtures> --- no SMART series for 30 min posts nothing while pyrite is silent, then one RuleSignalMissing for each NVMe rule 30 min after pyrite reports fleet-alert-fixtures> --- pyrite's NVMe critical warning on one run fires NvmeCriticalWarning pyrite at once at high fleet-alert-fixtures> --- pyrite, not collecting, reporting without SMART series for 35 min and then silent for 40 min posts nothing fleet-alert-fixtures> === mutant allDatabasesFreeze fleet-alert-fixtures> --- a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual fleet-alert-fixtures> --- /nix/store/3ksgnvzycis2fgfdfci3lnshpnwcfzqx-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.ItMxvPOg3t/actual 2026-11-01 05:48:49.889974485 +0000 fleet-alert-fixtures> @@ -1,4 +1,3 @@ fleet-alert-fixtures> 0 [FIRING] HostOOMKill magnetite high fleet-alert-fixtures> 4 [FIRING] StoreQueryFailing devloop high fleet-alert-fixtures> -5 [RESOLVED] HostOOMKill magnetite high fleet-alert-fixtures> 11 [RESOLVED] StoreQueryFailing devloop high fleet-alert-fixtures> MISS a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual: the alert posts differ from the expected list fleet-alert-fixtures> mutant allDatabasesFreeze rejected: a devloop failure raises StoreQueryFailing for devloop after 3 min while HostOOMKill on fleet fires and resolves as usual: the alert posts differ from the expected list fleet-alert-fixtures> === mutant alwaysOnMagnetiteOnly fleet-alert-fixtures> --- cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again fleet-alert-fixtures> --- /nix/store/aarcpx64r6vljqa73iglqd5hxmfbycl5-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.xLkVIKe2Sm/actual 2026-11-01 05:48:49.912974529 +0000 fleet-alert-fixtures> @@ -1,2 +0,0 @@ fleet-alert-fixtures> -3 [FIRING] RuleSignalMissing cinnabar default fleet-alert-fixtures> -10 [RESOLVED] RuleSignalMissing cinnabar default fleet-alert-fixtures> MISS cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again: the alert posts differ from the expected list fleet-alert-fixtures> MISS cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again: no posted message carries: selector=system_memory_usage_bytes fleet-alert-fixtures> mutant alwaysOnMagnetiteOnly rejected: cinnabar silent for 40 min while magnetite reports raises RuleSignalMissing cinnabar 10 min after it last reported, and resolves when it reports again: the alert posts differ from the expected list fleet-alert-fixtures> === mutant everyQueryToFleet fleet-alert-fixtures> --- one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL fleet-alert-fixtures> MISS one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL: the server did not record exactly one: REQUEST 0 /v1/sql devloop SELECT CAST(sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) AS DOUBLE) / count(*) AS value FROM ci_build c JOIN (SELECT build_id, min(finished_at) AS first_finished FROM ci_build GROUP BY build_id) f ON c.build_id = f.build_id AND c.finished_at = f.first_finished WHERE c.pr_number IS NOT NULL AND c.started_at IS NOT NULL AND c.finished_at >= now() - INTERVAL '24 hours' HAVING count(*) >= 10 AND sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) * 10 > count(*) fleet-alert-fixtures> mutant everyQueryToFleet rejected: one run queries HostOOMKill on fleet through PromQL and PullRequestQueueWaitBurn on devloop through SQL: the server did not record exactly one: REQUEST 0 /v1/sql devloop SELECT CAST(sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) AS DOUBLE) / count(*) AS value FROM ci_build c JOIN (SELECT build_id, min(finished_at) AS first_finished FROM ci_build GROUP BY build_id) f ON c.build_id = f.build_id AND c.finished_at = f.first_finished WHERE c.pr_number IS NOT NULL AND c.started_at IS NOT NULL AND c.finished_at >= now() - INTERVAL '24 hours' HAVING count(*) >= 10 AND sum(CASE WHEN c.started_at > c.created_at + INTERVAL '1 hour' THEN 1 ELSE 0 END) * 10 > count(*) fleet-alert-fixtures> === mutant forZero fleet-alert-fixtures> --- HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears fleet-alert-fixtures> --- /nix/store/2ngrim690axpydj1bk1ghjafymwc0v6c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.G2xMANtdtP/actual 2026-11-01 05:49:21.571035554 +0000 fleet-alert-fixtures> @@ -1,2 +1,2 @@ fleet-alert-fixtures> -5 [FIRING] HostMemoryPressure magnetite default fleet-alert-fixtures> +0 [FIRING] HostMemoryPressure magnetite default fleet-alert-fixtures> 11 [RESOLVED] HostMemoryPressure magnetite default fleet-alert-fixtures> MISS HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list fleet-alert-fixtures> mutant forZero rejected: HostMemoryPressure fires once after its for of 5 min and resolves once 5 min after it clears: the alert posts differ from the expected list fleet-alert-fixtures> === mutant noAbsenceFreeze fleet-alert-fixtures> --- stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once fleet-alert-fixtures> --- /nix/store/2w5szkyqg1flnpplqw9sd3fy1894ks1c-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.e6zWwGXtPW/actual 2026-11-01 05:49:51.990093955 +0000 fleet-alert-fixtures> @@ -1,2 +1,4 @@ fleet-alert-fixtures> 0 [FIRING] HostOOMKill stibnite high fleet-alert-fixtures> +5 [RESOLVED] HostOOMKill stibnite high fleet-alert-fixtures> +11 [FIRING] HostOOMKill stibnite high fleet-alert-fixtures> 16 [RESOLVED] HostOOMKill stibnite high fleet-alert-fixtures> MISS stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list fleet-alert-fixtures> mutant noAbsenceFreeze rejected: stibnite falls silent with HostOOMKill firing, posts nothing until it reports, then resolves once: the alert posts differ from the expected list fleet-alert-fixtures> === mutant noHostsUpdate fleet-alert-fixtures> --- magnetite and cinnabar silent for 40 min while the store answers raise RuleSignalMissing for each 10 min after they last reported, none for node_vmstat_oom_kill, and resolve when they report again fleet-alert-fixtures> --- /nix/store/kbmhj18ibd15nbjl6jgr9dkrqlclx8a2-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.tk62OQlomD/actual 2026-11-01 05:50:09.462127400 +0000 fleet-alert-fixtures> @@ -1,4 +1,4 @@ fleet-alert-fixtures> -3 [FIRING] RuleSignalMissing cinnabar default fleet-alert-fixtures> -3 [FIRING] RuleSignalMissing magnetite default fleet-alert-fixtures> +2 [FIRING] RuleSignalMissing cinnabar default fleet-alert-fixtures> +2 [FIRING] RuleSignalMissing magnetite default fleet-alert-fixtures> 10 [RESOLVED] RuleSignalMissing cinnabar default fleet-alert-fixtures> 10 [RESOLVED] RuleSignalMissing magnetite default fleet-alert-fixtures> MISS magnetite and cinnabar silent for 40 min while the store answers raise RuleSignalMissing for each 10 min after they last reported, none for node_vmstat_oom_kill, and resolve when they report again: the alert posts differ from the expected list fleet-alert-fixtures> mutant noHostsUpdate rejected: magnetite and cinnabar silent for 40 min while the store answers raise RuleSignalMissing for each 10 min after they last reported, none for node_vmstat_oom_kill, and resolve when they report again: the alert posts differ from the expected list fleet-alert-fixtures> === mutant noStateWrite fleet-alert-fixtures> --- HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears fleet-alert-fixtures> MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.zBfLp9OWxo/state.json: No such file or directory fleet-alert-fixtures> MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 1 exited 2, expected 0: jq: error: Could not open file /build/tmp.zBfLp9OWxo/state.json: No such file or directory fleet-alert-fixtures> --- /nix/store/qfv7fkygc10w316wiynys329fsmlx32w-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.zBfLp9OWxo/actual 2026-11-01 05:50:13.364134860 +0000 fleet-alert-fixtures> @@ -1,2 +1,2 @@ fleet-alert-fixtures> 0 [FIRING] HostOOMKill magnetite high fleet-alert-fixtures> -6 [RESOLVED] HostOOMKill magnetite high fleet-alert-fixtures> +1 [FIRING] HostOOMKill magnetite high fleet-alert-fixtures> MISS HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: the alert posts differ from the expected list fleet-alert-fixtures> mutant noStateWrite rejected: HostOOMKill fires once after its for of 0 min and resolves once 5 min after it clears: run 0 exited 2, expected 0: jq: error: Could not open file /build/tmp.zBfLp9OWxo/state.json: No such file or directory fleet-alert-fixtures> === mutant noStoreFreeze fleet-alert-fixtures> --- a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes fleet-alert-fixtures> --- /nix/store/yc63wbqnip77jf3vlvryjari8p7bs33s-fleet-alert-expected 1970-01-01 00:00:01.000000000 +0000 fleet-alert-fixtures> +++ /build/tmp.Gt6VKq3ZpG/actual 2026-11-01 05:50:36.118178293 +0000 fleet-alert-fixtures> @@ -1,6 +1,6 @@ fleet-alert-fixtures> 0 [FIRING] HostOOMKill magnetite high fleet-alert-fixtures> 4 [FIRING] StoreQueryFailing devloop high fleet-alert-fixtures> 4 [FIRING] StoreQueryFailing fleet high fleet-alert-fixtures> +5 [RESOLVED] HostOOMKill magnetite high fleet-alert-fixtures> 13 [RESOLVED] StoreQueryFailing devloop high fleet-alert-fixtures> 13 [RESOLVED] StoreQueryFailing fleet high fleet-alert-fixtures> -14 [RESOLVED] HostOOMKill magnetite high fleet-alert-fixtures> MISS a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes: the alert posts differ from the expected list fleet-alert-fixtures> mutant noStoreFreeze rejected: a store error on fleet and devloop freezes a firing HostOOMKill, raises one StoreQueryFailing per database after 3 min, heartbeats, and resumes: the alert posts differ from the expected list fleet-alert-fixtures> ok: every leg passed building '/nix/store/cza475sjfq5hp1d5z62gg3xn4fp7b48c-fleet-alert.drv' upload niks3: ok