these 9 derivations will be built: /nix/store/1y68hs61jwmq16g7ml5md239blxiz1br-stub-cluster-apply.drv /nix/store/2099as6xliyi7jpkc28nn2cz3282902d-omnigraph.drv /nix/store/5djajn2s9n6a41d8alsjgksh4df2d5bj-omnigraph-cluster-bootstrap.drv /nix/store/5vyc8f5w0k327ysy6wc9lkdmm4pz8b9h-omnigraph-cluster-bootstrap.drv /nix/store/b1shcsz7q20pfgi3mmbl9vcw7bbgagda-omnigraph-cluster-bootstrap.drv /nix/store/b8pk60m8y3862s2if5rccxdiqicnfwif-omnigraph-cluster-bootstrap.drv /nix/store/nd7gv2ccfa7bijk7823gz5dgh1lmvcf6-s3-stub.py.drv /nix/store/yjm3lzk42d212b9sy1cz7xjgv8h1q1m1-omnigraph-cluster-bootstrap.drv /nix/store/9x6m45piixs7y2ic06xlrc74danaakl3-omnigraph-bootstrap-rehearsal.drv building '/nix/store/nd7gv2ccfa7bijk7823gz5dgh1lmvcf6-s3-stub.py.drv' s3-stub.py> structuredAttrs is enabled building '/nix/store/2099as6xliyi7jpkc28nn2cz3282902d-omnigraph.drv' building '/nix/store/1y68hs61jwmq16g7ml5md239blxiz1br-stub-cluster-apply.drv' omnigraph> structuredAttrs is enabled stub-cluster-apply> structuredAttrs is enabled building '/nix/store/5vyc8f5w0k327ysy6wc9lkdmm4pz8b9h-omnigraph-cluster-bootstrap.drv' building '/nix/store/b1shcsz7q20pfgi3mmbl9vcw7bbgagda-omnigraph-cluster-bootstrap.drv' building '/nix/store/b8pk60m8y3862s2if5rccxdiqicnfwif-omnigraph-cluster-bootstrap.drv' omnigraph-cluster-bootstrap> structuredAttrs is enabled omnigraph-cluster-bootstrap> structuredAttrs is enabled building '/nix/store/5djajn2s9n6a41d8alsjgksh4df2d5bj-omnigraph-cluster-bootstrap.drv' building '/nix/store/yjm3lzk42d212b9sy1cz7xjgv8h1q1m1-omnigraph-cluster-bootstrap.drv' omnigraph-cluster-bootstrap> structuredAttrs is enabled omnigraph-cluster-bootstrap> structuredAttrs is enabled omnigraph-cluster-bootstrap> structuredAttrs is enabled building '/nix/store/9x6m45piixs7y2ic06xlrc74danaakl3-omnigraph-bootstrap-rehearsal.drv' omnigraph-bootstrap-rehearsal> --- s3: state found, marker present omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state and marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized present; nothing to do omnigraph-bootstrap-rehearsal> --- s3: state found, marker absent omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state present without a marker; adopted s3://rehearsal/clusters/dev-graph/ and wrote s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized omnigraph-bootstrap-rehearsal> --- s3: state missing, marker absent omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: no cluster state and no marker: initializing s3://rehearsal/clusters/dev-graph/ omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: initialized s3://rehearsal/clusters/dev-graph/ and wrote s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized omnigraph-bootstrap-rehearsal> --- s3: state missing, marker present omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: REFUSING TO INITIALIZE s3://rehearsal/clusters/dev-graph/ omnigraph-bootstrap-rehearsal> The marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized exists, so this storage root was initialized before, omnigraph-bootstrap-rehearsal> but it no longer holds cluster state (__cluster/state.json): the root was omnigraph-bootstrap-rehearsal> lost or emptied. Applying now would create a new, empty cluster in its omnigraph-bootstrap-rehearsal> place, so nothing was applied. omnigraph-bootstrap-rehearsal> Restore the root from backup, then: systemctl restart omnigraph-cluster-bootstrap omnigraph-bootstrap-rehearsal> To deliberately start over with an empty cluster, delete the marker omnigraph-bootstrap-rehearsal> s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized and then: systemctl restart omnigraph-cluster-bootstrap omnigraph-bootstrap-rehearsal> --- s3: apply fails omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: no cluster state and no marker: initializing s3://rehearsal/clusters/dev-graph/ omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: initializing s3://rehearsal/clusters/dev-graph/ failed; the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized was not written, so the next start retries omnigraph-bootstrap-rehearsal> --- s3: status fails omnigraph-bootstrap-rehearsal> {"ok":false,"diagnostics":[{"severity":"error","code":"state_read_error"}]} omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: omnigraph cluster status failed for s3://rehearsal/clusters/dev-graph/; nothing was applied omnigraph-bootstrap-rehearsal> --- local: state found, marker present omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state and marker store/_omnigraph-markers/root.initialized present; nothing to do omnigraph-bootstrap-rehearsal> --- local: state found, marker absent omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state present without a marker; adopted store/root/ and wrote store/_omnigraph-markers/root.initialized omnigraph-bootstrap-rehearsal> --- local: state missing, marker absent omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: no cluster state and no marker: initializing store/root/ omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: initialized store/root/ and wrote store/_omnigraph-markers/root.initialized omnigraph-bootstrap-rehearsal> --- local: state missing, marker present omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: REFUSING TO INITIALIZE store/root/ omnigraph-bootstrap-rehearsal> The marker store/_omnigraph-markers/root.initialized exists, so this storage root was initialized before, omnigraph-bootstrap-rehearsal> but it no longer holds cluster state (__cluster/state.json): the root was omnigraph-bootstrap-rehearsal> lost or emptied. Applying now would create a new, empty cluster in its omnigraph-bootstrap-rehearsal> place, so nothing was applied. omnigraph-bootstrap-rehearsal> Restore the root from backup, then: systemctl restart omnigraph-cluster-bootstrap omnigraph-bootstrap-rehearsal> To deliberately start over with an empty cluster, delete the marker omnigraph-bootstrap-rehearsal> store/_omnigraph-markers/root.initialized and then: systemctl restart omnigraph-cluster-bootstrap omnigraph-bootstrap-rehearsal> --- local: apply fails omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: no cluster state and no marker: initializing store/root/ omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: initializing store/root/ failed; the marker store/_omnigraph-markers/root.initialized was not written, so the next start retries omnigraph-bootstrap-rehearsal> --- local: status fails omnigraph-bootstrap-rehearsal> {"ok":false,"diagnostics":[{"severity":"error","code":"state_read_error"}]} omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: omnigraph cluster status failed for store/root/; nothing was applied omnigraph-bootstrap-rehearsal> --- s3: HEAD answers 500 omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: HEAD http://127.0.0.1:32871/rehearsal/clusters/_omnigraph-markers/dev-graph.initialized for the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized answered HTTP 500; cannot tell whether the root was initialized, nothing was applied omnigraph-bootstrap-rehearsal> --- s3: endpoint from AWS_ENDPOINT_URL omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state present without a marker; adopted s3://rehearsal/clusters/dev-graph/ and wrote s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized omnigraph-bootstrap-rehearsal> --- s3: endpoint unset omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cannot read the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized: s3_endpoint is unset (the endpoint comes from AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL); nothing was applied omnigraph-bootstrap-rehearsal> --- rejected root s3 bucket omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: the storage root s3://rehearsal/ has no basename after stripping trailing slashes, so it has no parent to hold its marker; nothing was applied omnigraph-bootstrap-rehearsal> --- rejected root slash omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: the storage root / has no basename after stripping trailing slashes, so it has no parent to hold its marker; nothing was applied omnigraph-bootstrap-rehearsal> --- rejected root gs omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: unsupported storage scheme in gs://rehearsal/root; only s3://, file:// and plain paths are supported; nothing was applied omnigraph-bootstrap-rehearsal> --- every S3 request is path-style and SigV4-signed for the configured key and region upload niks3: ok