these 6 derivations will be built: /nix/store/16b47ywqz6rizgf9cr4w5h12zkxxlsar-omnigraph-cluster-bootstrap.drv /nix/store/3zdigjy4gky5k4jpppvrq21avqisfny5-omnigraph-cluster-bootstrap.drv /nix/store/9gc0mvbahjr3ssr69p9yfahicx9h13zm-omnigraph-cluster-bootstrap.drv /nix/store/iasyhbhnp8hywiv6c5sjrm6x2p486x79-omnigraph-cluster-bootstrap.drv /nix/store/z4nw8ngipr68w4c5zs80601ckyci1k7r-omnigraph-cluster-bootstrap.drv /nix/store/z8jrdhyihii5inw6swq9hbcn4gsq3jq6-omnigraph-bootstrap-rehearsal.drv these 3 paths will be fetched (0.0 KiB download, 2.9 KiB unpacked): /nix/store/dqijsr7h13yf5q1vyy487wnmgkwivpjz-omnigraph /nix/store/z8l8a43jidj2wjcrk5mpz1lrx7n0yljb-s3-stub.py /nix/store/88spxgh6179yjjpfg1w4rla8h17jxiq8-stub-cluster-apply building '/nix/store/16b47ywqz6rizgf9cr4w5h12zkxxlsar-omnigraph-cluster-bootstrap.drv' building '/nix/store/3zdigjy4gky5k4jpppvrq21avqisfny5-omnigraph-cluster-bootstrap.drv' building '/nix/store/9gc0mvbahjr3ssr69p9yfahicx9h13zm-omnigraph-cluster-bootstrap.drv' building '/nix/store/iasyhbhnp8hywiv6c5sjrm6x2p486x79-omnigraph-cluster-bootstrap.drv' building '/nix/store/z4nw8ngipr68w4c5zs80601ckyci1k7r-omnigraph-cluster-bootstrap.drv' omnigraph-cluster-bootstrap> structuredAttrs is enabled omnigraph-cluster-bootstrap> structuredAttrs is enabled omnigraph-cluster-bootstrap> structuredAttrs is enabled omnigraph-cluster-bootstrap> structuredAttrs is enabled warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy omnigraph-cluster-bootstrap> structuredAttrs is enabled building '/nix/store/z8jrdhyihii5inw6swq9hbcn4gsq3jq6-omnigraph-bootstrap-rehearsal.drv' omnigraph-bootstrap-rehearsal> --- s3: state found, marker present omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state and marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized present; nothing to do omnigraph-bootstrap-rehearsal> --- s3: state found, marker absent omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state present without a marker; adopted s3://rehearsal/clusters/dev-graph/ and wrote s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized omnigraph-bootstrap-rehearsal> --- s3: state missing, marker absent omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: no cluster state and no marker: initializing s3://rehearsal/clusters/dev-graph/ omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: initialized s3://rehearsal/clusters/dev-graph/ and wrote s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized omnigraph-bootstrap-rehearsal> --- s3: state missing, marker present omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: REFUSING TO INITIALIZE s3://rehearsal/clusters/dev-graph/ omnigraph-bootstrap-rehearsal> The marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized exists, so this storage root was initialized before, omnigraph-bootstrap-rehearsal> but it no longer holds cluster state (__cluster/state.json): the root was omnigraph-bootstrap-rehearsal> lost or emptied. Applying now would create a new, empty cluster in its omnigraph-bootstrap-rehearsal> place, so nothing was applied. omnigraph-bootstrap-rehearsal> Restore the root from backup, then: systemctl restart omnigraph-cluster-bootstrap omnigraph-bootstrap-rehearsal> To deliberately start over with an empty cluster, delete the marker omnigraph-bootstrap-rehearsal> s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized and then: systemctl restart omnigraph-cluster-bootstrap omnigraph-bootstrap-rehearsal> --- s3: apply fails omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: no cluster state and no marker: initializing s3://rehearsal/clusters/dev-graph/ omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: initializing s3://rehearsal/clusters/dev-graph/ failed; the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized was not written, so the next start retries omnigraph-bootstrap-rehearsal> --- s3: status fails omnigraph-bootstrap-rehearsal> {"ok":false,"diagnostics":[{"severity":"error","code":"state_read_error"}]} omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: omnigraph cluster status failed for s3://rehearsal/clusters/dev-graph/; nothing was applied omnigraph-bootstrap-rehearsal> --- local: state found, marker present omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state and marker store/_omnigraph-markers/root.initialized present; nothing to do omnigraph-bootstrap-rehearsal> --- local: state found, marker absent omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state present without a marker; adopted store/root/ and wrote store/_omnigraph-markers/root.initialized omnigraph-bootstrap-rehearsal> --- local: state missing, marker absent omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: no cluster state and no marker: initializing store/root/ omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: initialized store/root/ and wrote store/_omnigraph-markers/root.initialized omnigraph-bootstrap-rehearsal> --- local: state missing, marker present omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: REFUSING TO INITIALIZE store/root/ omnigraph-bootstrap-rehearsal> The marker store/_omnigraph-markers/root.initialized exists, so this storage root was initialized before, omnigraph-bootstrap-rehearsal> but it no longer holds cluster state (__cluster/state.json): the root was omnigraph-bootstrap-rehearsal> lost or emptied. Applying now would create a new, empty cluster in its omnigraph-bootstrap-rehearsal> place, so nothing was applied. omnigraph-bootstrap-rehearsal> Restore the root from backup, then: systemctl restart omnigraph-cluster-bootstrap omnigraph-bootstrap-rehearsal> To deliberately start over with an empty cluster, delete the marker omnigraph-bootstrap-rehearsal> store/_omnigraph-markers/root.initialized and then: systemctl restart omnigraph-cluster-bootstrap omnigraph-bootstrap-rehearsal> --- local: apply fails omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: no cluster state and no marker: initializing store/root/ omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: initializing store/root/ failed; the marker store/_omnigraph-markers/root.initialized was not written, so the next start retries omnigraph-bootstrap-rehearsal> --- local: status fails omnigraph-bootstrap-rehearsal> {"ok":false,"diagnostics":[{"severity":"error","code":"state_read_error"}]} omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: omnigraph cluster status failed for store/root/; nothing was applied omnigraph-bootstrap-rehearsal> --- s3: HEAD answers 500 omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: HEAD http://127.0.0.1:33821/rehearsal/clusters/_omnigraph-markers/dev-graph.initialized for the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized answered HTTP 500; cannot tell whether the root was initialized, nothing was applied omnigraph-bootstrap-rehearsal> --- s3: endpoint from AWS_ENDPOINT_URL omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state present without a marker; adopted s3://rehearsal/clusters/dev-graph/ and wrote s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized omnigraph-bootstrap-rehearsal> --- s3: endpoint unset omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cannot read the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized: s3_endpoint is unset (the endpoint comes from AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL); nothing was applied omnigraph-bootstrap-rehearsal> --- rejected root s3 bucket omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: the storage root s3://rehearsal/ has no basename after stripping trailing slashes, so it has no parent to hold its marker; nothing was applied omnigraph-bootstrap-rehearsal> --- rejected root slash omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: the storage root / has no basename after stripping trailing slashes, so it has no parent to hold its marker; nothing was applied omnigraph-bootstrap-rehearsal> --- rejected root gs omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: unsupported storage scheme in gs://rehearsal/root; only s3://, file:// and plain paths are supported; nothing was applied omnigraph-bootstrap-rehearsal> --- every S3 request is path-style and SigV4-signed for the configured key and region upload niks3: ok