these 9 derivations will be built: /nix/store/3352l03wlkbgjhvxy92vn2hi4bk11q2z-stub-cluster-apply.drv /nix/store/kmy16blqxkaj3997bv006ah33sg3pszd-omnigraph.drv /nix/store/77lw1va4kv37xfs8l80kyh25crvw6m1k-omnigraph-cluster-bootstrap.drv /nix/store/7zz44a11n8nrr6f8z2di7qmld93wkcsh-omnigraph-cluster-bootstrap.drv /nix/store/dbr87q3aqxjll46491fd5k5c91zwi3a3-omnigraph-cluster-bootstrap.drv /nix/store/g4jvcfkcml14pliqmgaws204b7dkv93l-omnigraph-cluster-bootstrap.drv /nix/store/sdqi7nrs7vzkiq8861bwswwwnjm3rllx-omnigraph-cluster-bootstrap.drv /nix/store/xcl05wg7rmikjnp00nvhzhbqw4lfr19f-s3-stub.py.drv /nix/store/q9ix9ki8cna2gxajg48bn7259drvd95i-omnigraph-bootstrap-rehearsal.drv building '/nix/store/xcl05wg7rmikjnp00nvhzhbqw4lfr19f-s3-stub.py.drv' s3-stub.py> structuredAttrs is enabled building '/nix/store/kmy16blqxkaj3997bv006ah33sg3pszd-omnigraph.drv' building '/nix/store/3352l03wlkbgjhvxy92vn2hi4bk11q2z-stub-cluster-apply.drv' omnigraph> structuredAttrs is enabled warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy stub-cluster-apply> structuredAttrs is enabled building '/nix/store/7zz44a11n8nrr6f8z2di7qmld93wkcsh-omnigraph-cluster-bootstrap.drv' building '/nix/store/dbr87q3aqxjll46491fd5k5c91zwi3a3-omnigraph-cluster-bootstrap.drv' building '/nix/store/g4jvcfkcml14pliqmgaws204b7dkv93l-omnigraph-cluster-bootstrap.drv' omnigraph-cluster-bootstrap> structuredAttrs is enabled omnigraph-cluster-bootstrap> structuredAttrs is enabled building '/nix/store/77lw1va4kv37xfs8l80kyh25crvw6m1k-omnigraph-cluster-bootstrap.drv' building '/nix/store/sdqi7nrs7vzkiq8861bwswwwnjm3rllx-omnigraph-cluster-bootstrap.drv' omnigraph-cluster-bootstrap> structuredAttrs is enabled omnigraph-cluster-bootstrap> structuredAttrs is enabled omnigraph-cluster-bootstrap> structuredAttrs is enabled building '/nix/store/q9ix9ki8cna2gxajg48bn7259drvd95i-omnigraph-bootstrap-rehearsal.drv' omnigraph-bootstrap-rehearsal> --- s3: state found, marker present omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state and marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized present; nothing to do omnigraph-bootstrap-rehearsal> --- s3: state found, marker absent omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state present without a marker; adopted s3://rehearsal/clusters/dev-graph/ and wrote s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized omnigraph-bootstrap-rehearsal> --- s3: state missing, marker absent omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: no cluster state and no marker: initializing s3://rehearsal/clusters/dev-graph/ omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: initialized s3://rehearsal/clusters/dev-graph/ and wrote s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized omnigraph-bootstrap-rehearsal> --- s3: state missing, marker present omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: REFUSING TO INITIALIZE s3://rehearsal/clusters/dev-graph/ omnigraph-bootstrap-rehearsal> The marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized exists, so this storage root was initialized before, omnigraph-bootstrap-rehearsal> but it no longer holds cluster state (__cluster/state.json): the root was omnigraph-bootstrap-rehearsal> lost or emptied. Applying now would create a new, empty cluster in its omnigraph-bootstrap-rehearsal> place, so nothing was applied. omnigraph-bootstrap-rehearsal> Restore the root from backup, then: systemctl restart omnigraph-cluster-bootstrap omnigraph-bootstrap-rehearsal> To deliberately start over with an empty cluster, delete the marker omnigraph-bootstrap-rehearsal> s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized and then: systemctl restart omnigraph-cluster-bootstrap omnigraph-bootstrap-rehearsal> --- s3: apply fails omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: no cluster state and no marker: initializing s3://rehearsal/clusters/dev-graph/ omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: initializing s3://rehearsal/clusters/dev-graph/ failed; the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized was not written, so the next start retries omnigraph-bootstrap-rehearsal> --- s3: status fails omnigraph-bootstrap-rehearsal> {"ok":false,"diagnostics":[{"severity":"error","code":"state_read_error"}]} omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: omnigraph cluster status failed for s3://rehearsal/clusters/dev-graph/; nothing was applied omnigraph-bootstrap-rehearsal> --- local: state found, marker present omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state and marker store/_omnigraph-markers/root.initialized present; nothing to do omnigraph-bootstrap-rehearsal> --- local: state found, marker absent omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state present without a marker; adopted store/root/ and wrote store/_omnigraph-markers/root.initialized omnigraph-bootstrap-rehearsal> --- local: state missing, marker absent omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: no cluster state and no marker: initializing store/root/ omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: initialized store/root/ and wrote store/_omnigraph-markers/root.initialized omnigraph-bootstrap-rehearsal> --- local: state missing, marker present omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: REFUSING TO INITIALIZE store/root/ omnigraph-bootstrap-rehearsal> The marker store/_omnigraph-markers/root.initialized exists, so this storage root was initialized before, omnigraph-bootstrap-rehearsal> but it no longer holds cluster state (__cluster/state.json): the root was omnigraph-bootstrap-rehearsal> lost or emptied. Applying now would create a new, empty cluster in its omnigraph-bootstrap-rehearsal> place, so nothing was applied. omnigraph-bootstrap-rehearsal> Restore the root from backup, then: systemctl restart omnigraph-cluster-bootstrap omnigraph-bootstrap-rehearsal> To deliberately start over with an empty cluster, delete the marker omnigraph-bootstrap-rehearsal> store/_omnigraph-markers/root.initialized and then: systemctl restart omnigraph-cluster-bootstrap omnigraph-bootstrap-rehearsal> --- local: apply fails omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: no cluster state and no marker: initializing store/root/ omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: initializing store/root/ failed; the marker store/_omnigraph-markers/root.initialized was not written, so the next start retries omnigraph-bootstrap-rehearsal> --- local: status fails omnigraph-bootstrap-rehearsal> {"ok":false,"diagnostics":[{"severity":"error","code":"state_read_error"}]} omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: omnigraph cluster status failed for store/root/; nothing was applied omnigraph-bootstrap-rehearsal> --- s3: HEAD answers 500 omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: HEAD http://127.0.0.1:43149/rehearsal/clusters/_omnigraph-markers/dev-graph.initialized for the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized answered HTTP 500; cannot tell whether the root was initialized, nothing was applied omnigraph-bootstrap-rehearsal> --- s3: endpoint from AWS_ENDPOINT_URL omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cluster state present without a marker; adopted s3://rehearsal/clusters/dev-graph/ and wrote s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized omnigraph-bootstrap-rehearsal> --- s3: endpoint unset omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: cannot read the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized: s3_endpoint is unset (the endpoint comes from AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL); nothing was applied omnigraph-bootstrap-rehearsal> --- rejected root s3 bucket omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: the storage root s3://rehearsal/ has no basename after stripping trailing slashes, so it has no parent to hold its marker; nothing was applied omnigraph-bootstrap-rehearsal> --- rejected root slash omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: the storage root / has no basename after stripping trailing slashes, so it has no parent to hold its marker; nothing was applied omnigraph-bootstrap-rehearsal> --- rejected root gs omnigraph-bootstrap-rehearsal> omnigraph-cluster-bootstrap: unsupported storage scheme in gs://rehearsal/root; only s3://, file:// and plain paths are supported; nothing was applied omnigraph-bootstrap-rehearsal> --- every S3 request is path-style and SigV4-signed for the configured key and region upload niks3: ok