--- s3: state found, marker present omnigraph-cluster-bootstrap: cluster state and marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized present; nothing to do --- s3: state found, marker absent omnigraph-cluster-bootstrap: cluster state present without a marker; adopted s3://rehearsal/clusters/dev-graph/ and wrote s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized --- s3: state missing, marker absent omnigraph-cluster-bootstrap: no cluster state and no marker: initializing s3://rehearsal/clusters/dev-graph/ omnigraph-cluster-bootstrap: initialized s3://rehearsal/clusters/dev-graph/ and wrote s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized --- s3: state missing, marker present omnigraph-cluster-bootstrap: REFUSING TO INITIALIZE s3://rehearsal/clusters/dev-graph/ The marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized exists, so this storage root was initialized before, but it no longer holds cluster state (__cluster/state.json): the root was lost or emptied. Applying now would create a new, empty cluster in its place, so nothing was applied. Restore the root from backup, then: systemctl restart omnigraph-cluster-bootstrap To deliberately start over with an empty cluster, delete the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized and then: systemctl restart omnigraph-cluster-bootstrap --- s3: apply fails omnigraph-cluster-bootstrap: no cluster state and no marker: initializing s3://rehearsal/clusters/dev-graph/ omnigraph-cluster-bootstrap: initializing s3://rehearsal/clusters/dev-graph/ failed; the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized was not written, so the next start retries --- s3: status fails {"ok":false,"diagnostics":[{"severity":"error","code":"state_read_error"}]} omnigraph-cluster-bootstrap: omnigraph cluster status failed for s3://rehearsal/clusters/dev-graph/; nothing was applied --- local: state found, marker present omnigraph-cluster-bootstrap: cluster state and marker store/_omnigraph-markers/root.initialized present; nothing to do --- local: state found, marker absent omnigraph-cluster-bootstrap: cluster state present without a marker; adopted store/root/ and wrote store/_omnigraph-markers/root.initialized --- local: state missing, marker absent omnigraph-cluster-bootstrap: no cluster state and no marker: initializing store/root/ omnigraph-cluster-bootstrap: initialized store/root/ and wrote store/_omnigraph-markers/root.initialized --- local: state missing, marker present omnigraph-cluster-bootstrap: REFUSING TO INITIALIZE store/root/ The marker store/_omnigraph-markers/root.initialized exists, so this storage root was initialized before, but it no longer holds cluster state (__cluster/state.json): the root was lost or emptied. Applying now would create a new, empty cluster in its place, so nothing was applied. Restore the root from backup, then: systemctl restart omnigraph-cluster-bootstrap To deliberately start over with an empty cluster, delete the marker store/_omnigraph-markers/root.initialized and then: systemctl restart omnigraph-cluster-bootstrap --- local: apply fails omnigraph-cluster-bootstrap: no cluster state and no marker: initializing store/root/ omnigraph-cluster-bootstrap: initializing store/root/ failed; the marker store/_omnigraph-markers/root.initialized was not written, so the next start retries --- local: status fails {"ok":false,"diagnostics":[{"severity":"error","code":"state_read_error"}]} omnigraph-cluster-bootstrap: omnigraph cluster status failed for store/root/; nothing was applied --- s3: HEAD answers 500 omnigraph-cluster-bootstrap: HEAD http://127.0.0.1:32871/rehearsal/clusters/_omnigraph-markers/dev-graph.initialized for the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized answered HTTP 500; cannot tell whether the root was initialized, nothing was applied --- s3: endpoint from AWS_ENDPOINT_URL omnigraph-cluster-bootstrap: cluster state present without a marker; adopted s3://rehearsal/clusters/dev-graph/ and wrote s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized --- s3: endpoint unset omnigraph-cluster-bootstrap: cannot read the marker s3://rehearsal/clusters/_omnigraph-markers/dev-graph.initialized: s3_endpoint is unset (the endpoint comes from AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL); nothing was applied --- rejected root s3 bucket omnigraph-cluster-bootstrap: the storage root s3://rehearsal/ has no basename after stripping trailing slashes, so it has no parent to hold its marker; nothing was applied --- rejected root slash omnigraph-cluster-bootstrap: the storage root / has no basename after stripping trailing slashes, so it has no parent to hold its marker; nothing was applied --- rejected root gs omnigraph-cluster-bootstrap: unsupported storage scheme in gs://rehearsal/root; only s3://, file:// and plain paths are supported; nothing was applied --- every S3 request is path-style and SigV4-signed for the configured key and region