HA storage and recovery

HA remains opt-in and experimental. Kubernetes Lease ownership and PostgreSQL advisory locks do not fence a suspended former owner. Resource-enforced ownership now covers the database, broker, media, dispatch, and admission boundaries, described in High availability. The failure envelope is qualified by the HA qualification harness, and the qualification report records measured recovery times and the remaining limitations. These checks add storage and recovery prerequisites to that qualification.

Supported storage

• Configure one global PostgreSQL database and leave component connection strings empty or identical to the global connection string. Separate component databases and SQLite are not supported in HA mode.
• Enable only the experimental MSCs qualified for HA: msc2444, msc2753, and msc2836. Startup refuses any other MSC, including msc3814 and msc3861. With msc2836, leave mscs.database.connection_string empty or identical to the global connection string, so the relationship index is fenced like every other table.
• Configure external JetStream addresses, a nonempty homeserver-specific topic prefix, and global.jetstream.replicas: 3 or 5. HA uses file storage for every stream, including typing and presence streams. Startup waits up to 30 seconds per stream for every configured replica to become current and online. It does not require a zero-lag snapshot while publishers are active.
• Use S3 media with the same endpoint, region, bucket, and prefix on every candidate, or explicitly set media_api.storage.local.shared: true. The shared-filesystem option asserts that every candidate mounts the same durable POSIX filesystem at the configured path. It requires synchronous client writes and synchronous server-side persistence, atomic rename, and coherent reads across clients, for example a synchronously mounted NFS export backed by redundant stable storage. An ordinary local volume, asynchronously written network mount, per-candidate PVC, or merely identical path strings do not satisfy this contract. The application cannot verify filesystem mount or storage-controller durability settings.

PostgreSQL must use synchronous commit and synchronously replicated storage with a failover policy that promotes only a replica containing acknowledged transactions. Set connect_timeout in the connection string, for example postgres://zendrite@db-primary/zendrite?connect_timeout=5. When the primary pod is deleted, connections to its address get no reply until the operator repoints the service, so without a timeout each new connection attempt blocks the request that needed it for as long as the failover takes. The timeout does not shorten the failover itself; it makes those requests fail and be retried instead of hanging. JetStream nodes must use independent persistent volumes in separate failure domains and sync_interval: always for the documented acknowledged-write guarantee across operating-system failures. Replication alone does not imply immediate fsync; see the NATS sync_interval reference. Set max_payload: 8MB on every JetStream node to match the embedded server. With the 1 MiB NATS default, a room event whose state delta exceeds the limit, such as a join to a room with tens of thousands of members, fails to publish. The roomserver stores that event but does not apply it to the room’s current state and does not retry the publication. S3-compatible services must preserve acknowledged writes across their supported failover and provide read-after-write consistency. These are deployment guarantees, not settings that Zendrite can establish through a stream API.

The guarantee covers committed database writes and acknowledged JetStream publications within configured retention limits. It does not promise an atomic transaction spanning PostgreSQL, JetStream, media, and remote HTTP delivery. The room input stream expires messages after 24 hours, typing after 60 seconds, and presence after five minutes. Interest retention removes messages after all interested consumers acknowledge them, and operator-supplied size/count limits still apply. Federation, appservice, and push retries can produce duplicate remote deliveries.

Existing streams and consumers

Stream updates use the fully prefixed name and preserve operator settings such as size limits. Explicit replication changes are applied in place, then verified against broker state. In standalone mode, an omitted or zero replicas value preserves existing replica counts and defaults newly created streams to one replica. HA rejects storage, retention, age, or subject changes rather than risking queued data. A failed stream update never deletes and recreates the stream, in either HA or standalone mode. Standalone retains its explicit memory-storage option and degraded memory fallback for new streams; HA never uses that fallback. The package stream templates are never mutated by memory-mode startup.

Before enabling HA on an existing installation, migrate its memory-based typing/presence streams and legacy push/ack-all consumers during an offline maintenance window. Drain or back up their data and durable acknowledgement positions before an operator changes them. HA rejects incompatible existing durable consumer storage, replication, or acknowledgement settings and blocks application stream/consumer deletion paths. The roomserver’s headers-only ephemeral push consumer is allowed because it only wakes durable room workers and owns no recoverable state. Consumer validation uses paginated listing with error reporting and cancels further enumeration requests and drains the consumer channel before returning on failure. Do not delete occupied streams or consumers to make validation pass.

Candidate compatibility and upgrades

Candidates compute a deployment identity from the public signing identities, effective PostgreSQL target and session parameters, broker endpoints/domain/prefix/replication, media namespace, search settings, and the set of enabled MSCs. Every candidate must present the same identity; enabling or disabling an MSC is therefore an offline upgrade. The executable is not part of it: which builds may take over from each other is declared by their upgrade contract. Disabling msc2836 leaves its MSC2836RoomServerConsumerPull durable, named with the topic prefix, on the roomserver output stream, which HA will not delete; remove it during the same maintenance window, or it retains every roomserver output message. Signing private keys and database passwords, NATS credentials, and S3 credentials are excluded from the stored record. Error messages never print the fingerprint inputs. Password/token rotation within the same database user, NATS account, and S3 namespace does not require resetting compatibility. NATS credentials on every candidate must resolve to the same account; changing an account is an offline namespace migration. Use the same homeserver configuration on every candidate. Different builds, including local builds, may share a deployment only where their upgrade contracts admit each other.

Ownership authority itself lives in zendrite_ha_ownership, separately from the compatibility records described here. The Lease annotation zendrite.codefloe.com/compatibility pins the election scope before ownership is granted. An existing Lease with a missing or mismatched annotation fails closed, including a manually precreated Lease. The initial candidate creates the annotated Lease; do not replace an existing Lease to bypass a mismatch. Owner history is retained independently of the Lease in zendrite_ha_ownership_history. After fenced ownership, a dedicated PostgreSQL connection obtains an advisory migration lock, checks the deployment identity and upgrade contract against zendrite_ha_compatibility, and records its own level and the raised floor there before component constructors run. Standbys read this record, but never write it. That connection holds the lock through initialization, recovery, and listener binding. It carries the activation marker as well, so a successor’s sweep terminates it like any other application backend. Application initialization failure leaves the compatibility record pinned, stops readiness, and requires a fresh process. Migration transactions retain their existing rollback behavior; a partially completed sequence is retried only by a build the record admits, under safe ownership. Incompatible schema errors during initialization prevent readiness.

Migrations during a rolling upgrade

Migrations run only in an activation, after the previous owner has been fenced and under the migration lock, so the old and new builds never write the schema at the same time and a standby never changes storage the running owner uses. A migration therefore delays the successor’s readiness, and a planned handover holds requests for that long: keep migrations that run during a rolling upgrade short, and move long data backfills to background work that tolerates restarts. A migration that an older level cannot run against, such as dropping or renaming a column it reads, must raise rollback_to; one that the new level needs before it can read older data must raise upgrade_from, which makes the upgrade offline. Split such changes over several levels where possible: first add the new structure and write both, then stop using the old one, and drop it only in a later level whose rollback_to excludes builds that still read it. Every level’s schema, migrations and streams are recorded in setup/ha/testdata/surface.txt, and CI refuses a change to them that does not declare a new level.

Offline upgrades

Offline upgrades are needed when the upgrade contract does not admit the new build, or the deployment identity changes: stop and fence every candidate, back up the database/broker/media, verify the intended schema migration with the new binary, and deploy the same artifact everywhere. Only after verifying that no former activation can return may an operator deliberately reset both compatibility records, by deleting the Lease and the zendrite_ha_compatibility row, and bootstrap the new configuration. A failed or ambiguous migration must leave candidates stopped until the schema is repaired or the coordinated backup is restored. Resetting the records is not a schema migration and does not make an old binary safe to run against a newer schema. Automatic schema downgrade is unsupported.

Promotion recovery and search

A fresh process initializes caches and synchronously restores sync stream positions and notifier membership from PostgreSQL. Federation destination enumeration must succeed in HA mode; recovered destination workers retain the existing staggered retry schedule, so readiness does not wait for remote homeservers to become available. Durable consumers resume their persisted acknowledgement positions. Before opening public listeners, startup waits for the sync consumers to apply their captured stream backlog, including already delivered but unacknowledged messages. The presence consumer participates only when inbound or outbound presence is enabled. Cancellation, broker errors, or a 120-second sync recovery timeout prevent readiness. Typing state, login/SSO intermediates, and sliding-sync sessions remain transient process state; clients may need to retry or establish a fresh session after promotion. Client transaction IDs are recorded in the database, so retrying a message after promotion does not store it twice.

When search is enabled, set sync_api.search.rebuild_on_startup: true. Every activation builds a fresh Bleve index from PostgreSQL in the background, so readiness and the public listeners do not wait for it. Until the rebuild finishes, /search answers HTTP 503 with Retry-After instead of results that would silently miss history. Sync consumers index new events during the rebuild. Deletions they make for redactions and edits are replayed once the rebuild has written its last document, so a rebuilt batch read before a redaction cannot restore the redacted text. A database read or index write error is logged and the rebuild restarts from the beginning; ownership cancellation stops it. The rebuild advances by the maximum stream position in each batch, independent of map iteration order. It starts from an empty index so deleted or redacted documents cannot survive from a candidate’s older local index.

sync_api.search.index_path is a candidate-local workspace for disposable recovery indexes, separate from shared media. It does not require a shared index or persistent volume and must not point to a concurrently opened Bleve index. Normal shutdown removes the activation’s temporary index after closing it; a killed process can leave unused recovery-* directories for offline cleanup. Provision space for a complete rebuilt index and possible crash leftovers. Rebuild time grows with the searchable history and delays only search; HACHAOS_SEARCH_HISTORY makes the HA qualification harness seed a history of a chosen size to measure it.

Media fencing boundary and qualification

Media originals use content-hash keys and PostgreSQL metadata controls their publication. A repeated original upload is idempotent, but thumbnail replacements and media/thumbnail deletion are mutable operations. The fencing provider must drain or otherwise exclude previously dispatched mutable media operations before authorizing a successor, in addition to its PostgreSQL and broker barriers. Shared media does not itself supply that barrier or solve local search-index persistence. Deployments without those proofs remain blocked by the fencing release gate.

Automated tests cover configuration rejection, credential-independent compatibility, Lease mismatches, PostgreSQL migration serialization, search recovery errors/cancellation/cursors, namespace-safe stream updates, replication upgrades, and preservation of acknowledged messages and pending consumer state across a three-node broker leader failure. They also cover the ownership failure envelope. A leader paused beyond expiry and resumed after takeover is refused at the database, broker, media, admission, and dispatch boundaries, including on a connection pool it had already opened and used. A leader partitioned from the Kubernetes API discovers that it was replaced through the database alone, while its Lease context is still live and it still holds data-plane access. A sweep that completes a pass without converging blocks promotion and reports how many backends still hold the retired generation. An application that will not stop within the shutdown budget forces process termination. A transaction already dispatched completes while newly admitted dispatch is refused. PostgreSQL failover and complete Matrix client, sync, media, search and federation recovery, including against the ADR reference workload, are covered by the HA qualification harness. Infrastructure power-off, correlated storage failures, and S3 or shared-filesystem failover remain outside it and should be exercised on the target infrastructure.