High availability
[!NOTE] Clustering needs a valid license on every node.
Several NullWard nodes can run as one cluster. Each node keeps its own SurrealDB and Valkey. There's no shared database. Nodes exchange configuration, blocks and certificates with each other over one port, protected by mutual TLS 1.3 with the cluster's own certificate authority.
- A change made in the dashboard of any node reaches all the others.
- Requests never wait for another node. If nodes lose contact, each keeps serving with what it has and catches up when the connection returns.
- Put the nodes behind a load balancer or DNS that spreads traffic across them.
Since version 0.6.109 new clusters run on the SurrealHA cluster engine: configuration is replicated inside SurrealDB, row by row, and everything else (blocks, rate limits, edit locks, ACME challenges, cluster-wide views) runs over the same port. Clusters created on an earlier version keep the WireGuard mesh engine until you move them; see Move a mesh cluster to SurrealHA.
Before you start
- Deploy each node as described in Deploy with Docker Compose, with its own SurrealDB and Valkey. SurrealDB 2 and 3 both work.
- Give every node the same
NULLWARD_ENCRYPTION_KEYandNULLWARD_JWT_SECRET. Each node can have its own admin password. The bootstrap admin account is local to each node. - Activate a license on every node.
- Allow the cluster port, 7059, over both TCP and UDP between the nodes, and publish it in Docker Compose:
ports:
- "7059:7059/tcp"
- "7059:7059/udp"
Only the nodes need to reach this port. TCP connections on it must present a certificate of your cluster; the UDP gossip is encrypted with the cluster's key.
Set up the cluster
On each node, go to Administration > Cluster.
Administration > Cluster
- Open Cluster Configuration and turn on clustering. Leave Cluster Engine on Automatic (SurrealHA for a new cluster).
- Set Advertise Address: the host name or address other nodes reach this node on, for example
waf1.example.com. The cluster port is added to it. - Set Public Admin URL, and choose a Network Profile: LAN (nodes on one low-latency network: a node that stops answering counts as down within seconds) or WAN (nodes at different sites: about half a minute). Change Cluster Port only if 7059 is taken; every node may use its own.
- Click Save Settings, then restart the node. Cluster settings take effect after a restart.
Then join the nodes together:
- On the node that holds your configuration, click Generate Join Token and choose how long the token is valid. The first token creates the cluster; this node's configuration becomes the cluster's.
- On each other node, click Join Cluster and paste a token. A token works once.
[!WARNING] Joining replaces the joining node's shared configuration with the cluster's. Each node keeps its own node settings (cluster, admin TLS and similar), its bootstrap admin account and its own events and audit log.
The joining node copies the cluster's configuration, then keeps up with every change. The Cluster page shows each member, its version and health, and whether this node is still copying or is the leader (the node that runs cluster-wide jobs). Event pages such as WAF events and the audit log gain a node filter.
To take a node out, click Remove next to it on any other node: its cluster certificate is revoked, so no node talks to it any more. Removing the node you're on makes it leave the cluster and run alone with the configuration it has; to come back, it joins again with a new token.
A node that was switched off or cut off for longer than 7 days drops its shared state (blocks, rate limits and similar) when it comes back and takes the cluster's, so nothing the cluster removed meanwhile returns. The Cluster page shows it as stale for the few seconds this takes.
Certificates in a cluster
Any node may renew a certificate, so use the DNS-01 challenge for ACME in a cluster. See TLS certificates. By default each certificate's renewal node is chosen automatically, and another node takes over if it goes down. You can pin a renewal node per service.
FIPS mode
Turn on FIPS mode to restrict the cluster port's TLS 1.3 to FIPS 140-3 approved cryptography. The nodes switch one at a time, in a fixed order, each waiting until the one before it is back and serving, so the cluster keeps serving. Turn FIPS mode on before joining a node to a FIPS cluster.
Move a mesh cluster to SurrealHA
A cluster formed before 0.6.109 runs on the WireGuard mesh (UDP 7057). The Cluster page says so and lists these steps. A cluster moves in one go: nodes on different engines don't exchange anything, so plan a short configuration freeze.
- Upgrade every node to 0.6.109 or later and check the cluster is healthy.
- Open the cluster port (7059, TCP and UDP) between the nodes.
- Pause configuration changes until the last node has joined. A change made meanwhile on a node that hasn't moved yet is replaced by the cluster's copy when it joins.
- On the node whose configuration should win: set Cluster Engine to SurrealHA, save, restart it, then click Generate Join Token.
- On every other node, one at a time: set Cluster Engine to SurrealHA, save, restart, then Join Cluster with a new token from the first node.
- Check every node shows as alive on the Cluster page, then close UDP 7057.
Blocks, the tunnel peer list and the tunnel signing key live in the shared state that carries over to SurrealHA, so tunnel connectors need no new configuration. Check that your connectors show as connected after the move.

