Creating a cluster > Setting up replication > Understanding cluster status and node states

When a whole multi-node replication cluster is down, one node must be started first so the other nodes know which copy of the cluster to join.

There is one normal-startup exception: if the cluster had already become a clean one-node cluster, for example after the other nodes left with EXIT CLUSTER, start that remaining node normally. After a clean shutdown, Manticore restores that self-only cluster as primary / synced without --new-cluster.

That first-start decision is based on grastate.dat, the small replication state file stored in the cluster data directory. The most important fields are:

seqno - the last transaction number known to that node
safe_to_bootstrap - whether that node is marked as safe to start first after a clean shutdown

Example of what grastate.dat can look like after a clean shutdown:

# saved replication state
version: 2.1
uuid:    <cluster-uuid>
seqno:   12345
safe_to_bootstrap: 1

In this example:

seqno: 12345 means this node knows about transactions up to sequence number 12345
safe_to_bootstrap: 1 means this node is marked as safe to start first

If the whole cluster was shut down cleanly, start the node that was stopped last. In practice, this is usually the node with:

the most advanced seqno
safe_to_bootstrap: 1

Start that node first. This tells Manticore to start a new copy of the cluster from that node. After that, start the remaining nodes normally so they can rejoin.

Use this after a clean full cluster shutdown.

‹›

Bash
Systemd

📋

searchd --new-cluster

If another node is started first without the required clean-shutdown state, startup is refused to protect the cluster from being restored from an older copy.

If all nodes crashed or were shut down uncleanly, grastate.dat may no longer be trustworthy for normal bootstrap selection. In that case, find the node with the most recent data, usually the one with the largest seqno, and start it with --new-cluster-force. This overrides the normal protection and forces the cluster to start from the chosen node.

Use this after a crash or unclean full cluster shutdown.

‹›

Bash
Systemd

📋

searchd --new-cluster-force

Cluster recovery

Last modified: June 30, 2026

If a replication node or an entire cluster becomes unavailable, the correct recovery procedure depends on how many nodes are still reachable and whether the shutdown was clean or abrupt.

A replication cluster should be treated as one logical system rather than a set of unrelated servers. That gives you multi-master writes and consistent data, but it also means you must recover quorum carefully. In particular, do not run the manual recovery command that restores writes on the surviving side until you are sure the missing nodes are really gone. That command is shown later in this page as SET CLUSTER <name> GLOBAL 'pc.bootstrap' = 1. If you run it too early, you can create split-brain and end up with two independent clusters.

For the examples below, assume a cluster with nodes A, B, and C unless noted otherwise.

First identify which situation you are in:

Is at least one node still online?
Was the node stopped cleanly, or did it crash? A clean stop means searchd was shut down normally and had time to save its replication state before exiting. A crash, power loss, or kill -9 is not a clean stop.
Does the surviving part of the cluster still have quorum? Quorum means enough nodes can still see each other to safely remain the writable cluster.
If all nodes are down, which node should be started first to bring the cluster back?

Useful checks:

SHOW STATUS LIKE 'cluster_<name>_status'
SHOW STATUS LIKE 'cluster_<name>_size'
SHOW STATUS LIKE 'cluster_<name>_node_state'
if all nodes are down, inspect grastate.dat, the small replication state file stored in the cluster data directory. Look especially at seqno and safe_to_bootstrap: on a clean shutdown, the best node to start first is usually the one with the most advanced seqno and safe_to_bootstrap: 1. For the full bootstrap procedure, see Restarting a cluster.

Example of what grastate.dat can look like after a clean shutdown:

# saved replication state
version: 2.1
uuid:    <cluster-uuid>
seqno:   12345
safe_to_bootstrap: 1

In this example:

seqno: 12345 means this node knows about transactions up to sequence number 12345
safe_to_bootstrap: 1 means this node is marked as safe to start first

In a clean all-nodes-down recovery, this is usually the kind of node you start first with --new-cluster to bring the cluster back.

After recovery, wait until the restarted node reports cluster_<name>_status=primary and cluster_<name>_node_state=synced before treating it as fully writable again. You can check this with SHOW STATUS LIKE 'cluster_<name>_status' and SHOW STATUS LIKE 'cluster_<name>_node_state'. In local tests, restarted nodes sometimes spent a short time with cluster_<name>_node_state=joining and cluster_<name>_status=disconnected before reaching synced/primary.

If node A is stopped normally, nodes B and C keep serving writes. You can confirm that the cluster is still healthy on those nodes with SHOW STATUS LIKE 'cluster_<name>_status' and SHOW STATUS LIKE 'cluster_<name>_size'.

When node A starts again, it rejoins the cluster automatically. Until synchronization finishes, do not send writes to that node. Check SHOW STATUS LIKE 'cluster_<name>_status' and SHOW STATUS LIKE 'cluster_<name>_node_state' and wait for primary / synced.

If donor nodes B or C still have all the transactions that node A missed in their replication cache, node A can catch up using an incremental state transfer (IST). IST stands for incremental state transfer. It means the node receives only the transactions it missed, so recovery is usually faster and lighter. Otherwise it will require a snapshot state transfer (SST). SST stands for snapshot state transfer. It means copying table files from another node instead of just replaying the missing transactions. SST is heavier: it is usually slower, moves more data, and can make recovery more disruptive on large clusters.

If nodes A and B are stopped cleanly and node C remains online, node C can continue accepting writes. Check SHOW STATUS LIKE 'cluster_<name>_status' and SHOW STATUS LIKE 'cluster_<name>_size' on node C if you want to confirm it is now the only active node.

When nodes A and B start again, they rejoin automatically and synchronize from node C. While they are rejoining, check SHOW STATUS LIKE 'cluster_<name>_status', SHOW STATUS LIKE 'cluster_<name>_node_state', and SHOW STATUS LIKE 'cluster_<name>_size'. Wait until all nodes show primary / synced and the expected cluster size before treating recovery as complete.

If nodes A and B were intentionally removed from the cluster, for example with EXIT CLUSTER, and node C is now the only persisted node in the cluster, node C can also recover after its own clean restart with a normal daemon start. In that self-only case, --new-cluster is not required.

If all nodes in a multi-node cluster were stopped normally, the cluster is fully offline and must be started again in a special way so one node can become the first running node of the cluster.

On clean shutdown, each node writes its last transaction number to grastate.dat. The node that was stopped last is the safest node to start first:

it has the most advanced seqno
it has safe_to_bootstrap: 1

Start that node with --new-cluster. This tells Manticore to start a new copy of the cluster from that node. If you run Manticore via systemd on Linux, use manticore_new_cluster. It starts Manticore in --new-cluster mode for you.

After that, start the remaining nodes normally and let them rejoin. Verify recovery with SHOW STATUS LIKE 'cluster_<name>_status', SHOW STATUS LIKE 'cluster_<name>_node_state', and SHOW STATUS LIKE 'cluster_<name>_size'.

If you bootstrap a less advanced node first, a more advanced node may later join it and receive a full SST from an older state, which can discard transactions that existed only on the more advanced node. That is why the node with safe_to_bootstrap: 1 should be your first choice.

Use this procedure as a last resort when no node can restore the cluster through the normal recovery methods above, but the cluster's tables are still present in the local data directories. This can happen when saved cluster metadata is incomplete or incompatible with the running version, or when another startup failure leaves the old cluster membership unusable.

This procedure creates a new cluster. It discards the old cluster membership and replication history, then copies the tables from one selected node to the other nodes. Do not use it while any node still has a healthy primary / synced copy of the old cluster. In that case, keep the healthy cluster and recover the other nodes through JOIN CLUSTER.

Stop Manticore on every node that belonged to the failed cluster. Do not write to the local tables during this procedure.
Back up the full data_dir on every node. Keep a copy of the old cluster descriptor if it contains custom path, nodes, or provider options that you need when recreating the cluster.
Choose the node whose local tables contain the data you want to keep. This node will become the source for the new cluster. Adding its tables to the cluster later replaces tables with the same names on the other nodes.
On every node, edit <data_dir>/manticore.json. From the top-level clusters object, remove only the entry for the failed cluster. For example, remove clusters.posts for a cluster named posts.

Leave the indexes object and every table directory in place. Do not delete or recreate the tables.
Start Manticore normally on every node. The former cluster tables should now appear as local tables. Check the tables on the selected source node before continuing. If they are missing or incomplete, stop and restore the backup.
On the selected source node, create the cluster. Include any saved path, nodes, or provider options if you still need them. If authentication is enabled, add '<replication-user>' AS user to this statement and the JOIN CLUSTER statements below. The account must have matching stored authentication data and replication permission on every node. See setting up replication.
```
CREATE CLUSTER posts
```
On every other node, join the new cluster through the first node:
```
JOIN CLUSTER posts AT 'node-a.example:9312'
```
On the selected source node, attach the existing local tables to the new cluster:
```
ALTER CLUSTER posts ADD table_a, table_b
```
Wait until every node reports cluster_posts_status=primary and cluster_posts_node_state=synced before resuming writes.

If node A disappears because of a crash or a network problem, nodes B and C will first try to reconnect to it. If that fails, they remove it from the cluster, recalculate quorum, and continue working as the primary cluster.

In local tests this peer removal was not immediate: the surviving nodes stayed at the old cluster size for a few seconds before dropping the failed peer and switching to the smaller primary cluster.

When node A is started again, it rejoins automatically and catches up the same way as after a clean one-node shutdown. Again, use SHOW STATUS LIKE 'cluster_<name>_status', SHOW STATUS LIKE 'cluster_<name>_node_state', and SHOW STATUS LIKE 'cluster_<name>_size' to confirm recovery is finished.

If nodes A and B are lost and only node C is still running, node C no longer has quorum in a three-node cluster. It switches to non-primary and rejects writes.

The write error is explicit:

ERROR 1064 (42000): cluster '<name>' is not ready, not primary state (synced)

If nodes A and B are only temporarily disconnected but can still see each other, they may continue accepting writes while node C remains isolated. Use SHOW STATUS LIKE 'cluster_<name>_status' on each side if you need to see which side is still writable.

If nodes A and B really crashed and node C is the only surviving copy you want to keep working with, run this command on node C to make it writable again:

If you have confirmed that the other nodes are truly offline, run:

‹›

SQL
JSON

📋

SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1

Important:

run this only after you are sure the other nodes are unreachable
run it only on the side that must survive
after bootstrapping, the node can accept writes again and the other nodes can later rejoin from it

If every node crashed, grastate.dat is typically no longer trustworthy for normal bootstrap selection. In local tests, all nodes showed:

seqno: -1
safe_to_bootstrap: 0

In this situation, choose the node with the most recent data and start it with --new-cluster-force. This forces Manticore to start a new copy of the cluster from that node even though the usual clean-shutdown metadata is not trustworthy. If you run Manticore via systemd on Linux, use manticore_new_cluster --force. It starts Manticore in --new-cluster-force mode for you.

Then start the remaining nodes normally and let them rejoin. Verify recovery with SHOW STATUS LIKE 'cluster_<name>_status', SHOW STATUS LIKE 'cluster_<name>_node_state', and SHOW STATUS LIKE 'cluster_<name>_size'.

Split-brain risk is highest in even-sized clusters. For example, imagine four nodes split into two isolated pairs across two data centers. Each side has exactly half of the original members, so neither side has quorum and both sides stop accepting writes.

If you must restore writes before connectivity is fixed, choose only one side of the split and run the same recovery command there so that side becomes writable again. Before doing that, check SHOW STATUS LIKE 'cluster_<name>_status' on both sides so you know which side is currently non-primary.

Choose the side that should remain the writable cluster, then run:

‹›

SQL
JSON

📋

SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1

Never issue that statement on both sides. If you do, you will create two separate primary clusters, and they will not merge back automatically when the network recovers.

In local testing, abruptly losing half of a four-node cluster reproduced the same non-primary behavior and the same recovery command brought one surviving half back to primary.

Restarting a cluster Understanding Replication Statuses

Last modified: July 14, 2026

The cluster status shows two key values that help you understand your cluster's health:

cluster_%_status - whether the cluster can accept writes
cluster_%_node_state - what this specific node is doing

‹›

SQL

📋

SHOW STATUS LIKE 'cluster_%_status';
SHOW STATUS LIKE 'cluster_%_node_state';

‹›

Response

+----------------------------+-------+
| Counter                    | Value |
+----------------------------+-------+
| cluster_posts_status       | primary |
| cluster_posts_node_state   | synced |
+----------------------------+-------+

primary - Normal operation, can read and write
non-primary - Lost quorum, writes blocked
disconnected - Node isolated, trying to reconnect

synced - Normal operation
joining - Node joining cluster (temporary)
donor - Helping another node join (temporary)
closed - Node stopped
destroyed - Node crashed, needs restart

What happened	cluster_%_status	cluster_%_node_state	What to do
Normal operation	`primary`	`synced`	Nothing
Most nodes down	`non-primary`	`synced`	Bootstrap or wait
Node starting	`disconnected`	`joining` → `synced`	Wait
Node crashed	varies	`destroyed`	Restart node
Network partition (majority)	`primary`	`synced`	Continue normally
Network partition (minority)	`non-primary`	`synced`	Wait or bootstrap

When most nodes are down and status shows non-primary:

‹›

SQL
JSON

📋

-- Check which node has highest sequence number
SHOW STATUS LIKE 'cluster_%_last_committed';
-- Bootstrap the cluster (run on best node only)
SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1;

‹›

Response

+----------------------------+-------+
| Counter                    | Value |
+----------------------------+-------+
| cluster_posts_last_committed | 1547 |
+----------------------------+-------+

Cluster recovery 4️⃣ Connecting to the server

Last modified: September 01, 2026

With default configuration, Manticore is waiting for your connections on:

port 9306 for MySQL clients
port 9308 for HTTP/HTTPS connections
port 9312 for HTTP/HTTPS, and connections from other Manticore nodes and clients based on Manticore binary API

If authentication and authorization is enabled, MySQL clients must connect with a Manticore user name and password, and HTTP/HTTPS clients must use Basic authentication or a bearer token. User names are exact-case. See Authentication and authorization for bootstrap and bearer token creation or rotation.

‹›

SQL
HTTP
PHP
Python
Python-asyncio
Javascript
Java
C#
Rust
docker

📋

mysql -h0 -P9306

HTTP is a stateless protocol, so it doesn't require any special connection phase:

curl -s "http://localhost:9308/search"

require_once __DIR__ . '/vendor/autoload.php';
$config = ['host'=>'127.0.0.1','port'=>9308];
$client = new \Manticoresearch\Client($config);

import manticoresearch
config = manticoresearch.Configuration(
    host = "http://127.0.0.1:9308"
)
client =  manticoresearch.ApiClient(config)
indexApi = manticoresearch.IndexApi(client)
searchApi = manticoresearch.searchApi(client)
utilsApi = manticoresearch.UtilsApi(client)

import manticoresearch
config = manticoresearch.Configuration(
    host = "http://127.0.0.1:9308"
)
async with manticoresearch.ApiClient(config) as client:
    indexApi = manticoresearch.IndexApi(client)
    searchApi = manticoresearch.searchApi(client)
    utilsApi = manticoresearch.UtilsApi(client)

var Manticoresearch = require('manticoresearch');
var client= new Manticoresearch.ApiClient()
client.basePath="http://127.0.0.1:9308";
indexApi = new Manticoresearch.IndexApi(client);
searchApi = new Manticoresearch.SearchApi(client);
utilsApi = new Manticoresearch.UtilsApi(client);

import com.manticoresearch.client.ApiClient;
import com.manticoresearch.client.ApiException;
import com.manticoresearch.client.Configuration;
import com.manticoresearch.client.model.*;
import com.manticoresearch.client.api.IndexApi;
import com.manticoresearch.client.api.UtilsApi;
import com.manticoresearch.client.api.SearchApi;
ApiClient client = Configuration.getDefaultApiClient();
client.setBasePath("http://127.0.0.1:9308");
IndexApi indexApi = new IndexApi(client);
SearchApi searchApi = new UtilsApi(client);
UtilsApi utilsApi = new UtilsApi(client);

using ManticoreSearch.Client;
using ManticoreSearch.Api;
using ManticoreSearch.Model;
string basePath = "http://127.0.0.1:9308";
IndexApi indexApi = new IndexApi(basePath);
SearchApi searchApi = new UtilsApi(basePath);
UtilsApi utilsApi = new UtilsApi(basePath);

use std::sync::Arc;
use manticoresearch::{
    apis::{
        {configuration::Configuration,IndexApi,IndexApiClient,SearchApi,SearchApiClient,UtilsApi,UtilsApiClient}
    },
};
async fn maticore_connect {
    let configuration = Configuration {
        base_path: "http://127.0.0.1:9308".to_owned(),
        ..Default::default(),
    };
    let api_config = Arc::new(configuration);
    let utils_api = UtilsApiClient::new(api_config.clone());
    let index_api = IndexApiClient::new(api_config.clone());
    let search_api = SearchApiClient::new(api_config.clone());

Run Manticore container and use built-in MySQL client to connect to the node.

docker run --name manticore -d manticoresearch/manticore && docker exec -it manticore mysql

Understanding Replication Statuses MySQL protocol

Last modified: June 16, 2026

Restarting a cluster

Cluster recovery

Before you recover anything

One node was shut down cleanly

Two nodes were shut down cleanly and one node stayed online

All nodes were shut down cleanly

Recreate a cluster from existing local tables

One node crashed or became unreachable

Two nodes are gone and only one node is still running

All nodes crashed

An even-sized cluster lost quorum, for example after a split

Understanding cluster status and node states

Quick reference

Status values

cluster_%_status

cluster_%_node_state

Common scenarios

Recovery from quorum loss

Connecting to the server