When a whole replication cluster is down, one node must be started first so the other nodes know which copy of the cluster to join.
That first-start decision is based on grastate.dat, the small replication state file stored in the cluster data directory. The most important fields are:
seqno- the last transaction number known to that nodesafe_to_bootstrap- whether that node is marked as safe to start first after a clean shutdown
Example of what grastate.dat can look like after a clean shutdown:
# saved replication state
version: 2.1
uuid: <cluster-uuid>
seqno: 12345
safe_to_bootstrap: 1
In this example:
seqno: 12345means this node knows about transactions up to sequence number 12345safe_to_bootstrap: 1means this node is marked as safe to start first
If the whole cluster was shut down cleanly, start the node that was stopped last. In practice, this is usually the node with:
- the most advanced
seqno safe_to_bootstrap: 1
Start that node first. This tells Manticore to start a new copy of the cluster from that node. After that, start the remaining nodes normally so they can rejoin.
Use this after a clean full cluster shutdown.
- Bash
- Systemd
searchd --new-clustermanticore_new_clusterIf another node is started first without the required clean-shutdown state, startup is refused to protect the cluster from being restored from an older copy.
If all nodes crashed or were shut down uncleanly, grastate.dat may no longer be trustworthy for normal bootstrap selection. In that case, find the node with the most recent data, usually the one with the largest seqno, and start it with --new-cluster-force. This overrides the normal protection and forces the cluster to start from the chosen node.
Use this after a crash or unclean full cluster shutdown.
- Bash
- Systemd
searchd --new-cluster-forcemanticore_new_cluster --forceIf a replication node or an entire cluster becomes unavailable, the correct recovery procedure depends on how many nodes are still reachable and whether the shutdown was clean or abrupt.
A replication cluster should be treated as one logical system rather than a set of unrelated servers. That gives you multi-master writes and consistent data, but it also means you must recover quorum carefully. In particular, do not run the manual recovery command that restores writes on the surviving side until you are sure the missing nodes are really gone. That command is shown later in this page as SET CLUSTER <name> GLOBAL 'pc.bootstrap' = 1. If you run it too early, you can create split-brain and end up with two independent clusters.
For the examples below, assume a cluster with nodes A, B, and C unless noted otherwise.
First identify which situation you are in:
- Is at least one node still online?
- Was the node stopped cleanly, or did it crash? A clean stop means
searchdwas shut down normally and had time to save its replication state before exiting. A crash, power loss, orkill -9is not a clean stop. - Does the surviving part of the cluster still have quorum? Quorum means enough nodes can still see each other to safely remain the writable cluster.
- If all nodes are down, which node should be started first to bring the cluster back?
Useful checks:
SHOW STATUS LIKE 'cluster_<name>_status'SHOW STATUS LIKE 'cluster_<name>_size'SHOW STATUS LIKE 'cluster_<name>_node_state'- if all nodes are down, inspect
grastate.dat, the small replication state file stored in the cluster data directory. Look especially atseqnoandsafe_to_bootstrap: on a clean shutdown, the best node to start first is usually the one with the most advancedseqnoandsafe_to_bootstrap: 1. For the full bootstrap procedure, see Restarting a cluster.
Example of what grastate.dat can look like after a clean shutdown:
# saved replication state
version: 2.1
uuid: <cluster-uuid>
seqno: 12345
safe_to_bootstrap: 1
In this example:
seqno: 12345means this node knows about transactions up to sequence number 12345safe_to_bootstrap: 1means this node is marked as safe to start first
In a clean all-nodes-down recovery, this is usually the kind of node you start first with --new-cluster to bring the cluster back.
After recovery, wait until the restarted node reports cluster_<name>_status=primary and cluster_<name>_node_state=synced before treating it as fully writable again. You can check this with SHOW STATUS LIKE 'cluster_<name>_status' and SHOW STATUS LIKE 'cluster_<name>_node_state'. In local tests, restarted nodes sometimes spent a short time with cluster_<name>_node_state=joining and cluster_<name>_status=disconnected before reaching synced/primary.
If node A is stopped normally, nodes B and C keep serving writes. You can confirm that the cluster is still healthy on those nodes with SHOW STATUS LIKE 'cluster_<name>_status' and SHOW STATUS LIKE 'cluster_<name>_size'.
When node A starts again, it rejoins the cluster automatically. Until synchronization finishes, do not send writes to that node. Check SHOW STATUS LIKE 'cluster_<name>_status' and SHOW STATUS LIKE 'cluster_<name>_node_state' and wait for primary / synced.
If donor nodes B or C still have all the transactions that node A missed in their replication cache, node A can catch up using an incremental state transfer (IST). IST stands for incremental state transfer. It means the node receives only the transactions it missed, so recovery is usually faster and lighter. Otherwise it will require a snapshot state transfer (SST). SST stands for snapshot state transfer. It means copying table files from another node instead of just replaying the missing transactions. SST is heavier: it is usually slower, moves more data, and can make recovery more disruptive on large clusters.
If nodes A and B are stopped cleanly and node C remains online, node C can continue accepting writes. Check SHOW STATUS LIKE 'cluster_<name>_status' and SHOW STATUS LIKE 'cluster_<name>_size' on node C if you want to confirm it is now the only active node.
When nodes A and B start again, they rejoin automatically and synchronize from node C. While they are rejoining, check SHOW STATUS LIKE 'cluster_<name>_status', SHOW STATUS LIKE 'cluster_<name>_node_state', and SHOW STATUS LIKE 'cluster_<name>_size'. Wait until all nodes show primary / synced and the expected cluster size before treating recovery as complete.
If all nodes were stopped normally, the cluster is fully offline and must be started again in a special way so it can become the first running node of the cluster.
On clean shutdown, each node writes its last transaction number to grastate.dat. The node that was stopped last is the safest node to start first:
- it has the most advanced
seqno - it has
safe_to_bootstrap: 1
Start that node with --new-cluster. This tells Manticore to start a new copy of the cluster from that node. If you run Manticore via systemd on Linux, use manticore_new_cluster. It starts Manticore in --new-cluster mode for you.
After that, start the remaining nodes normally and let them rejoin. Verify recovery with SHOW STATUS LIKE 'cluster_<name>_status', SHOW STATUS LIKE 'cluster_<name>_node_state', and SHOW STATUS LIKE 'cluster_<name>_size'.
If you bootstrap a less advanced node first, a more advanced node may later join it and receive a full SST from an older state, which can discard transactions that existed only on the more advanced node. That is why the node with safe_to_bootstrap: 1 should be your first choice.
If node A disappears because of a crash or a network problem, nodes B and C will first try to reconnect to it. If that fails, they remove it from the cluster, recalculate quorum, and continue working as the primary cluster.
In local tests this peer removal was not immediate: the surviving nodes stayed at the old cluster size for a few seconds before dropping the failed peer and switching to the smaller primary cluster.
When node A is started again, it rejoins automatically and catches up the same way as after a clean one-node shutdown. Again, use SHOW STATUS LIKE 'cluster_<name>_status', SHOW STATUS LIKE 'cluster_<name>_node_state', and SHOW STATUS LIKE 'cluster_<name>_size' to confirm recovery is finished.
If nodes A and B are lost and only node C is still running, node C no longer has quorum in a three-node cluster. It switches to non-primary and rejects writes.
The write error is explicit:
ERROR 1064 (42000): cluster '<name>' is not ready, not primary state (synced)
If nodes A and B are only temporarily disconnected but can still see each other, they may continue accepting writes while node C remains isolated. Use SHOW STATUS LIKE 'cluster_<name>_status' on each side if you need to see which side is still writable.
If nodes A and B really crashed and node C is the only surviving copy you want to keep working with, run this command on node C to make it writable again:
If you have confirmed that the other nodes are truly offline, run:
- SQL
- JSON
SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1POST /cli -d "
SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1
"Important:
- run this only after you are sure the other nodes are unreachable
- run it only on the side that must survive
- after bootstrapping, the node can accept writes again and the other nodes can later rejoin from it
If every node crashed, grastate.dat is typically no longer trustworthy for normal bootstrap selection. In local tests, all nodes showed:
seqno: -1safe_to_bootstrap: 0
In this situation, choose the node with the most recent data and start it with --new-cluster-force. This forces Manticore to start a new copy of the cluster from that node even though the usual clean-shutdown metadata is not trustworthy. If you run Manticore via systemd on Linux, use manticore_new_cluster --force. It starts Manticore in --new-cluster-force mode for you.
Then start the remaining nodes normally and let them rejoin. Verify recovery with SHOW STATUS LIKE 'cluster_<name>_status', SHOW STATUS LIKE 'cluster_<name>_node_state', and SHOW STATUS LIKE 'cluster_<name>_size'.
Split-brain risk is highest in even-sized clusters. For example, imagine four nodes split into two isolated pairs across two data centers. Each side has exactly half of the original members, so neither side has quorum and both sides stop accepting writes.
If you must restore writes before connectivity is fixed, choose only one side of the split and run the same recovery command there so that side becomes writable again. Before doing that, check SHOW STATUS LIKE 'cluster_<name>_status' on both sides so you know which side is currently non-primary.
Choose the side that should remain the writable cluster, then run:
- SQL
- JSON
SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1POST /cli -d "
SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1
"Never issue that statement on both sides. If you do, you will create two separate primary clusters, and they will not merge back automatically when the network recovers.
In local testing, abruptly losing half of a four-node cluster reproduced the same non-primary behavior and the same recovery command brought one surviving half back to primary.
With default configuration, Manticore is waiting for your connections on:
- port 9306 for MySQL clients
- port 9308 for HTTP/HTTPS connections
- port 9312 for HTTP/HTTPS, and connections from other Manticore nodes and clients based on Manticore binary API
- SQL
- HTTP
- PHP
- Python
- Python-asyncio
- Javascript
- Java
- C#
- Rust
- docker
mysql -h0 -P9306HTTP is a stateless protocol, so it doesn't require any special connection phase:
curl -s "http://localhost:9308/search"require_once __DIR__ . '/vendor/autoload.php';
$config = ['host'=>'127.0.0.1','port'=>9308];
$client = new \Manticoresearch\Client($config);import manticoresearch
config = manticoresearch.Configuration(
host = "http://127.0.0.1:9308"
)
client = manticoresearch.ApiClient(config)
indexApi = manticoresearch.IndexApi(client)
searchApi = manticoresearch.searchApi(client)
utilsApi = manticoresearch.UtilsApi(client)import manticoresearch
config = manticoresearch.Configuration(
host = "http://127.0.0.1:9308"
)
async with manticoresearch.ApiClient(config) as client:
indexApi = manticoresearch.IndexApi(client)
searchApi = manticoresearch.searchApi(client)
utilsApi = manticoresearch.UtilsApi(client)var Manticoresearch = require('manticoresearch');
var client= new Manticoresearch.ApiClient()
client.basePath="http://127.0.0.1:9308";
indexApi = new Manticoresearch.IndexApi(client);
searchApi = new Manticoresearch.SearchApi(client);
utilsApi = new Manticoresearch.UtilsApi(client);import com.manticoresearch.client.ApiClient;
import com.manticoresearch.client.ApiException;
import com.manticoresearch.client.Configuration;
import com.manticoresearch.client.model.*;
import com.manticoresearch.client.api.IndexApi;
import com.manticoresearch.client.api.UtilsApi;
import com.manticoresearch.client.api.SearchApi;
ApiClient client = Configuration.getDefaultApiClient();
client.setBasePath("http://127.0.0.1:9308");
IndexApi indexApi = new IndexApi(client);
SearchApi searchApi = new UtilsApi(client);
UtilsApi utilsApi = new UtilsApi(client);using ManticoreSearch.Client;
using ManticoreSearch.Api;
using ManticoreSearch.Model;
string basePath = "http://127.0.0.1:9308";
IndexApi indexApi = new IndexApi(basePath);
SearchApi searchApi = new UtilsApi(basePath);
UtilsApi utilsApi = new UtilsApi(basePath);use std::sync::Arc;
use manticoresearch::{
apis::{
{configuration::Configuration,IndexApi,IndexApiClient,SearchApi,SearchApiClient,UtilsApi,UtilsApiClient}
},
};
async fn maticore_connect {
let configuration = Configuration {
base_path: "http://127.0.0.1:9308".to_owned(),
..Default::default(),
};
let api_config = Arc::new(configuration);
let utils_api = UtilsApiClient::new(api_config.clone());
let index_api = IndexApiClient::new(api_config.clone());
let search_api = SearchApiClient::new(api_config.clone());Run Manticore container and use built-in MySQL client to connect to the node.
docker run --name manticore -d manticoresearch/manticore && docker exec -it manticore mysqlManticore Search implements an SQL interface using the MySQL protocol, allowing any MySQL library or connector and many MySQL clients to be used to connect to Manticore Search and work with it as if it were a MySQL server, not Manticore.
However, the SQL dialect is different and implements only a subset of the SQL commands or functions available in MySQL. Additionally, there are clauses and functions that are specific to Manticore Search, such as the MATCH() clause for full-text search.
Manticore Search supports server-side prepared statements over the MySQL protocol. Client-side prepared statements can also be used. It is important to note that Manticore implements the multi-value (MVA) and float_vector data types, which have no equivalents in MySQL or libraries implementing prepared statements. In these cases, values must be crafted in the raw query as comma-separated list of numbers.
Some MySQL clients/connectors require values for user/password and/or database name. Since Manticore Search does not have the concept of databases and there is no user access control implemented, these values can be set arbitrarily as Manticore will simply ignore them.
The default port for the SQL interface is 9306 and it's enabled by default.
You can configure the MySQL port in the searchd section of the configuration file using the listen directive like this:
searchd {
...
listen = 127.0.0.1:9306:mysql
...
}
Keep in mind that Manticore doesn't have user authentication, so make sure that the MySQL port is not accessible to anyone outside of your network.
A separate MySQL port can be used for performing "VIP" connections. When connecting to this port, the thread pool is bypassed, and a new dedicated thread is always created. This is useful in cases of severe overload, where the server would either stall or prevent a connection through the regular port.
searchd {
...
listen = 127.0.0.1:9306:mysql
listen = 127.0.0.1:9307:mysql_vip
...
}
The easiest way to connect to Manticore is by using a standard MySQL client:
mysql -P9306 -h0
The MySQL protocol supports SSL encryption. Secure connections can be made on the same mysql listening port.
Compression can be used with MySQL connections and is available to clients by default. The client just needs to specify that the connection should use compression.
An example using the MySQL client:
mysql -P9306 -h0 -C
Compression can be used in both secured and non-secured connections.
The official MySQL connectors can be used to connect to Manticore Search, however they might require certain settings passed in the DSN string as the connector can try running certain SQL commands not implemented yet in Manticore.
JDBC Connector 6.x and above require Manticore Search 2.8.2 or greater and the DSN string should contain the following options:
jdbc:mysql://IP:PORT/DB/?characterEncoding=utf8&maxAllowedPacket=512000&serverTimezone=XXX
By default Manticore Search will report it's own version to the connector, however this may cause some troubles. To overcome that mysql_version_string directive in searchd section of the configuration should be set to a version lower than 5.1.1:
searchd {
...
mysql_version_string = 5.0.37
...
}
.NET MySQL connector uses connection pools by default. To correctly get the statistics of SHOW META, queries along with SHOW META command should be sent as a single multistatement (SELECT ...;SHOW META). If pooling is enabled option Allow Batch=True is required to be added to the connection string to allow multistatements:
Server=127.0.0.1;Port=9306;Database=somevalue;Uid=somevalue;Pwd=;Allow Batch=True;
Manticore can be accessed using ODBC. It's recommended to set charset=UTF8 in the ODBC string. Some ODBC drivers will not like the reported version by the Manticore server as they will see it as a very old MySQL server. This can be overridden with mysql_version_string option.
Manticore SQL over MySQL supports C-style comment syntax. Everything from an opening /* sequence to a closing */ sequence is ignored. Comments can span multiple lines, can not nest, and should not get logged. MySQL specific /*! ... */ comments are also currently ignored. (As the comments support was rather added for better compatibility with mysqldump produced dumps, rather than improving general query interoperability between Manticore and MySQL.)
SELECT /*! SQL_CALC_FOUND_ROWS */ col1 FROM table1 WHERE ...
Manticore supports prepared statements over MySQL protocol.
In databases, prepared statements are a way to run queries where the SQL code is kept separate from the input data. Instead of writing a full SQL query with the values included directly in it, you write the query once using placeholders and then supply the values separately.
Some client libraries (for example, sqlx) communicate with databases only in this way.
-
Security (Preventing SQL Injection): This is the most critical reason. SQL injection is a common web vulnerability in which attackers insert malicious SQL into queries. Prepared statements ensure that user input is treated strictly as data, not executable code, preventing it from being interpreted as part of the SQL command.
-
Readability & Maintainability: Prepared statements improve code clarity by separating SQL logic from input data, making the code easier to read and maintain.
-
Performance: The database can cache query plans. If the same parameterized query runs multiple times with different values, the database can reuse the cached execution plan instead of parsing and optimizing the query each time, which can improve performance.
- Prepare the query: Send the SQL statement with placeholders (such as
?or?VEC?) to Manticore. Manticore parses and compiles the statement, stores it, and returns a unique ID that can be used to execute it later. - Bind parameters: Send the values for the placeholders separately. Manticore treats these values strictly as data (not SQL code) and safely inserts them into the stored statement.
- Execute the query: Manticore executes the statement using the precompiled plan together with the provided parameter values.
- SQL
- PHP
INSERT INTO table (id, floatvec) VALUES (?, (?VEC?))$stmt = $mysqli->prepare("INSERT INTO tbl (id, str, floatvec) VALUES (0, ?, (?VEC?))");
$str = "I'm a string";
$vec = "0.1,0.2,0.3";
$stmt->bind_param("ss", $str, $vec);
$stmt->execute();This will execute:
INSERT INTO tbl (id, str, floatvec) VALUES (0, 'I\'m a string', (0.1,0.2,0.3))?(question mark): Represents a single parameter (integer, float, or string). String values are processed automatically — for example, special characters such as single quotes are escaped, and the value is enclosed in quotes. Boolean values can be passed as strings ('true','false').?VEC?: Represents a list of numeric values provided as a string (for example,1,2,3or0.3,15E+3, 20 ,INF). Only numbers, commas, and spaces are allowed. After validation, the values are inserted exactly as provided (without additional escaping or quotes).
In the standard Manticore SQL syntax MVA or float vectors are written as a list of values enclosed in parentheses, for example (1, 2, 3).
When using prepared statements, include the parentheses directly in the query, and use the ?VEC? placeholder only for the values inside them. For example:
$stmt = $mysqli->prepare("INSERT INTO tbl (id, floatvec) VALUES (0, (?VEC?))");
$vec = "0.1,0.2,0.3";
$stmt->bind_param("s", $vec);
- Only one SQL statement is allowed per prepared statement. Multi-queries (for example,
SELECT ...; SHOW META) are not supported. If you need to run multiple statements, prepare a separate statement for each one and execute them in sequence in the same session. - Some drivers send numeric parameters as DOUBLE (e.g.,
mysql2for Node.js). If you need strict integer behavior (such as rejecting negative IDs), send integers as strings or use driver-specific integer types (e.g.,BigInt). - Rust sqlx: When reading result set rows, use column indices rather than column names. Column names are present in the result set, but sqlx does not use them for mapping. For example, use
row.try_get(0)?instead ofrow.try_get("id")?.