There are cases of multiple possible scenarios where searchd service got stopped with no node in the cluster being able to serve requests. In these cases someone needs to recover the cluster or part of it. Due to multi-master nature of Galera library used for replication, a cluster is like one logical entity and takes care about each its node and node's data consistency and keeps a cluster's status as a whole. This allows safe writes on multiple nodes at the same time and maintains cluster integrity unlike a traditional asynchronous replication.

Here we take a cluster of nodes A, B and C as an example and consider scenarios where some or all nodes are out of service and what to do to bring them back.

Node A is stopped as usual. The other nodes receive "normal shutdown" message from node A. The cluster size is reduced and a quorum re-calculation is issued.

After node A got started as usual, it joins the cluster nodes. Node A will not serve any write transaction until the join is finished and it's fully synchronized with the cluster. If a writeset cache on a donor node B or C, which is set with a Galera cluster's option gcache.size, still has all transactions missed at node A, node A will receive a fast incremental state transfer (IST), that is, a transfer of only missed transactions. Otherwise, a snapshot state transfer (SST) will start, that is, a transfer of index files.

Nodes A and B are stopped as usual. That is the same situation as in previous case but the cluster's size is reduced to 1 and node C itself forms primary component that allows it to handle write transactions.

Nodes A and B may be started as usual and will join the cluster after the start. Node C becomes a "donor" and provides the transfer of the state to nodes A and B.

All nodes are stopped as usual and the cluster is off.

The problem now is how to initialize cluster and it is important that on a clean shutdown of searchd nodes write the number of the last executed transaction into the cluster directory grastate.dat file along with the safe_to_bootstrap key. The node stopped last will have the safe_to_bootstrap: 1 option and the most advanced seqno number.

It is important that this node should start first to form a cluster. To bootstrap, a cluster server should be started on this node with the --new-cluster command line key.

If another node starts first and bootstraps the cluster, then the most advanced node joins that cluster, performs full SST and receives an index file where some transactions are missed in comparison with index files it got before. That is why it is important to start the node that shut down last or to look at the cluster directory grastate.dat file to find the node with the safe_to_bootstrap: 1 option.

Node A disappears from the cluster due to crash or network failure.

Nodes B and C try to reconnect to missed node A and after failure remove node A from the cluster. The cluster quorum is valid as 2 out of 3 nodes are running and the cluster works as usual.

After node A restarts it will join the cluster automatically the same way as in Case 1.

Nodes A and B disappear. Node C is not able to form the quorum alone as 1 node is less than 1.5 (half of 3). So the cluster on node C is switched to non-primary state and node C rejects any write transactions with an error message.

Meanwhile, the single node C is waiting for other nodes to connect and try to connect them itself. If this happens, after the network is restored and nodes A and B are running again, the cluster will be formed again automatically. If nodes A and B are just cut from node C, but they can still reach each other, they keep working as usual because they still form the quorum.

SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1

POST /sql -d "mode=raw&query=
SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1
"

All nodes crashed. In this case the grastate.dat file at cluster directory is not updated and does not contain a valid sequence number seqno.

If this happened, someone should find the most advanced node and start the server on it with the --new-cluster-force command line key. All other nodes will start as usual as in Case 3).

Split-brain causes a cluster to get into non-primary state. For example, the cluster consists of even number of nodes (four), two couple of nodes being located in separate datacenters, and network failure interrupts the connection these datacenters. Split-brain happens as each group of nodes has exactly half of quorum. Both groups stop to handle write transactions as Galera replication model cares about data consistency and the cluster can not accept write transactions without quorum. But nodes in both groups try to re-connect to the nodes nodes from the other group to restore the cluster.

SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1

POST /sql -d "mode=raw&query=
SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1
"

However, we want to notice that if the statement gets issued at both groups this will end up with two separate clusters made, so the following network restoration will not make the groups to rejoin.

Manticore Search implements an SQL interface using MySQL protocol, which allows any MySQL client, library or connector to be used for connecting to Manticore Search and work with it as if it would be MySQL server, not Manticore.

However the SQL dialect is different. It implements only a subset of SQL commands or functions available in MySQL. In addition, there are clauses and functions that are specific to Manticore Search. The most eloquent example is the MATCH() clause which allows setting the full-text search.

Manticore Search doesn't support server-side prepared statements. Client-side prepared statements can be used with Manticore. It must be noted that Manticore implements the multi value (MVA) data type for which there is no equivalent in MySQL or libraries implementing prepared statements. In these cases, the MVA values will need to be crafted in the raw query.

Some MySQL clients/connectors demand values for user/password and/or database name. Since Manticore Search does not have the concept of database and there is no user access control yet implemented, these can be set arbitrarily as Manticore will simply ignore the values.

The default port for the SQL interface is 9306 and it's enabled by default.

In the searchd section of the configuration file the MySQL port can be defined by listen directive like this:

searchd {
...
   listen = 127.0.0.1:9306:mysql
...
}

Because Manticore doesn't have yet user authentication implemented make sure the MySQL port can't be accessed by anyone outside your network.

A separate MySQL port can be used to perform 'VIP' connections. A connection to this port bypasses the thread pool and always forcibly creates a new dedicated thread. That's useful for managing in case of a severe overload when the server would either stall or not let you connect via a regular port.

searchd {
...
   listen = 127.0.0.1:9306:mysql
   listen = 127.0.0.1:9307:mysql_vip
...
}

The easiest way to connect to Manticore is by using a standard MySQL client:

mysql -P9306 -h0

The MySQL protocol supports SSL encryption. The secured connections can be made on the same mysql listening port.

Compression can be used with MySQL Connections and available to clients by default. The client just need to specify the connection to use compression.

An example with the MySQL client:

mysql -P9306 -h0 -C

Compression can be used in both secured and non-secured connections.

The official MySQL connectors can be used to connect to Manticore Search, however they might require certain settings passed in the DSN string as the connector can try running certain SQL commands not implemented yet in Manticore.

JDBC Connector 6.x and above require Manticore Search 2.8.2 or greater and the DSN string should contain the following options:

jdbc:mysql://IP:PORT/DB/?characterEncoding=utf8&maxAllowedPacket=512000&serverTimezone=XXX

By default Manticore Search will report it's own version to the connector, however this may cause some troubles. To overcome that mysql_version_string directive in searchd section of the configuration should be set to a version lower than 5.1.1:

searchd {
...
   mysql_version_string = 5.0.37
...
}

.NET MySQL connector uses connection pools by default. To correctly get the statistics of SHOW META, queries along with SHOW META command should be sent as a single multistatement (SELECT ...;SHOW META). If pooling is enabled option Allow Batch=True is required to be added to the connection string to allow multistatements:

Server=127.0.0.1;Port=9306;Database=somevalue;Uid=somevalue;Pwd=;Allow Batch=True;

Manticore can be accessed using ODBC. It's recommended to set charset=UTF8 in the ODBC string. Some ODBC drivers will not like the reported version by the Manticore server as they will see it as a very old MySQL server. This can be overridden with mysql_version_string option.

Manticore SQL over MySQL supports C-style comment syntax. Everything from an opening /* sequence to a closing */ sequence is ignored. Comments can span multiple lines, can not nest, and should not get logged. MySQL specific /*! ... */ comments are also currently ignored. (As the comments support was rather added for better compatibility with mysqldump produced dumps, rather than improving general query interoperability between Manticore and MySQL.)

SELECT /*! SQL_CALC_FOUND_ROWS */ col1 FROM table1 WHERE ...

You can connect to Manticore Search over HTTP/HTTPS.

searchd {
...
   listen = 127.0.0.1:9308
   listen = 127.0.0.1:9312:http
...
}

All HTTP endpoints respond with application/json content type. Most endpoints use JSON payload for requests, however there are some exceptions that use NDJSON or simple URL encoded payload.

There is no user authentication implemented at the moment, so make sure the HTTP interface is not reachable by anyone outside your network. Since Manticore acts like any other web server, you can use a reverse proxy like Nginx to add HTTP authentication or caching.

searchd {
...
   listen = 127.0.0.1:9308
   listen = 127.0.0.1:9443:https
...
}

searchd {
...
   listen = 127.0.0.1:9308
   listen = 127.0.0.1:9318:_vip
...
}

curl -sX POST http://localhost:9308/search -d ' {"index":"test","query":{"match":{"title":"keyword"}}}'

POST /sql
--data-urlencode "query=select id,subject,author_id  from forum where match('@subject php manticore') group by
author_id order by id desc limit 0,5"

{
  "took":10,
  "timed_out": false,
  "hits":
  {
    "total": 2,
    "hits":
    [
      {
        "_id": "1",
        "_score": 1,
        "_source": { "gid": 11 }
      },
      {
        "_id": "2",
        "_score": 1,
        "_source": { "gid": 12 }
      }
    ]
  }
}

POST /sql -d "query=select id,packedfactors() from movies where match('star') option ranker=expr('1')"

{"error":"query missing"}

POST /sql -d "mode=raw&query=select id,packedfactors() from movies where match('star') option ranker=expr('1')"

{
  "took":0,
  "timed_out":false,
  "hits":{
    "total":72,
    "hits":[
      {
        "_id":"5",
        "_score":1,
        "_source":{
          "packedfactors()":{
            "bm25":612,
            "bm25a":0.69104159,
            "field_mask":32,
            "doc_word_count":1,
            "fields":[
              {
                "field":5,
                "lcs":1,
                "hit_count":1,
                "word_count":1,
                "tf_idf":0.24835411,
                "min_idf":0.24835411,
                "max_idf":0.24835411,
                "sum_idf":0.24835411,
                "min_hit_pos":1,
                "min_best_span_pos":1,
                "exact_hit":0,
                "max_window_hits":1,
                "min_gaps":0,
                "exact_order":1,
                "lccs":1,
                "wlccs":0.24835411,
                "atc":0.000000
              }
            ],
            "words":[
              {
                "tf":1,
                "idf":0.24835411
              }
            ]
          }
        }
      },
...
    ]
  }
}