Creating a cluster > Setting up replication > Replication cluster status

ALTER CLUSTER <cluster_name> ADD <table_name>[, <table_name>] adds one or more existing local tables to the cluster. The node that receives the ALTER query sends the table(s) to the other nodes in the cluster. All the local tables with the same name on the other nodes of the cluster are replaced with the new table(s).

Once the tables are replicated, write statements can be performed on any node, but the table names must be prefixed with the cluster name, like INSERT INTO <clusterName>:<table_name>.

‹›

SQL
JSON
PHP
Python
Python-asyncio
javascript
Java
C#
Rust

📋

ALTER CLUSTER click_query ADD clicks_daily_index

POST /cli -d "
ALTER CLUSTER click_query ADD clicks_daily_index
"

$params = [
  'cluster' => 'click_query',
  'body' => [
     'operation' => 'add',
     'table' => 'clicks_daily_index'
  ]
];
$response = $client->cluster()->alter($params);

utilsApi.sql('ALTER CLUSTER click_query ADD clicks_daily_index')

await utilsApi.sql('ALTER CLUSTER click_query ADD clicks_daily_index')

res = await utilsApi.sql('ALTER CLUSTER click_query ADD clicks_daily_index');

utilsApi.sql("ALTER CLUSTER click_query ADD clicks_daily_index");

utilsApi.Sql("ALTER CLUSTER click_query ADD clicks_daily_index");

utils_api.sql("ALTER CLUSTER click_query ADD clicks_daily_index", Some(true)).await;

‹›

Response

{u'error': u'', u'total': 0, u'warning': u''}

ALTER CLUSTER <cluster_name> DROP <table_name>[, <table_name>] forgets about one or more existing table(s), meaning it does not remove the table(s) files on the nodes, but rather just makes them inactive, non-replicated table(s).

Once a table is removed from a cluster, it becomes a local table, and write statements must use just the table name, like INSERT INTO <table_name>, without the cluster prefix.

‹›

SQL
JSON
PHP
Python
Python-asyncio
javascript
Java
C#
Rust

📋

ALTER CLUSTER posts DROP weekly_index

POST /cli -d "
ALTER CLUSTER posts DROP weekly_index
"

$params = [
  'cluster' => 'posts',
  'body' => [
     'operation' => 'drop',
     'table' => 'weekly_index'
  ]
];
$response = $client->cluster->alter($params);

utilsApi.sql('ALTER CLUSTER posts DROP weekly_index')

await utilsApi.sql('ALTER CLUSTER posts DROP weekly_index')

res = await utilsApi.sql('ALTER CLUSTER posts DROP weekly_index');

utilsApi.sql("ALTER CLUSTER posts DROP weekly_index");

utilsApi.Sql("ALTER CLUSTER posts DROP weekly_index");

utils_api.sql("ALTER CLUSTER posts DROP weekly_index", Some(true)).await;

‹›

Response

{u'error': u'', u'total': 0, u'warning': u''}

{u'error': u'', u'total': 0, u'warning': u''}

{"total":0,"error":"","warning":""}

Managing replication nodes

The ALTER CLUSTER <cluster_name> UPDATE nodes statement updates the node lists on each node within the specified cluster to include all active nodes in the cluster. For more information on node lists, see Joining a cluster.

‹›

SQL
JSON
PHP
Python
Python-asyncio
javascript
Java
C#
Rust

📋

ALTER CLUSTER posts UPDATE nodes

POST /cli -d "
ALTER CLUSTER posts UPDATE nodes
"

$params = [
  'cluster' => 'posts',
  'body' => [
     'operation' => 'update',
  ]
];
$response = $client->cluster()->alter($params);

utilsApi.sql('ALTER CLUSTER posts UPDATE nodes')

await utilsApi.sql('ALTER CLUSTER posts UPDATE nodes')

res = await utilsApi.sql('ALTER CLUSTER posts UPDATE nodes');

utilsApi.sql("ALTER CLUSTER posts UPDATE nodes");

utilsApi.Sql("ALTER CLUSTER posts UPDATE nodes");

utils_api.sql("ALTER CLUSTER posts UPDATE nodes", Some(true)).await;

‹›

Response

{u'error': u'', u'total': 0, u'warning': u''}

{u'error': u'', u'total': 0, u'warning': u''}

{"total":0,"error":"","warning":""}

For instance, when the cluster was initially established, the list of nodes used to rejoin the cluster was 10.10.0.1:9312,10.10.1.1:9312. Since then, other nodes joined the cluster and now the active nodes are 10.10.0.1:9312,10.10.1.1:9312,10.15.0.1:9312,10.15.0.3:9312.However, the list of nodes used to rejoin the cluster has not been updated.

To rectify this, you can run the ALTER CLUSTER ... UPDATE nodes statement to copy the list of active nodes to the list of nodes used to rejoin the cluster. After this, the list of nodes used to rejoin the cluster will include all the active nodes in the cluster.

Both lists of nodes can be viewed using the Cluster status statement (cluster_post_nodes_set and cluster_post_nodes_view).

To remove a node from the replication cluster, follow these steps:

Stop the node
Remove the information about the cluster from <data_dir>/manticore.json (usually /var/lib/manticore/manticore.json) on the node that has been stopped.
Run ALTER CLUSTER cluster_name UPDATE nodes on any other node.

After these steps, the other nodes will forget about the detached node and the detached node will forget about the cluster. This action will not impact the tables in the cluster or on the detached node.

Adding and removing a table from a replication cluster Replication cluster status

You can view the cluster status information by checking the node status. This can be done using the Node status command, which displays various information about the node, including the cluster status variables.

The output format for the cluster status variables is as follows: cluster_name_variable_name variable_value. Most of the variables are described in the Galera Documentation Status Variables. In addition to these variables, Manticore Search also displays:

cluster_name - the name of the cluster, as defined in the replication setup
node_state - the current state of the node: closed, destroyed, joining, donor, synced
indexes_count - the number of tables managed by the cluster
indexes - a list of table names managed by the cluster
nodes_set - the list of nodes in the cluster defined using the CREATE, JOIN or ALTER UPDATE commands
nodes_view - the actual list of nodes in the cluster that the current node can see.
state_uuid - UUID state of the cluster. If it matches the value in local_state_uuid, the local and cluster nodes are in sync.
conf_id - total number of cluster membership changes that have taken place.
status - cluster component status. Possible values are primary (primary group configuration, quorum present), non_primary (non-primary group configuration, quorum lost), or disconnected (not connected to group, retrying).
size - number of nodes currently in the cluster.
local_index - the node's index in the cluster.
last_error - the last recorded error message related to a cluster operation. The message provides a high-level summary of the problem. For more detailed context, you should consult the searchd.log file.

During a State Snapshot Transfer (SST), a node provisions another by transferring a full data copy. This happens when a new node joins the cluster JOIN CLUSTER or when new tables are added ALTER CLUSTER ADD. While an SST is active, the following additional status variables will be available on both the donor and joiner nodes, with their progress kept in sync.

cluster_name_sst_total - The overall progress of the entire SST operation, from 0 to 100. This is the primary counter to watch.
cluster_name_sst_stage - The name of the current work phase. The process cycles through these stages for each table being transferred:
- await nodes sync
- block checksum calculate
- analyze remote
- send files
- activate tables
cluster_name_sst_stage_total - The progress of the current stage, from 0 to 100.
cluster_name_sst_tables - The total number of tables being transferred in the SST.
cluster_name_sst_table - The name and index of the table currently being processed (e.g., 3 (products)).

For most use cases, cluster_name_sst_total is sufficient. However, the other counters can be useful for investigating stalls or performance issues during a specific SST stage or on a particular table.

‹›

SQL
JSON
PHP
Python
Python-asyncio
javascript
Java
C#
Rust

📋

SHOW STATUS

POST /cli -d "
SHOW STATUS
"

$params = [
    'body' => []
];
$response = $client->nodes()->status($params);

utilsApi.sql('SHOW STATUS')

await utilsApi.sql('SHOW STATUS')

res = await utilsApi.sql('SHOW STATUS');

utilsApi.sql("SHOW STATUS");

utilsApi.sql("SHOW STATUS");

utils_api.sql("SHOW STATUS", Some(true)).await;

‹›

Response

+---------------------------------+-------------------------------------------------------------------------------------+
| Counter                         | Value                                                                               |
+---------------------------------+-------------------------------------------------------------------------------------+
| cluster_name                    | post                                                                                |
| cluster_post_state_uuid         | fba97c45-36df-11e9-a84e-eb09d14b8ea7                                                |
| cluster_post_conf_id            | 1                                                                                   |
| cluster_post_status             | primary                                                                             |
| cluster_post_size               | 5                                                                                   |
| cluster_post_local_index        | 0                                                                                   |
| cluster_post_node_state         | donor                                                                               |
| cluster_post_indexes_count      | 2                                                                                   |
| cluster_post_indexes            | pq1,pq_posts                                                                        |
| cluster_post_nodes_set          | 10.10.0.1:9312                                                                      |
| cluster_post_nodes_view         | 10.10.0.1:9312,10.10.0.1:9320:replication,10.10.1.1:9312,10.10.1.1:9320:replication |
| cluster_post_sst_total          | 65                                                                                  |
| cluster_post_sst_stage          | send files                                                                       |
| cluster_post_sst_stage_total    | 78                                                                                  |
| cluster_post_sst_tables         | 5                                                                                   |
| cluster_post_sst_table          | 3 (products)                                                                        |
+---------------------------------+-------------------------------------------------------------------------------------+

"
{"columns":[{"Counter":{"type":"string"}},{"Value":{"type":"string"}}],
"data":[
{"Counter":"cluster_name", "Value":"post"},
{"Counter":"cluster_post_state_uuid", "Value":"fba97c45-36df-11e9-a84e-eb09d14b8ea7"},
{"Counter":"cluster_post_conf_id", "Value":"1"},
{"Counter":"cluster_post_status", "Value":"primary"},
{"Counter":"cluster_post_size", "Value":"5"},
{"Counter":"cluster_post_local_index", "Value":"0"},
{"Counter":"cluster_post_node_state", "Value":"donor"},
{"Counter":"cluster_post_indexes_count", "Value":"2"},
{"Counter":"cluster_post_indexes", "Value":"pq1,pq_posts"},
{"Counter":"cluster_post_nodes_set", "Value":"10.10.0.1:9312"},
{"Counter":"cluster_post_nodes_view", "Value":"10.10.0.1:9312,10.10.0.1:9320:replication,10.10.1.1:9312,10.10.1.1:9320:replication"},
{"Counter":"cluster_post_sst_total", "Value":"65"},
{"Counter":"cluster_post_sst_stage", "Value":"send files"},
{"Counter":"cluster_post_sst_stage_total", "Value":"78"},
{"Counter":"cluster_post_sst_tables", "Value":"5"},
{"Counter":"cluster_post_sst_table", "Value":"3 (products)"}
],
"total":0,
"error":"",
"warning":""
}

(
"cluster_name" => "post",
"cluster_post_state_uuid" => "fba97c45-36df-11e9-a84e-eb09d14b8ea7",
"cluster_post_conf_id" => 1,
"cluster_post_status" => "primary",
"cluster_post_size" => 5,
"cluster_post_local_index" => 0,
"cluster_post_node_state" => "donor",
"cluster_post_indexes_count" => 2,
"cluster_post_indexes" => "pq1,pq_posts",
"cluster_post_nodes_set" => "10.10.0.1:9312",
"cluster_post_nodes_view" => "10.10.0.1:9312,10.10.0.1:9320:replication,10.10.1.1:9312,10.10.1.1:9320:replication",
"cluster_post_sst_total" => 65,
"cluster_post_sst_stage" => "send files",
"cluster_post_sst_stage_total" => 78,
"cluster_post_sst_tables" => 5,
"cluster_post_sst_table" => "3 (products)"
)

{u'columns': [{u'Key': {u'type': u'string'}},
              {u'Value': {u'type': u'string'}}],
 u'data': [
    {u'Key': u'cluster_name', u'Value': u'post'},
    {u'Key': u'cluster_post_state_uuid', u'Value': u'fba97c45-36df-11e9-a84e-eb09d14b8ea7'},
    {u'Key': u'cluster_post_conf_id', u'Value': u'1'},
    {u'Key': u'cluster_post_status', u'Value': u'primary'},
    {u'Key': u'cluster_post_size', u'Value': u'5'},
    {u'Key': u'cluster_post_local_index', u'Value': u'0'},
    {u'Key': u'cluster_post_node_state', u'Value': u'donor'},
    {u'Key': u'cluster_post_indexes_count', u'Value': u'2'},
    {u'Key': u'cluster_post_indexes', u'Value': u'pq1,pq_posts'},
    {u'Key': u'cluster_post_nodes_set', u'Value': u'10.10.0.1:9312'},
    {u'Key': u'cluster_post_nodes_view', u'Value': u'10.10.0.1:9312,10.10.0.1:9320:replication,10.10.1.1:9312,10.10.1.1:9320:replication'},
    {u'Key': u'cluster_post_sst_total', u'Value': u'65'},
    {u'Key': u'cluster_post_sst_stage', u'Value': u'send files'},
    {u'Key': u'cluster_post_sst_stage_total', u'Value': u'78'},
    {u'Key': u'cluster_post_sst_tables', u'Value': u'5'},
    {u'Key': u'cluster_post_sst_table', u'Value': u'3 (products)'}],
 u'error': u'',
 u'total': 0,
 u'warning': u''}

{u'columns': [{u'Key': {u'type': u'string'}},
              {u'Value': {u'type': u'string'}}],
 u'data': [
    {u'Key': u'cluster_name', u'Value': u'post'},
    {u'Key': u'cluster_post_state_uuid', u'Value': u'fba97c45-36df-11e9-a84e-eb09d14b8ea7'},
    {u'Key': u'cluster_post_conf_id', u'Value': u'1'},
    {u'Key': u'cluster_post_status', u'Value': u'primary'},
    {u'Key': u'cluster_post_size', u'Value': u'5'},
    {u'Key': u'cluster_post_local_index', u'Value': u'0'},
    {u'Key': u'cluster_post_node_state', u'Value': u'donor'},
    {u'Key': u'cluster_post_indexes_count', u'Value': u'2'},
    {u'Key': u'cluster_post_indexes', u'Value': u'pq1,pq_posts'},
    {u'Key': u'cluster_post_nodes_set', u'Value': u'10.10.0.1:9312'},
    {u'Key': u'cluster_post_nodes_view', u'Value': u'10.10.0.1:9312,10.10.0.1:9320:replication,10.10.1.1:9312,10.10.1.1:9320:replication'},
    {u'Key': u'cluster_post_sst_total', u'Value': u'65'},
    {u'Key': u'cluster_post_sst_stage', u'Value': u'send files'},
    {u'Key': u'cluster_post_sst_stage_total', u'Value': u'78'},
    {u'Key': u'cluster_post_sst_tables', u'Value': u'5'},
    {u'Key': u'cluster_post_sst_table', u'Value': u'3 (products)'}],
 u'error': u'',
 u'total': 0,
 u'warning': u''}

{"columns": [{"Key": {"type": "string"}},
              {"Value": {"type": "string"}}],
 "data": [
    {"Key": "cluster_name", "Value": "post"},
    {"Key": "cluster_post_state_uuid", "Value": "fba97c45-36df-11e9-a84e-eb09d14b8ea7"},
    {"Key": "cluster_post_conf_id", "Value": "1"},
    {"Key": "cluster_post_status", "Value": "primary"},
    {"Key": "cluster_post_size", "Value": "5"},
    {"Key": "cluster_post_local_index", "Value": "0"},
    {"Key": "cluster_post_node_state", "Value": "donor"},
    {"Key": "cluster_post_indexes_count", "Value": "2"},
    {"Key": "cluster_post_indexes", "Value": "pq1,pq_posts"},
    {"Key": "cluster_post_nodes_set", "Value": "10.10.0.1:9312"},
    {"Key": "cluster_post_nodes_view", "Value": "10.10.0.1:9312,10.10.0.1:9320:replication,10.10.1.1:9312,10.10.1.1:9320:replication"},
    {"Key": "cluster_post_sst_total", "Value": "65"},
    {"Key": "cluster_post_sst_stage", "Value": "send files"},
    {"Key": "cluster_post_sst_stage_total", "Value": "78"},
    {"Key": "cluster_post_sst_tables", "Value": "5"},
    {"Key": "cluster_post_sst_table", "Value": "3 (products)"}],
 "error": "",
 "total": 0,
 "warning": ""}

{columns=[{ Key : { type=string }},
              { Value : { type=string }}],
  data : [
    { Key=cluster_name, Value=post},
    { Key=cluster_post_state_uuid, Value=fba97c45-36df-11e9-a84e-eb09d14b8ea7},
    { Key=cluster_post_conf_id, Value=1},
    { Key=cluster_post_status, Value=primary},
    { Key=cluster_post_size, Value=5},
    { Key=cluster_post_local_index, Value=0},
    { Key=cluster_post_node_state, Value=donor},
    { Key=cluster_post_indexes_count, Value=2},
    { Key=cluster_post_indexes, Value=pq1,pq_posts},
    { Key=cluster_post_nodes_set, Value=10.10.0.1:9312},
    { Key=cluster_post_nodes_view, Value=10.10.0.1:9312,10.10.0.1:9320:replication,10.10.1.1:9312,10.10.1.1:9320:replication},
    { Key=cluster_post_sst_total, Value=65},
    { Key=cluster_post_sst_stage, Value=send files},
    { Key=cluster_post_sst_stage_total, Value=78},
    { Key=cluster_post_sst_tables, Value=5},
    { Key=cluster_post_sst_table, Value=3 (products)}],
  error= ,
  total=0,
  warning= }

{columns=[{ Key : { type=String }},
              { Value : { type=String }}],
  data : [
    { Key=cluster_name, Value=post},
    { Key=cluster_post_state_uuid, Value=fba97c45-36df-11e9-a84e-eb09d14b8ea7},
    { Key=cluster_post_conf_id, Value=1},
    { Key=cluster_post_status, Value=primary},
    { Key=cluster_post_size, Value=5},
    { Key=cluster_post_local_index, Value=0},
    { Key=cluster_post_node_state, Value=donor},
    { Key=cluster_post_indexes_count, Value=2},
    { Key=cluster_post_indexes, Value=pq1,pq_posts},
    { Key=cluster_post_nodes_set, Value=10.10.0.1:9312},
    { Key=cluster_post_nodes_view, Value=10.10.0.1:9312,10.10.0.1:9320:replication,10.10.1.1:9312,10.10.1.1:9320:replication},
    { Key=cluster_post_sst_total, Value=65},
    { Key=cluster_post_sst_stage, Value=send files},
    { Key=cluster_post_sst_stage_total, Value=78},
    { Key=cluster_post_sst_tables, Value=5},
    { Key=cluster_post_sst_table, Value=3 (products)}],
  error="" ,
  total=0,
  warning="" }

{columns=[{ Key : { type=String }},
              { Value : { type=String }}],
  data : [
    { Key=cluster_name, Value=post},
    { Key=cluster_post_state_uuid, Value=fba97c45-36df-11e9-a84e-eb09d14b8ea7},
    { Key=cluster_post_conf_id, Value=1},
    { Key=cluster_post_status, Value=primary},
    { Key=cluster_post_size, Value=5},
    { Key=cluster_post_local_index, Value=0},
    { Key=cluster_post_node_state, Value=donor},
    { Key=cluster_post_indexes_count, Value=2},
    { Key=cluster_post_indexes, Value=pq1,pq_posts},
    { Key=cluster_post_nodes_set, Value=10.10.0.1:9312},
    { Key=cluster_post_nodes_view, Value=10.10.0.1:9312,10.10.0.1:9320:replication,10.10.1.1:9312,10.10.1.1:9320:replication},
    { Key=cluster_post_sst_total, Value=65},
    { Key=cluster_post_sst_stage, Value=send files},
    { Key=cluster_post_sst_stage_total, Value=78},
    { Key=cluster_post_sst_tables, Value=5},
    { Key=cluster_post_sst_table, Value=3 (products)}],
  error="" ,
  total=0,
  warning="" }

Managing replication nodes Restarting a cluster

In a multi-master replication cluster, a reference point must be established before other nodes can join and form the cluster. This is called cluster bootstrapping and involves starting a single node as the primary component. Restarting a single node or reconnecting after a shutdown can be done normally.

In case of a full cluster shutdown, the server that was stopped last should be started first with the --new-cluster command line option or by running manticore_new_cluster through systemd. To ensure that the server is capable of being the reference point, the grastate.dat file located at the cluster path should be updated with a value of 1 for the safe_to_bootstrap option. Both conditions, --new-cluster and safe_to_bootstrap=1, must be met. If any other node is started without these options set, an error will occur. The --new-cluster-force command line option can be used to override this protection and start the cluster from another server forcibly. Alternatively, you can run manticore_new_cluster --force to use systemd.

In the event of a hard crash or an unclean shutdown of all servers in the cluster, the most advanced node with the largest seqno in the grastate.dat file located at the cluster path must be identified and started with the --new-cluster-force command line key.

Replication cluster status Cluster recovery

Adding and removing a table from a replication cluster

Managing replication nodes

Removing node from cluster

Replication cluster status

SST Progress Metrics

Restarting a cluster