HTTP

You can connect to Manticore Search over HTTP/HTTPS.

Configuration

By default Manticore listens for HTTP, HTTPS and binary requests on ports 9308 and 9312.

In section "searchd" of your configuration file the HTTP port can be defined with directive listen like this:

Both lines are valid and equal by meaning (except for the port number), they both define listeners that will serve all api/http/https protocols. There are no special requirements and any HTTP client can be used to connect to Manticore.

  • HTTP
HTTP
📋
searchd {
...
   listen = 127.0.0.1:9308
   listen = 127.0.0.1:9312:http
...
}

All HTTP endpoints respond with application/json content type. Most endpoints use JSON payload for requests, however there are some exceptions that use NDJSON or simple URL encoded payload.

There is no user authentication implemented at the moment, so make sure the HTTP interface is not reachable by anyone outside your network. Since Manticore acts like any other web server, you can use a reverse proxy like Nginx to add HTTP authentication or caching.

The HTTP protocol also supports SSL encryption: If you specify :https instead of :http only secured connections will be accepted. Otherwise in case of no valid key/certificate provided, but the client trying to connect via https - the connection will be dropped. If you make not HTTPS, but an HTTP request to 9443 it will respond with HTTP code 400.

  • HTTPS
HTTPS
📋
searchd {
...
   listen = 127.0.0.1:9308
   listen = 127.0.0.1:9443:https
...
}

VIP connection

Separate HTTP interface can be used to perform 'VIP' connections. A connection in this case bypasses a thread pool and always forcibly creates a new dedicated thread. That's useful for managing Manticore Search in case of a severe overload when the server would either stall or not let you connect via a regular port otherwise.

Read more about listen in this section.

  • VIP
VIP
📋
searchd {
...
   listen = 127.0.0.1:9308
   listen = 127.0.0.1:9318:_vip
...
}

SQL over HTTP

Endpoints /sql and /cli allow running SQL queries via HTTP.

  • /sql accepts only SELECTs and returns response in HTTP JSON format. Parameter query should be url-encoded.
  • /sql?mode=raw accepts any SQL query and returns response in raw format similar to what you get via mysql. Parameter query should be url-encoded.
  • /cli accepts any SQL query and returns response in raw format similar to what you get via mysql. The query should not be url-encoded. This endpoint is to be used for doing manual actions via browser or command line http clients like curl.

/sql

/sql accepts an SQL SELECT query via HTTP JSON interface.

Query payload must be URL encoded, otherwise query statements with = (filtering or setting options) will result in an error.

It returns a JSON response which contains hits information and execution time. The response has the same format as json/search endpoint. Note, that /sql endpoint supports only single search requests. If you are looking for processing a multi-query see below.

  • HTTP
HTTP
📋
POST /sql -d "query=select%20id%2Csubject%2Cauthor_id%20%20from%20forum%20where%20match%28%27%40subject%20php%20manticore%27%29%20group%20by%20author_id%20order%20by%20id%20desc%20limit%200%2C5"
Response
{
  "took": 0,
  "timed_out": false,
  "hits": {
    "total": 2,
    "total_relation": "eq",
    "hits": [
      {
        "_id": "2",
        "_score": 2356,
        "_source": {
          "subject": "php manticore",
          "author_id": 12
        }
      },
      {
        "_id": "1",
        "_score": 2356,
        "_source": {
          "subject": "php manticore",
          "author_id": 11
        }
      }
    ]
  }
}

/sql?mode=raw

/sql endpoint also has a special mode "raw", which allows to send any valid sphinxql queries including multi-queries. The returned value is a json array of one or more result sets.

  • HTTP
HTTP
📋
POST /sql?mode=raw -d "query=desc%20test"
Response
[
  {
    "columns": [
      {
        "Field": {
          "type": "string"
        }
      },
      {
        "Type": {
          "type": "string"
        }
      },
      {
        "Properties": {
          "type": "string"
        }
      }
    ],
    "data": [
      {
        "Field": "id",
        "Type": "bigint",
        "Properties": ""
      },
      {
        "Field": "title",
        "Type": "text",
        "Properties": "indexed"
      },
      {
        "Field": "gid",
        "Type": "uint",
        "Properties": ""
      },
      {
        "Field": "title",
        "Type": "string",
        "Properties": ""
      },
      {
        "Field": "j",
        "Type": "json",
        "Properties": ""
      },
      {
        "Field": "new1",
        "Type": "uint",
        "Properties": ""
      }
    ],
    "total": 6,
    "error": "",
    "warning": ""
  }
]

/cli

While the /sql endpoint is useful to control Manticore programmatically from your application, there's also endpoint /cli which makes it eaiser to maintain a Manticore instance via curl or your browser manually. It accepts POST and GET HTTP methods. Everything after /cli? is taken by Manticore as is even if you don't escape it manually via curl or let the browser encode it automaticaly. The + sign is not decoded to space as well, eliminating the necessity of encoding it. The response format is the same as in the /sql?mode=raw.

  • HTTP
  • Browser
📋
POST /cli -d "select id,1+2 as a, packedfactors() from test where match('tes*') option ranker=expr('1')"
Response
[
  {
    "columns": [
      {
        "id": {
          "type": "long long"
        }
      },
      {
        "a": {
          "type": "long"
        }
      },
      {
        "packedfactors()": {
          "type": "string"
        }
      }
    ],
    "data": [
      {
        "id": 1,
        "a": 3,
        "packedfactors()": "bm25=616, bm25a=0.69689077, field_mask=1, doc_word_count=1, field0=(lcs=1, hit_count=1, word_count=1, tf_idf=0.25595802, min_idf=0.25595802, max_idf=0.25595802, sum_idf=0.25595802, min_hit_pos=1, min_best_span_pos=1, exact_hit=0, max_window_hits=1, min_gaps=0, exact_order=1, lccs=1, wlccs=0.25595802, atc=0.000000), word0=(tf=1, idf=0.25595802)"
      },
      {
        "id": 2,
        "a": 3,
        "packedfactors()": "bm25=616, bm25a=0.69689077, field_mask=1, doc_word_count=1, field0=(lcs=1, hit_count=1, word_count=1, tf_idf=0.25595802, min_idf=0.25595802, max_idf=0.25595802, sum_idf=0.25595802, min_hit_pos=1, min_best_span_pos=1, exact_hit=0, max_window_hits=1, min_gaps=0, exact_order=1, lccs=1, wlccs=0.25595802, atc=0.000000), word0=(tf=1, idf=0.25595802)"
      },
      {
        "id": 8,
        "a": 3,
        "packedfactors()": "bm25=616, bm25a=0.69689077, field_mask=1, doc_word_count=1, field0=(lcs=1, hit_count=1, word_count=1, tf_idf=0.25595802, min_idf=0.25595802, max_idf=0.25595802, sum_idf=0.25595802, min_hit_pos=2, min_best_span_pos=2, exact_hit=0, max_window_hits=1, min_gaps=0, exact_order=1, lccs=1, wlccs=0.25595802, atc=0.000000), word0=(tf=1, idf=0.25595802)"
      }
    ],
    "total": 3,
    "error": "",
    "warning": ""
  }
]

Keep-alive

HTTP keep-alive is also supported, which makes working via the HTTP JSON interface stateful as long as the client supports keep-alive too. For example, using the new /cli endpoint you can call SHOW META after SELECT and it will work the same way it works via mysql.

▪️ Adding documents to an index

Adding documents to a real-time index

If you are looking for information about adding documents to a plain index please read section about adding data from external storages.

Adding documents in a real-time manner is only supported for Real-Time and percolate indexes. Corresponding SQL command or HTTP endpoint or a client's functions inserts new rows (documents) into an existing index with provided field values.

You can insert a single or multiple documents with values for all fields of the index or only part of them. In this case the other fields will be filled with their default values (0 for scalar types, empty string for text types).

Expressions are currently not supported in INSERT and the values should be explicitly specified.

The ID field/value can be omitted as RT and PQ indexes support auto-id functionality. You can also use 0 as the id value to force automatic ID generation. Rows with duplicate IDs will not be overwritten by INSERT. You can use REPLACE for that.

Note, when you use the HTTP JSON protocol node doc is mandatory, all the values should be provided inside it.

  • SQL
  • HTTP
  • PHP
  • Python
  • Javascript
  • Java
📋

General syntax:

INSERT INTO <index name> [(column, ...)]
VALUES (value, ...)
[, (...)]
INSERT INTO products(title,price) VALUES ('Crossbody Bag with Tassel', 19.85);
INSERT INTO products(title) VALUES ('Crossbody Bag with Tassel');
INSERT INTO products VALUES (0,'Yellow bag', 4.95);
Response
Query OK, 1 rows affected (0.00 sec)
Query OK, 1 rows affected (0.00 sec)
Query OK, 1 rows affected (0.00 sec)

Auto ID

There is an auto ID generation functionality for column ID of documents inserted or replaced into an real-time or a Percolate index. The generator produces a unique ID of a document with some guarantees and should not be considered an auto-incremented ID.

The value of ID generated is guaranteed to be unique under the following conditions:

  • server_id value of the current server is in range of 0 to 127 and is unique among nodes in the cluster or it uses the default value generated from MAC address as a seed
  • system time does not change for the Manticore node between server restarts
  • auto ID is generated fewer than 16 million times per second between search server restarts

The auto ID generator creates 64 bit integer for a document ID and uses the following schema:

  • 0 to 23 bits is a counter that gets incremented on every call to auto ID generator
  • 24 to 55 bits is a unix timestamp of the server start
  • 56 to 63 bits is a server_id

This schema allows to be sure that the generated ID is unique among all nodes at the cluster and that data inserted into different cluster nodes does not create collisions between the nodes.

That is why the first ID from the generator used for auto ID is NOT 1 but a larger number. Also documents stream inserted into an index might have not sequential ID values if inserts into other indexes happen between the calls as the ID generator is single in the server and shared between all its indexes.

  • SQL
  • HTTP
  • PHP
  • Python
  • Javascript
  • Java
📋
INSERT INTO products(title,price) VALUES ('Crossbody Bag with Tassel', 19.85);
INSERT INTO products VALUES (0,'Yello bag', 4.95);
select * from products;
Response
+---------------------+-----------+---------------------------+
| id                  | price     | title                     |
+---------------------+-----------+---------------------------+
| 1657860156022587404 | 19.850000 | Crossbody Bag with Tassel |
| 1657860156022587405 |  4.950000 | Yello bag                 |
+---------------------+-----------+---------------------------+

Bulk adding documents

You can insert into a real-time index not just a single document, but as many as you want. It's ok to insert into a real-time index in batches of tens of thousands of documents. What's important to know in this case:

  • the larger the batch the higher is the latency of each insert operation
  • the larger the batch the higher indexation speed you can expect
  • each batch insert operation is considered a single transaction with atomicity guarantee, so you will either have all the new documents in the index at once or in case of a failure none of them will be added
  • you might want to increase max_packet_size value to allow bigger batches
  • SQL
  • HTTP
  • PHP
  • Python
  • Javascript
  • Java
📋

For bulk insert just provide more documents in brackets after VALUES(). The syntax is:

INSERT INTO <index name>[(column1, column2, ...)] VALUES ()[,(value1,[value2, ...])]

Optional column name list lets you explicitly specify values for some of the columns present in the index. All the other columns will be filled with their default values (0 for scalar types, empty string for string types).

For example:

INSERT INTO products(title,price) VALUES ('Crossbody Bag with Tassel', 19.85), ('microfiber sheet set', 19.99), ('Pet Hair Remover Glove', 7.99);
Response
Query OK, 3 rows affected (0.01 sec)

Expressions are not currently supported in INSERT and values should be explicitly specified.

Inserting multi-value attributes (MVA) values

Multi-value attributes (MVA) are inserted as arrays of numbers.

  • SQL
  • HTTP
  • PHP
  • Python
  • Javascript
  • Java
📋
INSERT INTO products(title, sizes) VALUES('shoes', (40,41,42,43));

Inserting JSON

JSON value can be inserted as as an escaped string (via SQL, HTTP, PHP) or as a JSON object (via HTTP).

  • SQL
  • HTTP
  • PHP
  • Python
  • Javascript
  • Java
📋
INSERT INTO products VALUES (1, 'shoes', '{"size": 41, "color": "red"}');