Pagination of search results

Manticore Search returns the top 20 matched documents in the result set by default.

SQL

In SQL, you can navigate through the result set using the LIMIT clause.

LIMIT can accept either one number as the size of the returned set with a zero offset, or a pair of offset and size values.

HTTP JSON

When using HTTP JSON, the nodes offset and limit control the offset of the result set and the size of the returned set. Alternatively, you can use the pair size and from instead.

‹›
  • SQL
  • JSON
📋
SELECT  ... FROM ...  [LIMIT [offset,] row_count]
SELECT  ... FROM ...  [LIMIT row_count][ OFFSET offset]

Result set window

By default, Manticore Search uses a result set window of 1000 best-ranked documents that can be returned in the result set. If the result set is paginated beyond this value, the query will end in error.

This limitation can be adjusted with the query option max_matches.

Increasing the max_matches to very high values should only be done if it's necessary for the navigation to reach such points. A high max_matches value requires more memory and can increase the query response time. One way to work with deep result sets is to set max_matches as the sum of the offset and limit.

Lowering max_matches below 1000 has the benefit of reducing the memory used by the query. It can also reduce the query time, but in most cases, it might not be a noticeable gain.

‹›
  • SQL
  • JSON
📋
SELECT  ... FROM ...   OPTION max_matches=<value>

Scroll Search Option

The scroll search option provides an efficient and reliable way to paginate through large result sets. Unlike traditional offset-based pagination, scroll search offers better performance for deep pagination and provides an easier way to implement pagination.

Scrolling via SQL

Initial Query with Sorting Criteria

Start by executing an initial query with your desired sorting criteria. The only requirement is that id must be included in the ORDER BY clause to ensure consistent pagination. The query will return both your results and a scroll token for subsequent pages.

SELECT ... ORDER BY [... ,] id {ASC|DESC};
‹›
  • Initial Query Example
Initial Query Example
📋
SELECT weight(), id FROM test WHERE match('hello') ORDER BY weight() desc, id asc limit 2;
‹›
Response
+----------+------+
| weight() | id   |
+----------+------+
|     1281 |    1 |
|     1281 |    2 |
+----------+------+
2 rows in set (0.00 sec)

Retrieving the Scroll Token

After executing the initial query, retrieve the scroll token by executing the SHOW SCROLL command.

SHOW SCROLL;

Response:

| scroll_token                       |
|------------------------------------|
| <base64 encoded scroll token>      |
‹›
  • Scroll Token Example
Scroll Token Example
📋
SHOW SCROLL;
‹›
Response
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| scroll_token                                                                                                                                                                                                             |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| eyJvcmRlcl9ieV9zdHIiOiJ3ZWlnaHQoKSBkZXNjLCBpZCBhc2MiLCJvcmRlcl9ieSI6W3siYXR0ciI6IndlaWdodCgpIiwiZGVzYyI6dHJ1ZSwidmFsdWUiOjEyODEsInR5cGUiOiJpbnQifSx7ImF0dHIiOiJpZCIsImRlc2MiOmZhbHNlLCJ2YWx1ZSI6MiwidHlwZSI6ImludCJ9XX0= |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Paginated Query Using scroll

To retrieve the next page of results, include the scroll token in the subsequent query as an option. When the scroll option is provided, specifying the sort criteria is optional.

SELECT ... [ORDER BY [... ,] id {ASC|DESC}] OPTION scroll='<base64 encoded scroll token>'[, ...];

This ensures that pagination continues seamlessly, maintaining the sorting context established in the initial query.

‹›
  • Paginated Query Example
Paginated Query Example
📋
SELECT weight(), id FROM test WHERE match('hello') limit 2
OPTION scroll='eyJvcmRlcl9ieV9zdHIiOiJ3ZWlnaHQoKSBkZXNjLCBpZCBhc2MiLCJvcmRlcl9ieSI6W3siYXR0ciI6IndlaWdodCgpIiwiZGVzYyI6dHJ1ZSwidmFsdWUiOjEyODEsInR5cGUiOiJpbnQifSx7ImF0dHIiOiJpZCIsImRlc2MiOmZhbHNlLCJ2YWx1ZSI6MiwidHlwZSI6ImludCJ9XX0=';
‹›
Response
+----------+------+
| weight() | id   |
+----------+------+
|     1281 |    3 |
|     1281 |    4 |
+----------+------+
2 rows in set (0.00 sec)

Scrolling via JSON

Initial Request

In the initial request, specify "scroll": true in the options and the desired sorting criteria. Note that id must be present in the sort array. The response will include a scroll token, which can be used for pagination in subsequent requests.

POST /search
{ 
  "table": "<table_names>",
  "options": {
      "scroll": true
  }, 

  ...

  "sort": [
    ...
    { "id":{ "order":"{asc|desc}"} }
  ]
}

Example output:

{
    "timed_out": false,
    "hits": {

        ...

    },
    "scroll": "<base64 encoded scroll token>"
}
‹›
  • Initial Request Example
Initial Request Example
📋
POST /search
{ 
  "table": "test",
  "options":
  {
    "scroll": true
  },
  "query":
  {  
    "query_string":"hello"
  },
  "sort":
  [
    { "_score":{ "order":"desc"} },
    { "id":{ "order":"asc"} }
  ],
  "track_scores": true,
  "limit":2
}
‹›
Response
{
  "took": 0,
  "timed_out": false,
  "hits":
  {
    "total": 10,
    "total_relation": "eq",
    "hits":
    [
      {
        "_id": 1,
        "_score": 1281,
        "_source":
        {
          "title": "hello world1"
        }
      },
      {
        "_id": 2,
        "_score": 1281,
        "_source":
        {
          "title": "hello world2"
        }
      }
    ]
  },
  "scroll": "eyJvcmRlcl9ieV9zdHIiOiJAd2VpZ2h0IGRlc2MsIGlkIGFzYyIsIm9yZGVyX2J5IjpbeyJhdHRyIjoid2VpZ2h0KCkiLCJkZXNjIjp0cnVlLCJ2YWx1ZSI6MTI4MSwidHlwZSI6ImludCJ9LHsiYXR0ciI6ImlkIiwiZGVzYyI6ZmFsc2UsInZhbHVlIjoyLCJ0eXBlIjoiaW50In1dfQ=="
}

Paginated Request Using scroll

To continue pagination, include the scroll token obtained from the previous response within the options object of the next request. Specifying the sort criteria is optional.

POST /search
{ 
  "table": "<table_names>",
  "options": {
    "scroll": "<base64 encoded scroll token>"
  },

  ...

}
‹›
  • Paginated Request Example
Paginated Request Example
📋
POST /search
{ 
  "table": "test",
  "options":
  {
    "scroll": "eyJvcmRlcl9ieV9zdHIiOiJAd2VpZ2h0IGRlc2MsIGlkIGFzYyIsIm9yZGVyX2J5IjpbeyJhdHRyIjoid2VpZ2h0KCkiLCJkZXNjIjp0cnVlLCJ2YWx1ZSI6MTI4MSwidHlwZSI6ImludCJ9LHsiYXR0ciI6ImlkIiwiZGVzYyI6ZmFsc2UsInZhbHVlIjoyLCJ0eXBlIjoiaW50In1dfQ=="
  },
  "query":
  {  
    "query_string":"hello"
  },
  "track_scores": true,
  "limit":2
}
‹›
Response
{
  "took": 0,
  "timed_out": false,
  "hits":
  {
    "total": 8,
    "total_relation": "eq",
    "hits":
   [
      {
        "_id": 3,
        "_score": 1281,
        "_source":
        {
          "title": "hello world3"
        }
      },
      {
        "_id": 4,
        "_score": 1281,
        "_source":
        {
          "title": "hello world4"
        }
      }
    ]
  },
  "scroll": "eyJvcmRlcl9ieV9zdHIiOiJAd2VpZ2h0IGRlc2MsIGlkIGFzYyIsIm9yZGVyX2J5IjpbeyJhdHRyIjoid2VpZ2h0KCkiLCJkZXNjIjp0cnVlLCJ2YWx1ZSI6MTI4MSwidHlwZSI6ImludCJ9LHsiYXR0ciI6ImlkIiwiZGVzYyI6ZmFsc2UsInZhbHVlIjo0LCJ0eXBlIjoiaW50In1dfQ=="
}
Sorting and ranking Distributed searching