Highlighting enables you to obtain highlighted text fragments (referred to as snippets) from documents containing matching keywords.
The SQL HIGHLIGHT() function, the "highlight" property in JSON queries via HTTP, and the highlight() function in the PHP client all utilize the built-in document storage to retrieve the original field contents (enabled by default).
- SQL
- JSON
- PHP
- Python
- Javascript
- Java
- C#
- TypeScript
- Go
SELECT HIGHLIGHT() FROM books WHERE MATCH('try');+----------------------------------------------------------+
| highlight() |
+----------------------------------------------------------+
| Don`t <strong>try</strong> to compete in childishness, said Bliss. |
+----------------------------------------------------------+
1 row in set (0.00 sec)When using SQL for highlighting search results, you will receive snippets from various fields combined into a single string due to the limitations of the MySQL protocol. You can adjust the concatenation separators with the field_separator and snippet_separator options, as detailed below.
When executing JSON queries through HTTP or using the PHP client, there are no such constraints, and the result set includes an array of fields containing arrays of snippets (without separators).
Keep in mind that snippet generation options like limit, limit_words, and limit_snippets apply to each field individually by default. You can alter this behavior using the limits_per_field option, but it could lead to unwanted results. For example, one field may have matching keywords, but no snippets from that field are included in the result set because they didn't rank as high as snippets from other fields in the highlighting engine.
The highlighting algorithm currently prioritizes better snippets (with closer phrase matches) and then snippets with keywords not yet included in the result. Generally, it aims to highlight the best match for the query and to highlight all query keywords, as allowed by the limits. If no matches are found in the current field, the beginning of the document will be trimmed according to the limits and returned by default. To return an empty string instead, set the allow_empty option to 1.
Highlighting is performed during the so-called post limit stage, which means that snippet generation is deferred not only until the entire final result set is prepared but also after the LIMIT clause is applied. For instance, with a LIMIT 20,10 clause, the HIGHLIGHT() function will be called a maximum of 10 times.
There are several optional highlighting options that can be used to fine-tune snippet generation, which are common to SQL, HTTP, and PHP clients.
A string to insert before a keyword match. The %SNIPPET_ID% macro can be used in this string. The first occurrence of the macro is replaced with an incrementing snippet number within the current snippet. Numbering starts at 1 by default but can be overridden with the start_snippet_id option. %SNIPPET_ID% restarts at the beginning of each new document. The default is <strong>.
A string to insert after a keyword match. The default is </strong>.
The maximum snippet size, in symbols (codepoints). The default is 256. This is applied per-field by default, see limits_per_field.
Limits the maximum number of words that can be included in the result. Note that this limit applies to all words, not just the matched keywords to highlight. For example, if highlighting Mary and a snippet Mary had a little lamb is selected, it contributes 5 words to this limit, not just 1. The default is 0 (no limit). This is applied per-field by default, see limits_per_field.
Limits the maximum number of snippets that can be included in the result. The default is 0 (no limit). This is applied per-field by default, see limits_per_field.
Determines whether limit, limit_words, and limit_snippets operate as individual limits in each field of the document being highlighted or as global limits for the entire document. Setting this option to 0 means that all combined highlighting results for one document must be within the specified limits. The downside is that you may have several snippets highlighted in one field and none in another if the highlighting engine decides they are more relevant. The default is 1 (use per-field limits).
The number of words to select around each matching keyword block. The default is 5.
Determines whether to additionally break snippets by phrase boundary characters, as configured in table settings with the phrase_boundary directive. The default is 0 (don't use boundaries).
Specifies whether to sort the extracted snippets in order of relevance (decreasing weight) or in order of appearance in the document (increasing position). The default is 0 (don't use weight order).
Ignores the length limit until the result includes all keywords. The default is 0 (don't force all keywords).
Sets the starting value of the %SNIPPET_ID% macro (which is detected and expanded in before_match, after_match strings). The default is 1.
Defines the HTML stripping mode setting. Defaults to index, meaning that table settings will be used. Other values include none and strip, which forcibly skip or apply stripping regardless of table settings; and retain, which retains HTML markup and protects it from highlighting. The retain mode can only be used when highlighting full documents and therefore requires that no snippet size limits are set. The allowed string values are none, strip, index, and retain.
Permits an empty string to be returned as the highlighting result when no snippets could be generated in the current field (no keyword match or no snippets fit the limit). By default, the beginning of the original text would be returned instead of an empty string. The default is 0 (don't allow an empty result).
Ensures that snippets do not cross a sentence, paragraph, or zone boundary (when used with a table that has the respective indexing settings enabled). The allowed values are sentence, paragraph, and zone.
Emits an HTML tag with the enclosing zone name before each snippet. The default is 0 (don't emit zone names).
Determines whether to force snippet generation even if limits allow highlighting the entire text. The default is 0 (don't force snippet generation).
- SQL
- JSON
- PHP
- Python
- Javascript
- Java
- C#
- TypeScript
- Go
SELECT HIGHLIGHT({limit=50}) FROM books WHERE MATCH('try|gets|down|said');+---------------------------------------------------------------------------+
| highlight({limit=50}) |
+---------------------------------------------------------------------------+
| ... , "It <strong>gets</strong> infantile pleasure ... to knock it <strong>down</strong>." |
| Don`t <strong>try</strong> to compete in childishness, <strong>said</strong> Bliss. |
| ... a small room. Bander <strong>said</strong>, "Come, half-humans, I ... |
+---------------------------------------------------------------------------+
3 rows in set (0.00 sec)The HIGHLIGHT() function can be used to highlight search results. Here's the syntax:
HIGHLIGHT([options], [field_list], [query] )By default, it works with no arguments.
- SQL
SELECT HIGHLIGHT() FROM books WHERE MATCH('before');+-----------------------------------------------------------+
| highlight() |
+-----------------------------------------------------------+
| A door opened <strong>before</strong> them, revealing a small room. |
+-----------------------------------------------------------+
1 row in set (0.00 sec)HIGHLIGHT() retrieves all available full-text fields from document storage and highlights them against the provided query. Field syntax in queries is supported. Field text is separated by field_separator, which can be modified in the options.
- SQL
SELECT HIGHLIGHT() FROM books WHERE MATCH('@title one');+-----------------+
| highlight() |
+-----------------+
| Book <strong>one</strong> |
+-----------------+
1 row in set (0.00 sec)Optional first argument in HIGHLIGHT() is the list of options.
- SQL
SELECT HIGHLIGHT({before_match='[match]',after_match='[/match]'}) FROM books WHERE MATCH('@title one');+------------------------------------------------------------+
| highlight({before_match='[match]',after_match='[/match]'}) |
+------------------------------------------------------------+
| Book [match]one[/match] |
+------------------------------------------------------------+
1 row in set (0.00 sec)The optional second argument is a string containing a single field or a comma-separated list of fields. If this argument is present, only the specified fields will be fetched from document storage and highlighted. An empty string as the second argument signifies "fetch all available fields."
- SQL
SELECT HIGHLIGHT({},'title,content') FROM books WHERE MATCH('one|robots');+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| highlight({},'title,content') |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Book <strong>one</strong> | They followed Bander. The <strong>robots</strong> remained at a polite distance, but their presence was a constantly felt threat. |
| Bander ushered all three into the room. <strong>One</strong> of the <strong>robots</strong> followed as well. Bander gestured the other <strong>robots</strong> away and entered itself. The door closed behind it. |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)Alternatively, you can use the second argument to specify a string attribute or field name without quotes. In this case, the supplied string will be highlighted against the provided query, but field syntax will be ignored.
- SQL
SELECT HIGHLIGHT({}, title) FROM books WHERE MATCH('one');+---------------------+
| highlight({},title) |
+---------------------+
| Book <strong>one</strong> |
| Book five |
+---------------------+
2 rows in set (0.00 sec)The optional third argument is the query. This is used to highlight search results against a query different from the one used for searching.
- SQL
SELECT HIGHLIGHT({},'title', 'five') FROM books WHERE MATCH('one');+-------------------------------+
| highlight({},'title', 'five') |
+-------------------------------+
| Book one |
| Book <strong>five</strong> |
+-------------------------------+
2 rows in set (0.00 sec)Although HIGHLIGHT() is designed to work with stored full-text fields and string attributes, it can also be used to highlight arbitrary text. Keep in mind that if the query contains any field search operators (e.g., @title hello @body world), the field part of them is ignored in this case.
- SQL
SELECT HIGHLIGHT({},TO_STRING('some text to highlight'), 'highlight') FROM books WHERE MATCH('@title one');+----------------------------------------------------------------+
| highlight({},TO_STRING('some text to highlight'), 'highlight') |
+----------------------------------------------------------------+
| some text to <strong>highlight</strong> |
+----------------------------------------------------------------+
1 row in set (0.00 sec)Several options are relevant only when generating a single string as a result (not an array of snippets). This applies exclusively to the SQL HIGHLIGHT() function:
A string to insert between snippets. The default is ....
A string to insert between fields. The default is |.
Another way to highlight text is to use the CALL SNIPPETS statement. This mostly duplicates the HIGHLIGHT() functionality but cannot use built-in document storage. However, it can load source text from files.
To highlight full-text search results in JSON queries via HTTP, field contents must be stored in document storage (enabled by default). In the example, full-text fields content and title are fetched from document storage and highlighted against the query specified in the query clause.
Highlighted snippets are returned in the highlight property of the hits array.
- JSON
- PHP
- Python
- Javascript
- Java
- C#
- TypeScript
- Go
POST /search
{
"table": "books",
"query": { "match": { "*": "one|robots" } },
"highlight":
{
"fields": ["content"]
}
}{
"took": 0,
"timed_out": false,
"hits": {
"total": 1,
"hits": [
{
"_id": 1,
"_score": 2788,
"_source": {
"title": "Books one",
"content": "They followed Bander. The robots remained at a polite distance, but their presence was a constantly felt threat. Bander ushered all three into the room. One of the robots followed as well. Bander gestured the other robots away and entered itself. The door closed behind it. "
},
"highlight": {
"content": [
"They followed Bander. The <strong>robots</strong> remained at a polite distance, ",
" three into the room. <strong>One</strong> of the <strong>robots</strong> followed as well. Bander",
" gestured the other <strong>robots</strong> away and entered itself. The"
]
}
}
]
}
}To highlight all possible fields, pass an empty object as the highlight property.
- JSON
- PHP
- Python
- Javascript
- Java
- C#
- TypeScript
- Go
POST /search
{
"table": "books",
"query": { "match": { "*": "one|robots" } },
"highlight": {}
}{
"took": 0,
"timed_out": false,
"hits": {
"total": 1,
"hits": [
{
"_id": 1,
"_score": 2788,
"_source": {
"title": "Books one",
"content": "They followed Bander. The robots remained at a polite distance, but their presence was a constantly felt threat. Bander ushered all three into the room. One of the robots followed as well. Bander gestured the other robots away and entered itself. The door closed behind it. "
},
"highlight": {
"title": [
"Books <strong>one</strong>"
],
"content": [
"They followed Bander. The <strong>robots</strong> remained at a polite distance, ",
" three into the room. <strong>One</strong> of the <strong>robots</strong> followed as well. Bander",
" gestured the other <strong>robots</strong> away and entered itself. The"
]
}
}
]
}
}In addition to common highlighting options, several synonyms are available for JSON queries via HTTP:
The fields object contains attribute names with options. It can also be an array of field names (without any options).
Note that by default, highlighting attempts to highlight the results following the full-text query. In a general case, when you don't specify fields to highlight, the highlight is based on your full-text query. However, if you specify fields to highlight, it highlights only if the full-text query matches the selected fields.
The encoder can be set to default or html. When set to html, it retains HTML markup when highlighting. This works similarly to the html_strip_mode=retain option.
The highlight_query option allows you to highlight against a query other than your search query. The syntax is the same as in the main query.
- JSON
- PHP
- Python
- Javascript
- Java
- C#
- TypeScript
- Go
POST /search
{
"table": "books",
"query": { "match": { "content": "one|robots" } },
"highlight":
{
"fields": [ "content"],
"highlight_query": { "match": { "*":"polite distance" } }
}
}{'aggregations': None,
'hits': {'hits': [{u'_id': u'1',
u'_score': 1788,
u'_source': {u'content': u'They followed Bander. The robots remained at a polite distance, but their presence was a constantly felt threat. Bander ushered all three into the room. One of the robots followed as well. Bander gestured the other robots away and entered itself. The door closed behind it. ',
u'title': u'Books one'},
u'highlight': {u'content': [u'. The robots remained at a <strong>polite distance</strong>, but their presence was a']}}],
'max_score': None,
'total': 1},
'profile': None,
'timed_out': False,
'took': 0}pre_tags and post_tags set the opening and closing tags for highlighted text snippets. They function similarly to the before_match and after_match options. These are optional, with default values of <strong> and </strong>.
- JSON
- PHP
- Python
- Javascript
- Java
- C#
- TypeScript
- Go
POST /search
{
"table": "books",
"query": { "match": { "*": "one|robots" } },
"highlight":
{
"fields": [ "content", "title" ],
"pre_tags": "before_",
"post_tags": "_after"
}
}Document: 1
title : Books one
content : They followed Bander. The robots remained at a polite distance, but their presence was a constantly felt threat. Bander ushered all three into the room. One of the robots followed as well. Bander gestured the other robots away and entered itself. The door closed behind it.
Highlight for content:
- They followed Bander. The before_robots_after remained at a polite distance,
- three into the room. before_One_after of the before_robots_after followed as well. Bander
- gestured the other before_robots_after away and entered itself. The
Highlight for title:
- Books before_one_after
no_match_size functions similarly to the allow_empty option. If set to 0, it acts as allow_empty=1, allowing an empty string to be returned as a highlighting result when a snippet could not be generated. Otherwise, the beginning of the field will be returned. This is optional, with a default value of 1.
- JSON
- PHP
- Python
- Javascript
- Java
- C#
- TypeScript
- Go
POST /search
{
"table": "books",
"query": { "match": { "*": "one|robots" } },
"highlight":
{
"fields": [ "content", "title" ],
"no_match_size": 0
}
}Document: 1
title : Books one
content : They followed Bander. The robots remained at a polite distance, but their presence was a constantly felt threat. Bander ushered all three into the room. One of the robots followed as well. Bander gestured the other robots away and entered itself. The door closed behind it.
Highlight for content:
- They followed Bander. The <strong>robots</strong> remained at a polite distance,
- three into the room. <strong>One</strong> of the <strong>robots</strong> followed as well. Bander
- gestured the other <strong>robots</strong> away and entered itself. The
Highlight for title:
- Books <strong>one</strong>order sets the sorting order of extracted snippets. If set to "score", it sorts the extracted snippets in order of relevance. This is optional and works similarly to the weight_order option.
- JSON
- PHP
- Python
- Javascript
- Java
- C#
- TypeScript
- Go
POST /search
{
"table": "books",
"query": { "match": { "*": "one|robots" } },
"highlight":
{
"fields": [ "content", "title" ],
"order": "score"
}
}Document: 1
title : Books one
content : They followed Bander. The robots remained at a polite distance, but their presence was a constantly felt threat. Bander ushered all three into the room. One of the robots followed as well. Bander gestured the other robots away and entered itself. The door closed behind it.
Highlight for content:
- three into the room. <strong>One</strong> of the <strong>robots</strong> followed as well. Bander
- gestured the other <strong>robots</strong> away and entered itself. The
- They followed Bander. The <strong>robots</strong> remained at a polite distance,
Highlight for title:
- Books <strong>one</strong>fragment_size sets the maximum snippet size in symbols. It can be global or per-field. Per-field options override global options. This is optional, with a default value of 256. It works similarly to the limit option.
- JSON
- PHP
- Python
- Javascript
- Java
- C#
- TypeScript
- Go
POST /search
{
"table": "books",
"query": { "match": { "*": "one|robots" } },
"highlight":
{
"fields": [ "content", "title" ],
"fragment_size": 100
}
}Document: 1
title : Books one
content : They followed Bander. The robots remained at a polite distance, but their presence was a constantly felt threat. Bander ushered all three into the room. One of the robots followed as well. Bander gestured the other robots away and entered itself. The door closed behind it.
Highlight for content:
- the room. <strong>One</strong> of the <strong>robots</strong> followed as well
- Bander gestured the other <strong>robots</strong> away and entered
Highlight for title:
- Books <strong>one</strong>number_of_fragments limits the maximum number of snippets in the result. Like fragment_size, it can be global or per-field. This is optional, with a default value of 0 (no limit). It works similarly to the limit_snippets option.
- JSON
- PHP
- Python
- Javascript
- Java
- C#
- TypeScript
- Go
POST /search
{
"table": "books",
"query": { "match": { "*": "one|robots" } },
"highlight":
{
"fields": [ "content", "title" ],
"number_of_fragments": 10
}
}Document: 1
title : Books one
content : They followed Bander. The robots remained at a polite distance, but their presence was a constantly felt threat. Bander ushered all three into the room. One of the robots followed as well. Bander gestured the other robots away and entered itself. The door closed behind it.
Highlight for content:
- They followed Bander. The <strong>robots</strong> remained at a polite distance,
- three into the room. <strong>One</strong> of the <strong>robots</strong> followed as well. Bander
- gestured the other <strong>robots</strong> away and entered itself. The
Highlight for title:
- Books <strong>one</strong>
Options like limit, limit_words, and limit_snippets can be set as global or per-field options. Global options are used as per-field limits unless per-field options override them. In the example, the title field is highlighted with default limit settings, while the content field uses a different limit.
- JSON
- PHP
- Python
- Javascript
- Java
- C#
- TypeScript
- Go
POST /search
{
"table": "books",
"query": { "match": { "*": "one|robots" } },
"highlight":
{
"fields":
{
"title": {},
"content" : { "limit": 50 }
}
}
}Document: 1
title : Books one
content : They followed Bander. The robots remained at a polite distance, but their presence was a constantly felt threat. Bander ushered all three into the room. One of the robots followed as well. Bander gestured the other robots away and entered itself. The door closed behind it.
Highlight for content:
- into the room. <strong>One</strong> of the <strong>robots</strong> followed as well
Highlight for title:
- Books <strong>one</strong>Global limits can also be enforced by specifying limits_per_field=0. Setting this option means that all combined highlighting results must be within the specified limits. The downside is that you may get several snippets highlighted in one field and none in another if the highlighting engine decides that they are more relevant.
- JSON
- PHP
- Python
- Javascript
- Java
- C#
- TypeScript
- Go
POST /search
{
"table": "books",
"query": { "match": { "content": "and first" } },
"highlight":
{
"limits_per_field": false,
"fields":
{
"content" : { "limit": 50 }
}
}
}Document: 1
title : Books one
content : They followed Bander. The robots remained at a polite distance, but their presence was a constantly felt threat. Bander ushered all three into the room. One of the robots followed as well. Bander gestured the other robots away and entered itself. The door closed behind it.
Highlight for content:
- gestured the other robots away <strong>and</strong> entered itself. The door closedThe CALL SNIPPETS statement builds a snippet from provided data and query using specified table settings. It can't access built-in document storage, which is why it's recommended to use the HIGHLIGHT() function instead.
The syntax is:
CALL SNIPPETS(data, table, query[, opt_value AS opt_name[, ...]])data serves as the source from which a snippet is extracted. It can either be a single string or a list of strings enclosed in curly brackets.
table refers to the name of the table that provides the text processing settings for snippet generation.
query is the full-text query used to build the snippets.
opt_value and opt_name represent the snippet generation options.
- SQL
CALL SNIPPETS(('this is my document text','this is my another text'), 'forum', 'is text', 5 AS around, 200 AS limit);+----------------------------------------+
| snippet |
+----------------------------------------+
| this <strong>is</strong> my document <strong>text</strong> |
| this <strong>is</strong> my another <strong>text</strong> |
+----------------------------------------+
2 rows in set (0.02 sec)Most options are the same as in the HIGHLIGHT() function. There are, however, several options that can only be used with CALL SNIPPETS.
The following options can be used to highlight text stored in separate files:
This option, when enabled, treats the first argument as file names instead of data to extract snippets from. The specified files on the server side will be loaded for data. Up to max_threads_per_query worker threads per request will be used to parallelize the work when this flag is enabled. Default is 0 (no limit). To distribute snippet generation between remote agents, invoke snippets generation in a distributed table containing only one(!) local agent and several remotes. The snippets_file_prefix option is used to generate the final file name. For example, when searchd is configured with snippets_file_prefix = /var/data_ and text.txt is provided as a file name, snippets will be generated from the content of /var/data_text.txt.
This option only works with distributed snippets generation with remote agents. Source files for snippet generation can be distributed among different agents, and the main server will merge all non-erroneous results. For example, if one agent of the distributed table has file1.txt, another agent has file2.txt, and you use CALL SNIPPETS with both of these files, searchd will merge agent results, so you will get results from both file1.txt and file2.txt. Default is 0.
If the load_files option is also enabled, the request will return an error if any of the files is not available anywhere. Otherwise (if load_files is not enabled), it will return empty strings for all absent files. Searchd does not pass this flag to agents, so agents do not generate a critical error if the file does not exist. If you want to be sure that all source files are loaded, set both load_files_scattered and load_files to 1. If the absence of some source files on some agent is not critical, set only load_files_scattered to 1.
- SQL
CALL SNIPPETS(('data/doc1.txt','data/doc2.txt'), 'forum', 'is text', 1 AS load_files);+----------------------------------------+
| snippet |
+----------------------------------------+
| this <strong>is</strong> my document <strong>text</strong> |
| this <strong>is</strong> my another <strong>text</strong> |
+----------------------------------------+
2 rows in set (0.02 sec)