Manticore doesn't store text exactly as it is for full-text searching. Instead, it breaks the text into words (called tokens) and builds several internal structures to enable fast full-text searches. These structures include a dictionary that helps quickly check if a word exists in the index. Other structures track which documents and fields contain the word, and even where exactly in the field it appears. These are all used during a search to find relevant results.

The process of splitting and handling text like this is called tokenization. Tokenization happens both when adding data to the index and when running a search. It works at both the character and word level.

At the character level, only certain characters are allowed. This is controlled by the charset_table. Any other characters are replaced with a space (which is treated as a word separator). The charset_table also supports things like turning characters into lowercase or replacing one character with another. It can also define characters to be ignored, blended, or treated as a phrase boundary.

At the word level, the engine uses the min_word_len setting to decide the minimum word length (in characters) that should be indexed.

Manticore also supports matching words with different forms. For example, to treat "car" and "cars" as the same word, you can use morphology processors.

If you want different words to be treated as the same—for example, "USA" and "United States" — you can define them using the word forms feature.

Very common words (like "the", "and", "is") can slow down searches and increase index size. You can filter them out using stop words. This can make searches faster and the index smaller.

A more advanced filtering method is bigrams, which creates special tokens by combining a common word with an uncommon one. This can significantly speed up phrase searches when common words are involved.

If you're indexing HTML, it's usually best not to include the HTML tags in the index, since they add a lot of unnecessary content. You can use HTML stripping to remove the tags, but still index certain tag attributes or skip specific elements entirely.

Keep in mind that Manticore has a maximum token length of 42 bytes after token normalization. Any token longer than this will be truncated. This limit applies during both indexing and searching, so both indexed data and search queries are affected.

Tables that use dict=keywords_32k are the exception. They can index normalized tokens up to 32768 bytes in plain and RT tables. Tokens longer than 32768 bytes are skipped with a warning instead of being indexed as truncated terms.

Long keywords_32k tokens are indexed as original tokens. Tokens longer than 42 bytes after normalization do not go through stemming or lemmatization. Tokens up to 42 bytes can still use the configured morphology.

Snippets and highlighting still use the regular token limit. Tokens up to 42 bytes can be highlighted; longer keywords_32k tokens are skipped by snippet/highlight processing.

Manticore supports a wide range of languages, with basic support enabled for most languages via charset_table = non_cont (which is the default value). The non_cjk option which is an alias for non_cont can be used as well: charset_table = non_cjk.

For many languages, Manticore provides a stopwords file that can be used to improve search relevance.

Additionally, advanced morphology is available for a few languages that can significantly improve search relevance by using dictionary-based lemmatization or stemming algorithms for better segmentation and normalization.

The table below lists all supported languages and indicates how to enable:

• basic support (column "Supported")
• stopwords (column "Stopwords file name")
• advanced morphology (column "Advanced morphology")

The default non_cont charset maps German umlauts to their ASCII base letters and maps ß to s before morphology runs. To let German AOT morphology receive these characters, add the built-in german alias after non_cont:

charset_table = non_cont, german

The alias preserves ä, ö, ü, and ß, and maps their uppercase variants to lowercase, including ẞ to ß. It changes only the tokenizer's character mappings; it does not select a morphology mode automatically.

A v2 configuration therefore includes both the morphology value and a charset that retains German characters, for example:

morphology = lemmatize_de_v2
charset_table = non_cont, german

The v2 variants normalize ß and ẞ to ss before dictionary lookup and also normalize ß contained in generated dictionary lemmas to ss. With v2, forms such as Straße/Strasse/STRAẞE and GUSSE/GÜSSE/GUß/GUSS are equivalent in ordinary whole-word searches. This intentionally also removes distinctions such as Maße versus Masse for ordinary full-text matching.

With index_exact_words=1, Manticore also stores the tokenized form from before morphology. Exact-word queries such as MATCH('=straße') and MATCH('=strasse') can therefore distinguish the ß and ss forms. Exact words still follow charset_table mappings: with the german alias, for example, Straße and STRAẞE both have the exact form straße because case is folded before morphology.

Existing tables that keep lemmatize_de or lemmatize_de_all require no action after a daemon upgrade. Migrating an existing table to either v2 value changes its generated terms, so rebuild a plain table or replay all documents into a new RT table before serving it. Changing the morphology of a populated RT table, or using merge, attach, or optimize, does not convert already indexed terms. Existing dictionary-settings checks can reject an attach between tables with different morphology values where those checks apply, but they are not a migration mechanism.

Both v2 values use the same de.pak as the existing values; no language-pack update is required. Daemons that predate the v2 option names cannot load a v2 table, so rolling back a v2 table requires restoring or rebuilding a table with an existing morphology value. Tables that keep lemmatize_de or lemmatize_de_all are unaffected by this limitation.

Manticore provides built-in support for indexing languages with continuous scripts (i.e., languages that do not use spaces or other marks between words or sentences). This allows you to process texts in these languages in two different ways:

CREATE TABLE products(title text, price float) charset_table = 'cont' morphology = 'icu_chinese'

POST /cli -d "
CREATE TABLE products(title text, price float) charset_table = 'cont' morphology = 'icu_chinese'"

$index = new \Manticoresearch\Index($client);
$index->setName('products');
$index->create([
            'title'=>['type'=>'text'],
            'price'=>['type'=>'float']
        ],[
            'charset_table' => 'cont',
            'morphology' => 'icu_chinese'
        ]);

utilsApi.sql('CREATE TABLE products(title text, price float) charset_table = \'cont\' morphology = \'icu_chinese\'')

await utilsApi.sql('CREATE TABLE products(title text, price float) charset_table = \'cont\' morphology = \'icu_chinese\'')

res = await utilsApi.sql('CREATE TABLE products(title text, price float) charset_table = \'cont\' morphology = \'icu_chinese\'');

utilsApi.sql("CREATE TABLE products(title text, price float) charset_table = 'cont' morphology = 'icu_chinese'", true);

utilsApi.Sql("CREATE TABLE products(title text, price float) charset_table = 'cont' morphology = 'icu_chinese'", true);

utils_api.sql("CREATE TABLE products(title text, price float) charset_table = 'cont' morphology = 'icu_chinese'", Some(true)).await;

table products {
  charset_table = cont
  morphology = icu_chinese
  type = rt
  path = tbl
  rt_field = title
  rt_attr_uint = price
}

CREATE TABLE products(title text, price float) charset_table = 'cont' morphology = 'jieba_chinese'

POST /cli -d "
CREATE TABLE products(title text, price float) charset_table = 'cont' morphology = 'jieba_chinese'"

$index = new \Manticoresearch\Index($client);
$index->setName('products');
$index->create([
            'title'=>['type'=>'text'],
            'price'=>['type'=>'float']
        ],[
            'charset_table' => 'cont',
            'morphology' => 'jieba_chinese'
        ]);

utilsApi.sql('CREATE TABLE products(title text, price float) charset_table = \'cont\' morphology = \'jieba_chinese\'')

await utilsApi.sql('CREATE TABLE products(title text, price float) charset_table = \'cont\' morphology = \'jieba_chinese\'')

res = await utilsApi.sql('CREATE TABLE products(title text, price float) charset_table = \'cont\' morphology = \'jieba_chinese\'');

utilsApi.sql("CREATE TABLE products(title text, price float) charset_table = 'cont' morphology = 'jieba_chinese'", true);

utilsApi.Sql("CREATE TABLE products(title text, price float) charset_table = 'cont' morphology = 'jieba_chinese'", true);

utils_api.sql("CREATE TABLE products(title text, price float) charset_table = 'cont' morphology = 'jieba_chinese'", Some(true)).await;

table products {
  charset_table = cont
  morphology = jieba_chinese
  type = rt
  path = tbl
  rt_field = title
  rt_attr_uint = price
}

CREATE TABLE products(title text, price float) charset_table = 'non_cont' ngram_len = '1' ngram_chars = 'cont'
/* Or, alternatively */
CREATE TABLE products(title text, price float) charset_table = 'non_cont' ngram_len = '1' ngram_chars = 'cjk,thai'

POST /cli -d "
CREATE TABLE products(title text, price float) charset_table = 'non_cont' ngram_len = '1' ngram_chars = 'cont'"
/* Or, alternatively */
POST /cli -d "
CREATE TABLE products(title text, price float) charset_table = 'non_cont' ngram_len = '1' ngram_chars = 'cjk,thai'"

$index = new \Manticoresearch\Index($client);
$index->setName('products');
$index->create([
            'title'=>['type'=>'text'],
            'price'=>['type'=>'float']
        ],[
             'charset_table' => 'non_cont',
             'ngram_len' => '1',
             'ngram_chars' => 'cont'
        ]);

utilsApi.sql('CREATE TABLE products(title text, price float) charset_table = \'non_cont\' ngram_len = \'1\' ngram_chars = \'cont\'')

await utilsApi.sql('CREATE TABLE products(title text, price float) charset_table = \'non_cont\' ngram_len = \'1\' ngram_chars = \'cont\'')

res = await utilsApi.sql('CREATE TABLE products(title text, price float) charset_table = \'non_cont\' ngram_len = \'1\' ngram_chars = \'cont\'');

utilsApi.sql("CREATE TABLE products(title text, price float) charset_table = 'non_cont' ngram_len = '1' ngram_chars = 'cont'", true);

utilsApi.Sql("CREATE TABLE products(title text, price float) charset_table = 'non_cont' ngram_len = '1' ngram_chars = 'cont'", true);

utils_api.sql("CREATE TABLE products(title text, price float) charset_table = 'non_cont' ngram_len = '1' ngram_chars = 'cont'", Some(true)).await;

table products {
  charset_table = non_cont
  ngram_len = 1
  ngram_chars = cont
  type = rt
  path = tbl
  rt_field = title
  rt_attr_uint = price
}

CREATE TABLE products(title text, price float) charset_table = 'chinese' morphology = 'icu_chinese' stopwords = 'zh'

POST /cli -d "
CREATE TABLE products(title text, price float) charset_table = 'chinese' morphology = 'icu_chinese' stopwords = 'zh'"

$index = new \Manticoresearch\Index($client);
$index->setName('products');
$index->create([
            'title'=>['type'=>'text'],
            'price'=>['type'=>'float']
        ],[
            'charset_table' => 'chinese',
            'morphology' => 'icu_chinese',
            'stopwords' => 'zh'
        ]);

utilsApi.sql('CREATE TABLE products(title text, price float) charset_table = \'chinese\' morphology = \'icu_chinese\' stopwords = \'zh\'')

await utilsApi.sql('CREATE TABLE products(title text, price float) charset_table = \'chinese\' morphology = \'icu_chinese\' stopwords = \'zh\'')

res = await utilsApi.sql('CREATE TABLE products(title text, price float) charset_table = \'chinese\' morphology = \'icu_chinese\' stopwords = \'zh\'');

utilsApi.sql("CREATE TABLE products(title text, price float) charset_table = 'chinese' morphology = 'icu_chinese' stopwords = 'zh'", true);

utilsApi.Sql("CREATE TABLE products(title text, price float) charset_table = 'chinese' morphology = 'icu_chinese' stopwords = 'zh'", true);

utils_api.sql("CREATE TABLE products(title text, price float) charset_table = 'chinese' morphology = 'icu_chinese' stopwords = 'zh'", Some(true)).await;

table products {
  charset_table = chinese
  morphology = icu_chinese
  stopwords = zh
  type = rt
  path = tbl
  rt_field = title
  rt_attr_uint = price
}