- Plain
- Real-time
table <table name> {
type = plain
path = /path/to/table
source = <source_name>
source = <another source_name>
[stored_fields = <comma separated list of full-text fields that should be stored, all are stored by default, can be empty>]
}
type = plain
type = rt
Table type: "plain" or "rt" (real-time)
Value: plain (default), rt
path = path/to/table
Absolute or relative path without extension where to store the table or where to look for it
Value: path to the table, mandatory
stored_fields = title, content
By default when a table is defined in a configuration file, full-text fields' original content is both indexed and stored. This setting lets you specify the fields that should have their original values stored.
Value: comma separated list of full-text fields that should be stored. Empty value (i.e. stored_fields =
) disables storing original values for all the fields.
Note, in case of a real-time table the fields listed in stored_fields
should be also declared as rt_field.
Note also, that you don't need to list attributes in stored_fields
, since their original values are stored anyway. stored_fields
can be only used for full-text fields.
See also docstore_block_size, docstore_compression for document storage compression options.
- SQL
- JSON
- PHP
- Python
- Javascript
- Java
- CONFIG
CREATE TABLE products(title text, content text stored indexed, name text indexed, price float)
stored_only_fields = title,content
List of fields that will be stored in the table, but will not be indexed. Similar to stored_fields except when a field is specified in stored_only_fields
it is only stored, not indexed and can't be searched with full-text queries. It can only be returned with search results.
Value: comma separated list of fields that should be stored only, not indexed. Default is empty. Note, in case of a real-time table the fields listed in stored_only_fields
should be also declared as rt_field.
Note also, that you don't need to list attributes in stored_only_fields
, since their original values are stored anyway. If to compare stored_only_fields
to string attributes the former (stored field):
- is stored on disk and doesn't require memory
- is stored compressed
- can be only fetched, you can't sort/filter/group by the value
The latter (string attribute) is:
- stored on disk and in memory
- stored uncompressed
- can be used for sorting, grouping, filtering and anything else you want to do with attributes.
Max number of RT table disk chunks. Read more here.
rt_field = subject
Full-text fields to be indexed. The names must be unique. The order is preserved; and so field values in INSERT
statements without an explicit list of inserted columns will have to be in the same order as configured.
Full-text field declaration. Multi-value, optional.
rt_attr_uint = gid
Unsigned integer attribute declaration
Value: field_name or field_name:N, can be multiple records. N is the max number of bits to keep.
rt_attr_bigint = gid
BIGINT attribute declaration
Value: field name, multiple records allowed
rt_attr_multi = tags
Multi-valued attribute (MVA) declaration. Declares the UNSIGNED INTEGER (unsigned 32-bit) MVA attribute. Multi-value (ie. there may be more than one such attribute declared), optional.
Value: field name, multiple records allowed.
rt_attr_multi_64 = wide_tags
Multi-valued attribute (MVA) declaration. Declares the BIGINT (signed 64-bit) MVA attribute. Multi-value (ie. there may be more than one such attribute declared), optional.
Value: field name, multiple records allowed.
rt_attr_float = lat
rt_attr_float = lon
Floating point attribute declaration. Multi-value (an arbitrary number of attributes is allowed), optional. Declares a single precision, 32-bit IEEE 754 format float attribute.
Value: field name, multiple records allowed.
rt_attr_bool = available
Boolean attribute declaration. Multi-value (there might be multiple attributes declared), optional. Declares a 1-bit unsigned integer attribute.
Value: field name, multiple records allowed.
rt_attr_string = title
String attribute declaration. Multi-value (an arbitrary number of attributes is allowed), optional.
Value: field name, multiple records allowed.
rt_attr_json = properties
JSON attribute declaration. Multi-value (ie. there may be more than one such attribute declared), optional.
Value: field name, multiple records allowed.
rt_attr_timestamp = date_added
Timestamp attribute declaration. Multi-value (an arbitrary number of attributes is allowed), optional.
Value: field name, multiple records allowed.
rt_mem_limit = 512M
RAM chunk size limit. Optional, default is 128M.
RT table keeps some data in memory ("RAM chunk") and also maintains a number of on-disk tables ("disk chunks"). This directive lets you control the RAM chunk size. Once there’s too much data to keep in RAM, RT table will flush it to disk, activate a newly created disk chunk, and reset the RAM chunk.
The limit is pretty strict: RT table never allocates more memory than it’s limited to. The memory is not preallocated either, hence, specifying 512 MB limit and only inserting 3 MB of data should result in allocating 3 MB, not 512 MB.
The rt_mem_limit
is never exceeded, but the actual RAM chunk can be significantly lower than the limit. Real-time table learns by your data insertion pace and adapts the actual limit to decrease RAM consumption and increase data write speed. How it works:
- By default RAM chunk size is 50% of
rt_mem_limit
. It's called "rt_mem_limit
rate". - As soon as RAM chunk accumulates
rt_mem_limit * rate
data (50% ofrt_mem_limit
by default) Manticore starts saving the RAM chunk as a new disk chunk. - While a new disk chunk is being saved, Manticore checks how many new/replaced documents have appeared.
- Upon saving a new disk chunk we update the
rt_mem_limit
rate. - The rate is reset to 50% as soon as you restart the searchd.
For example, if we saved 90M docs to a disk chunk and 10M more docs arrived while saving, the rate is 90%, so next time we collect up to 90% of rt_mem_limit
before starting flushing. The higher is the speed of insertion, the lower is the rt_mem_limit
rate. The rate varies in the range of 33.3% to 95%. You can see table's current rate in SHOW TABLE
In the RT mode RAM chunk size limit and max number of disk chunks can be changed using ALTER TABLE
. To set rt_mem_limit
to 1 gigabyte for table 't' run query ALTER TABLE t rt_mem_limit='1G'
. To change max number of chunks - ALTER TABLE t optimize_cutoff='5'
.
In the plain mode rt_mem_limit
and optimize_cutoff
can be changed so:
- change the value in the table configuration
- run
ALTER TABLE <index_name> RECONFIGURE
- RT table is quite similar to a distributed table consisting of multiple local tables. The local tables are called "disk chunks".
- RAM chunk internally consists of multiple "segments".
- While disk chunks are stored on disk, the segments of RAM chunk are special RAM-only "tables".
- Any transaction you make to a real-time table generates a new segment. RAM chunk segments are merged after each transaction commit. Therefore it is beneficial to do bulk INSERTs of hundreds/thousands documents rather than hundreds/thousands different inserts with 1 document to avoid the overhead from merging RAM chunk segments.
- When the number of segments gets greater than 32, the segments get merged, so the count is not greater than 32.
- RT table always has a single RAM-chunk (may be empty) and one or multiple disk chunks.
- Merging larger segments take longer, that's why it may be suboptimal to have very large RAM chunk (and therefore
rt_mem_limit
). - Number of disk chunks depends on the amount of data in the table and
rt_mem_limit
setting. - Searchd flushes RAM chunk to disk (not as a disk chunk, just persists) on shutdown and periodically according to rt_flush_period. Flushing several gigabytes to disk may take some time.
- Large RAM chunk will put more pressure on the storage:
- when flushing the RAM chunk to disk into the
.ram
file - when the RAM chunk is full and is dumped to disk as a disk chunk.
- when flushing the RAM chunk to disk into the
- Until flushed RAM chunk is not persisted on disk there's a binary log as its backup for the case of a sudden daemon shutdown. In this case the larger you have
rt_mem_limit
, the longer will it take to replay the binlog on start to recover the RAM chunk. - RAM chunk may be performing slightly slower than a disk chunk.
- Even though a RAM chunk doesn't take more memory than
rt_mem_limit
Manticore itself can take more in some cases, e.g. if you begin a transaction to insert data and don't commit it for some time, then the data you have already transmitted within the transaction to Manticore is kept in memory.
source = srcpart1
source = srcpart2
source = srcpart3
Specifies document source to get documents from when the current table is indexed. There must be at least one source. The sources can be of different types (e.g. one - mysql, another - postgresql). Read more about indexing from external storages here
Value: name of the source to build the table from, mandatory. Can be multiple records.
killlist_target = main:kl
Sets the table(s) that the kill-list will be applied to. Suppresses matches in the targeted table that are updated or deleted in the current table. In :kl
mode the documents to suppress are taken from the kill-list. In :id
mode all document ids from the current table are suppressed in the targeted one. If neither is specified the both modes take effect. Read more about kill-lists here
Value: not specified (default), target_index_name:kl, target_index_name:id, target_index_name. Multiple values are allowed
columnar_attrs = *
columnar_attrs = id, attr1, attr2, attr3
Specifies what attributes should be stored in the columnar storage instead of the default row-wise storage.
You can do columnar_attrs = *
to store fields of all supported data types in the columnar storage.
id
is also supported.
columnar_strings_no_hash = attr1, attr2, attr3
By default, all string attributes stored in columnar storage store pre-calculated hashes. These hashes are used for grouping and filtering. However, they occupy extra space, and if you don't need to group by that attribute, you can save space by disabling hash generation.
CREATE TABLE [IF NOT EXISTS] name ( <field name> <field data type> [data type options] [, ...]) [table_options]
Read more about data types here.
Type | Equivalent in a configuration file | Notes | Aliases |
---|---|---|---|
text | rt_field | Options: indexed, stored. Default - both. To keep text stored, but indexed specify "stored" only. To keep text indexed only specify only "indexed". | string |
integer | rt_attr_uint | integer | int, uint |
bigint | rt_attr_bigint | big integer | |
float | rt_attr_float | float | |
multi | rt_attr_multi | multi-integer | |
multi64 | rt_attr_multi_64 | multi-bigint | |
bool | rt_attr_bool | boolean | |
json | rt_attr_json | JSON | |
string | rt_attr_string | string. Option indexed, attribute will make the value full-text indexed and filterable, sortable and groupable at the same time |
|
timestamp | rt_attr_timestamp | timestamp | |
bit(n) | rt_attr_uint field_name:N | N is the max number of bits to keep |
- SQL
CREATE TABLE products (title text, price float) morphology='stem_en'
creates table "products" with two fields: "title" (full-text) and "price" (float) and setting "morphology" with value "stem_en"
CREATE TABLE products (title text indexed, description text stored, author text, price float)
creates table "products" with three fields:
- field "title" - indexed, but not stored
- field "description" - stored, but not indexed
- field "author" - both stored and indexed
create table ... engine='columnar';
create table ... engine='rowwise';
Changes default attribute storage for all attributes in the table. Can be overridden by specifying engine
separately for each attribute.
See columnar_attrs on how to enable columnar storage for a plain table.
Values:
- columnar - enables columnar storage for all table attributes except for json
- rowwise (default) - doesn't change anything, i.e. makes Manticore use the traditional row-wise storage for the table
The following settings are similar for both real-time and plain table in either mode: whether specified in a configuration file or online via CREATE
or ALTER
command.
Manticore uses two access modes to read table data - seek+read and mmap.
In seek+read mode the server performs system call pread
to read document lists and keyword positions, i.e. *.spd
and *.spp
files. Internal read buffers are used to optimize reading. The size of these buffers can be tuned with options read_buffer_docs and read_buffer_hits. There is also option preopen that allows to control how Manticore opens files at start.
In the mmap access mode the search server just maps table's file into memory with mmap
system call and OS caches file contents by itself. Options read_buffer_docs and read_buffer_hits have no effect for corresponding files in this mode. The mmap reader can also lock table's data in memory via mlock
privileged call which prevents swapping out the cached data to disk by OS.
To control what access mode will be used access_plain_attrs, access_blob_attrs, access_doclists and access_hitlists options are available with the following values:
Value | Description |
---|---|
file | server reads the table files from disk with seek+read using internal buffers on file access |
mmap | server maps the table files into memory and OS caches up its contents on file access |
mmap_preread | server maps the table files into memory and a background thread reads it once to warm up the cache |
mlock | server maps the table files into memory and then executes the mlock() system call to cache up the file contents and lock it into memory to prevent it being swapped out |
Setting | Values | Description |
---|---|---|
access_plain_attrs | mmap, mmap_preread (default), mlock | controls how *.spa (plain attributes) *.spe (skip lists) *.spi (word lists) *.spt (lookups) *.spm (killed docs) will be read |
access_blob_attrs | mmap, mmap_preread (default), mlock | controls how *.spb (blob attributes) (string, mva and json attributes) will be read |
access_doclists | file (default), mmap, mlock | controls how *.spd (doc lists) data will be read |
access_hitlists | file (default), mmap, mlock | controls how *.spp (hit lists) data will be read |
Here is a table which can help you select your desired mode:
table part | keep it on disk | keep it in memory | cached in memory on server start | lock it in memory |
---|---|---|---|---|
plain attributes in row-wise (non-columnar) storage, skip lists, word lists, lookups, killed docs | mmap | mmap | mmap_preread (default) | mlock |
row-wise string, multi-value attributes (MVA) and json attributes | mmap | mmap | mmap_preread (default) | mlock |
columnar numeric, string and multi-value attributes | always | only by means of OS | no | not supported |
doc lists | file (default) | mmap | no | mlock |
hit lists | file (default) | mmap | no | mlock |
- If you want the best search response time and have enough memory - use row-wise attributes and
mlock
for attributes and for doclists/hitlists - If you can't afford lower performance on start and are ready to wait longer on start until it's warmed up - use --force-preread. If you want searchd to be able to restart faster - stay with
mmap_preread
- If you want to save RAM, but still have enough RAM for all the attributes - do not use
mlock
, then your OS will decide what should be in memory at any given moment of time depending on what is read from disk more frequently - If row-wise attributes don't fit into RAM - use columnar attributes
- If full-text search performance is not a priority and you want to save RAM - use
access_doclists/access_hitlists=file
The default mode is to:
- mmap
- preread non-columnar attributes
- seek+read columnar attributes with no preread
- seek+read doclists/hitlists with no preread
which provides decent search performance, optimal memory usage and faster searchd restart in most cases.
attr_update_reserve = 256k
Sets the space to be reserved for blob attribute updates. Optional, default value is 128k. When blob attributes (multi-value attributes (MVA), strings, JSON) are updated, their length may change. If the updated string (or MVA or JSON) is shorter than the old one, it overwrites the old one in the *.spb
file. But if the updated string is longer, the updates are written to the end of the *.spb
file. This file is memory mapped, that's why resizing it may be a rather slow process, depending on the OS implementation of memory mapped files. To avoid frequent resizes, you can specify the extra space to be reserved at the end of the .spb file by using this setting.
Value: size, default 128k.
docstore_block_size = 32k
Size of the block of documents used by document storage. Optional, default is 16kb. When stored_fields or stored_only_fields are specified, original document text is stored inside the table. To use less disk space, documents are compressed. To get more efficient disk access and better compression ratios on small documents, documents are concatenated into blocks. When indexing, documents are collected until their total size reaches the threshold. After that, this block of documents is compressed. This option can be used to get better compression ratio (by increasing block size) or to get faster access to document text (by decreasing block size).
Value: size, default 16k.
docstore_compression = lz4hc
Type of compression used to compress blocks of documents used by document storage. When stored_fields or stored_only_fields are specified, document storage stores compressed document blocks. 'lz4' has fast compression and decompression speeds, 'lz4hc' (high compression) has the same fast decompression but compression speed is traded for better compression ratio. 'none' disables compression.
Value: lz4 (default), lz4hc, none.
docstore_compression_level = 12
Compression level in document storage when 'lz4hc' compression is used. When 'lz4hc' compression is used, compression level can be fine-tuned to get better performance or better compression ratio. Does not work with 'lz4' compression.
Value: 1-12 (default 9).
preopen = 1
This option tells searchd that it should pre-open all table files on startup (or rotation) and keep them open while it runs. Currently, the default mode is not to pre-open the files. Pre-opened tables take a few (currently 2) file descriptors per table. However, they save on per-query open() calls; and also they are invulnerable to subtle race conditions that may happen during table rotation under high load. On the other hand, when serving many tables (100s to 1000s), it still might be desired to open them on per-query basis in order to save file descriptors
Value: 0 (default), 1.
read_buffer_docs = 1M
Per-keyword read buffer size for document lists. The higher the value the higher per-query RAM use is, but possibly lower IO time
Value: size, default 256k, min 8k.
read_buffer_hits = 1M
Per-keyword read buffer size for hit lists. The higher the value the higher per-query RAM use is, but possibly lower IO time
Value: size, default 256k, min 8k.
inplace_enable = {0|1}
Whether to enable in-place table inversion. Optional, default is 0 (use separate temporary files).
inplace_enable
greatly reduces indexing disk footprint for a plain table, at a cost of slightly slower indexing (it uses around 2x less disk, but yields around 90-95% the original performance).
Indexing involves two major phases. The first phase collects, processes, and partially sorts documents by keyword, and writes the intermediate result to temporary files (.tmp*). The second phase fully sorts the documents, and creates the final table files. Thus, rebuilding a production table on the fly involves around 3x peak disk footprint: 1st copy for the intermediate temporary files, 2nd copy for newly constructed copy, and 3rd copy for the old table that will be serving production queries in the meantime. (Intermediate data is comparable in size to the final table.) That might be too much disk footprint for big data collections, and inplace_enable
allows to reduce it. When enabled, it reuses the temporary files, outputs the final data back to them, and renames them on completion. However, this might require additional temporary data chunk relocation, which is where the performance impact comes from.
This directive does not affect searchd in any way, it only affects indexer.
- CONFIG
table products {
inplace_enable = 1
path = products
source = src_base
}
inplace_hit_gap = size
In-place inversion fine-tuning option. Controls preallocated hitlist gap size. Optional, default is 0.
This directive does not affect searchd in any way, it only affects indexer.
- CONFIG
table products {
inplace_hit_gap = 1M
inplace_enable = 1
path = products
source = src_base
}
inplace_reloc_factor = 0.1
Controls relocation buffer size within indexing memory arena. Optional, default is 0.1.
This directive does not affect searchd in any way, it only affects indexer.
- CONFIG
table products {
inplace_reloc_factor = 0.1
inplace_enable = 1
path = products
source = src_base
}
inplace_write_factor = 0.1
Controls in-place write buffer size within indexing memory arena. Optional, default is 0.1.
This directive does not affect searchd in any way, it only affects indexer.
- CONFIG
table products {
inplace_write_factor = 0.1
inplace_enable = 1
path = products
source = src_base
}
The following settings are supported. They are all described in section NLP and tokenization.
- bigram_freq_words
- bigram_index
- blend_chars
- blend_mode
- charset_table
- dict
- embedded_limit
- exceptions
- expand_keywords
- global_idf
- hitless_words
- html_index_attrs
- html_remove_elements
- html_strip
- ignore_chars
- index_exact_words
- index_field_lengths
- index_sp
- index_token_filter
- index_zones
- infix_fields
- killlist_target
- max_substring_len
- min_infix_len
- min_prefix_len
- min_stemming_len
- min_word_len
- morphology
- morphology_skip_fields
- ngram_chars
- ngram_len
- overshort_step
- phrase_boundary
- phrase_boundary_step
- prefix_fields
- regexp_filter
- stopwords
- stopword_step
- stopwords_unstemmed
- stored_fields
- stored_only_fields
- wordforms
Percolate table is a special table which stores queries instead of documents. It is used for prospective searches (or "search in reverse").
- See section Percolate query for more details about performing a search query against a percolate table.
- See section Adding rules to a percolate table to learn how to prepare a table for searching.
The schema of a percolate table is fixed and contains the following fields:
Field | Description |
---|---|
ID | Unsigned 64-bit integer with autoincrement functionality therefore it can be omitted when you add a PQ rule |
Query | Full-text query of the rule. You can think of it as the value of a MATCH clause or JSON /search. If per field operators are used inside the query, the full text fields need to be declared in the percolate table configuration. If the stored query is supposed to do only attribute filtering (no full-text querying), the query value can be empty or simply omitted. The value of this field should correspond to the expected document schema which you specify when you create a percolate table |
Filters | Filters is an optional string containing attribute filters and/or expressions the same way they are defined in the WHERE clause or JSON filtering. The value of this field should correspond to the expected document schema which you specify when you create a percolate table |
Tags | Optional. Tags represent a list of string labels separated by comma that can be used for filtering/deleting PQ rules. The tags can be returned along with matching documents when you Percolate query |
You don't need to worry about adding the above fields when you create a percolate table.
What you need to take care of when you add a new table is to specify the expected schema of a document which is to be checked against the rules you will add later. This is done the same way as for any other local table.
Template table is a pseudo-table since it does not store any data and does not create any files on your disk. At the same time it can have the same NLP settings as a plain or a real-time table. Template tables can be used for a few purposes:
- as templates to inherit plain/real-time tables in the Plain mode just to minify Manticore configuration file
- keywords generation with help of CALL KEYWORDS
- highlighting of an arbitrary string using CALL SNIPPETS