Real-time index

Real-time index is a main type of Manticore indexes. It allows adding, updating and deleting documents with immediate availability of the changes. Real-time index settings can be defined in a configuration file or online via CREATE/UPDATE/DELETE/ALTER commands.

Real-time index internally consists of one or multiple plain indexes called chunks. There can be:

  • multiple disk chunks. They are stored on disk with the same structure as any plain index
  • single ram chunk. Stored in memory and used as an accumulator of changes

RAM chunk size is controlled by rt_mem_limit. Once the limit is exceeded the RAM chunk is flushed to disk in a form of a disk chunk. When there are too many disk chunks they can be merged into one for better performance using command OPTIMIZE.

📋
create table products(title text, price float) morphology='stem_en';
Response
Query OK, 0 rows affected (0.00 sec)
Creating a real-time index via JSON over HTTP:

👍 What you can do with a real-time index:

⛔ What you cannot do with a real-time index:

  • Index data with help of indexer
  • Link it with sources for easy indexing from external storages
  • Update it's killlist_target, it's just not needed as the real-time index takes controls of it automatically

Real-time index files structure

Extension Description
.lock lock file
.ram RAM chunk
.meta RT index headers
.*.sp* disk chunks (see plain index format)

Plain index

Plain index is a basic element for non-percolate searching. It can be specified only in a configuration file in Plain mode. It is not supported in RT mode. It's normally used together with a source to index data from an external storage and afterwards can be attached to a real-time index.

👍 What you can do with a plain index:

⛔ What you cannot do with a plain index:

  • insert more data into an index after it's built
  • update it
  • delete from it
  • create/delete/alter a plain index online (you need to define it in a configuration file)
  • use UUID for automatic ID generation. When you fetch data from an external storage it must include a unique identifier for each document

Except numeric attributes (including MVA), the rest of the data in a plain index is immutable. If you need to update/add new records you need to rebuild the index. While index is being rebuilt, existing index is still available for serving requests. When a new version of the index is ready, a process called rotation is performed which puts the new version online and discards the old one.

Plain index example
📋

A plain index can be only defined in a configuration file. It's not supported by command CREATE TABLE

source source {
  type             = mysql
  sql_host         = localhost
  sql_user         = myuser
  sql_pass         = mypass
  sql_db           = mydb
  sql_query        = SELECT id, title, description, category_id  from mytable
  sql_attr_uint    = category_id
  sql_field_string = title
 }

index idx {
  type   = plain
  source = source
  path   = /path/to/index
 }

Plain indexing performance

Speed of plain indexing depends on several factors:

  • how fast the source can be providing the data
  • tokenization settings
  • your hardware (CPU, amount of RAM, disk performance)

Plain indexing scenarios

Rebuild fully when needed

In the simplest usage scenario, we would use a single plain index which we just fully rebuild from time to time. It works fine for smaller data sets and if you are ready that:

  • the index will be not as fresh as data in the source
  • indexing duration grows with the data, the more data you have in the source the longer it will take to build the index
Main+delta

If you have a bigger data set and still want to use a plain index rather than Real-Time what you can do is:

  • make another smaller index for incremental indexing
  • combine the both using a distributed index

What it can give is you can rebuild the bigger index seldom (say once per week), save the position of the freshest indexed document and after that use the smaller index to index anything new or updated from your source. Since you will only need to fetch the updates from your storage you can do it much more frequently (say once per minute or even each few seconds).

But after a while the smaller indexing duration will become too high and that will be the moment when you need to rebuild the bigger index and empty the smaller one.

This is called main+delta schema and you can learn more about it in this interactive course.

When you build a smaller "delta" index it can get documents that are already in the "main" index. To let Manticore know that documents from the current index should take precedence there's a mechanism called kill list and corresponding directive killlist_target.

More information on this topic can be found here.

Plain index files structure

Extension Description
.spa stores document attributes
.spb stores blob attributes: strings, MVA, json
.spd stores matching document ID lists for each word ID
.sph stores index header information
.sphi stores histograms of attribute values
.spi stores word lists (word IDs and pointers to .spd file)
.spk stores kill-lists
.spl lock file
.spm stores a bitmap of killed documents
.spp stores hit (aka posting, aka word occurrence) lists for each word ID
.spt stores additional data structures to speed up lookups by document ids
.spe stores skip-lists to speed up doc-list filtering
.spds stores document texts
.tmp* temporary files during index_settings_and_status
.new.sp* during indexing new version of the index is written by default in the same folder
.old.sp* after rotation previous version files are saved with .old extension

Plain and real-time index settings

Defining index schema in a configuration file

index <index_name>[:<parent index name>] {
...
}
📋
index <index_name> {
  type = plain
  path = /path/to/index
  source = <source_name>
  source = <another source_name>
  [stored_fields = <comma separated list of full-text fields that should be stored>]
}

Common plain and real-time indexes settings

type

type = plain

type = rt

Index type: "plain" or "rt" (real-time)

Value: plain (default), rt

path

path = path/to/index

Absolute or relative path without extension where to store the index or where to look for it

Value: path to the index, mandatory

stored_fields

stored_fields = title, content

By default when an index is defined in a configuration file, original full-text field's content is not stored, but just indexed. If this option is set, the field's contents will be both indexes and stored

Value: comma separated list of full-text fields that should be stored. Default is empty.

A list of fields to be stored in the index. Optional, default is empty (do not store original field text) for Plain mode, but enabled for every field for RT mode.

By default, original document text is not stored in the index in the Plain mode. If stored_fields option is set (or RT mode is used), the field's full text is stored in the index. It can be returned with search results.

See also docstore_block_size, docstore_compression for document storage compression options.

📋
create table products(title text stored indexed, content text stored indexed, name text indexed, price float)

stored_only_fields

stored_only_fields = title,content

A list of fields that will be stored in the index but will be not indexed. Similar to stored_fields except when a field is specified in stored_only_fields it is only stored, not indexed and can’t be searched with fulltext queries. It can only be returned with search results.

Value: comma separated list of fields that should be stored only, not indexed. Default is empty.

Real-time index settings:

rt_field

rt_field = subject

Full-text fields to be indexed. The names must be unique. The order is preserved; and so field values in INSERT statements without an explicit list of inserted columns will have to be in the same order as configured.

Value: at least one full-text field should be specified in an index, multiple records allowed.

rt_attr_uint

rt_attr_uint = gid

Unsigned integer attribute declaration

Value: field_name or field_name:N, can be multiple records. N is the max number of bits to keep.

rt_attr_bigint

rt_attr_bigint = gid

BIGINT attribute declaration

Value: field name, multiple records allowed

rt_attr_multi

rt_attr_multi = tags

Multi-valued attribute (MVA) declaration. Declares the UNSIGNED INTEGER (unsigned 32-bit) MVA attribute. Multi-value (ie. there may be more than one such attribute declared), optional.

Value: field name, multiple records allowed.

rt_attr_multi_64

rt_attr_multi_64 = wide_tags

Multi-valued attribute (MVA) declaration. Declares the BIGINT (signed 64-bit) MVA attribute. Multi-value (ie. there may be more than one such attribute declared), optional.

Value: field name, multiple records allowed.

rt_attr_float

rt_attr_float = lat
rt_attr_float = lon

Floating point attribute declaration. Multi-value (an arbitrary number of attributes is allowed), optional. Declares a single precision, 32-bit IEEE 754 format float attribute.

Value: field name, multiple records allowed.

rt_attr_bool

rt_attr_bool = available

Boolean attribute declaration. Multi-value (there might be multiple attributes declared), optional. Declares a 1-bit unsigned integer attribute.

Value: field name, multiple records allowed.

rt_attr_string

rt_attr_string = title

String attribute declaration. Multi-value (an arbitrary number of attributes is allowed), optional.

Value: field name, multiple records allowed.

rt_attr_json

rt_attr_json = properties

JSON attribute declaration. Multi-value (ie. there may be more than one such attribute declared), optional.

Value: field name, multiple records allowed.

rt_attr_timestamp

rt_attr_timestamp = date_added

Timestamp attribute declaration. Multi-value (an arbitrary number of attributes is allowed), optional.

Value: field name, multiple records allowed.

rt_mem_limit

rt_mem_limit = 512M

RAM chunk size limit. Optional, default is 128M.

RT index keeps some data in memory (so-called RAM chunk) and also maintains a number of on-disk indexes (so-called disk chunks). This directive lets you control the RAM chunk size. Once there’s too much data to keep in RAM, RT index will flush it to disk, activate a newly created disk chunk, and reset the RAM chunk.

The limit is pretty strict; RT index should never allocate more memory than it’s limited to. The memory is not preallocated either, hence, specifying 512 MB limit and only inserting 3 MB of data should result in allocating 3 MB, not 512 MB.

Value: size

Plain index settings:

source

source = srcpart1
source = srcpart2
source = srcpart3

Specifies document source to get documents from when the current index is indexed. There must be at least one source. The sources can be of different types (e.g. one - mysql, another - postgresql). Read more about indexing from external storages here

Value: name of the source to build the index from, mandatory. Can be multiple records.

killlist_target

killlist_target = main:kl

Sets the index(es) that the kill-list will be applied to. Suppresses matches in the targeted index that are updated or deleted in the current index. In :kl mode the documents to suppress are taken from the kill-list. In :id mode - all document ids from the current index are suppressed in the targeted one. If neither is specified the both modes take effect. Read more about kill-lists here

Value: not specified (default), target_index_name:kl, target_index_name:id, target_index_name. Multiple values are allowed

Creating a real-time index online via CREATE TABLE

General syntax of CREATE TABLE
CREATE TABLE [IF NOT EXISTS] name ( <field name> <field data type> [data type options] [, ...]) [table_options]
CREATE TABLE via /sql HTTP endpoint

Besides using CREATE TABLE via MySQL protocol using any MySQL client you can also create a table via HTTP if you use the /sql endpoint:

http[s]://manticore_host:port/sql
POST: mode=raw&query=CREATE TABLE [IF NOT EXISTS] name ( <field name> <field data type> [data type options] [, ...]) [table_options]
Data types:

Read more about data types here.

Type Equivalent in a configuration file Notes Aliases
text rt_field Options: indexed, stored. Default - both. To keep text stored, but indexed specify "stored" only. To keep text indexed only specify only "indexed". At least one "text" field should be specified in an index
integer rt_attr_uint integer int, uint
float rt_attr_float float
multi rt_attr_multi multi-integer
multi64 rt_attr_multi_64 multi-bigint
bool rt_attr_bool boolean
json rt_attr_json JSON
string rt_attr_string string. Option: indexed - also index the strings in a full-text field with same name.
timestamp rt_attr_timestamp timestamp
bit(n) rt_attr_uint field_name:N N is the max number of bits to keep
SQL
📋
CREATE TABLE products (title text, price float) morphology='stem_en'

creates table "products" with two fields: "title" (full-text) and "price" (float) and setting "morphology" with value "stem_en"

CREATE TABLE products (title text indexed, description text stored, author text, price float)

creates table "products" with three fields:

  • field "title" - indexed, but not stored
  • field "description" - stored, but not indexed
  • field "author" - both stored and indexed

Other settings

The following settings are similar for both real-time and plain index in either mode: whether specified in a configuration file or online via CREATE or ALTER command.

Performance related

Accessing index files

Manticore uses two access modes to read index data - seek+read and mmap.

In seek+read mode the server performs system call pread(2) to read document lists and keyword positions, i.e. *.spd and *.spp files. Internal read buffers are used to optimize reading. The size of these buffers can be tuned with options read_buffer_docs and read_buffer_hits. There is also option preopen that allows to control how Manticore opens files at start.

In the mmap access mode the search server just maps index's file into memory with mmap(2) system call and OS caches file contents by itself. Options read_buffer_docs and read_buffer_hits have no effect for corresponding files in this mode. The mmap reader can also lock index's data in memory via mlock(2) privileged call which prevents swapping out the cached data to disk by OS.

To control what access mode will be used access_plain_attrs, access_blob_attrs, access_doclists and access_hitlists options are available with the following values:

Value Description
file server reads index file from disk with seek+read using internal buffers on file access
mmap server maps index file into memory and OS caches up its contents on file access
mmap_preread server maps index file into memory and a background thread reads it once to warm up the cache
mlock server maps index file into memory and then issues mlock system call to cache up the file contents and lock it into memory to prevent it being swapped out
Setting Values Description
access_plain_attrs mmap, mmap_preread (default), mlock controls how *.spa (plain attributes) *.spe (skip lists) *.spi (word lists) *.spt (lookups) *.spm (killed docs) will be read
access_blob_attrs mmap, mmap_preread (default), mlock controls how *.spb (blob attributes) (string, mva and json attributes) will be read
access_doclists file (default), mmap, mlock controls how *.spd (doc lists) data will be read
access_hitlists file (default), mmap, mlock controls how *.spp (doc lists) data will be read

Here is a table which can help you select your desired mode:

index part keep it on disk keep it in memory cached in memory on server start lock it in memory
plain attributes, skip lists, word lists, lookups, killed docs mmap mmap mmap_preread (default) mlock
string, multi-value attributes (MVA) and json attributes mmap mmap mmap_preread (default) mlock
doc lists file (default) mmap no mlock
hit lists file (default) mmap no mlock
The recommendations are:
  • If you want the best search response time and have enough memory - use mlock for attributes and for doclists/hitlists. Be aware mlock is a privileged system call and the user running searchd should have enough privileges.
  • If you can't afford lower performance on start and ready to wait longer on start until it's warmed up - use --force-preread If you want searchd to be able to restart faster - stay with mmap_preread
  • If you want to save RAM - do not use mlock, then your OS will decide what should be in memory at any given moment of time depending on what is read from disk more frequently
  • If search performance doesn't matter at all and you want to save maximum RAM - use access_doclists/access_hitlists=file and access_plain_attrs/access_blob_attrs=mmap

The default mode is to mmap and pre-read attributes and access doclists/hitlists directly from disk which provides decent search performance, optimal memory usage and faster searchd restart in most cases.

Other performance related settings

attr_update_reserve

attr_update_reserve = 256k

Sets the space to be reserved for blob attribute updates. Optional, default value is 128k. When blob attributes (multi-value attributes (MVA), strings, JSON) are updated, their length may change. If the updated string (or MVA or JSON) is shorter than the old one, it overwrites the old one in the *.spb file. But if the updated string is longer, the updates are written to the end of the *.spb file. This file is memory mapped, that's why resizing it may be a rather slow process, depending on the OS implementation of memory mapped files. To avoid frequent resizes, you can specify the extra space to be reserved at the end of the .spb file by using this setting.

Value: size, default 128k.

docstore_block_size

docstore_block_size = 32k

Size of the block of documents used by document storage. Optional, default is 16kb. When stored_fields or stored_only_fields are specified, original document text is stored inside the index. To use less disk space, documents are compressed. To get more efficient disk access and better compression ratios on small documents, documents are concatenated into blocks. When indexing, documents are collected until their total size reaches the threshold. After that, this block of documents is compressed. This option can be used to get better compression ratio (by increasing block size) or to get faster access to document text (by decreasing block size).

Value: size, default 16k.

docstore_compression

docstore_compression = lz4hc

Type of compression used to compress blocks of documents used by document storage. When stored_fields or stored_only_fields are specified, document storage stores compressed document blocks. 'lz4' has fast compression and decompression speeds, 'lz4hc' (high compression) has the same fast decompression but compression speed is traded for better compression ratio. 'none' disables compression.

Value: lz4 (default), lz4hc, none.

docstore_compression_level

docstore_compression_level = 12

Compression level in document storage when 'lz4hc' compression is used. When 'lz4hc' compression is used, compression level can be fine-tuned to get better performance or better compression ratio. Does not work with 'lz4' compression.

Value: 1-12 (default 9).

preopen

preopen = 1

This option tells searchd that it should pre-open all index files on startup (or rotation) and keep them open while it runs. Currently, the default mode is not to pre-open the files. Pre-opened indexes take a few (currently 2) file descriptors per index. However, they save on per-query open() calls; and also they are invulnerable to subtle race conditions that may happen during index rotation under high load. On the other hand, when serving many indexes (100s to 1000s), it still might be desired to open them on per-query basis in order to save file descriptors

Value: 0 (default), 1.

read_buffer_docs

read_buffer_docs = 1M

Per-keyword read buffer size for document lists. The higher the value the higher per-query RAM use is, but possibly lower IO time

Value: size, default 256k, min 8k.

read_buffer_hits

read_buffer_hits = 1M

Per-keyword read buffer size for hit lists. The higher the value the higher per-query RAM use is, but possibly lower IO time

Value: size, default 256k, min 8k.

Plain index disk footprint settings

inplace_enable

inplace_enable = {0|1}

Whether to enable in-place index inversion. Optional, default is 0 (use separate temporary files).

inplace_enable greatly reduces indexing disk footprint for a plain index, at a cost of slightly slower indexing (it uses around 2x less disk, but yields around 90-95% the original performance).

Indexing involves two major phases. The first phase collects, processes, and partially sorts documents by keyword, and writes the intermediate result to temporary files (.tmp*). The second phase fully sorts the documents, and creates the final index files. Thus, rebuilding a production index on the fly involves around 3x peak disk footprint: 1st copy for the intermediate temporary files, 2nd copy for newly constructed copy, and 3rd copy for the old index that will be serving production queries in the meantime. (Intermediate data is comparable in size to the final index.) That might be too much disk footprint for big data collections, and inplace_enable allows to reduce it. When enabled, it reuses the temporary files, outputs the final data back to them, and renames them on completion. However, this might require additional temporary data chunk relocation, which is where the performance impact comes from.

This directive does not affect searchd in any way, it only affects indexer.

CONFIG
📋
index products {
  inplace_enable = 1

  path = products
  source = src_base
}

inplace_hit_gap

inplace_hit_gap = size

In-place inversion fine-tuning option. Controls preallocated hitlist gap size. Optional, default is 0.

This directive does not affect searchd in any way, it only affects indexer.

CONFIG
📋
index products {
  inplace_hit_gap = 1M
  inplace_enable = 1

  path = products
  source = src_base
}

inplace_reloc_factor

inplace_reloc_factor = 0.1

Controls relocation buffer size within indexing memory arena. Optional, default is 0.1.

This directive does not affect searchd in any way, it only affects indexer.

CONFIG
📋
index products {
  inplace_reloc_factor = 0.1
  inplace_enable = 1

  path = products
  source = src_base
}

inplace_write_factor

inplace_write_factor = 0.1

Controls in-place write buffer size within indexing memory arena. Optional, default is 0.1.

This directive does not affect searchd in any way, it only affects indexer.

CONFIG
📋
index products {
  inplace_write_factor = 0.1
  inplace_enable = 1

  path = products
  source = src_base
}

Natural language processing specific settings

The following settings are supported. They are all described in section NLP and tokenization.