When calling Update to update document attributes in real-time, the changes are first written to in-memory copy of attributes. The updates are done in a memory mapped file, which means that the OS decides when to write these changes to disk. Once searchd shuts down normally (via SIGTERM being sent) it forces writing all the changes to disk.

It is also possible to tell searchd to periodically write these changes back to disk to avoid them being lost. The time between those intervals is set with attr_flush_period, in seconds (or special_suffixes).

It defaults to 0, which disables the periodic flushing, but flushing will still occur at normal shut-down.

Disables or throttles automatic OPTIMIZE.

Since Manticore 4 tables compaction happens automatically. You can change it with help of the setting auto_optimize by changing it to:

• 0 to disable automatic tables compaction (you can still call OPTIMIZE manually)
• 1 to enable it explicitly
• N to enable it, but let OPTIMIZE start as soon as the number of disk chunks is greater than # of CPU cores * 2 * N

Note turning auto_optimize on/off doesn't prevent you from running OPTIMIZE manually.

Manticore supports the automatic creation of tables that do not yet exist but are specified in INSERT statements. This feature is enabled by default. To disable it, set auto_schema = 0 explicitly in your configuration. To re-enable it, set auto_schema = 1 or remove the auto_schema setting from the configuration.

Binary log transaction flush/sync mode. Optional, default is 2 (flush every transaction, sync every second).

This directive controls how frequently will binary log be flushed to OS and synced to disk. Three modes are supported:

• 0, flush and sync every second. Best performance, but up to 1 second worth of committed transactions can be lost both on server crash, or OS/hardware crash.
• 1, flush and sync every transaction. Worst performance, but every committed transaction data is guaranteed to be saved.
• 2, flush every transaction, sync every second. Good performance, and every committed transaction is guaranteed to be saved in case of server crash. However, in case of OS/hardware crash up to 1 second worth of committed transactions can be lost.

For those familiar with MySQL and InnoDB, this directive is entirely similar to innodb_flush_log_at_trx_commit. In most cases, the default hybrid mode 2 provides a nice balance of speed and safety, with full RT table data protection against server crashes, and some protection against hardware ones.

Maximum binary log file size. Optional, default is 268435456, or 256Mb.

A new binlog file will be forcibly opened once the current binlog file reaches this limit. This achieves a finer granularity of logs and can yield more efficient binlog disk usage under certain borderline workloads. 0 means do not reopen binlog file based on size.

Binary log (aka transaction log) files path. Optional, default is build-time configured data directory (/var/lib/manticore/data/binlog.* in Linux).

Binary logs are used for crash recovery of RT table data, and also for attributes updates of plain disk indices that would otherwise only be stored in RAM until flush. When logging is enabled, every transaction COMMIT-ted into an RT table gets written into a log file. Logs are then automatically replayed on startup after an unclean shutdown, recovering the logged changes.

binlog_path directive specifies the binary log files location. It should contain just the path; searchd will create and unlink multiple binlog.* files in the directory as necessary (binlog data, metadata, and lock files, etc).

Empty value disables binary logging. That improves performance, but puts the RT table data at risk.

Maximum time to wait between requests (in seconds or special_suffixes) when using persistent connections. Optional, default is five minutes.

Server libc locale. Optional, default is C.

Specifies the libc locale, affecting the libc-based collations. Refer to collations section for the details.

Default server collation. Optional, default is libc_ci.

Specifies the default collation used for incoming requests. The collation can be overridden on a per-query basis. Refer to collations section for the list of available collations and other details.

When specified, enables the real-time mode (imperative way of managing data schema). The value should be a path to the directory where you want to store all your tables, binary log and everything else needed for the proper functioning of Manticore Search in this mode. Indexation of plain tables is not allowed when the data_dir is specified. Read more about the difference between the RT mode and the plain mode in this section.

Maximum size of document blocks from document storage that are held in memory. Optional, default is 16m (16 megabytes).

When stored_fields is used, document blocks are read from disk and uncompressed. Since every block typically holds several documents, it may be reused when processing the next document. For this purpose, the block is held in a server-wide cache. The cache holds uncompressed blocks.

The maximum number of expanded keywords for a single wildcard. Optional, default is 0 (no limit).

When doing substring searches against tables built with dict = keywords enabled, a single wildcard may potentially result in thousands and even millions of matched keywords (think of matching 'a*' against the entire Oxford dictionary). This directive lets you limit the impact of such expansions. Setting expansion_limit = N restricts expansions to no more than N of the most frequent matching keywords (per each wildcard in the query).

Agent mirror statistics window size, in seconds (or special_suffixes). Optional, default is 60.

For a distributed table with agent mirrors in it (see more in agent, master tracks several different per-mirror counters. These counters are then used for failover and balancing. (Master picks the best mirror to use based on the counters.) Counters are accumulated in blocks of ha_period_karma seconds.

After beginning a new block, master may still use the accumulated values from the previous one, until the new one is half full. Thus, any previous history stops affecting the mirror choice after 1.5 times ha_period_karma seconds at most.

Despite that at most 2 blocks are used for mirror selection, upto 15 last blocks are actually stored, for instrumentation purposes. They can be inspected using SHOW AGENT STATUS statement.

Interval between agent mirror pings, in milliseconds (or special_suffixes). Optional, default is 1000.

For a distributed table with agent mirrors in it (see more in agent), master sends all mirrors a ping command during the idle periods. This is to track the current agent status (alive or dead, network roundtrip, etc). The interval between such pings is defined by this directive. To disable pings, set ha_ping_interval to 0.

TCP listen backlog. Optional, default is 5.

Windows builds currently can only process the requests one by one. Concurrent requests will be enqueued by the TCP stack on OS level, and requests that can not be enqueued will immediately fail with "connection refused" message. listen_backlog directive controls the length of the connection queue. Non-Windows builds should work fine with the default value.

This setting lets you specify IP address and port, or Unix-domain socket path, that Manticore will accept connections on.

The general syntax for listen is:

listen = ( address ":" port | port | path | address ":" port start - port end ) [ ":" protocol [ "_vip" ] [ "_readonly" ] ]

You can specify:

• either an IP address (or hostname) and a port number
• or just a port number
• or Unix socket path (not supported on Windows)
• or an IP address and ports range

If you specify a port number, but not an address, searchd will listen on all network interfaces. Unix path is identified by a leading slash. Ports range could be set only for the replication protocol.

You can also specify a protocol handler (listener) to be used for connections on this socket. The listeners are:

• Not specified - Manticore will accept connections at this port from:
  • other Manticore agents (i.e. a remote distributed table)
  • clients via HTTP and HTTPS
• mysql - MySQL protocol for connections from MySQL clients. Note:
  • Compressed protocol is also supported.
  • If SSL is enabled you can make an encrypted connection.
• replication - replication protocol, used for nodes communication. More details can be found in the replication section. You can specify multiple replication listeners, but they must all listen on the same IP, only the ports can be different.
• http - same as Not specified. Manticore will accept connections at this port from remote agents and clients via HTTP and HTTPS.
• https - HTTPS protocol. Manticore will accept only HTTPS connections at this port. More details can be found in section SSL.
• sphinx - legacy binary protocol. Used to serve connections from remote SphinxSE clients. Some Sphinx API clients implementations (an example is the Java one) require the explicit declaration of the listener.

Adding suffix _vip to client protocols (that is all, except replication, for instance mysql_vip or http_vip or just _vip) forces creating a dedicated thread for the connection to bypass different limitations. That's useful for node maintenance in case of a severe overload when the server would either stall or not let you connect via a regular port otherwise.

Suffix _readonly sets read-only mode for the listener and limits it to accept only read queries.

Log file name. Optional, default is 'searchd.log'. All searchd run time events will be logged in this file.

Also you can use the 'syslog' as the file name. In this case the events will be sent to syslog daemon. To use the syslog option the sphinx must be configured -–with-syslog on building.

Limits the amount of queries per batch. Optional, default is 32.

Makes searchd perform a sanity check of the amount of the queries submitted in a single batch when using multi-queries. Set it to 0 to skip the check.

Maximum number of simultaneous client connections. Unlimited by default. That is usually noticeable only when using any kind of persistent connections, like cli mysql sessions or persistent remote connections from remote distributed tables. When the limit is exceeded you can still connect to the server using the VIP connection. VIP connections are not counted towards the limit.

Instance-wide limit of threads one operation can use. By default appropriate operations can occupy all CPU cores, leaving no room for other operations. Let's say, call pq against considerably big percolate table can utilize all threads for tens of seconds. Setting max_threads_per_query to, say, half of threads will ensure that you can run couple of such call pq in parallel.

You can also set this setting as a session or a global variable during the runtime.

You can also control the behaviour on per-query with help of the threads OPTION.

Maximum allowed per-query filter count. Only used for internal sanity checks, does not directly affect RAM use or performance. Optional, default is 256.

Maximum allowed per-filter values count. Only used for internal sanity checks, does not directly affect RAM use or performance. Optional, default is 4096.

The maximum number of files that are allowed to be opened by the server is called "soft limit". Note that serving large fragmented real-time tables may require this limit to be set high, as each disk chunk may occupy a dozen or more files. For example, a real-time table with 1000 chunks may require thousands of files to be opened simultaneously. If you encounter the error 'Too many open files' in the logs, try adjusting this option, as it may help resolve the issue.

There is also a "hard limit" that cannot be exceeded by the option. This limit is defined by the system and can be changed in the file /etc/security/limits.conf on Linux. Other operating systems may have different approaches, so consult your manuals for more information.

Apart direct numeric values, you can use magic word 'max', to set the limit equal to available current hard limit.

Maximum allowed network packet size. Limits both query packets from clients, and response packets from remote agents in distributed environment. Only used for internal sanity checks, does not directly affect RAM use or performance. Optional, default is 8M.

A server version string to return via MySQL protocol. Optional, default is empty (return Manticore version).

Several picky MySQL client libraries depend on a particular version number format used by MySQL, and moreover, sometimes choose a different execution path based on the reported version number (rather than the indicated capabilities flags). For instance, Python MySQLdb 1.2.2 throws an exception when the version number is not in X.Y.ZZ format; MySQL .NET connector 6.3.x fails internally on version numbers 1.x along with a certain combination of flags, etc. To workaround that, you can use mysql_version_string directive and have searchd report a different version to clients connecting over MySQL protocol. (By default, it reports its own version.)

Network client request read/write timeout, in seconds (or special_suffixes). Optional, default is 5 seconds. searchd will forcibly close a client connection which fails to send a query or read a result within this timeout.

This setting lets you specify the network address of the node. By default it is set to replication listen address. That is correct in most cases, however there are situations where you have to specify it manually:

• node behind a firewall
• network address translation enabled (NAT)
• container deployments, such as Docker or cloud deployments
• clusters with nodes in more than one region

Whether to allow queries with only negation full-text operator. Optional, default is 0 (fail queries with only NOT operator).

Sets default table compaction threshold. Read more here - Number of optimized disk chunks. Can be overridden with per-query option cutoff. Can be changed dynamically via SET GLOBAL.

The maximum # of simultaneous persistent connections to remote persistent agents. Each time connecting an agent defined under agent_persistent we try to reuse existing connection (if any), or connect and save the connection for the future. However in some cases it makes sense to limit # of such persistent connections. This directive defines the number. It affects the number of connections to each agent's host across all distributed tables.

It is reasonable to set the value equal or less than max_connections option of the agent's config.

searchd process ID file name. Mandatory.

PID file will be re-created (and locked) on startup. It will contain head server process ID while the server is running, and it will be unlinked on server shutdown. It's mandatory because Manticore uses it internally for a number of things: to check whether there already is a running instance of searchd; to stop searchd; to notify it that it should rotate the tables. Can also be used for different external automation scripts.

Costs for the query time prediction model, in nanoseconds. Optional, default is "doc=64, hit=48, skip=2048, match=64" (without the quotes).

Terminating queries before completion based on their execution time (with max query time setting) is a nice safety net, but it comes with an inborn drawback: indeterministic (unstable) results. That is, if you repeat the very same (complex) search query with a time limit several times, the time limit will get hit at different stages, and you will get different result sets.

Whether to forcibly preopen all tables on startup. Optional, default is 1 (preopen everything).

When set to 1, this directive overrides and enforces preopen on all tables. They will be preopened, no matter what is the per-table preopen setting. When set to 0, per-table settings can take effect. (And they default to 0.)

Pre-opened tables avoid races between search queries and rotations that can cause queries to fail occasionally. They also make searchd use more file handles. In most scenarios it's therefore preferred and recommended to preopen tables.

Enables pseudo-sharding for search queries to plain and real-time tables, no matter if they are queried directly or through a distributed table. Any search query to a local table will be automatically parallelized to up to searchd.threads # of threads.

Note that if your worker threads are already busy, because you have:

• high query concurrency
• physical sharding of any kind:
  • distributed table of multiple plain/real-time tables
  • real-time table consisting of too many disk chunks

then the pseudo-sharding may not bring any positive effect and in some cases can even cause slight throughput decrease. If you are looking for a higher throughput rather than lower latency it's recommended to disable it.

Enabled by default.

Integer, in bytes. The maximum RAM allocated for cached result sets. Default is 16777216, or 16Mb. 0 means disabled. Refer to query cache for details.

Query log format. Optional, allowed values are plain and sphinxql, default is sphinxql.

The sphinxql mode logs valid SQL statements. The plain mode logs queries in a plain text format (mostly suitable for purely full-text use cases). This directive allows to switch between the two formats on search server startup. The log format can also be altered on the fly, using SET GLOBAL query_log_format=sphinxql syntax. Refer to Query logging for more details.

Query log file name. Optional, default is empty (do not log queries). All search queries (such as SELECT ... but not INSERT/REPLACE/UPDATE queries) will be logged in this file. The format is described in Query logging. In case of 'plain' format, you can use the 'syslog' as the path to the log file. In this case all search queries will be sent to syslog daemon with LOG_INFO priority, prefixed with '[query]' instead of timestamp. To use the syslog option the sphinx must be configured -–with-syslog on building.

By default the searchd and query log files are created with 600 permission, so only the user under which server runs and root users can read the log files. query_log_mode allows settings a different permission. This can be handy to allow other users to be able to read the log files (for example monitoring solutions running on non-root users).

Per-keyword read buffer size for document lists. Optional, default is 256K, minimal is 8K.

For every keyword occurrence in every search query, there are two associated read buffers (one for document list and one for hit list). This setting lets you control the document list buffer size. Bigger buffer size might increase per-query RAM use, but possibly decrease IO time. Large values make sense in general for slow storage. For storage capable of high IOPS, experimenting should be done in the low values area.

You may also want to set read_buffer_docs on per-table basis; that value will override anything set on server's config level.

Per-keyword read buffer size for hit lists. Optional, default is 256K, minimal is 8K.

For every keyword occurrence in every search query, there are two associated read buffers (one for document list and one for hit list). This setting lets you control the hit list buffer size. Bigger buffer size might increase per-query RAM use, but possibly decrease IO time. Large values make sense in general for slow storage. For storage capable of high IOPS, experimenting should be done in the low values area.

You may also want to set read_buffer_hits on per-table basis; that valuewill override anything set on server's config level.

Unhinted read size. Optional, default is 32K, minimal 1K

When querying, some reads know in advance exactly how much data is there to be read, but some currently do not. Most prominently, hit list size in not currently known in advance. This setting lets you control how much data to read in such cases. It impacted hit list IO time, reducing it for lists larger than unhinted read size, but raising it for smaller lists. It not affects RAM usage because read buffer will be already allocated. So it should be not greater than read_buffer.

RT tables RAM chunk flush check period, in seconds (or special_suffixes). Optional, default is 10 hours.

Actively updated RT tables that however fully fit in RAM chunks can result in ever-growing binlogs, impacting disk use and crash recovery time. With this directive the search server performs periodic flush checks, and eligible RAM chunks can get saved, enabling consequential binlog cleanup. See Binary logging for more details.

A maximum number of I/O operations (per second) that the RT chunks merge thread is allowed to start. Optional, default is 0 (no limit).

This directive lets you throttle down the I/O impact arising from the OPTIMIZE statements. It is guaranteed that all the RT optimization activity will not generate more disk iops (I/Os per second) than the configured limit. Limiting rt_merge_iops can reduce search performance degradation caused by merging.

A maximum size of an I/O operation that the RT chunks merge thread is allowed to start. Optional, default is 0 (no limit).

This directive lets you throttle down the I/O impact arising from the OPTIMIZE statements. I/Os bigger than this limit will be broken down into 2 or more I/Os, which will then be accounted as separate I/Os with regards to the rt_merge_iops limit. Thus, it is guaranteed that all the optimization activity will not generate more than (rt_merge_iops * rt_merge_maxiosize) bytes of disk I/O per second.

Prevents searchd stalls while rotating tables with huge amounts of data to precache. Optional, default is 1 (enable seamless rotation). On Windows systems seamless rotation is disabled by default.

Tables may contain some data that needs to be precached in RAM. At the moment, .spa, .spb, .spi and .spm files are fully precached (they contain attribute data, blob attribute data, keyword table and killed row map, respectively.) Without seamless rotate, rotating a table tries to use as little RAM as possible and works as follows:

1. new queries are temporarily rejected (with "retry" error code);
2. searchd waits for all currently running queries to finish;
3. old table is deallocated and its files are renamed;
4. new table files are renamed and required RAM is allocated;
5. new table attribute and dictionary data is preloaded to RAM;
6. searchd resumes serving queries from new table.

However, if there's a lot of attribute or dictionary data, then preloading step could take noticeable time - up to several minutes in case of preloading 1-5+ GB files.

With seamless rotate enabled, rotation works as follows:

1. new table RAM storage is allocated;
2. new table attribute and dictionary data is asynchronously preloaded to RAM;
3. on success, old table is deallocated and both tables' files are renamed;
4. on failure, new table is deallocated;
5. at any given moment, queries are served either from old or new table copy.

Seamless rotate comes at the cost of higher peak memory usage during the rotation (because both old and new copies of .spa/.spb/.spi/.spm data need to be in RAM while preloading new copy). Average usage stays the same.

This option enables/disables the use of secondary indexes for search queries. It is optional and the default is 1 (enabled). Note that you don't need to enable it for indexing as it is always enabled as long as the Manticore Columnar Library is installed. The latter is also required for using the indexes on searching. There are three modes available:

• 0: disable the use of secondary indexes on search. They can be enabled for individual queries using analyzer hints
• 1: enable the use of secondary indexes on search. They can be disabled for individual queries using analyzer hints
• force: same as enable, but any errors during the loading of secondary indexes will be reported and the whole index will not be loaded into the daemon

Note that secondary indexes are not effective for full-text queries.

Integer number that serves as server identificator used as seed to generate an unique short UUID for nodes that are part of a replication cluster. The server_id must be unique across the nodes of a cluster and in range from 0 to 127. If server_id is not set, MAC address or a random number will be used as seed for the short UUID.

searchd --stopwait waiting time, in seconds (or special_suffixes). Optional, default is 60 seconds.

When you run searchd --stopwait your server needs to perform some activities before stopping like finishing queries, flushing RT RAM chunk, flushing attributes and updating binlog. And it requires some time. searchd --stopwait will wait up to shutdown_time seconds for server to finish its jobs. Suitable time depends on your table size and load.

A prefix to prepend to the local file names when generating snippets. Optional, default is current working folder.

This prefix can be used in distributed snippets generation along with load_files or load_files_scattered options.

Note how this is a prefix, and not a path! Meaning that if a prefix is set to "server1" and the request refers to "file23", searchd will attempt to open "server1file23" (all of that without quotes). So if you need it to be a path, you have to mention the trailing slash.

After constructing final file path, server unwinds all relative dirs and compares final result with the value of snippet_file_prefix. If result is not begin with the prefix, such file will be rejected with error message.

So, if you set it to '/mnt/data' and somebody calls snippet generation with file '../../../etc/passwd', as the source, it will get error message

File '/mnt/data/../../../etc/passwd' escapes '/mnt/data/' scope

instead of content of the file.

Also, with non-set parameter and reading '/etc/passwd' it will actually read /daemon/working/folder/etc/passwd since default for param is exactly server's working folder.

Note also that this is a local option, it does not affect the agents anyhow. So you can safely set a prefix on a master server. The requests routed to the agents will not be affected by the master's setting. They will however be affected by the agent's own settings.

This might be useful, for instance, when the document storage locations (be those local storage or NAS mountpoints) are inconsistent across the servers.

Path to a file where current SQL state will be serialized.

On server startup, this file gets replayed. On eligible state changes (eg. SET GLOBAL), this file gets rewritten automatically. This can prevent a hard-to-diagnose problem: If you load UDF functions, but Manticore crashes, when it gets (automatically) restarted, your UDF and global variables will no longer be available; using persistent state helps a graceful recovery with no such surprises.

Maximum time to wait between requests (in seconds, or special_suffixes) when using SQL interface. Optional, default is 15 minutes.

Path to the SSL Certificate Authority (CA) certificate file (aka root certificate). Optional, default is empty. When not empty the certificate in ssl_cert should be signed by this root certificate.

Server uses the CA file to verify the signature on the certificate. The file must be in PEM format.

Path to the server's SSL certificate. Optional, default is empty.

Server uses this certificate as self-signed public key encrypting HTTP traffic over SSL. The file must be in PEM format.

Path to the SSL certificate key. Optional, default is empty.

Server uses this private key to encrypt HTTP traffic over SSL. The file must be in PEM format.

Max common subtree document cache size, per-query. Optional, default is 0 (disabled).

Limits RAM usage of a common subtree optimizer (see multi-queries). At most this much RAM will be spent to cache document entries per each query. Setting the limit to 0 disables the optimizer.

Max common subtree hit cache size, per-query. Optional, default is 0 (disabled).

Limits RAM usage of a common subtree optimizer (see multi-queries). At most this much RAM will be spent to cache keyword occurrences (hits) per each query. Setting the limit to 0 disables the optimizer.

Number of working threads (or, size of thread pool) of Manticore daemon. Manticore creates this number of OS threads on start, and they perform all jobs inside the daemon such as executing queries, creating snippets, etc. Some operations may be split into sub-tasks and executed in parallel, for example:

• Search in a real-time table
• Search in a distributed table consisting of local tables
• Percolate query call
• and others

By default it's set to the number of CPU cores on the server. Manticore creates the threads on start and keep them until it's stopped. Each sub-task can use one of the threads when it needs it, when the sub-task finishes it releases the thread so another sub-task can use it.

In case of intensive I/O type of load it might make sense to set the value higher than the # of CPU cores.

Maximum stack size for a job (coroutine, one search query may cause multiple jobs/coroutines). Optional, default is 128K.

Each job has it's own stack of 128K. When you run a query it's checked for how much stack it requires. If the default 128K is enough, it's just processed. If it needs more we schedule another job with an increased stack, which continues processing. The maximum size of such advanced stack is limited by this setting.

Setting the value to a reasonably high rate will help with processing very deep queries without implication, that overall RAM consumption will grow too high. For example, setting it to 1G does not imply that every new job will take 1G of RAM, but if we see that it requires let's say 100M stack, we just allocate 100M for the job. Other jobs at the same time will be running with their default 128K stack. The same way we can run even more complex queries that need 500M. And if only if we see internally that the job requires more than 1G of stack we will fail and report about too low thread_stack.

However in practice even a query which needs 16M of stack is often too complex for parsing, and consumes too much time and resources to be processed. So, the daemon will process it, but limiting such queries by the thread_stack setting looks quite reasonable.

Whether to unlink .old table copies on successful rotation. Optional, default is 1 (do unlink).

Threaded server watchdog. Optional, default is 1 (watchdog enabled).

When Manticore query crashes it can take down the entire server. With the watchdog feature enabled, searchd additionally keeps a separate lightweight process that monitors the main server process, and automatically restarts the latter in case of abnormal termination. Watchdog is enabled by default.