Reporting bugs

Unfortunately, Manticore is not yet 100% bug free (even though we're working hard towards that), so you might occasionally run into some issues.

Reporting as much as possible about each bug is very important - because to fix it, we need to be able to either reproduce and fix the bug, or to deduce what's causing it from the information that you provide. So here are some instructions how to do that.

Bug-tracker

We track bugs and feature requests in Github. Feel free to create a new ticket and describe your bug in details so both you and developers can save their time.

Documentation updates

Updates to the documentation (what you are reading now) is also done on Github.

Crashes

Manticore is written in C++ - low level programming language allowing to speak to the computer with not so many intermediate layers for faster performance. The drawback of that is that in rare cases there is no way to handle a bug elegantly writing the error about it to a log and skipping processing the command which caused the problem. Instead of that the program can just crash which means it would stop completely and would have to be restarted.

When Manticore Search crashes you need to let Manticore team know about that by making a bug report on github or if you use Manticore's professional services in your private helpdesk. Manticore team needs the following:

  1. searchd log
  2. coredump
  3. query log

It will be great if you additionally do the following:

  1. run gdb to inspect the coredump:
    gdb /usr/bin/searchd </path/to/coredump>
  2. Find crashed thread id in the coredump file name (make sure you have %p in /proc/sys/kernel/core_pattern), e.g. core.work_6.29050.server_name.1637586599 means thread_id=29050
  3. In gdb run:
    set pagination off
    info threads
    # find thread number by it's id (e.g. for `LWP 29050` it will be thread number 8
    thread apply all bt
    thread <thread number>
    bt full
    info locals
    quit
  4. Provide the outputs

What do I do when Manticore Search hangs?

You need to run gdb manually and collect some info that may be useful to understand why it's hanging.

  1. show threads option format=all run trough a VIP port

  2. collect lsof output since hanging can be caused by too many connections or open file descriptors

    lsof -p `cat /var/run/manticore/searchd.pid`
  3. dump core

    gcore `cat /var/run/manticore/searchd.pid`

    (it will save the dump to the current dir)

  4. Install and run gdb:

    gdb /usr/bin/searchd `cat /var/run/manticore/searchd.pid`

    Note it will halt your running searchd, but if it's alredy hanging it shouldn't be a problem.

  5. In gdb run:

    set pagination off
    info threads
    thread apply all bt
    quit
  6. Collect all the outputs and files and provide them in a bug report.

For experts: the macros added in this commit can be helpful to debug.

How to enable saving coredumps on crash?

[root@srv lib]# systemctl set-environment _ADDITIONAL_SEARCHD_PARAMS='--coredump'
[root@srv lib]# systemctl restart manticore
[root@srv lib]# ps aux|grep searchd
mantico+  1955  0.0  0.0  61964  1580 ?        S    11:02   0:00 /usr/bin/searchd --config /etc/manticoresearch/manticore.conf --coredump
mantico+  1956  0.6  0.0 392744  2664 ?        Sl   11:02   0:00 /usr/bin/searchd --config /etc/manticoresearch/manticore.conf --coredump
  • make sure that your OS allows you to save coredumps: /proc/sys/kernel/core_pattern should be non-empty - it is where it will save them. If you do:
    echo "/cores/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern

    it will instruct your kernel to save them to file like core.searchd.1773.centos-4gb-hel1-1.1636454937

  • searchd should be started with ulimit -c unlimited, but if you start Manticore via systemctl it does it for yourself since it does:
    [root@srv lib]# grep CORE /lib/systemd/system/manticore.service
    LimitCORE=infinity

How do I install debug symbols?

Manticore Search and Manticore Columnar Library are written in C++, which means that what you get is a compiled compact binary file which executes in your OS optimal way. When you run a binary your system doesn't have full access to the names of variables, functions, methods, classes etc that are implemented. All that is provided separately in so called "debuginfo" packages or "symbol packages". Debug symbols are useful for troubleshooting and other debugging purposes, since when you have symbols and your binary crashes there's a way to visualize the state it crashed at including function names. Manticore Search provides such backtrace in searchd log and also generates coredump if it was run with --coredump. Without symbols all you get is just internal offsets that is difficult/impossible to decode. So if you make a bug report about a crash in most cases Manticore team will need debug symbols to be able to help you.

To install Manticore Search / Manticore Columnar Library debug symbols just install package *debuginfo* (centos), *dbgsym* (ubuntu, debian), *dbgsymbols* (windows, macos) of exactly the same version you are running. For example if you've installed Manticore Search in Centos 8 from package https://repo.manticoresearch.com/repository/manticoresearch/release/centos/8/x86_64/manticore-4.0.2_210921.af497f245-1.el8.x86_64.rpm the corresponding package with symbols is https://repo.manticoresearch.com/repository/manticoresearch/release/centos/8/x86_64/manticore-debuginfo-4.0.2_210921.af497f245-1.el8.x86_64.rpm

Note they have the same commit id af497f245 which corresponds to the commit this version was built from.

If you installed Manticore from a Manticore APT/YUM repo you can one of the following tools:

  • debuginfo-install in centos 7
  • dnf debuginfo-install centos 8
  • find-dbgsym-packages in Debian and Ubuntu

to find a debug symbols package for you.

How to check the debug symbols exist?

  1. Find build id in file /usr/bin/searchd output:
[root@srv lib]# file /usr/bin/searchd
/usr/bin/searchd: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=2c582e9f564ea1fbeb0c68406c271ba27034a6d3, stripped

In this case it's 2c582e9f564ea1fbeb0c68406c271ba27034a6d3.

  1. Find symbols in /usr/lib/debug/.build-id like this:
[root@srv ~]# ls -la /usr/lib/debug/.build-id/2c/582e9f564ea1fbeb0c68406c271ba27034a6d3*
lrwxrwxrwx. 1 root root 23 Nov  9 10:42 /usr/lib/debug/.build-id/2c/582e9f564ea1fbeb0c68406c271ba27034a6d3 -> ../../../../bin/searchd
lrwxrwxrwx. 1 root root 27 Nov  9 10:42 /usr/lib/debug/.build-id/2c/582e9f564ea1fbeb0c68406c271ba27034a6d3.debug -> ../../usr/bin/searchd.debug

Uploading your data

To fix your bug developers often need to reproduce it locally. To do it they need your configuration file, index files, binlog (if present), sometimes data to index (like data from external storages or XML/CSV files) and queries.

Attach your data when you create a ticket on Github. In case it's too big or the data is sensitive feel free to upload it to our write-only FTP server:

  • ftp: dev.manticoresearch.com
  • user: manticorebugs
  • pass: shithappens
  • mode: only active mode is supported
  • directory: create directory github-issue-N so we understand what data is related with what issue on Github.

It's convenient to mirror your directory with our FTP using tool lftp which is available for Linux, Mac and Windows. For example, if you want to sync your current directory ftp to directory github-ussue-123, here's what you should do and what you will get:

➜  ~ lftp -e "set ftp:passive-mode off; mkdir github-issue-123; mirror -LR ftp/ github-issue-123/" -u manticorebugs,shithappens dev.manticoresearch.com
mkdir ok, `github-issue-123' created
Total: 2 directories, 1 file, 0 symlinks
New: 1 file, 0 symlinks
lftp manticorebugs@dev.manticoresearch.com:/> quit

DEBUG

DEBUG [ subcommand ]

DEBUG statement is designed to call different internal or vip commands for dev/testing purposes. It is not intended for production automation, since the syntax of subcommand part may be freely changed in any build.

Call DEBUG without params to show list of useful commands (in general) and subcommands (of DEBUG statement) available at the moment.

However you can invoke DEBUG without params to know which subcommands of the statement are available in any particular case:

MySQL [(none)]> debug;
+------------------------------------------------------------------+----------------------------------------------------------------------------------------+
| command                                                          | meaning                                                                                |
+------------------------------------------------------------------+----------------------------------------------------------------------------------------+
| flush logs                                                       | emulate USR1 signal                                                                    |
| reload indexes                                                   | emulate HUP signal                                                                     |
| debug token <password>                                           | calculate token for password                                                           |
| debug malloc_stats                                               | perform 'malloc_stats', result in searchd.log                                          |
| debug malloc_trim                                                | pefrorm 'malloc_trim' call                                                             |
| debug sleep <N>                                                  | sleep for <N> seconds                                                                  |
| debug tasks                                                      | display global tasks stat (use select from @@system.tasks instead)                     |
| debug systhreads                                                 | display task manager threads (use select from @@system.systhreads instead)             |
| debug sched                                                      | display task manager schedule (use select from @@system.sched instead)                 |
| debug merge <IDX> [chunk] <X> [into] [chunk] <Y> [option sync=1] | For RT index <IDX> merge disk chunk X into disk chunk Y                                |
| debug drop [chunk] <X> [from] <IDX> [option sync=1]              | For RT index <IDX> drop disk chunk X                                                   |
| debug files <IDX> [option format=all|external]                   | list files belonging to <IDX>. 'all' - including external (wordforms, stopwords, etc.) |
| debug close                                                      | ask server to close connection from it's side                                          |
| debug compress <IDX> [chunk] <X> [option sync=1]                 | Compress disk chunk X of RT index <IDX> (wipe out deleted documents)                   |
| debug split <IDX> [chunk] <X> on @<uservar> [option sync=1]      | Split disk chunk X of RT index <IDX> using set of DocIDs from @uservar                 |
+------------------------------------------------------------------+----------------------------------------------------------------------------------------+
15 rows in set (0.00 sec)

(these commands are already documented, but such short help just remind about them).

If you connect via 'VIP' connection (see listen for details) the output might be a bit different:

MySQL [(none)]> debug;
+------------------------------------------------------------------+----------------------------------------------------------------------------------------+
| command                                                          | meaning                                                                                |
+------------------------------------------------------------------+----------------------------------------------------------------------------------------+
| flush logs                                                       | emulate USR1 signal                                                                    |
| reload indexes                                                   | emulate HUP signal                                                                     |
| debug shutdown <password>                                        | emulate TERM signal                                                                    |
| debug crash <password>                                           | crash daemon (make SIGSEGV action)                                                     |
| debug token <password>                                           | calculate token for password                                                           |
| debug malloc_stats                                               | perform 'malloc_stats', result in searchd.log                                          |
| debug malloc_trim                                                | pefrorm 'malloc_trim' call                                                             |
| debug procdump                                                   | ask watchdog to dump us                                                                |
| debug setgdb on|off                                              | enable or disable potentially dangerous crash dumping with gdb                         |
| debug setgdb status                                              | show current mode of gdb dumping                                                       |
| debug sleep <N>                                                  | sleep for <N> seconds                                                                  |
| debug tasks                                                      | display global tasks stat (use select from @@system.tasks instead)                     |
| debug systhreads                                                 | display task manager threads (use select from @@system.systhreads instead)             |
| debug sched                                                      | display task manager schedule (use select from @@system.sched instead)                 |
| debug merge <IDX> [chunk] <X> [into] [chunk] <Y> [option sync=1] | For RT index <IDX> merge disk chunk X into disk chunk Y                                |
| debug drop [chunk] <X> [from] <IDX> [option sync=1]              | For RT index <IDX> drop disk chunk X                                                   |
| debug files <IDX> [option format=all|external]                   | list files belonging to <IDX>. 'all' - including external (wordforms, stopwords, etc.) |
| debug close                                                      | ask server to close connection from it's side                                          |
| debug compress <IDX> [chunk] <X> [option sync=1]                 | Compress disk chunk X of RT index <IDX> (wipe out deleted documents)                   |
| debug split <IDX> [chunk] <X> on @<uservar> [option sync=1]      | Split disk chunk X of RT index <IDX> using set of DocIDs from @uservar                 |
+------------------------------------------------------------------+----------------------------------------------------------------------------------------+
20 rows in set (0.00 sec)

Here you can see additional commands available only in the current context (namely, if you connected on a VIP port). Two additional subcommands available right now are token and shutdown. The first one just calculates a hash (SHA1) of the (which, in turn, may be empty, or a word, or num/phrase enclosed in '-quotes) like:

mysql> debug token hello;
+-------------+------------------------------------------+
| command     | result                                   |
+-------------+------------------------------------------+
| debug token | aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d |
+-------------+------------------------------------------+
1 row in set (0,00 sec)

Another debug subcommand shutdown will send a TERM signal to the server and so will make it shut down. Since it is quite dangerous (nobody wants accidentally stop a production service), it:

  1. needs a VIP connection, and
  2. needs the password

For the chosen password you need to generate a token with debug token subcommand, and put it into shutdown_token param of searchd section of the config file. If no such section exists, or if a hash of the provided password does not match with the token stored in the config, the subcommand will do nothing. Otherwise it will cause 'clean' shutdown of the server.

References

SQL commands

Schema management
Data management
  • INSERT - Adds new documents
  • UPDATE - Replaces existing documents with new ones
  • UPDATE - Does in-place update in documents
  • DELETE - Deletes documents
  • TRUNCATE TABLE - Deletes all documents from index
SELECT
Flushing misc things
  • FLUSH ATTRIBUTES - Forces flushing updated attributes to disk
  • FLUSH HOSTNAMES - Renews IPs associates to agent host names
  • FLUSH LOGS - Initiates reopen of searchd log and query log files (similar to USR1)
Real-time index optimization
Importing to a real-time index
  • ATTACH INDEX - Moves data from a plain index to a real-time index
  • IMPORT TABLE - Imports previously created RT or PQ index into a server running in RT mode
Replication
Plain index rotate
Transactions
  • BEGIN - Begins a transaction
  • COMMIT - Finishes a transaction
  • ROLLBACK - Rolls back a transaction
CALL
Plugins
Server status

HTTP endpoints

  • /sql - Allows running an SQL statement over HTTP
  • /cli - HTTP command line interface
  • /insert - Inserts a document into a real-time index
  • /pq/idx/doc - Inserts a PQ rule into a percolate index
  • /update - Updates a document in a real-time index
  • /replace - Replaces a document in a real-time index
  • /pq/idx/doc/N?refresh=1 - Replaces a PQ rule in a percolate index
  • /delete - Deletes a document in an index
  • /bulk - Perform several insert, update or delete operations in a single call
  • /search - Performs search
  • /pq/idx/search - Performs reverse search in a percolate index

Common things

Common index settings
Plain index settings
Distributed index settings
RT index settings

Full-text search operators

Functions

Mathematical
  • ABS() - Returns absolute value
  • ATAN2() - Returns arctangent function of two arguments
  • BITDOT() - Returns sum of products of an each bit of a mask multiplied with its weight
  • CEIL() - Returns smallest integer value greater or equal to the argument
  • COS() - Returns cosine of the argument
  • CRC32() - Returns CRC32 value of the argument
  • EXP() - Returns exponent of the argument
  • FIBONACCI() - Returns the N-th Fibonacci number, where N is the integer argument
  • FLOOR() - Returns the largest integer value lesser or equal to the argument
  • GREATEST() - Takes JSON/MVA array as the argument and returns the greatest value in that array
  • IDIV() - Returns result of an integer division of the first argument by the second argument
  • LEAST() - Takes JSON/MVA array as the argument, and returns the least value in that array
  • LN() - Returns natural logarithm of the argument
  • LOG10() - Returns common logarithm of the argument
  • LOG2() - Returns binary logarithm of the argument
  • MAX() - Returns the bigger of two arguments
  • MIN() - Returns the smaller of two arguments
  • POW() - Returns the first argument raised to the power of the second argument
  • RAND() - Returns random float between 0..1
  • SIN() - Returns sine of the argument
  • SQRT() - Returns square root of the argument
Searching and ranking
  • BM25F() - Returns precise BM25F formula value
  • EXIST() - Replaces non-existing columns with default values
  • GROUP_CONCAT() - Produces a comma-separated list of the attribute values of all documents in the group
  • HIGHLIGHT() - Highlights search results
  • MIN_TOP_SORTVAL() - Returns sort key value of the worst found element in the current top-N matches
  • MIN_TOP_WEIGHT() - Returns weight of the worst found element in the current top-N matches
  • PACKEDFACTORS() - Outputs weighting factors
  • REMOVE_REPEATS() - Removes repeated adjusted rows with the same 'column' value
  • WEIGHT() - Returns fulltext match score
  • ZONESPANLIST() - Returns pairs of matched zone spans
  • QUERY() - Returns current full-text query
Type casting
  • BIGINT() - Forcibly promotes the integer argument to 64-bit type
  • DOUBLE() - Forcibly promotes given argument to floating point type
  • INTEGER() - Forcibly promotes given argument to 64-bit signed type
  • TO_STRING() - Forcibly promotes the argument to string type
  • UINT() - Forcibly reinterprets given argument to 64-bit unsigned type
  • SINT() - Interprets 32-bit unsigned integer as signed 64-bit integer
Arrays and conditions
  • ALL() - Returns 1 if condition is true for all elements in the array
  • ANY() - Returns 1 if condition is true for any element in the array
  • CONTAINS() - Checks whether the (x,y) point is within the given polygon
  • IF() - Checks whether the 1st argument is equal to 0.0, returns the 2nd argument if it is not zero or the 3rd one when it is
  • IN() - Returns 1 if the first argument is equal to any of the other arguments, or 0 otherwise
  • INDEXOF() - Iterates through all elements in the array and returns index of the first matching element
  • INTERVAL() - Returns index of the argument that is less than the first argument
  • LENGTH() - Returns number of elements in MVA
  • REMAP() - Allows to make some exceptions of expression values depending on the condition values
Date and time
  • NOW() - Returns current timestamp as an INTEGER
  • CURTIME() - Returns current time in local timezone
  • UTC_TIME() - Returns current time in UTC timezone
  • UTC_TIMESTAMP() - Returns current date/time in UTC timezone
  • SECOND() - Returns integer second from the timestamp argument
  • MINUTE() - Returns integer minute from the timestamp argument
  • HOUR() - Returns integer hour from the timestamp argument
  • DAY() - Returns integer day from the timestamp argument
  • MONTH() - Returns integer month from the timestamp argument
  • YEAR() - Returns integer year from the timestamp argument
  • YEARMONTH() - Returns integer year and month code from the timestamp argument
  • YEARMONTHDAY() - Returns integer year, month and day code from the timestamp argument
  • TIMEDIFF() - Returns difference between the timstamps
Geo-spatial
  • GEODIST() - Computes geosphere distance between two given points
  • GEOPOLY2D() - Creates a polygon that takes in account the Earth's curvature
  • POLY2D() - Creates a simple polygon in plain space
String
  • CONCAT() - Concatenates two or more strings
  • REGEX() - Returns 1 if regular expression matched to string of attribute and 0 otherwise
  • SNIPPET() - Highlights search results
  • SUBSTRING_INDEX() - Returns a substring of the string before the specified number of delimiter occurs
  • Other
  • LAST_INSERT_ID() - Returns ids of documents inserted or replaced by last statement in the current session

Common settings in configuration file

To be put to section common {} in configuration file:

indexer is a tool to create plain indexes

Indexer settings in configuration file

To be put to section indexer {} in configuration file:

Indexer start parameters
indexer [OPTIONS] [indexname1 [indexname2 [...]]]
  • --all - Rebuilds all indexes from the config
  • --buildstops - Reviews the index source, as if it were indexing the data, and produces a list of the terms that are being indexed.
  • --buildfreqs - Adds the quantity present in the index for --buildstops
  • --config, -c - Path to configuration file
  • --dump-rows - Dumps rows fetched by SQL source(s) into the specified file
  • --help - Lists all the parameters
  • --keep-attrs - Allows to reuse existing attributes on reindexing
  • --keep-attrs-names - Allows to specify attributes to reuse from the existing index
  • --merge-dst-range - Runs the filter range given upon merging
  • --merge-killlists - Changes the way kill lists are processed when merging indexes
  • --merge - Merges two plain indexes into one
  • --nohup - Indexer won't send SIGHUP if this option is on
  • --noprogress - Prevents displaying progress details
  • --print-queries - Prints out SQL queries that indexer sends to the database
  • --print-rt - Outputs data fetched from sql source(s) as INSERTs to a real-time index
  • --quiet - Prevents displaying anything
  • --rotate - Forces indexes rotation after all the indexes are built
  • --sighup-each - Forces rotation of each index after it's built
  • -v - Shows indexer version

Index converter from Manticore v2 / Sphinx v2

index_converter is a tool for converting indexes created with Sphinx/Manticore Search 2.x to Manticore Search 3.x index format.

index_converter {--config /path/to/config|--path}
Index converter start parameters
  • --config, -c - Path to indexes configuration file
  • --index - Specifies which index should be converted
  • --path - Defines path containing index(es) instead of the configuration file
  • --strip-path - Strips path from filenames referenced by index
  • --large-docid - Allows to convert documents with ids larger than 2^63
  • --output-dir - Writes the new files in a chosen folder
  • --all - Converts all indexes from the configuration file / path
  • --killlist-target - Sets the target indexes for which kill-lists will be applied

searchd is a Manticore server.

Searchd settings in a configuration file

To be put to section searchd {} in configuration file:

Searchd start parameters
searchd [OPTIONS]
  • --config, -c - Path to configuration file
  • --console - Forces running in console mode
  • --coredump - Enables saving core dump on crash
  • --cpustats - Enables CPU time reporting
  • --delete - Removes Manticore service from Microsoft Management Console and other places where the services are registered
  • --force-preread - Forbids the server to serve any incoming connection until pre-reading of the index files completes
  • --help, -h - Lists all the parameters
  • --index - Forces serving only the specified index
  • --install - Installs searchd as a service into Microsoft Management Console
  • --iostats - Enables input/output reporting
  • --listen, -l - Overrides listen from the configuration file
  • --logdebug, --logdebugv, --logdebugvv - Enables additional debug output in the server log
  • --logreplication - Enables additional replication debug output in the server log
  • --new-cluster - Bootstraps a replication cluster and makes the server a reference node with cluster restart protection
  • --new-cluster-force - Bootstraps a replication cluster and makes the server a reference node bypassing cluster restart protection
  • --nodetach - Leaves searchd in foreground
  • --ntservice - Passed by Microsoft Management Console to searchd to invoke it as a service on Windows platforms
  • --pidfile - Overrides pid_file from the configuration file
  • --port, p - Specifies port searchd should listen on disregarding the port specified in the configuration file
  • --replay-flags - Specifies extra binary log replay options
  • --servicename - Applies the given name to searchd when installing or deleting the service, as would appear in Microsoft Management Console
  • --status - Queries running search to return its status
  • --stop - Stops Manticore server
  • --stopwait - Stops Manticore server gracefully
  • --strip-path - Strips path names from all the file names referenced from the index
  • -v - shows version information
Searchd environment variables

Miscellaneous index maintenance functionality useful for troubleshooting.

indextool <command> [options]
Indextool start parameters

Used to dump miscellaneous debug information about the physical index

indextool <command> [options]
  • --config, -c - Path to configuration file
  • --quiet, -q - Keeps indextool quiet - it will not output banner, etc
  • --help, -h - Lists all the parameters
  • -v - Shows version information
  • Indextool - Verifies configuration file
  • --buildidf - Builds IDF file from one or several dictionary dumps
  • --build-infixes - Build infixes for an existing dict=keywords index
  • --dumpheader - Quickly dumps the provided index header file
  • --dumpconfig - Dumps index definition from the given index header file in almost compliant manticore.conf file format
  • --dumpheader - Dumps index header by index name with looking up the header path in the configuration file
  • --dumpdict - Dumps index dictionary
  • --dumpdocids - Dumps document IDs by index name
  • --dumphitlist - Dumps all occurrences of the given keyword/id in the given index
  • --fold - Tests tokenization based on index's settings
  • --htmlstrip - Filters STDIN using HTML stripper settings for the given index
  • --mergeidf - Merges several .idf files into a single one
  • --morph - Applies morphology to the given STDIN and prints the result to stdout
  • --check - Checks the index data files for consistency
  • --check-id-dups - Checks if there are duplicate ids
  • --check-disk-chunk - Checks one disk chunk of an RT index
  • --strip-path - Strips path names from all the file names referenced from the index
  • --rotate - Defines whether to check index waiting for rotation in --check
  • --apply-killlists - Applies kill-lists for all indexes listed in the configuration file

Splits compound words into components.

wordbreaker [-dict path/to/dictionary_file] {split|test|bench}
Wordbreaker start parameters.

Used to extract contents of a dictionary file that uses ispell or MySpell format.

spelldump [options] <dictionary> <affix> [result] [locale-name]
  • dictionary - Dictionary's main file
  • affix - Dictionary's affix file
  • result - Specifies where the dictionary data should be output to
  • locale-name - Specifies the locale details to use

List of reserved keywords

A complete alphabetical list of keywords that are currently reserved in Manticore SQL syntax (and therefore can not be used as identifiers).

AND, AS, BY, DISTINCT, DIV, EXPLAIN, FACET, FALSE, FORCE, FROM, IGNORE, IN, INDEXES, IS, LIMIT, LOGS, MOD, NOT, NULL, OFFSET, OR, ORDER, REGEX, RELOAD, SELECT, SYSFILTERS, TRUE, USE

Documentation for old Manticore versions

References

SQL commands

Schema management
Data management
  • INSERT - Adds new documents
  • UPDATE - Replaces existing documents with new ones
  • UPDATE - Does in-place update in documents
  • DELETE - Deletes documents
  • TRUNCATE TABLE - Deletes all documents from index
SELECT
Flushing misc things
  • FLUSH ATTRIBUTES - Forces flushing updated attributes to disk
  • FLUSH HOSTNAMES - Renews IPs associates to agent host names
  • FLUSH LOGS - Initiates reopen of searchd log and query log files (similar to USR1)
Real-time index optimization
Importing to a real-time index
  • ATTACH INDEX - Moves data from a plain index to a real-time index
  • IMPORT TABLE - Imports previously created RT or PQ index into a server running in RT mode
Replication
Plain index rotate
Transactions
  • BEGIN - Begins a transaction
  • COMMIT - Finishes a transaction
  • ROLLBACK - Rolls back a transaction
CALL
Plugins
Server status

HTTP endpoints

  • /sql - Allows running an SQL statement over HTTP
  • /cli - HTTP command line interface
  • /insert - Inserts a document into a real-time index
  • /pq/idx/doc - Inserts a PQ rule into a percolate index
  • /update - Updates a document in a real-time index
  • /replace - Replaces a document in a real-time index
  • /pq/idx/doc/N?refresh=1 - Replaces a PQ rule in a percolate index
  • /delete - Deletes a document in an index
  • /bulk - Perform several insert, update or delete operations in a single call
  • /search - Performs search
  • /pq/idx/search - Performs reverse search in a percolate index

Common things

Common index settings
Plain index settings
Distributed index settings
RT index settings

Full-text search operators

Functions

Mathematical
  • ABS() - Returns absolute value
  • ATAN2() - Returns arctangent function of two arguments
  • BITDOT() - Returns sum of products of an each bit of a mask multiplied with its weight
  • CEIL() - Returns smallest integer value greater or equal to the argument
  • COS() - Returns cosine of the argument
  • CRC32() - Returns CRC32 value of the argument
  • EXP() - Returns exponent of the argument
  • FIBONACCI() - Returns the N-th Fibonacci number, where N is the integer argument
  • FLOOR() - Returns the largest integer value lesser or equal to the argument
  • GREATEST() - Takes JSON/MVA array as the argument and returns the greatest value in that array
  • IDIV() - Returns result of an integer division of the first argument by the second argument
  • LEAST() - Takes JSON/MVA array as the argument, and returns the least value in that array
  • LN() - Returns natural logarithm of the argument
  • LOG10() - Returns common logarithm of the argument
  • LOG2() - Returns binary logarithm of the argument
  • MAX() - Returns the bigger of two arguments
  • MIN() - Returns the smaller of two arguments
  • POW() - Returns the first argument raised to the power of the second argument
  • RAND() - Returns random float between 0..1
  • SIN() - Returns sine of the argument
  • SQRT() - Returns square root of the argument
Searching and ranking
  • BM25F() - Returns precise BM25F formula value
  • EXIST() - Replaces non-existing columns with default values
  • GROUP_CONCAT() - Produces a comma-separated list of the attribute values of all documents in the group
  • HIGHLIGHT() - Highlights search results
  • MIN_TOP_SORTVAL() - Returns sort key value of the worst found element in the current top-N matches
  • MIN_TOP_WEIGHT() - Returns weight of the worst found element in the current top-N matches
  • PACKEDFACTORS() - Outputs weighting factors
  • REMOVE_REPEATS() - Removes repeated adjusted rows with the same 'column' value
  • WEIGHT() - Returns fulltext match score
  • ZONESPANLIST() - Returns pairs of matched zone spans
  • QUERY() - Returns current full-text query
Type casting
  • BIGINT() - Forcibly promotes the integer argument to 64-bit type
  • DOUBLE() - Forcibly promotes given argument to floating point type
  • INTEGER() - Forcibly promotes given argument to 64-bit signed type
  • TO_STRING() - Forcibly promotes the argument to string type
  • UINT() - Forcibly reinterprets given argument to 64-bit unsigned type
  • SINT() - Interprets 32-bit unsigned integer as signed 64-bit integer
Arrays and conditions
  • ALL() - Returns 1 if condition is true for all elements in the array
  • ANY() - Returns 1 if condition is true for any element in the array
  • CONTAINS() - Checks whether the (x,y) point is within the given polygon
  • IF() - Checks whether the 1st argument is equal to 0.0, returns the 2nd argument if it is not zero or the 3rd one when it is
  • IN() - Returns 1 if the first argument is equal to any of the other arguments, or 0 otherwise
  • INDEXOF() - Iterates through all elements in the array and returns index of the first matching element
  • INTERVAL() - Returns index of the argument that is less than the first argument
  • LENGTH() - Returns number of elements in MVA
  • REMAP() - Allows to make some exceptions of expression values depending on the condition values
Date and time
  • NOW() - Returns current timestamp as an INTEGER
  • CURTIME() - Returns current time in local timezone
  • UTC_TIME() - Returns current time in UTC timezone
  • UTC_TIMESTAMP() - Returns current date/time in UTC timezone
  • SECOND() - Returns integer second from the timestamp argument
  • MINUTE() - Returns integer minute from the timestamp argument
  • HOUR() - Returns integer hour from the timestamp argument
  • DAY() - Returns integer day from the timestamp argument
  • MONTH() - Returns integer month from the timestamp argument
  • YEAR() - Returns integer year from the timestamp argument
  • YEARMONTH() - Returns integer year and month code from the timestamp argument
  • YEARMONTHDAY() - Returns integer year, month and day code from the timestamp argument
  • TIMEDIFF() - Returns difference between the timstamps
Geo-spatial
  • GEODIST() - Computes geosphere distance between two given points
  • GEOPOLY2D() - Creates a polygon that takes in account the Earth's curvature
  • POLY2D() - Creates a simple polygon in plain space
String
  • CONCAT() - Concatenates two or more strings
  • REGEX() - Returns 1 if regular expression matched to string of attribute and 0 otherwise
  • SNIPPET() - Highlights search results
  • SUBSTRING_INDEX() - Returns a substring of the string before the specified number of delimiter occurs
  • Other
  • LAST_INSERT_ID() - Returns ids of documents inserted or replaced by last statement in the current session

Common settings in configuration file

To be put to section common {} in configuration file:

indexer is a tool to create plain indexes

Indexer settings in configuration file

To be put to section indexer {} in configuration file:

Indexer start parameters
indexer [OPTIONS] [indexname1 [indexname2 [...]]]
  • --all - Rebuilds all indexes from the config
  • --buildstops - Reviews the index source, as if it were indexing the data, and produces a list of the terms that are being indexed.
  • --buildfreqs - Adds the quantity present in the index for --buildstops
  • --config, -c - Path to configuration file
  • --dump-rows - Dumps rows fetched by SQL source(s) into the specified file
  • --help - Lists all the parameters
  • --keep-attrs - Allows to reuse existing attributes on reindexing
  • --keep-attrs-names - Allows to specify attributes to reuse from the existing index
  • --merge-dst-range - Runs the filter range given upon merging
  • --merge-killlists - Changes the way kill lists are processed when merging indexes
  • --merge - Merges two plain indexes into one
  • --nohup - Indexer won't send SIGHUP if this option is on
  • --noprogress - Prevents displaying progress details
  • --print-queries - Prints out SQL queries that indexer sends to the database
  • --print-rt - Outputs data fetched from sql source(s) as INSERTs to a real-time index
  • --quiet - Prevents displaying anything
  • --rotate - Forces indexes rotation after all the indexes are built
  • --sighup-each - Forces rotation of each index after it's built
  • -v - Shows indexer version

Index converter from Manticore v2 / Sphinx v2

index_converter is a tool for converting indexes created with Sphinx/Manticore Search 2.x to Manticore Search 3.x index format.

index_converter {--config /path/to/config|--path}
Index converter start parameters
  • --config, -c - Path to indexes configuration file
  • --index - Specifies which index should be converted
  • --path - Defines path containing index(es) instead of the configuration file
  • --strip-path - Strips path from filenames referenced by index
  • --large-docid - Allows to convert documents with ids larger than 2^63
  • --output-dir - Writes the new files in a chosen folder
  • --all - Converts all indexes from the configuration file / path
  • --killlist-target - Sets the target indexes for which kill-lists will be applied

searchd is a Manticore server.

Searchd settings in a configuration file

To be put to section searchd {} in configuration file:

Searchd start parameters
searchd [OPTIONS]
  • --config, -c - Path to configuration file
  • --console - Forces running in console mode
  • --coredump - Enables saving core dump on crash
  • --cpustats - Enables CPU time reporting
  • --delete - Removes Manticore service from Microsoft Management Console and other places where the services are registered
  • --force-preread - Forbids the server to serve any incoming connection until pre-reading of the index files completes
  • --help, -h - Lists all the parameters
  • --index - Forces serving only the specified index
  • --install - Installs searchd as a service into Microsoft Management Console
  • --iostats - Enables input/output reporting
  • --listen, -l - Overrides listen from the configuration file
  • --logdebug, --logdebugv, --logdebugvv - Enables additional debug output in the server log
  • --logreplication - Enables additional replication debug output in the server log
  • --new-cluster - Bootstraps a replication cluster and makes the server a reference node with cluster restart protection
  • --new-cluster-force - Bootstraps a replication cluster and makes the server a reference node bypassing cluster restart protection
  • --nodetach - Leaves searchd in foreground
  • --ntservice - Passed by Microsoft Management Console to searchd to invoke it as a service on Windows platforms
  • --pidfile - Overrides pid_file from the configuration file
  • --port, p - Specifies port searchd should listen on disregarding the port specified in the configuration file
  • --replay-flags - Specifies extra binary log replay options
  • --servicename - Applies the given name to searchd when installing or deleting the service, as would appear in Microsoft Management Console
  • --status - Queries running search to return its status
  • --stop - Stops Manticore server
  • --stopwait - Stops Manticore server gracefully
  • --strip-path - Strips path names from all the file names referenced from the index
  • -v - shows version information
Searchd environment variables

Miscellaneous index maintenance functionality useful for troubleshooting.

indextool <command> [options]
Indextool start parameters

Used to dump miscellaneous debug information about the physical index

indextool <command> [options]
  • --config, -c - Path to configuration file
  • --quiet, -q - Keeps indextool quiet - it will not output banner, etc
  • --help, -h - Lists all the parameters
  • -v - Shows version information
  • Indextool - Verifies configuration file
  • --buildidf - Builds IDF file from one or several dictionary dumps
  • --build-infixes - Build infixes for an existing dict=keywords index
  • --dumpheader - Quickly dumps the provided index header file
  • --dumpconfig - Dumps index definition from the given index header file in almost compliant manticore.conf file format
  • --dumpheader - Dumps index header by index name with looking up the header path in the configuration file
  • --dumpdict - Dumps index dictionary
  • --dumpdocids - Dumps document IDs by index name
  • --dumphitlist - Dumps all occurrences of the given keyword/id in the given index
  • --fold - Tests tokenization based on index's settings
  • --htmlstrip - Filters STDIN using HTML stripper settings for the given index
  • --mergeidf - Merges several .idf files into a single one
  • --morph - Applies morphology to the given STDIN and prints the result to stdout
  • --check - Checks the index data files for consistency
  • --check-id-dups - Checks if there are duplicate ids
  • --check-disk-chunk - Checks one disk chunk of an RT index
  • --strip-path - Strips path names from all the file names referenced from the index
  • --rotate - Defines whether to check index waiting for rotation in --check
  • --apply-killlists - Applies kill-lists for all indexes listed in the configuration file

Splits compound words into components.

wordbreaker [-dict path/to/dictionary_file] {split|test|bench}
Wordbreaker start parameters.

Used to extract contents of a dictionary file that uses ispell or MySpell format.

spelldump [options] <dictionary> <affix> [result] [locale-name]
  • dictionary - Dictionary's main file
  • affix - Dictionary's affix file
  • result - Specifies where the dictionary data should be output to
  • locale-name - Specifies the locale details to use

List of reserved keywords

A complete alphabetical list of keywords that are currently reserved in Manticore SQL syntax (and therefore can not be used as identifiers).

AND, AS, BY, DISTINCT, DIV, EXPLAIN, FACET, FALSE, FORCE, FROM, IGNORE, IN, INDEXES, IS, LIMIT, LOGS, MOD, NOT, NULL, OFFSET, OR, ORDER, REGEX, RELOAD, SELECT, SYSFILTERS, TRUE, USE

Documentation for old Manticore versions