Unfortunately, Manticore is not yet 100% bug-free, although the development team is working hard towards that goal. You may encounter some issues from time to time. It is crucial to report as much information as possible about each bug to fix it effectively. To fix a bug, either it needs to be reproduced and fixed or its cause needs to be deduced based on the information you provide. To help with this, please follow the instructions below.
Bugs and feature requests are tracked on Github. You are welcome to create a new ticket and describe your bug in detail to save time for both you and the developers.
Updates to the documentation (what you are reading now) are also done on Github.
Manticore Search is written in C++, which is a low-level programming language that allows for direct communication with the computer for faster performance. However, there is a drawback to this approach as in rare cases, it may not be possible to elegantly handle a bug by writing an error to a log and skipping the processing of the command that caused the problem. Instead, the program may crash, resulting in it stopping completely and needing to be restarted.
When Manticore Search crashes, it is important to let the Manticore team know by submitting a bug report on GitHub or through Manticore's professional services in your private helpdesk. The Manticore team requires the following information:
- The searchd log
- The coredump
- The query log
Additionally, it would be helpful if you could do the following:
- Run gdb to inspect the coredump:
gdb /usr/bin/searchd </path/to/coredump>
- Find the crashed thread ID in the coredump file name (make sure you have
%p
in /proc/sys/kernel/core_pattern), e.g.core.work_6.29050.server_name.1637586599
means thread_id=29050 - In gdb run:
set pagination off info threads # find thread number by it's id (e.g. for `LWP 29050` it will be thread number 8 thread apply all bt thread <thread number> bt full info locals quit
- Provide the outputs
If Manticore Search hangs, you need to collect some information that may be useful in understanding the cause. Here's how you can do it:
-
Run
show threads option format=all
trough a VIP port -
collect the lsof output output, as hanging can be caused by too many connections or open file descriptors.
lsof -p `cat /var/run/manticore/searchd.pid`
-
Dump the core:
gcore `cat /var/run/manticore/searchd.pid`
(It will save the dump to the current directory.)
-
Install and run gdb:
gdb /usr/bin/searchd `cat /var/run/manticore/searchd.pid`
Note that this will halt your running searchd, but if it's already hanging, it shouldn't be a problem.
-
In gdb run:
set pagination off info threads thread apply all bt quit
-
Collect all the outputs and files and provide them in a bug report.
For experts: the macros added in this commit can be helpful in debugging.
- make sure you run searchd with
--coredump
. To avoid modifying scripts, you can use the https://manual.manticoresearch.com/Starting_the_server/Linux#Custom-startup-flags-using-systemd , method. For example::
[root@srv lib]# systemctl set-environment _ADDITIONAL_SEARCHD_PARAMS='--coredump'
[root@srv lib]# systemctl restart manticore
[root@srv lib]# ps aux|grep searchd
mantico+ 1955 0.0 0.0 61964 1580 ? S 11:02 0:00 /usr/bin/searchd --config /etc/manticoresearch/manticore.conf --coredump
mantico+ 1956 0.6 0.0 392744 2664 ? Sl 11:02 0:00 /usr/bin/searchd --config /etc/manticoresearch/manticore.conf --coredump
-
Ensure that your operating system allows you to save core dumps by checking that:
/proc/sys/kernel/core_pattern
is not empty. This is the location where the core dumps will be saved. To save core dumps to a file such ascore.searchd.1773.centos-4gb-hel1-1.1636454937
, run the following command:echo "/cores/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern
-
searchd should be started with
ulimit -c unlimited
. If you start Manticore using systemctl, it will automatically set the limit to infinity as indicated by the following line in the manticore.service file:[root@srv lib]# grep CORE /lib/systemd/system/manticore.service LimitCORE=infinity
Manticore Search and Manticore Columnar Library are written in C++, which results in compiled binary files that execute optimally on your operating system. However, when running a binary, your system does not have full access to the names of variables, functions, methods, and classes. This information is provided in separate "debuginfo" or "symbol packages."
Debug symbols are essential for troubleshooting and debugging, as they allow you to visualize the system state when it crashed, including the names of functions. Manticore Search provides a backtrace in the searchd log and generates a coredump if run with the --coredump flag. Without symbols, all you will see is internal offsets, making it difficult or impossible to decode the cause of the crash. If you need to make a bug report about a crash, the Manticore team will often require debug symbols to assist you.
To install Manticore Search/Manticore Columnar Library debug symbols, you will need to install the *debuginfo*
package for CentOS, the *dbgsym*
package for Ubuntu and Debian, or the *dbgsymbols*
package for Windows and macOS. These packages should be the same version as the installed Manticore. For example, if you installed Manticore Search in Centos 8 from the package https://repo.manticoresearch.com/repository/manticoresearch/release/centos/8/x86_64/manticore-4.0.2_210921.af497f245-1.el8.x86_64.rpm , the corresponding package with symbols would be https://repo.manticoresearch.com/repository/manticoresearch/release/centos/8/x86_64/manticore-debuginfo-4.0.2_210921.af497f245-1.el8.x86_64.rpm
Note that both packages have the same commit id af497f245
, which corresponds to the commit that this version was built from.
If you have installed Manticore from a Manticore APT/YUM repository, you can use one of the following tools:
debuginfo-install
in CentOS 7dnf debuginfo-install
CentOS 8find-dbgsym-packages
in Debian and Ubuntu
to find a debug symbols package for you.
- Find the build ID in the output of
file /usr/bin/searchd
:
[root@srv lib]# file /usr/bin/searchd
/usr/bin/searchd: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=2c582e9f564ea1fbeb0c68406c271ba27034a6d3, stripped
In this case, the build ID is 2c582e9f564ea1fbeb0c68406c271ba27034a6d3.
- Find symbols in
/usr/lib/debug/.build-id
like this:
[root@srv ~]# ls -la /usr/lib/debug/.build-id/2c/582e9f564ea1fbeb0c68406c271ba27034a6d3*
lrwxrwxrwx. 1 root root 23 Nov 9 10:42 /usr/lib/debug/.build-id/2c/582e9f564ea1fbeb0c68406c271ba27034a6d3 -> ../../../../bin/searchd
lrwxrwxrwx. 1 root root 27 Nov 9 10:42 /usr/lib/debug/.build-id/2c/582e9f564ea1fbeb0c68406c271ba27034a6d3.debug -> ../../usr/bin/searchd.debug
To fix your bug, developers often need to reproduce it locally. To do this, they need your configuration file, table files, binlog (if present), and sometimes source data (such as data from external storages or XML/CSV files) and queries.
Attach your data when you create a ticket on Github. If the data is too large or sensitive, you can upload it to our write-only S3 storage at s3://s3.manticoresearch.com/write-only/
. Here's how you can do it using the Minio client:
- Install the client https://min.io/docs/minio/linux/reference/minio-mc.html#install-mc For example on 64-bit Linux:
curl https://dl.min.io/client/mc/release/linux-amd64/mc \
--create-dirs \
-o $HOME/minio-binaries/mc
chmod +x $HOME/minio-binaries/mc
export PATH=$PATH:$HOME/minio-binaries/
- Add our s3 host (use full path to executable or change into its directory):
cd $HOME/minio-binaries
and then./mc config host add manticore http://s3.manticoresearch.com:9000 manticore manticore
- Copy your files (use full path to executable or change into its directory):
cd $HOME/minio-binaries
and then./mc cp -r issue-1234/ manticore/write-only/issue-1234
. Make sure the folder name is unique and best if it corresponds to the issue on GitHub where you described the bug.
DEBUG [ subcommand ]
The DEBUG
statement is designed for developers and testers to call various internal or VIP commands. However, it is not intended for production use as the syntax of the subcommand
component may change freely in any build.
To view a list of useful commands and DEBUG
statement subcommands available in the current context, simply call DEBUG
without any parameters.
mysql> debug;
+-------------------------------------------------------------------------+----------------------------------------------------------------------------------------+
| command | meaning |
+-------------------------------------------------------------------------+----------------------------------------------------------------------------------------+
| flush logs | emulate USR1 signal |
| reload indexes | emulate HUP signal |
| debug token <password> | calculate token for password |
| debug malloc_stats | perform 'malloc_stats', result in searchd.log |
| debug malloc_trim | pefrorm 'malloc_trim' call |
| debug sleep <N> | sleep for <N> seconds |
| debug tasks | display global tasks stat (use select from @@system.tasks instead) |
| debug sched | display task manager schedule (use select from @@system.sched instead) |
| debug merge <TBL> [chunk] <X> [into] [chunk] <Y> [option sync=1,byid=0] | For RT table <TBL> merge disk chunk X into disk chunk Y |
| debug drop [chunk] <X> [from] <TBL> [option sync=1] | For RT table <TBL> drop disk chunk X |
| debug files <TBL> [option format=all|external] | list files belonging to <TBL>. 'all' - including external (wordforms, stopwords, etc.) |
| debug close | ask server to close connection from it's side |
| debug compress <TBL> [chunk] <X> [option sync=1] | Compress disk chunk X of RT table <TBL> (wipe out deleted documents) |
| debug split <TBL> [chunk] <X> on @<uservar> [option sync=1] | Split disk chunk X of RT table <TBL> using set of DocIDs from @uservar |
| debug wait <cluster> [like 'xx'] [option timeout=3] | wait <cluster> ready, but no more than 3 secs. |
| debug wait <cluster> status <N> [like 'xx'] [option timeout=13] | wait <cluster> commit achieve <N>, but no more than 13 secs |
| debug meta | Show max_matches/pseudo_shards. Needs set profiling=1 |
| debug trace OFF|'path/to/file' [<N>] | trace flow to file until N bytes written, or 'trace OFF' |
| debug curl <URL> | request given url via libcurl |
+-------------------------------------------------------------------------+----------------------------------------------------------------------------------------+
19 rows in set (0.00 sec)
Same from VIP connection:
mysql> debug;
+-------------------------------------------------------------------------+----------------------------------------------------------------------------------------+
| command | meaning |
+-------------------------------------------------------------------------+----------------------------------------------------------------------------------------+
| flush logs | emulate USR1 signal |
| reload indexes | emulate HUP signal |
| debug shutdown <password> | emulate TERM signal |
| debug crash <password> | crash daemon (make SIGSEGV action) |
| debug token <password> | calculate token for password |
| debug malloc_stats | perform 'malloc_stats', result in searchd.log |
| debug malloc_trim | pefrorm 'malloc_trim' call |
| debug procdump | ask watchdog to dump us |
| debug setgdb on|off | enable or disable potentially dangerous crash dumping with gdb |
| debug setgdb status | show current mode of gdb dumping |
| debug sleep <N> | sleep for <N> seconds |
| debug tasks | display global tasks stat (use select from @@system.tasks instead) |
| debug sched | display task manager schedule (use select from @@system.sched instead) |
| debug merge <TBL> [chunk] <X> [into] [chunk] <Y> [option sync=1,byid=0] | For RT table <TBL> merge disk chunk X into disk chunk Y |
| debug drop [chunk] <X> [from] <TBL> [option sync=1] | For RT table <TBL> drop disk chunk X |
| debug files <TBL> [option format=all|external] | list files belonging to <TBL>. 'all' - including external (wordforms, stopwords, etc.) |
| debug close | ask server to close connection from it's side |
| debug compress <TBL> [chunk] <X> [option sync=1] | Compress disk chunk X of RT table <TBL> (wipe out deleted documents) |
| debug split <TBL> [chunk] <X> on @<uservar> [option sync=1] | Split disk chunk X of RT table <TBL> using set of DocIDs from @uservar |
| debug wait <cluster> [like 'xx'] [option timeout=3] | wait <cluster> ready, but no more than 3 secs. |
| debug wait <cluster> status <N> [like 'xx'] [option timeout=13] | wait <cluster> commit achieve <N>, but no more than 13 secs |
| debug meta | Show max_matches/pseudo_shards. Needs set profiling=1 |
| debug trace OFF|'path/to/file' [<N>] | trace flow to file until N bytes written, or 'trace OFF' |
| debug curl <URL> | request given url via libcurl |
+-------------------------------------------------------------------------+----------------------------------------------------------------------------------------+
24 rows in set (0.00 sec)
All debug XXX
commands should be regarded as non-stable and subject to modification at any time, so don't be surprised if they change. This example output may not reflect the actual available commands, so try it on your system to see what is available on your instance. Additionally, there is no detailed documentation provided aside from this short 'meaning' column.
As a quick illustration, two commands available only to VIP clients are described below - shutdown and crash. Both require a token, which can be generated with the debug token subcommand, and added to the shutdown_token param in the searchd section of the config file. If no such section exists, or if the provided password hash does not match the token stored in the config, the subcommands will do nothing.
mysql> debug token hello;
+-------------+------------------------------------------+
| command | result |
+-------------+------------------------------------------+
| debug token | aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d |
+-------------+------------------------------------------+
1 row in set (0,00 sec)
The subcommand shutdown
will send a TERM signal to the server, causing it to shut down. This can be dangerous, as nobody wants to accidentally stop a production service. Therefore, it requires a VIP connection and the password to be used.
The subcommand crash
literally causes a crash. It may be used for testing purposes, such as to test how the system manager maintains the service's liveness or to test the feasibility of tracking coredumps.
If some commands are found to be useful in a more general context, they may be moved from the debug subcommands to a more stable and generic location (as exemplified by the debug tasks
and debug sched
in the table).
- CREATE TABLE - Creates new table
- CREATE TABLE LIKE - Creates table using another one as a template
- CREATE TABLE LIKE ... WITH DATA - Copies a table
- DESCRIBE - Prints out table's field list and their types
- ALTER TABLE - Changes table schema / settings
- ALTER TABLE REBUILD SECONDARY - Updates/recovers secondary indexes
- ALTER TABLE type='distributed' - Updates/recovers secondary indexes
- ALTER TABLE RENAME
- DROP TABLE IF EXISTS - Deletes a table (if it exists)
- SHOW TABLES - Shows tables list
- SHOW CREATE TABLE - Shows SQL command how to create the table
- SHOW TABLE STATUS - Shows information about current table status
- SHOW TABLE SETTINGS - Shows table settings
- INSERT - Adds new documents
- REPLACE - Replaces existing documents with new ones
- REPLACE .. SET - Replaces one or multiple fields in a table
- UPDATE - Does in-place update in documents
- DELETE - Deletes documents
- TRUNCATE TABLE - Deletes all documents from table
- BACKUP - Backs up your tables
- SELECT - Searches
- WHERE - Filters
- GROUP BY - Groups search results
- GROUP BY ORDER - Orders groups
- GROUP BY HAVING - Filters groups
- OPTION - Query Options
- FACET - Faceted search
- SUB-SELECTS - About using SELECT sub-queries
- JOIN - Joining tables in SELECT
- EXPLAIN QUERY - Shows query execution plan without running the query itself
- SHOW META - Shows extended information about executed query
- SHOW PROFILE - Shows profiling information about executed query
- SHOW PLAN - Shows query execution plan after the query was executed
- SHOW WARNINGS - Shows warnings from the latest query
- FLUSH ATTRIBUTES - Forces flushing updated attributes to disk
- FLUSH HOSTNAMES - Renews IPs associates to agent host names
- FLUSH LOGS - Initiates reopen of searchd log and query log files (similar to USR1)
- FLUSH RAMCHUNK - Force creating a new disk chunk
- FLUSH TABLE - Flushes real-time table RAM chunk to disk
- OPTIMIZE TABLE - Enqueues real-time table for optimization
- ATTACH TABLE - Moves data from a plain table to a real-time table
- IMPORT TABLE - Imports previously created RT or PQ table into a server running in the RT mode
- JOIN CLUSTER - Joins a replication cluster
- ALTER CLUSTER - Adds/deletes a table to a replication cluster
- SET CLUSTER - Changes replication cluster settings
- DELETE CLUSTER - Deletes a replication cluster
- RELOAD TABLE - Rotates a plain table
- RELOAD TABLES - Rotates all plain tables
- CALL SUGGEST, CALL QSUGGEST - Suggests spell-corrected words
- CALL SNIPPETS - Builds a highlighted results snippet from provided data and query
- CALL PQ - Runs a percolate query
- CALL KEYWORDS - Used to check how keywords are tokenized. Also allows to retrieve tokenized forms of provided keywords
- CREATE FUNCTION - Installs a user-defined function (UDF)
- DROP FUNCTION - Drops a user-defined function (UDF)
- CREATE PLUGIN - Installs a plugin
- CREATE BUDDY PLUGIN - Installs a Buddy plugin
- DROP PLUGIN - Drops a plugin
- DROP BUDDY PLUGIN - Drops a Buddy plugin
- RELOAD PLUGINS - Reloads all plugins from a given library
- ENABLE BUDDY PLUGIN - Reactivates a previously disabled Buddy plugin
- DISABLE BUDDY PLUGIN - Deactivates an active Buddy plugin
- SHOW STATUS - Displays a number of useful performance counters
- SHOW THREADS - Lists all currently active client threads
- SHOW VARIABLES - Lists server-wide variables and their values
- SHOW VERSION - Provides detailed version information of various components of the instance.
- /sql - Execute an SQL statement over HTTP JSON
- /cli - Provides an HTTP command line interface
- /insert - Inserts a document into a real-time table
- /pq/tbl_name/doc - Adds a PQ rule to a percolate table
- /update - Updates a document in a real-time table
- /replace - Replaces an existing document in a real-time table or inserts it if it doesn't exist
- /pq/tbl_name/doc/N?refresh=1 - Replaces a PQ rule in a percolate table
- /delete - Removes a document from a table
- /bulk - Executes multiple insert, update, or delete operations in a single call. Learn more about bulk inserts here.
- /search - Performs a search
- /search -> knn - Performs a KNN vector search
- /pq/tbl_name/search - Performs a reverse search in a percolate table
- /tbl_name/_mapping - Creates a table schema in the Elasticsearch style
- access_plain_attrs
- access_blob_attrs
- access_doclists
- access_hitlists
- access_dict
- attr_update_reserve
- bigram_freq_words
- bigram_index
- blend_chars
- blend_mode
- charset_table
- dict
- docstore_block_size
- docstore_compression
- docstore_compression_level
- embedded_limit
- exceptions
- expand_keywords
- global_idf
- hitless_words
- html_index_attrs
- html_remove_elements
- html_strip
- ignore_chars
- index_exact_words
- index_field_lengths
- index_sp
- index_token_filter
- index_zones
- infix_fields
- inplace_enable
- inplace_hit_gap
- inplace_reloc_factor
- inplace_write_factor
- killlist_target
- max_substring_len
- min_infix_len
- min_prefix_len
- min_stemming_len
- min_word_len
- morphology
- morphology_skip_fields
- ngram_chars
- ngram_len
- overshort_step
- path
- phrase_boundary
- phrase_boundary_step
- prefix_fields
- preopen
- read_buffer_docs
- read_buffer_hits
- regexp_filter
- stopwords
- stopword_step
- stopwords_unstemmed
- type
- wordforms
- local
- agent
- agent_connect_timeout
- agent_blackhole
- agent_persistent
- agent_query_timeout
- agent_retry_count
- ha_strategy
- mirror_retry_count
- rt_attr_bigint
- rt_attr_bool
- rt_attr_float
- rt_attr_float_vector
- rt_attr_json
- rt_attr_multi_64
- rt_attr_multi
- rt_attr_string
- rt_attr_timestamp
- rt_attr_uint
- rt_field
- rt_mem_limit
- OR
- MAYBE
- NOT - NOT operator
- @field - field search operator
- @field[N] - field position limit modifier
- @(field1,field2,...) - multi-field search operator
- @!field - ignore field search operator
- @!(field1,field2,...) - ignore multi-field search operator
- @* - all-field search operator
- "word1 word2 ... " - phrase search operator
- "word1 word2 ... "~N - proximity search operator
- "word1 word2 ... "/N - quorum matching operator
- word1 << word2 << word3 - strict order operator
- =word1 - exact form modifier
- ^word1 - field-start modifier
- word2$ - field-end modifier
- word^N - keyword IDF boost modifier
- word1 NEAR/N word2 - NEAR, generalized proximity operator
- word1 NOTNEAR/N word2 - NOTNEAR, negative assertion operator
- word1 PARAGRAPH word2 PARAGRAPH "word3 word4" - PARAGRAPH operator
- word1 SENTENCE word2 SENTENCE "word3 word4" - SENTENCE operator
- ZONE:(h3,h4) - ZONE limit operator
- ZONESPAN:(h2) - ZONESPAN limit operator
- @@relaxed - suppresses errors about missing fields
- t?st - wildcard operators
- REGEX(/pattern/) - REGEX operator
- ABS() - Returns absolute value
- ATAN2() - Returns arctangent function of two arguments
- BITDOT() - Returns sum of products of each bit of a mask multiplied with its weight
- CEIL() - Returns smallest integer value greater or equal to the argument
- COS() - Returns cosine of the argument
- CRC32() - Returns CRC32 value of the argument
- EXP() - Returns exponent of the argument
- FIBONACCI() - Returns the N-th Fibonacci number, where N is the integer argument
- FLOOR() - Returns the largest integer value lesser or equal to the argument
- GREATEST() - Takes JSON/MVA array as the argument and returns the greatest value in that array
- IDIV() - Returns result of an integer division of the first argument by the second argument
- LEAST() - Takes JSON/MVA array as the argument, and returns the least value in that array
- LN() - Returns natural logarithm of the argument
- LOG10() - Returns common logarithm of the argument
- LOG2() - Returns binary logarithm of the argument
- MAX() - Returns the larger of two arguments
- MIN() - Returns the smaller of two arguments
- POW() - Returns the first argument raised to the power of the second argument
- RAND() - Returns random float between 0 and 1
- SIN() - Returns sine of the argument
- SQRT() - Returns square root of the argument
- BM25F() - Returns precise BM25F formula value
- EXIST() - Replaces non-existing columns with default values
- GROUP_CONCAT() - Produces a comma-separated list of the attribute values of all documents in the group
- HIGHLIGHT() - Highlights search results
- MIN_TOP_SORTVAL() - Returns sort key value of the worst found element in the current top-N matches
- MIN_TOP_WEIGHT() - Returns weight of the worst found element in the current top-N matches
- PACKEDFACTORS() - Outputs weighting factors
- REMOVE_REPEATS() - Removes repeated adjusted rows with the same 'column' value
- WEIGHT() - Returns fulltext match score
- ZONESPANLIST() - Returns pairs of matched zone spans
- QUERY() - Returns current full-text query
- BIGINT() - Forcibly promotes the integer argument to 64-bit type
- DOUBLE() - Forcibly promotes given argument to floating point type
- INTEGER() - Forcibly promotes given argument to 64-bit signed type
- TO_STRING() - Forcibly promotes the argument to string type
- UINT() - Converts the given argument to 32-bit unsigned integer type
- UINT64() - Converts the given argument to 64-bit unsigned integer type
- SINT() - Interprets 32-bit unsigned integer as signed 64-bit integer
- ALL() - Returns 1 if condition is true for all elements in the array
- ANY() - Returns 1 if condition is true for any element in the array
- CONTAINS() - Checks whether the (x,y) point is within the given polygon
- IF() - Checks whether the 1st argument is equal to 0.0, returns the 2nd argument if it is not zero or the 3rd one when it is
- IN() - Returns 1 if the first argument is equal to any of the other arguments, or 0 otherwise
- INDEXOF() - Iterates through all elements in the array and returns index of the first matching element
- INTERVAL() - Returns index of the argument that is less than the first argument
- LENGTH() - Returns number of elements in MVA
- REMAP() - Allows to make some exceptions of expression values depending on the condition values
- NOW() - Returns current timestamp as an INTEGER
- CURTIME() - Returns current time in local timezone
- CURDATE() - Returns current date in local timezone
- UTC_TIME() - Returns current time in UTC timezone
- UTC_TIMESTAMP() - Returns current date/time in UTC timezone
- SECOND() - Returns integer second from the timestamp argument
- MINUTE() - Returns integer minute from the timestamp argument
- HOUR() - Returns integer hour from the timestamp argument
- DAY() - Returns integer day from the timestamp argument
- MONTH() - Returns integer month from the timestamp argument
- QUARTER() - Returns the integer quarter of the year from a timestamp argument
- YEAR() - Returns integer year from the timestamp argument
- DAYNAME() - Returns the weekday name for a given timestamp argument
- MONTHNAME() - Returns the name of the month for a given timestamp argument
- DAYOFWEEK() - Returns the integer weekday index for a given timestamp argument
- DAYOFYEAR() - Returns the integer day of the year for a given timestamp argument
- YEARWEEK() - Returns the integer year and the day code of the first day of current week for a given timestamp argument
- YEARMONTH() - Returns integer year and month code from the timestamp argument
- YEARMONTHDAY() - Returns integer year, month and day code from the timestamp argument
- TIMEDIFF() - Returns difference between the timstamps
- DATEDIFF() - Returns the number of days between two given timestamps
- DATE() - Formats the date part from a timestamp argument
- TIME() - Formats the time part from a timestamp argument
- DATE_FORMAT() - Returns a formatted string based on the provided date and format arguments
- GEODIST() - Computes geosphere distance between two given points
- GEOPOLY2D() - Creates a polygon that takes in account the Earth's curvature
- POLY2D() - Creates a simple polygon in plain space
- CONCAT() - Concatenates two or more strings
- REGEX() - Returns 1 if regular expression matched to string of attribute and 0 otherwise
- SNIPPET() - Highlights search results
- SUBSTRING_INDEX() - Returns a substring of the string before the specified number of delimiter occurs
- CONNECTION_ID() - Returns the current connection ID
- KNN_DIST() - Returns KNN vector search distance
- LAST_INSERT_ID() - Returns ids of documents inserted or replaced by last statement in the current session
To be put to section common {}
in configuration file:
- lemmatizer_base - Lemmatizer dictionaries base path
- progressive_merge - Defines order of merging disk chunks in a real-time table
- json_autoconv_keynames - Whether and how to auto-convert key names within JSON attributes
- json_autoconv_numbers - Automatically detects and converts possible JSON strings that represent numbers into numeric attributes
- on_json_attr_error - What to do if JSON format errors are found
- plugin_dir - Location for the dynamic libraries and UDFs
indexer
is a tool to create plain tables
To be put to section indexer {}
in configuration file:
- lemmatizer_cache - Lemmatizer cache size
- max_file_field_buffer - Maximum file field adaptive buffer size
- max_iops - Maximum indexation I/O operations per second
- max_iosize - Maximum allowed I/O operation size
- max_xmlpipe2_field - Maximum allowed field size for XMLpipe2 source type
- mem_limit - Indexing RAM usage limit
- on_file_field_error - How to handle IO errors in file fields
- write_buffer - Write buffer size
- ignore_non_plain - To ignore warnings about non-plain tables
indexer [OPTIONS] [indexname1 [indexname2 [...]]]
- --all - Rebuilds all tables from the config
- --buildstops - Analyzes the table source as if indexing the data, generating a list of indexed terms
- --buildfreqs - Adds the frequency count to the table for --buildstops
- --config, -c - Specifies the path to the configuration file
- --dump-rows - Dumps rows retrieved by SQL source(s) into the specified file
- --help - Displays all available parameters
- --keep-attrs - Allows reuse of existing attributes when reindexing
- --keep-attrs-names - Specifies which attributes to reuse from the existing table
- --merge-dst-range - Applies the given filter range during merging
- --merge-killlists - Alters kill list processing when merging tables
- --merge - Combines two plain tables into one
- --nohup - Prevents indexer from sending SIGHUP when this option is enabled
- --noprogress - Hides progress details
- --print-queries - Outputs SQL queries sent by the indexer to the database
- --print-rt - Displays data fetched from SQL source(s) as INSERTs into a real-time table
- --quiet - Suppresses all output
- --rotate - Initiates table rotation after all tables are built
- --sighup-each - Triggers rotation of each table after it's built
- -v - Displays indexer version
index_converter
is a tool designed to convert tables created with Sphinx/Manticore Search 2.x into the Manticore Search 3.x table format.
index_converter {--config /path/to/config|--path}
- --config, -c - Path to table configuration file
- --index - Specifies which table to convert
- --path - Sets path containing table(s) instead of the configuration file
- --strip-path - Removes path from filenames referenced by table
- --large-docid - Allows conversion of documents with ids larger than 2^63
- --output-dir - Writes new files in a specified folder
- --all - Converts all tables from the configuration file / path
- --killlist-target - Sets target tables for applying kill-lists
searchd
is the Manticore server.
To be put in the searchd {}
section of the configuration file:
- access_blob_attrs - Defines how table's blob attributes file is accessed
- access_doclists - Defines how table's doclists file is accessed
- access_hitlists - Defines how table's hitlists file is accessed
- access_plain_attrs - Defines how search server accesses table's plain attributes
- access_dict - Defines how table's dictionary file is accessed
- agent_connect_timeout - Remote agent connection timeout
- agent_query_timeout - Remote agent query timeout
- agent_retry_count - Specifies the number of times Manticore tries to connect and query remote agents
- agent_retry_delay - Specifies the delay before retrying to query a remote agent in case of failure
- attr_flush_period - Sets the time period between flushing updated attributes to disk
- binlog_flush - Binary log transaction flush/sync mode
- binlog_max_log_size - Maximum binary log file size
- binlog_path - Binary log files path
- client_timeout - Maximum time to wait between requests when using persistent connections
- collation_libc_locale - Server libc locale
- collation_server - Default server collation
- data_dir - Path to data directory where Manticore stores everything (RT mode)
- docstore_cache_size - Maximum size of document blocks from document storage held in memory
- expansion_limit - Maximum number of expanded keywords for a single wildcard
- grouping_in_utc - Enables using UTC timezone for grouping time fields
- ha_period_karma - Agent mirror statistics window size
- ha_ping_interval - Interval between agent mirror pings
- hostname_lookup - Hostnames renew strategy
- jobs_queue_size - Defines the maximum number of "jobs" allowed in the queue simultaneously
- listen - Specifies IP address and port or Unix-domain socket path for searchd to listen on
- listen_backlog - TCP listen backlog
- listen_tfo - Enables TCP_FASTOPEN flag for all listeners
- log - Path to Manticore server log file
- max_batch_queries - Limits the number of queries per batch
- max_connections - Maximum number of active connections
- max_filters - Maximum allowed per-query filter count
- max_filter_values - Maximum allowed per-filter values count
- max_open_files - Maximum number of files allowed to be opened by server
- max_packet_size - Maximum allowed network packet size
- mysql_version_string - Server version string returned via MySQL protocol
- net_throttle_accept - Defines how many clients are accepted on each iteration of the network loop
- net_throttle_action - Defines how many requests are processed on each iteration of the network loop
- net_wait_tm - Controls busy loop interval of a network thread
- net_workers - Number of network threads
- network_timeout - Network timeout for client requests
- node_address - Specifies network address of the node
- persistent_connections_limit - Maximum number of simultaneous persistent connections to remote persistent agents
- pid_file - Path to Manticore server pid file
- predicted_time_costs - Costs for the query time prediction model
- preopen_tables - Determines whether to forcibly preopen all tables on startup
- pseudo_sharding - Enables pseudo-sharding for search queries to plain and real-time tables
- qcache_max_bytes - Maximum RAM allocated for cached result sets
- qcache_thresh_msec - Minimum wall time threshold for a query result to be cached
- qcache_ttl_sec - Expiration period for a cached result set
- query_log - Path to query log file
- query_log_format - Query log format
- query_log_min_msec - Prevents logging too fast queries
- query_log_mode - Query log file permissions mode
- read_buffer_docs - Per-keyword read buffer size for document lists
- read_buffer_hits - Per-keyword read buffer size for hit lists
- read_unhinted - Unhinted read size
- rt_flush_period - How often Manticore flushes real-time tables' RAM chunks to disk
- rt_merge_iops - Maximum number of I/O operations (per second) that real-time chunks merging thread is allowed to do
- rt_merge_maxiosize - Maximum size of an I/O operation that real-time chunks merging thread is allowed to do
- seamless_rotate - Prevents searchd stalls while rotating tables with huge amounts of data to precache
- secondary_indexes - Enables using secondary indexes for search queries
- server_id - Server identifier used as a seed to generate a unique document ID
- shutdown_timeout - Searchd
--stopwait
timeout - shutdown_token - SHA1 hash of the password required to invoke
shutdown
command from VIP SQL connection - snippets_file_prefix - Prefix to prepend to the local file names when generating snippets in
load_files
mode - sphinxql_state - Path to file where the current SQL state will be serialized
- sphinxql_timeout - Maximum time to wait between requests from a MySQL client
- ssl_ca - Path to SSL Certificate Authority certificate file
- ssl_cert - Path to server's SSL certificate
- ssl_key - Path to SSL certificate key of the server
- subtree_docs_cache - Maximum common subtree document cache size
- subtree_hits_cache - Maximum common subtree hit cache size, per-query
- timezone - Timezone used by date/time-related functions
- thread_stack - Maximum stack size for a job
- unlink_old - Whether to unlink .old table copies on successful rotation
- watchdog - Whether to enable or disable Manticore server watchdog
searchd [OPTIONS]
- --config, -c - Specifies the path to the configuration file
- --console - Forces the server to run in console mode
- --coredump - Enables core dump saving upon crash
- --cpustats - Enables CPU time reporting
- --delete - Removes the Manticore service from Microsoft Management Console and other locations where services are registered
- --force-preread - Prevents the server from serving incoming connections until table files are pre-read
- --help, -h - Displays all available parameters
- --table (--index) - Restricts the server to serve only the specified table
- --install - Installs searchd as a service in Microsoft Management Console
- --iostats - Enables input/output reporting
- --listen, -l - Overrides listen from the configuration file
- --logdebug, --logdebugv, --logdebugvv - Enables additional debug output in the server log
- --logreplication - Enables extra replication debug output in the server log
- --new-cluster - Initializes a replication cluster and sets the server as a reference node with cluster restart protection
- --new-cluster-force - Initializes a replication cluster and sets the server as a reference node, bypassing cluster restart protection
- --nodetach - Keeps searchd running in the foreground
- --ntservice - Used by Microsoft Management Console to launch searchd as a service on Windows platforms
- --pidfile - Overrides pid_file in the configuration file
- --port, p - Specifies the port searchd should listen on, ignoring the port specified in the configuration file
- --replay-flags - Sets additional binary log replay options
- --servicename - Assigns the given name to searchd when installing or deleting the service, as displayed in Microsoft Management Console
- --status - Queries the running search service to return its status
- --stop - Stops the Manticore server
- --stopwait - Stops the Manticore server gracefully
- --strip-path - Removes path names from all file names referenced in the table
- -v - Displays version information
- MANTICORE_TRACK_DAEMON_SHUTDOWN - Enables detailed logging during searchd shutdown
Assorted table maintenance features helpful for troubleshooting.
indextool <command> [options]
Utilized for dumping various debug information related to the physical table.
indextool <command> [options]
- --config, -c - Specifies the path to the configuration file
- --quiet, -q - Keeps indextool quiet; no banner output, etc.
- --help, -h - Lists all available parameters
- -v - Displays version information
- Indextool - Verifies the configuration file
- --buildidf - Builds an IDF file from one or more dictionary dumps
- --build-infixes - Builds infixes for an existing dict=keywords table
- --dumpheader - Quickly dumps the provided table header file
- --dumpconfig - Dumps table definition from the given table header file in a nearly compliant manticore.conf format
- --dumpheader - Dumps table header by table name while looking up the header path in the configuration file
- --dumpdict - Dumps the table dictionary
- --dumpdocids - Dumps document IDs by table name
- --dumphitlist - Dumps all occurrences of the given keyword/id in the specified table
- --docextract - Runs table check pass on entire dictionary/docs/hits and collects all words and hits belonging to the requested document
- --fold - Tests tokenization based on table settings
- --htmlstrip - Filters STDIN using HTML stripper settings for the specified table
- --mergeidf - Merges multiple .idf files into a single file
- --morph - Applies morphology to the provided STDIN and outputs the result to stdout
- --check - Checks table data files for consistency
- --check-id-dups - Checks for duplicate IDs
- --check-disk-chunk - Checks a single disk chunk of an RT table
- --strip-path - Removes path names from all file names referenced in the table
- --rotate - Determines whether to check a table waiting for rotation when using
--check
- --apply-killlists - Applies kill-lists for all tables listed in the configuration file
Splits compound words into their components.
wordbreaker [-dict path/to/dictionary_file] {split|test|bench}
- STDIN - Accepts a string to break into parts
- -dict - Specifies the dictionary file to use
- split|test|bench - Specifies the command
Extracts the contents of a dictionary file using ispell or MySpell format
spelldump [options] <dictionary> <affix> [result] [locale-name]
- dictionary - Main dictionary file
- affix - Affix file for the dictionary
- result - Specifies the output destination for the dictionary data
- locale-name - Specifies the locale details to use
A comprehensive alphabetical list of keywords currently reserved in Manticore SQL syntax (thus, they cannot be used as identifiers).
AND, AS, BY, COLUMNARSCAN, DATE_ADD, DATE_SUB, DAY, DISTINCT, DIV, DOCIDINDEX, EXPLAIN, FACET, FALSE, FORCE, FROM, HOUR, IGNORE, IN, INTERVAL, INDEXES, INNER, IS, JOIN, KNN, LEFT, LIMIT, MINUTE, MOD, MONTH, NOT, NO_COLUMNARSCAN, NO_DOCIDINDEX, NO_SECONDARYINDEX, NULL, OFFSET, ON, OR, ORDER, QUARTER, REGEX, RELOAD, SECOND, SECONDARYINDEX, SELECT, SYSFILTERS, TRUE, WEEK, YEAR
- 2.4.1
- 2.5.1
- 2.6.0
- 2.6.1
- 2.6.2
- 2.6.3
- 2.6.4
- 2.7.0
- 2.7.1
- 2.7.2
- 2.7.3
- 2.7.4
- 2.7.5
- 2.8.0
- 2.8.1
- 2.8.2
- 3.0.0
- 3.0.2
- 3.1.0
- 3.1.2
- 3.2.0
- 3.2.2
- 3.3.0
- 3.4.0
- 3.4.2
- 3.5.0
- 3.5.2
- 3.5.4
- 4.0.2
- 4.2.0
- 5.0.2. Installation page
- 6.0.0. Installation page
- 6.0.2. Installation page
- 6.0.4. Installation page
- 6.2.0. Installation page
- 6.2.12. Installation page
- 6.3.0. Installation page
- 6.3.2. Installation page
- 6.3.4. Installation page
- 6.3.6. Installation page
- CREATE TABLE - Creates new table
- CREATE TABLE LIKE - Creates table using another one as a template
- CREATE TABLE LIKE ... WITH DATA - Copies a table
- DESCRIBE - Prints out table's field list and their types
- ALTER TABLE - Changes table schema / settings
- ALTER TABLE REBUILD SECONDARY - Updates/recovers secondary indexes
- ALTER TABLE type='distributed' - Updates/recovers secondary indexes
- ALTER TABLE RENAME
- DROP TABLE IF EXISTS - Deletes a table (if it exists)
- SHOW TABLES - Shows tables list
- SHOW CREATE TABLE - Shows SQL command how to create the table
- SHOW TABLE STATUS - Shows information about current table status
- SHOW TABLE SETTINGS - Shows table settings
- INSERT - Adds new documents
- REPLACE - Replaces existing documents with new ones
- REPLACE .. SET - Replaces one or multiple fields in a table
- UPDATE - Does in-place update in documents
- DELETE - Deletes documents
- TRUNCATE TABLE - Deletes all documents from table
- BACKUP - Backs up your tables
- SELECT - Searches
- WHERE - Filters
- GROUP BY - Groups search results
- GROUP BY ORDER - Orders groups
- GROUP BY HAVING - Filters groups
- OPTION - Query Options
- FACET - Faceted search
- SUB-SELECTS - About using SELECT sub-queries
- JOIN - Joining tables in SELECT
- EXPLAIN QUERY - Shows query execution plan without running the query itself
- SHOW META - Shows extended information about executed query
- SHOW PROFILE - Shows profiling information about executed query
- SHOW PLAN - Shows query execution plan after the query was executed
- SHOW WARNINGS - Shows warnings from the latest query
- FLUSH ATTRIBUTES - Forces flushing updated attributes to disk
- FLUSH HOSTNAMES - Renews IPs associates to agent host names
- FLUSH LOGS - Initiates reopen of searchd log and query log files (similar to USR1)
- FLUSH RAMCHUNK - Force creating a new disk chunk
- FLUSH TABLE - Flushes real-time table RAM chunk to disk
- OPTIMIZE TABLE - Enqueues real-time table for optimization
- ATTACH TABLE - Moves data from a plain table to a real-time table
- IMPORT TABLE - Imports previously created RT or PQ table into a server running in the RT mode
- JOIN CLUSTER - Joins a replication cluster
- ALTER CLUSTER - Adds/deletes a table to a replication cluster
- SET CLUSTER - Changes replication cluster settings
- DELETE CLUSTER - Deletes a replication cluster
- RELOAD TABLE - Rotates a plain table
- RELOAD TABLES - Rotates all plain tables
- CALL SUGGEST, CALL QSUGGEST - Suggests spell-corrected words
- CALL SNIPPETS - Builds a highlighted results snippet from provided data and query
- CALL PQ - Runs a percolate query
- CALL KEYWORDS - Used to check how keywords are tokenized. Also allows to retrieve tokenized forms of provided keywords
- CREATE FUNCTION - Installs a user-defined function (UDF)
- DROP FUNCTION - Drops a user-defined function (UDF)
- CREATE PLUGIN - Installs a plugin
- CREATE BUDDY PLUGIN - Installs a Buddy plugin
- DROP PLUGIN - Drops a plugin
- DROP BUDDY PLUGIN - Drops a Buddy plugin
- RELOAD PLUGINS - Reloads all plugins from a given library
- ENABLE BUDDY PLUGIN - Reactivates a previously disabled Buddy plugin
- DISABLE BUDDY PLUGIN - Deactivates an active Buddy plugin
- SHOW STATUS - Displays a number of useful performance counters
- SHOW THREADS - Lists all currently active client threads
- SHOW VARIABLES - Lists server-wide variables and their values
- SHOW VERSION - Provides detailed version information of various components of the instance.
- /sql - Execute an SQL statement over HTTP JSON
- /cli - Provides an HTTP command line interface
- /insert - Inserts a document into a real-time table
- /pq/tbl_name/doc - Adds a PQ rule to a percolate table
- /update - Updates a document in a real-time table
- /replace - Replaces an existing document in a real-time table or inserts it if it doesn't exist
- /pq/tbl_name/doc/N?refresh=1 - Replaces a PQ rule in a percolate table
- /delete - Removes a document from a table
- /bulk - Executes multiple insert, update, or delete operations in a single call. Learn more about bulk inserts here.
- /search - Performs a search
- /search -> knn - Performs a KNN vector search
- /pq/tbl_name/search - Performs a reverse search in a percolate table
- /tbl_name/_mapping - Creates a table schema in the Elasticsearch style
- access_plain_attrs
- access_blob_attrs
- access_doclists
- access_hitlists
- access_dict
- attr_update_reserve
- bigram_freq_words
- bigram_index
- blend_chars
- blend_mode
- charset_table
- dict
- docstore_block_size
- docstore_compression
- docstore_compression_level
- embedded_limit
- exceptions
- expand_keywords
- global_idf
- hitless_words
- html_index_attrs
- html_remove_elements
- html_strip
- ignore_chars
- index_exact_words
- index_field_lengths
- index_sp
- index_token_filter
- index_zones
- infix_fields
- inplace_enable
- inplace_hit_gap
- inplace_reloc_factor
- inplace_write_factor
- killlist_target
- max_substring_len
- min_infix_len
- min_prefix_len
- min_stemming_len
- min_word_len
- morphology
- morphology_skip_fields
- ngram_chars
- ngram_len
- overshort_step
- path
- phrase_boundary
- phrase_boundary_step
- prefix_fields
- preopen
- read_buffer_docs
- read_buffer_hits
- regexp_filter
- stopwords
- stopword_step
- stopwords_unstemmed
- type
- wordforms
- local
- agent
- agent_connect_timeout
- agent_blackhole
- agent_persistent
- agent_query_timeout
- agent_retry_count
- ha_strategy
- mirror_retry_count
- rt_attr_bigint
- rt_attr_bool
- rt_attr_float
- rt_attr_float_vector
- rt_attr_json
- rt_attr_multi_64
- rt_attr_multi
- rt_attr_string
- rt_attr_timestamp
- rt_attr_uint
- rt_field
- rt_mem_limit
- OR
- MAYBE
- NOT - NOT operator
- @field - field search operator
- @field[N] - field position limit modifier
- @(field1,field2,...) - multi-field search operator
- @!field - ignore field search operator
- @!(field1,field2,...) - ignore multi-field search operator
- @* - all-field search operator
- "word1 word2 ... " - phrase search operator
- "word1 word2 ... "~N - proximity search operator
- "word1 word2 ... "/N - quorum matching operator
- word1 << word2 << word3 - strict order operator
- =word1 - exact form modifier
- ^word1 - field-start modifier
- word2$ - field-end modifier
- word^N - keyword IDF boost modifier
- word1 NEAR/N word2 - NEAR, generalized proximity operator
- word1 NOTNEAR/N word2 - NOTNEAR, negative assertion operator
- word1 PARAGRAPH word2 PARAGRAPH "word3 word4" - PARAGRAPH operator
- word1 SENTENCE word2 SENTENCE "word3 word4" - SENTENCE operator
- ZONE:(h3,h4) - ZONE limit operator
- ZONESPAN:(h2) - ZONESPAN limit operator
- @@relaxed - suppresses errors about missing fields
- t?st - wildcard operators
- REGEX(/pattern/) - REGEX operator
- ABS() - Returns absolute value
- ATAN2() - Returns arctangent function of two arguments
- BITDOT() - Returns sum of products of each bit of a mask multiplied with its weight
- CEIL() - Returns smallest integer value greater or equal to the argument
- COS() - Returns cosine of the argument
- CRC32() - Returns CRC32 value of the argument
- EXP() - Returns exponent of the argument
- FIBONACCI() - Returns the N-th Fibonacci number, where N is the integer argument
- FLOOR() - Returns the largest integer value lesser or equal to the argument
- GREATEST() - Takes JSON/MVA array as the argument and returns the greatest value in that array
- IDIV() - Returns result of an integer division of the first argument by the second argument
- LEAST() - Takes JSON/MVA array as the argument, and returns the least value in that array
- LN() - Returns natural logarithm of the argument
- LOG10() - Returns common logarithm of the argument
- LOG2() - Returns binary logarithm of the argument
- MAX() - Returns the larger of two arguments
- MIN() - Returns the smaller of two arguments
- POW() - Returns the first argument raised to the power of the second argument
- RAND() - Returns random float between 0 and 1
- SIN() - Returns sine of the argument
- SQRT() - Returns square root of the argument
- BM25F() - Returns precise BM25F formula value
- EXIST() - Replaces non-existing columns with default values
- GROUP_CONCAT() - Produces a comma-separated list of the attribute values of all documents in the group
- HIGHLIGHT() - Highlights search results
- MIN_TOP_SORTVAL() - Returns sort key value of the worst found element in the current top-N matches
- MIN_TOP_WEIGHT() - Returns weight of the worst found element in the current top-N matches
- PACKEDFACTORS() - Outputs weighting factors
- REMOVE_REPEATS() - Removes repeated adjusted rows with the same 'column' value
- WEIGHT() - Returns fulltext match score
- ZONESPANLIST() - Returns pairs of matched zone spans
- QUERY() - Returns current full-text query
- BIGINT() - Forcibly promotes the integer argument to 64-bit type
- DOUBLE() - Forcibly promotes given argument to floating point type
- INTEGER() - Forcibly promotes given argument to 64-bit signed type
- TO_STRING() - Forcibly promotes the argument to string type
- UINT() - Converts the given argument to 32-bit unsigned integer type
- UINT64() - Converts the given argument to 64-bit unsigned integer type
- SINT() - Interprets 32-bit unsigned integer as signed 64-bit integer
- ALL() - Returns 1 if condition is true for all elements in the array
- ANY() - Returns 1 if condition is true for any element in the array
- CONTAINS() - Checks whether the (x,y) point is within the given polygon
- IF() - Checks whether the 1st argument is equal to 0.0, returns the 2nd argument if it is not zero or the 3rd one when it is
- IN() - Returns 1 if the first argument is equal to any of the other arguments, or 0 otherwise
- INDEXOF() - Iterates through all elements in the array and returns index of the first matching element
- INTERVAL() - Returns index of the argument that is less than the first argument
- LENGTH() - Returns number of elements in MVA
- REMAP() - Allows to make some exceptions of expression values depending on the condition values
- NOW() - Returns current timestamp as an INTEGER
- CURTIME() - Returns current time in local timezone
- CURDATE() - Returns current date in local timezone
- UTC_TIME() - Returns current time in UTC timezone
- UTC_TIMESTAMP() - Returns current date/time in UTC timezone
- SECOND() - Returns integer second from the timestamp argument
- MINUTE() - Returns integer minute from the timestamp argument
- HOUR() - Returns integer hour from the timestamp argument
- DAY() - Returns integer day from the timestamp argument
- MONTH() - Returns integer month from the timestamp argument
- QUARTER() - Returns the integer quarter of the year from a timestamp argument
- YEAR() - Returns integer year from the timestamp argument
- DAYNAME() - Returns the weekday name for a given timestamp argument
- MONTHNAME() - Returns the name of the month for a given timestamp argument
- DAYOFWEEK() - Returns the integer weekday index for a given timestamp argument
- DAYOFYEAR() - Returns the integer day of the year for a given timestamp argument
- YEARWEEK() - Returns the integer year and the day code of the first day of current week for a given timestamp argument
- YEARMONTH() - Returns integer year and month code from the timestamp argument
- YEARMONTHDAY() - Returns integer year, month and day code from the timestamp argument
- TIMEDIFF() - Returns difference between the timstamps
- DATEDIFF() - Returns the number of days between two given timestamps
- DATE() - Formats the date part from a timestamp argument
- TIME() - Formats the time part from a timestamp argument
- DATE_FORMAT() - Returns a formatted string based on the provided date and format arguments
- GEODIST() - Computes geosphere distance between two given points
- GEOPOLY2D() - Creates a polygon that takes in account the Earth's curvature
- POLY2D() - Creates a simple polygon in plain space
- CONCAT() - Concatenates two or more strings
- REGEX() - Returns 1 if regular expression matched to string of attribute and 0 otherwise
- SNIPPET() - Highlights search results
- SUBSTRING_INDEX() - Returns a substring of the string before the specified number of delimiter occurs
- CONNECTION_ID() - Returns the current connection ID
- KNN_DIST() - Returns KNN vector search distance
- LAST_INSERT_ID() - Returns ids of documents inserted or replaced by last statement in the current session
To be put to section common {}
in configuration file:
- lemmatizer_base - Lemmatizer dictionaries base path
- progressive_merge - Defines order of merging disk chunks in a real-time table
- json_autoconv_keynames - Whether and how to auto-convert key names within JSON attributes
- json_autoconv_numbers - Automatically detects and converts possible JSON strings that represent numbers into numeric attributes
- on_json_attr_error - What to do if JSON format errors are found
- plugin_dir - Location for the dynamic libraries and UDFs
indexer
is a tool to create plain tables
To be put to section indexer {}
in configuration file:
- lemmatizer_cache - Lemmatizer cache size
- max_file_field_buffer - Maximum file field adaptive buffer size
- max_iops - Maximum indexation I/O operations per second
- max_iosize - Maximum allowed I/O operation size
- max_xmlpipe2_field - Maximum allowed field size for XMLpipe2 source type
- mem_limit - Indexing RAM usage limit
- on_file_field_error - How to handle IO errors in file fields
- write_buffer - Write buffer size
- ignore_non_plain - To ignore warnings about non-plain tables
indexer [OPTIONS] [indexname1 [indexname2 [...]]]
- --all - Rebuilds all tables from the config
- --buildstops - Analyzes the table source as if indexing the data, generating a list of indexed terms
- --buildfreqs - Adds the frequency count to the table for --buildstops
- --config, -c - Specifies the path to the configuration file
- --dump-rows - Dumps rows retrieved by SQL source(s) into the specified file
- --help - Displays all available parameters
- --keep-attrs - Allows reuse of existing attributes when reindexing
- --keep-attrs-names - Specifies which attributes to reuse from the existing table
- --merge-dst-range - Applies the given filter range during merging
- --merge-killlists - Alters kill list processing when merging tables
- --merge - Combines two plain tables into one
- --nohup - Prevents indexer from sending SIGHUP when this option is enabled
- --noprogress - Hides progress details
- --print-queries - Outputs SQL queries sent by the indexer to the database
- --print-rt - Displays data fetched from SQL source(s) as INSERTs into a real-time table
- --quiet - Suppresses all output
- --rotate - Initiates table rotation after all tables are built
- --sighup-each - Triggers rotation of each table after it's built
- -v - Displays indexer version
index_converter
is a tool designed to convert tables created with Sphinx/Manticore Search 2.x into the Manticore Search 3.x table format.
index_converter {--config /path/to/config|--path}
- --config, -c - Path to table configuration file
- --index - Specifies which table to convert
- --path - Sets path containing table(s) instead of the configuration file
- --strip-path - Removes path from filenames referenced by table
- --large-docid - Allows conversion of documents with ids larger than 2^63
- --output-dir - Writes new files in a specified folder
- --all - Converts all tables from the configuration file / path
- --killlist-target - Sets target tables for applying kill-lists
searchd
is the Manticore server.
To be put in the searchd {}
section of the configuration file:
- access_blob_attrs - Defines how table's blob attributes file is accessed
- access_doclists - Defines how table's doclists file is accessed
- access_hitlists - Defines how table's hitlists file is accessed
- access_plain_attrs - Defines how search server accesses table's plain attributes
- access_dict - Defines how table's dictionary file is accessed
- agent_connect_timeout - Remote agent connection timeout
- agent_query_timeout - Remote agent query timeout
- agent_retry_count - Specifies the number of times Manticore tries to connect and query remote agents
- agent_retry_delay - Specifies the delay before retrying to query a remote agent in case of failure
- attr_flush_period - Sets the time period between flushing updated attributes to disk
- binlog_flush - Binary log transaction flush/sync mode
- binlog_max_log_size - Maximum binary log file size
- binlog_path - Binary log files path
- client_timeout - Maximum time to wait between requests when using persistent connections
- collation_libc_locale - Server libc locale
- collation_server - Default server collation
- data_dir - Path to data directory where Manticore stores everything (RT mode)
- docstore_cache_size - Maximum size of document blocks from document storage held in memory
- expansion_limit - Maximum number of expanded keywords for a single wildcard
- grouping_in_utc - Enables using UTC timezone for grouping time fields
- ha_period_karma - Agent mirror statistics window size
- ha_ping_interval - Interval between agent mirror pings
- hostname_lookup - Hostnames renew strategy
- jobs_queue_size - Defines the maximum number of "jobs" allowed in the queue simultaneously
- listen - Specifies IP address and port or Unix-domain socket path for searchd to listen on
- listen_backlog - TCP listen backlog
- listen_tfo - Enables TCP_FASTOPEN flag for all listeners
- log - Path to Manticore server log file
- max_batch_queries - Limits the number of queries per batch
- max_connections - Maximum number of active connections
- max_filters - Maximum allowed per-query filter count
- max_filter_values - Maximum allowed per-filter values count
- max_open_files - Maximum number of files allowed to be opened by server
- max_packet_size - Maximum allowed network packet size
- mysql_version_string - Server version string returned via MySQL protocol
- net_throttle_accept - Defines how many clients are accepted on each iteration of the network loop
- net_throttle_action - Defines how many requests are processed on each iteration of the network loop
- net_wait_tm - Controls busy loop interval of a network thread
- net_workers - Number of network threads
- network_timeout - Network timeout for client requests
- node_address - Specifies network address of the node
- persistent_connections_limit - Maximum number of simultaneous persistent connections to remote persistent agents
- pid_file - Path to Manticore server pid file
- predicted_time_costs - Costs for the query time prediction model
- preopen_tables - Determines whether to forcibly preopen all tables on startup
- pseudo_sharding - Enables pseudo-sharding for search queries to plain and real-time tables
- qcache_max_bytes - Maximum RAM allocated for cached result sets
- qcache_thresh_msec - Minimum wall time threshold for a query result to be cached
- qcache_ttl_sec - Expiration period for a cached result set
- query_log - Path to query log file
- query_log_format - Query log format
- query_log_min_msec - Prevents logging too fast queries
- query_log_mode - Query log file permissions mode
- read_buffer_docs - Per-keyword read buffer size for document lists
- read_buffer_hits - Per-keyword read buffer size for hit lists
- read_unhinted - Unhinted read size
- rt_flush_period - How often Manticore flushes real-time tables' RAM chunks to disk
- rt_merge_iops - Maximum number of I/O operations (per second) that real-time chunks merging thread is allowed to do
- rt_merge_maxiosize - Maximum size of an I/O operation that real-time chunks merging thread is allowed to do
- seamless_rotate - Prevents searchd stalls while rotating tables with huge amounts of data to precache
- secondary_indexes - Enables using secondary indexes for search queries
- server_id - Server identifier used as a seed to generate a unique document ID
- shutdown_timeout - Searchd
--stopwait
timeout - shutdown_token - SHA1 hash of the password required to invoke
shutdown
command from VIP SQL connection - snippets_file_prefix - Prefix to prepend to the local file names when generating snippets in
load_files
mode - sphinxql_state - Path to file where the current SQL state will be serialized
- sphinxql_timeout - Maximum time to wait between requests from a MySQL client
- ssl_ca - Path to SSL Certificate Authority certificate file
- ssl_cert - Path to server's SSL certificate
- ssl_key - Path to SSL certificate key of the server
- subtree_docs_cache - Maximum common subtree document cache size
- subtree_hits_cache - Maximum common subtree hit cache size, per-query
- timezone - Timezone used by date/time-related functions
- thread_stack - Maximum stack size for a job
- unlink_old - Whether to unlink .old table copies on successful rotation
- watchdog - Whether to enable or disable Manticore server watchdog
searchd [OPTIONS]
- --config, -c - Specifies the path to the configuration file
- --console - Forces the server to run in console mode
- --coredump - Enables core dump saving upon crash
- --cpustats - Enables CPU time reporting
- --delete - Removes the Manticore service from Microsoft Management Console and other locations where services are registered
- --force-preread - Prevents the server from serving incoming connections until table files are pre-read
- --help, -h - Displays all available parameters
- --table (--index) - Restricts the server to serve only the specified table
- --install - Installs searchd as a service in Microsoft Management Console
- --iostats - Enables input/output reporting
- --listen, -l - Overrides listen from the configuration file
- --logdebug, --logdebugv, --logdebugvv - Enables additional debug output in the server log
- --logreplication - Enables extra replication debug output in the server log
- --new-cluster - Initializes a replication cluster and sets the server as a reference node with cluster restart protection
- --new-cluster-force - Initializes a replication cluster and sets the server as a reference node, bypassing cluster restart protection
- --nodetach - Keeps searchd running in the foreground
- --ntservice - Used by Microsoft Management Console to launch searchd as a service on Windows platforms
- --pidfile - Overrides pid_file in the configuration file
- --port, p - Specifies the port searchd should listen on, ignoring the port specified in the configuration file
- --replay-flags - Sets additional binary log replay options
- --servicename - Assigns the given name to searchd when installing or deleting the service, as displayed in Microsoft Management Console
- --status - Queries the running search service to return its status
- --stop - Stops the Manticore server
- --stopwait - Stops the Manticore server gracefully
- --strip-path - Removes path names from all file names referenced in the table
- -v - Displays version information
- MANTICORE_TRACK_DAEMON_SHUTDOWN - Enables detailed logging during searchd shutdown
Assorted table maintenance features helpful for troubleshooting.
indextool <command> [options]
Utilized for dumping various debug information related to the physical table.
indextool <command> [options]
- --config, -c - Specifies the path to the configuration file
- --quiet, -q - Keeps indextool quiet; no banner output, etc.
- --help, -h - Lists all available parameters
- -v - Displays version information
- Indextool - Verifies the configuration file
- --buildidf - Builds an IDF file from one or more dictionary dumps
- --build-infixes - Builds infixes for an existing dict=keywords table
- --dumpheader - Quickly dumps the provided table header file
- --dumpconfig - Dumps table definition from the given table header file in a nearly compliant manticore.conf format
- --dumpheader - Dumps table header by table name while looking up the header path in the configuration file
- --dumpdict - Dumps the table dictionary
- --dumpdocids - Dumps document IDs by table name
- --dumphitlist - Dumps all occurrences of the given keyword/id in the specified table
- --docextract - Runs table check pass on entire dictionary/docs/hits and collects all words and hits belonging to the requested document
- --fold - Tests tokenization based on table settings
- --htmlstrip - Filters STDIN using HTML stripper settings for the specified table
- --mergeidf - Merges multiple .idf files into a single file
- --morph - Applies morphology to the provided STDIN and outputs the result to stdout
- --check - Checks table data files for consistency
- --check-id-dups - Checks for duplicate IDs
- --check-disk-chunk - Checks a single disk chunk of an RT table
- --strip-path - Removes path names from all file names referenced in the table
- --rotate - Determines whether to check a table waiting for rotation when using
--check
- --apply-killlists - Applies kill-lists for all tables listed in the configuration file
Splits compound words into their components.
wordbreaker [-dict path/to/dictionary_file] {split|test|bench}
- STDIN - Accepts a string to break into parts
- -dict - Specifies the dictionary file to use
- split|test|bench - Specifies the command
Extracts the contents of a dictionary file using ispell or MySpell format
spelldump [options] <dictionary> <affix> [result] [locale-name]
- dictionary - Main dictionary file
- affix - Affix file for the dictionary
- result - Specifies the output destination for the dictionary data
- locale-name - Specifies the locale details to use
A comprehensive alphabetical list of keywords currently reserved in Manticore SQL syntax (thus, they cannot be used as identifiers).
AND, AS, BY, COLUMNARSCAN, DATE_ADD, DATE_SUB, DAY, DISTINCT, DIV, DOCIDINDEX, EXPLAIN, FACET, FALSE, FORCE, FROM, HOUR, IGNORE, IN, INTERVAL, INDEXES, INNER, IS, JOIN, KNN, LEFT, LIMIT, MINUTE, MOD, MONTH, NOT, NO_COLUMNARSCAN, NO_DOCIDINDEX, NO_SECONDARYINDEX, NULL, OFFSET, ON, OR, ORDER, QUARTER, REGEX, RELOAD, SECOND, SECONDARYINDEX, SELECT, SYSFILTERS, TRUE, WEEK, YEAR
- 2.4.1
- 2.5.1
- 2.6.0
- 2.6.1
- 2.6.2
- 2.6.3
- 2.6.4
- 2.7.0
- 2.7.1
- 2.7.2
- 2.7.3
- 2.7.4
- 2.7.5
- 2.8.0
- 2.8.1
- 2.8.2
- 3.0.0
- 3.0.2
- 3.1.0
- 3.1.2
- 3.2.0
- 3.2.2
- 3.3.0
- 3.4.0
- 3.4.2
- 3.5.0
- 3.5.2
- 3.5.4
- 4.0.2
- 4.2.0
- 5.0.2. Installation page
- 6.0.0. Installation page
- 6.0.2. Installation page
- 6.0.4. Installation page
- 6.2.0. Installation page
- 6.2.12. Installation page
- 6.3.0. Installation page
- 6.3.2. Installation page
- 6.3.4. Installation page
- 6.3.6. Installation page