Query logging can be enabled by setting the query_log
directive in the searchd section of the configuration file.
searchd {
...
query_log = /var/log/query.log
...
}
Queries can also be sent to syslog by setting syslog
instead of a file path.
In this case, all search queries will be sent to the syslog daemon with LOG_INFO
priority, prefixed with [query]
instead of a timestamp. Only the plain
log format is supported for syslog.
Two query log formats are supported:
plain
: recommended if most of your queries are primarily full-text, or if you don't care about non-full text components of your queries, such as filtering by attributes, sorting, grouping, etc. You can't replay queries logged in theplain
format.sphinxql
: recommended in all other cases (default). It also provides an easy way to replay logged queries.
With the plain
log format, Manticore logs all successfully executed search queries into a query log file. Here's an example:
[Fri Jun 29 21:17:58 2007] 0.004 sec 0.004 sec [all/0/rel 35254 (0,20)] [lj] test
[Fri Jun 29 21:20:34 2007] 0.024 sec 0.024 sec [all/0/rel 19886 (0,20) @channel_id] [lj] test
The log format is as follows:
[query-date] real-time wall-time [match-mode/filters-count/sort-mode total-matches (offset,limit) @groupby-attr] [table-name] query
- real-time is the time from the start to the finish of the query
- wall-time is similar to real-time, but excludes time spent waiting for agents and merging result sets from them
match-mode
can have one of the following values:
- "all" for
SPH_MATCH_ALL
mode; - "any" for
SPH_MATCH_ANY
mode; - "phr" for
SPH_MATCH_PHRASE
mode; - "bool" for
SPH_MATCH_BOOLEAN
mode; - "ext" for
SPH_MATCH_EXTENDED
mode; - "ext2" for
SPH_MATCH_EXTENDED2
mode; - "scan" if the full scan mode was used, either by being specified with
SPH_MATCH_FULLSCAN
or if the query was empty.
sort-mode
can have one of the following values:
- "rel" for
SPH_SORT_RELEVANCE
mode; - "attr-" for
SPH_SORT_ATTR_DESC
mode; - "attr+" for
SPH_SORT_ATTR_ASC
mode; - "tsegs" for
SPH_SORT_TIME_SEGMENTS
mode; - "ext" for
SPH_SORT_EXTENDED
mode.
Note: the SPH*
modes are specific to the sphinx
legacy interface. SQL and HTTP interfaces will log, in most cases, ext2
as match-mode
and ext
and rel
as sort-mode
.
If Manticore was started with --iostats
(or it was enabled via SET GLOBAL iostats=1
), the corresponding metrics will be included:
[Fri Jun 29 21:17:58 2021] 0.004 sec [all/0/rel 35254 (0,20)] [lj] [ios=6 kb=111.1 ms=0.5] test
where:
ios
- number of file I/O operations carried out;kb
- amount of data in kilobytes read from the table files;ioms
- time spent on I/O operations.
If Manticore was started with --cpustats
(ot it was enabled via SET GLOBAL cpustats=1
), the cpums
metric will be included in the log. The query log will then appear as follows:
[Fri Jun 29 21:17:58 2021] 0.004 sec [all/0/rel 35254 (0,20)] [lj] [ios=6 kb=111.1 ms=0.5 cpums=0.3] test
where cpums
is the time in milliseconds spent on CPU processing the query.
The SQL format is the default, but it can be changed by the searchd setting query_log_format:
searchd {
...
query_log = /var/log/query.log
query_log_format = sphinxql
...
}
In this format, the example from the previous section would appear as follows. (Wrapped below for readability, but with just one query per line in the actual log.)
/* Fri Jun 29 21:17:58.609 2007 2011 conn 2 real 0.004 wall 0.004 found 35254 */
SELECT * FROM test WHERE MATCH('test') OPTION ranker=proximity;
/* Fri Jun 29 21:20:34 2007.555 conn 3 real 0.024 wall 0.024 found 19886 */
SELECT * FROM test WHERE MATCH('test') GROUP BY channel_id OPTION ranker=proximity;
Note that all requests will be logged in this format, including those sent via SphinxAPI and SphinxSE, not just those sent via SQL. Also, keep in mind that this kind of logging works only with plain log files and will not work if you use the 'syslog' service for logging.
The features of Manticore SQL log format compared to the plain text one are as follows:
- Full statement data will be logged where possible.
- Errors and warnings are logged.
- Query log can be replayed.
- Additional performance counters (currently, per-agent distributed query times) are logged.
Use sphinxql:compact_in
to shorten your IN()
clauses in the log if you have too many values in it.
Every request (including both SphinxAPI and SQL) results in exactly one log line. Each log line must be a valid Manticore SQL statement that reconstructs the full request, except if the logged request is too big and needs shortening for performance reasons. Additional messages, counters, etc. can be logged in the comments section after the request.
By default, all queries are logged. If you want to log only queries with execution times exceeding a specified limit, the query_log_min_msec
directive can be used:
searchd {
...
query_log = /var/log/query.log
query_log_min_msec = 1000
...
}
The expected unit of measurement is milliseconds, but time suffix expressions can also be used, like:
searchd {
...
query_log = /var/log/query.log
query_log_min_msec = 1s
...
}
By default, the searchd and query log files are created with permission 600
, so only the user under which Manticore is running and root
can read the log files. The query_log_mode
option allows setting a different permission. This can be helpful for allowing other users to read the log files (for example, monitoring solutions running on non-root users).
searchd {
...
query_log = /var/log/query.log
query_log_mode = 666
...
}
By default, Manticore search daemon logs all runtime events in a searchd.log
file in the directory where searchd was started. In Linux by default, you can find the log at /var/log/manticore/searchd.log
.
The log file path/name can be overridden by setting log
in the searchd
section of the configuration file.
searchd {
...
log = /custom/path/to/searchd.log
...
}
- You can also use
syslog
as the file name. In this case, events will be sent to your server's syslog daemon. - In some cases, you might want to use
/dev/stdout
as the file name. In this case, on Linux, Manticore will simply output the events. This can be useful in Docker/Kubernetes environments.
Binary logging serves as a recovery mechanism for Real-Time table data, as well as attribute updates for plain tables that would otherwise only be stored in RAM until a flush occurs. When binary logs are enabled, searchd
records each transaction to the binlog file and utilizes it for recovery following an unclean shutdown. During a clean shutdown, RAM chunks are saved to disk, and all binlog files are subsequently unlinked.
By default, binary logging is enabled. On Linux systems, the default location for binlog.*
files is /var/lib/manticore/data/
.
In RT mode, binary logs are stored in the data_dir
folder, unless specified otherwise.
To disable binary logging, set binlog_path
to empty:
searchd {
...
binlog_path = # disable logging
...
Disabling binary logging can lead to better performance for Real-Time tables, but it also puts their data at risk.
You can use the following directive to set a custom path:
searchd {
...
binlog_path = /var/data
...
When logging is enabled, each transaction committed to an RT table is written to a log file. After an unclean shutdown, logs are automatically replayed upon startup, recovering any logged changes.
During normal operation, a new binlog file is opened whenever the binlog_max_log_size
limit is reached. Older, closed binlog files are retained until all transactions stored in them (from all tables) are flushed as a disk chunk. Setting the limit to 0 essentially prevents the binlog from being unlinked while searchd
is running; however, it will still be unlinked upon a clean shutdown. By default, there is no limit to the log file size.
binlog_max_log_size = 16M
There are 3 different binlog flushing strategies, controlled by the binlog_flush
directive:
- 0 - Flush and sync every second. This provides the best performance, but up to 1 second's worth of committed transactions may be lost in the event of a server crash or an OS/hardware crash.
- 1 - Flush and sync every transaction. This has the worst performance, but guarantees that every committed transaction's data is saved.
- 2 - Flush every transaction and sync every second. This offers good performance, and every committed transaction is guaranteed to be saved in the case of a server crash. However, up to 1 second's worth of committed transactions may be lost in the event of an OS/hardware crash.
The default mode is to flush every transaction and sync every second (mode 2).
searchd {
...
binlog_flush = 1 # ultimate safety, low speed
...
}
During recovery after an unclean shutdown, binlogs are replayed, and all logged transactions since the last good on-disk state are restored. Transactions are checksummed, so in case of binlog file corruption, garbage data will not be replayed; such a broken transaction will be detected and will stop the replay.
Intensive updates to a small RT table that fully fits into a RAM chunk can result in an ever-growing binlog that can never be unlinked until a clean shutdown. Binlogs essentially serve as append-only deltas against the last known good saved state on disk, and they cannot be unlinked unless the RAM chunk is saved. An ever-growing binlog is not ideal for disk usage and crash recovery time. To address this issue, you can configure searchd
to perform periodic RAM chunk flushes using the rt_flush_period
directive. With periodic flushes enabled, searchd
will maintain a separate thread that checks whether RT table RAM chunks need to be written back to disk. Once this occurs, the respective binlogs can be (and are) safely unlinked.
searchd {
...
rt_flush_period = 3600 # 1 hour
...
}
The default RT flush period is set to 10 hours.
It's important to note that rt_flush_period
only controls the frequency at which checks occur. There are no guarantees that a specific RAM chunk will be saved. For example, it doesn't make sense to regularly re-save a large RAM chunk that only receives a few rows worth of updates. Manticore automatically determines whether to perform the flush using a few heuristics.