Query logging can be enabled by setting the query_log directive in the searchd section of the configuration file.

Queries can also be sent to syslog by setting syslog instead of a file path. In this case, all search queries will be sent to the syslog daemon with LOG_INFO priority, prefixed with [query] instead of a timestamp. Only the plain log format is supported for syslog.

The SQL log format is the default setting. In this mode, Manticore logs all successful and unsuccessful select queries. Requests sent as SQL or via the binary API are logged in the SQL format, but JSON queries are logged as is. This type of logging only works with plain log files and does not support the 'syslog' service for logging.

The features of the Manticore SQL log format compared to the plain format include:

• Full statement data is logged where possible.
• Errors and warnings are logged.
• The query log can be replayed.
• Additional performance counters (currently, per-agent distributed query times) are logged.
• Each log entry is a valid Manticore SQL/JSON statement that reconstructs the full request, except if the logged request is too large and needs to be shortened for performance reasons.
• JSON requests and additional messages, counters, etc., are logged as comments.

With the plain log format, Manticore logs all successfully executed search queries in a simple text format. Non-full-text parts of the queries are not logged. JSON queries are recorded as single-line entries. Queries processed through Buddy are not logged.

The log format is as follows:

[query-date] real-time wall-time [match-mode/filters-count/sort-mode total-matches (offset,limit) @groupby-attr] [table-name] {perf-stats} query

where:

• real-time is the time from the start to the finish of the query.
• wall-time is similar to real-time, but excludes time spent waiting for agents and merging result sets from them.
• perf-stats includes CPU/IO stats when Manticore is started with --cpustats (or it was enabled via SET GLOBAL cpustats=1) and/or --iostats (or it was enabled via SET GLOBAL iostats=1):
  • ios is the number of file I/O operations carried out;
  • kb is the amount of data in kilobytes read from the table files;
  • ms is the time spent on I/O operations.
  • cpums is the time in milliseconds spent on CPU processing the query.
• match-mode can have one of the following values:
  • "all" for SPH_MATCH_ALL mode;
  • "any" for SPH_MATCH_ANY mode;
  • "phr" for SPH_MATCH_PHRASE mode;
  • "bool" for SPH_MATCH_BOOLEAN mode;
  • "ext" for SPH_MATCH_EXTENDED mode;
  • "ext2" for SPH_MATCH_EXTENDED2 mode;
  • "scan" if the full scan mode was used, either by being specified with SPH_MATCH_FULLSCAN or if the query was empty.
• sort-mode can have one of the following values:
  • "rel" for SPH_SORT_RELEVANCE mode;
  • "attr-" for SPH_SORT_ATTR_DESC mode;
  • "attr+" for SPH_SORT_ATTR_ASC mode;
  • "tsegs" for SPH_SORT_TIME_SEGMENTS mode;
  • "ext" for SPH_SORT_EXTENDED mode.

Note: the SPH* modes are specific to the sphinx legacy interface. SQL and JSON interfaces will log, in most cases, ext2 as match-mode and ext and rel as sort-mode.

By default, all queries are logged. If you want to log only queries with execution times exceeding a specified limit, the query_log_min_msec directive can be used.

The expected unit of measurement is milliseconds, but time suffix expressions can also be used.

By default, the searchd and query log files are created with permission 600, so only the user under which Manticore is running and root can read the log files. The query_log_mode option allows setting a different permission. This can be helpful for allowing other users to read the log files (for example, monitoring solutions running on non-root users).

To disable binary logging globally, set binlog_path to an empty value in the searchd configuration. Disabling binary logging requires a restart of the daemon and puts data at risk if the system shuts down unexpectedly.

You can use the following directive to set a custom path:

For more granular control, binary logging can be disabled at the table level for real-time tables by setting the binlog table parameter to 0. This option is not available for percolate tables.

For existing RT tables, binary logging can also be disabled by modifying the binlog parameter.

If binary logging was previously disabled, it can be re-enabled by setting the binlog parameter back to 1:

During normal operations, when the amount of data logged reaches a certain limit (set by binlog_max_log_size), a new log file starts. Old log files are kept until all changes in them are completely processed and saved to disk as a disk chunk. If this limit is set to 0, the log files are kept until the system is properly shut down. By default, there's no limit to how large these files can grow.

Each binlog file is named with a zero-padded number, like binlog.0000, binlog.0001, etc., typically showing four digits. You can change how many digits the number has with the setting binlog_filename_digits. If you have more binlog files than the number of digits can accommodate, the number of digits will be automatically increased to fit all files.

Important: To change the number of digits, you must first save all table data and properly shut down the system. Then, delete the old log files and restart the system.

You can choose between two ways to manage binary log files, which can be set with the binlog_common directive:

• Separate file for each table (default, 0): Each table saves its changes in its own log file. This setup is good if you have many tables that get updated at different times. It allows tables to be updated without waiting for others. Also, if there is a problem with one table's log file, it does not affect the others.
• Single file for all tables (1): All tables use the same binary log file. This method makes it easier to handle files because there are fewer of them. However, this could keep files longer than needed if one table still needs to save its updates. This setting might also slow things down if many tables need to update at the same time because all changes have to wait to be written to one file.

There are four different binlog flushing strategies, controlled by the binlog_flush directive:

• 0 - Data is written to disk (flushed) every second, and Manticore initiates making it secure on the disk (syncing) right after flushing. This method is the fastest, but if the server or computer crashes suddenly, some recently written data that hasn't been secured may be lost.
• 1 - Data is written to the binlog and synced immediately after each transaction. This method is the safest as it ensures that each change is immediately preserved, but it slows down writing.
• 2 - Data is written after each transaction, and a sync is initiated every second. This approach offers a balance, writing data regularly and quickly. However, if the computer fails, some of the data that was being secured might not finish saving. Also, syncing may take longer than one second depending on the disk.
• 3 - Similar to 2, but it also ensures the binlog file is synced before it is closed due to exceeding binlog_max_log_size.

The default mode is 2, which writes data after each transaction and starts syncing it every second, balancing speed and safety.

Intensive updates to a small RT table that fully fits into a RAM chunk can result in an ever-growing binlog that can never be unlinked until a clean shutdown. Binlogs essentially serve as append-only deltas against the last known good saved state on disk, and they cannot be unlinked unless the RAM chunk is saved. An ever-growing binlog is not ideal for disk usage and crash recovery time. To address this issue, you can configure searchd to perform periodic RAM chunk flushes using the rt_flush_period directive. With periodic flushes enabled, searchd will maintain a separate thread that checks whether RT table RAM chunks need to be written back to disk. Once this occurs, the respective binlogs can be (and are) safely unlinked.

The default RT flush period is set to 10 hours.