In many cases, you might want to encrypt traffic between your client and the server. To do that, you can specify that the server should use the HTTPS protocol rather than HTTP.
To enable HTTPS, at least the following two directives should be set in the searchd section of the config, and there should be at least one listener set to https
In addition to that, you can specify the certificate authority's certificate (aka root certificate) in:
- ssl_ca certificate authority's certificate file
- with CA
- without CA
Example with CA:
ssl_ca = ca-cert.pem
ssl_cert = server-cert.pem
ssl_key = server-key.pem
Example without CA:
ssl_cert = server-cert.pem
ssl_key = server-key.pem
These steps will help you generate the SSL certificates using the 'openssl' tool.
The server can use a Certificate Authority to verify the signature of certificates, but it can also work with just a private key and certificate (without the CA certificate).
openssl genrsa 2048 > ca-key.pem
To generate a self-signed CA (root) certificate from the private key (make sure to fill in at least the "Common Name"), use the following command:
openssl req -new -x509 -nodes -days 365 -key ca-key.pem -out ca-cert.pem
The server uses the server certificate to secure communication with the client. To generate the certificate request and server private key (ensure that you fill in at least the "Common Name" and that it is different from the root certificate's common name), execute the following commands:
openssl req -newkey rsa:2048 -days 365 -nodes -keyout server-key.pem -out server-req.pem
openssl rsa -in server-key.pem -out server-key.pem
openssl x509 -req -in server-req.pem -days 365 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem
Once completed, you can verify that the key and certificate files were generated correctly by running:
openssl verify -CAfile ca-cert.pem server-cert.pem
When your SSL configuration is valid, the following features are available:
- You can connect to the multi-protocol port (when no listener type is specified) over HTTPS and run queries. Both the request and response will be SSL encrypted.
- You can connect to the dedicated https port with HTTP and run queries. The connection will be secured (attempting to connect to this port via plain HTTP will be rejected with a 400 error code).
- You can connect to the MySQL port with a MySQL client using a secured connection. The session will be secured. Note that the Linux
mysql
client tries to use SSL by default, so a typical connection to Manticore with a valid SSL configuration will most likely be secured. You can check this by running the SQL 'status' command after connecting.
If your SSL configuration is not valid for any reason (which the daemon detects by the fact that a secured connection cannot be established), apart from an invalid configuration there may be other reasons, such as the inability to load the appropriate SSL library at all. In this case, the following things will not work or will work in a non-secured manner:
- You cannot connect to the multi-protocol port with HTTPS. The connection will be dropped.
- You cannot connect to the dedicated
https
port. The HTTPS connections will be dropped - Connection to the
mysql
port via a MySQL client will not support SSL securing. If the client requires SSL, the connection will fail. If SSL is not required, it will use plain MySQL or compressed connections.
- Binary API connections (such as connections from old clients or inter-daemons master-agent communication) are not secured.
- SSL for replication needs to be set up separately. However, since the SST stage of the replication is done through the binary API connection, it is not secured either.
- You can still use any external proxies (e.g., SSH tunneling) to secure your connections.
Read-only mode for a connection disables any table or global modifications. Therefore, queries like create
, drop
, various types of alter
, attach
, optimize
, and data modification queries such as insert
, replace
, delete
, update
, and others will all be rejected. Changing daemon-wide settings using SET GLOBAL
is also not possible in this mode.
However, you can still perform all search operations, generate snippets, and run CALL PQ
queries. Additionally, you can modify local (connection-wide) settings.
To check if your current connection is read-only or not, execute the show variables like 'session_read_only'
statement. A value of 1
indicates read-only, while 0
means not read-only (usual).
Typically, you define a separate listen directive in read-only mode by adding the suffix _readonly
to it. However, you can also do this interactively for the current connection by executing the SET ro=1
statement via SQL.
If you're connected to a VIP socket, you can execute SET ro=0
(even if the socket you are connected to was defined as read-only in the config and not interactively). This will switch the connection to the usual (not read-only) mode with all modifications allowed.
For standard (non-VIP) connections, escaping read-only mode is only possible by reconnecting if it was set interactively, or by updating the configuration file and restarting the daemon.
⪢ Logging
Query logging can be enabled by setting the query_log directive in the searchd section of the configuration file.
Queries can also be sent to syslog by setting syslog
instead of a file path. In this case, all search queries will be sent to the syslog daemon with LOG_INFO
priority, prefixed with [query]
instead of a timestamp. Only the plain
log format is supported for syslog.
- Config
searchd {
...
query_log = /var/log/query.log
query_log_format = sphinxql # default
...
}
Two query log formats are supported:
sphinxql
(default): Logs in SQL format. It also provides an easy way to replay logged queries.plain
: Logs full-text queries in a simple text format. This format is recommended if most of your queries are primarily full-text or if you do not need to log non-full-text components, such as filtering by attributes, sorting, or grouping. Queries logged in theplain
format cannot be replayed. Note that queries processed through Buddy are not logged in this mode.
To switch between the formats, you can use the searchd setting query_log_format.
The SQL log format is the default setting. In this mode, Manticore logs all successful and unsuccessful select queries. Requests sent as SQL or via the binary API are logged in the SQL format, but JSON queries are logged as is. This type of logging only works with plain log files and does not support the 'syslog' service for logging.
- Config
query_log_format = sphinxql # default
The features of the Manticore SQL log format compared to the plain format include:
- Full statement data is logged where possible.
- Errors and warnings are logged.
- The query log can be replayed.
- Additional performance counters (currently, per-agent distributed query times) are logged.
- Each log entry is a valid Manticore SQL/JSON statement that reconstructs the full request, except if the logged request is too large and needs to be shortened for performance reasons.
- JSON requests and additional messages, counters, etc., are logged as comments.
- Example
/* Sun Apr 28 12:38:02.808 2024 conn 2 (127.0.0.1:53228) real 0.000 wall 0.000 found 0 */ SELECT * FROM test WHERE MATCH('test') OPTION ranker=proximity;
/* Sun Apr 28 12:38:05.585 2024 conn 2 (127.0.0.1:53228) real 0.001 wall 0.001 found 0 */ SELECT * FROM test WHERE MATCH('test') GROUP BY channel_id OPTION ranker=proximity;
/* Sun Apr 28 12:40:57.366 2024 conn 4 (127.0.0.1:53256) real 0.000 wall 0.000 found 0 */ /*{
"table" : "test",
"query":
{
"match":
{
"*" : "test"
}
},
"_source": ["f"],
"limit": 30
} */
With the plain
log format, Manticore logs all successfully executed search queries in a simple text format. Non-full-text parts of the queries are not logged. JSON queries are recorded as single-line entries. Queries processed through Buddy are not logged.
- Config
query_log_format = plain
The log format is as follows:
[query-date] real-time wall-time [match-mode/filters-count/sort-mode total-matches (offset,limit) @groupby-attr] [table-name] {perf-stats} query
where:
real-time
is the time from the start to the finish of the query.wall-time
is similar to real-time, but excludes time spent waiting for agents and merging result sets from them.perf-stats
includes CPU/IO stats when Manticore is started with--cpustats
(or it was enabled viaSET GLOBAL cpustats=1
) and/or--iostats
(or it was enabled viaSET GLOBAL iostats=1
):ios
is the number of file I/O operations carried out;kb
is the amount of data in kilobytes read from the table files;ms
is the time spent on I/O operations.cpums
is the time in milliseconds spent on CPU processing the query.
match-mode
can have one of the following values:- "all" for
SPH_MATCH_ALL
mode; - "any" for
SPH_MATCH_ANY
mode; - "phr" for
SPH_MATCH_PHRASE
mode; - "bool" for
SPH_MATCH_BOOLEAN
mode; - "ext" for
SPH_MATCH_EXTENDED
mode; - "ext2" for
SPH_MATCH_EXTENDED2
mode; - "scan" if the full scan mode was used, either by being specified with
SPH_MATCH_FULLSCAN
or if the query was empty.
- "all" for
sort-mode
can have one of the following values:- "rel" for
SPH_SORT_RELEVANCE
mode; - "attr-" for
SPH_SORT_ATTR_DESC
mode; - "attr+" for
SPH_SORT_ATTR_ASC
mode; - "tsegs" for
SPH_SORT_TIME_SEGMENTS
mode; - "ext" for
SPH_SORT_EXTENDED
mode.
- "rel" for
Note: the SPH*
modes are specific to the sphinx
legacy interface. SQL and JSON interfaces will log, in most cases, ext2
as match-mode
and ext
and rel
as sort-mode
.
- Example
[Fri Jun 29 21:17:58 2021] 0.004 sec [all/0/rel 35254 (0,20)] [lj] [ios=6 kb=111.1 ms=0.5] test
[Fri Jun 29 21:17:58 2021] 0.004 sec [all/0/rel 35254 (0,20)] [lj] [ios=6 kb=111.1 ms=0.5 cpums=0.3] test
[Sun Apr 28 15:09:38.712 2024] 0.000 sec 0.000 sec [ext2/0/ext 0 (0,20)] [test] test
[Sun Apr 28 15:09:44.974 2024] 0.000 sec 0.000 sec [ext2/0/ext 0 (0,20) @channel_id] [test] test
[Sun Apr 28 15:24:32.975 2024] 0.000 sec 0.000 sec [ext2/0/ext 0 (0,30)] [test] { "table" : "test", "query": { "match": { "*" : "test" } }, "_source": ["f"], "limit": 30 }
By default, all queries are logged. If you want to log only queries with execution times exceeding a specified limit, the query_log_min_msec
directive can be used.
The expected unit of measurement is milliseconds, but time suffix expressions can also be used.
- Config
searchd {
...
query_log = /var/log/query.log
query_log_min_msec = 1000
# query_log_min_msec = 1s
...
}
By default, the searchd and query log files are created with permission 600
, so only the user under which Manticore is running and root
can read the log files. The query_log_mode
option allows setting a different permission. This can be helpful for allowing other users to read the log files (for example, monitoring solutions running on non-root users).
- Config
searchd {
...
query_log = /var/log/query.log
query_log_mode = 666
...
}