Manticore Search 2.x maintains compatibility with Sphinxsearch 2.x and can load existing indexes created by Sphinxsearch. In most cases, upgrading is just a matter of replacing the binaries.
Instead of sphinx.conf (in Linux normally located at /etc/sphinxsearch/sphinx.conf) Manticore by default uses /etc/manticoresearch/manticore.conf. It also runs under a different user and use different folders.
Systemd service name has changed from sphinx/sphinxsearch to manticore and the service runs under user manticore (Sphinx was using sphinx or sphinxsearch). It also uses a different folder for the PID file.
The folders used by default are /var/lib/manticore, /var/log/manticore, /var/run/manticore. You can still use the existing Sphinx config, but you need to manually change permissions for /var/lib/sphinxsearch and /var/log/sphinxsearch folders. Or, just rename globally 'sphinx' to 'manticore' in system files. If you use other folders (for data, wordforms files etc.) the ownership must be also switched to user manticore. The pid_file location should be changed to match the manticore.service to /var/run/manticore/searchd.pid.
If you want to use the Manticore folder instead, the index files need to be moved to the new data folder (/var/lib/manticore) and the permissions to be changed to user manticore.
Upgrading from Sphinx / Manticore 2.x to 3.x is not straightforward, because the index storage engine received a massive upgrade and the new searchd can't load older indexes and upgrade them to new format on-the-fly.
Manticore Search 3 got a redesigned index storage. Indexes created with Manticore/Sphinx 2.x cannot be loaded by Manticore Search 3 without a conversion. Because of the 4GB limitation, a real-time index in 2.x could still have several disk chunks after an optimize operation. After upgrading to 3.x, these indexes can now be optimized to 1-disk chunk with the usual OPTIMIZE command. Index files also changed. The only component that didn't get any structural changes is the .spp file (hitlists). .sps (strings/json) and .spm (MVA) are now held by .spb (var-length attributes). The new format has an .spm file present, but it's used for row map (previously it was dedicated for MVA attributes). The new extensions added are .spt (docid lookup), .sphi ( secondary index histograms), .spds (document storage). In case you are using scripts that manipulate index files, they should be adapted for the new file extensions.
The upgrade procedure may differ depending on your setup (number of servers in the cluster, whether you have HA or not etc.), but in general it's about creating new 3.x index versions and replacing your existing ones with them along with replacing older 2.x binaries with the new ones.
There are two special requirements to take care:
- Real-time indexes need to be flushed using FLUSH RAMCHUNK
- Plain indexes with kill-lists require adding a new directive in index configuration (see killlist_target)
Manticore Search 3 includes a new tool - index_converter - that can convert Sphinx 2.x / Manticore 2.x indexes to 3.x format. index_converter comes in a separate package which should be installed first. Using the convert tool create 3.x versions of your indexes. index_converter can write the new files in the existing data folder and backup the old files or it can write the new files to a chosen folder.
If you have a single server:
- install manticore-converter package
- use index_converter to create new versions of the indexes in a different folder than the existing data folder ( using –output-dir option)
- stop existing Manticore/Sphinx, upgrade to 3.0, move the new indexes to data folder, start Manticore
To get a minimal downtime, you can copy 2.x indexes, config (you'll need to edit paths here for indexes, logs and different ports) and binaries to a separate location and start this on a separate port and point your application to it. After upgrade is made to 3.0 and the new server is started, you can point back the application to the normal ports. If all is good, stop the 2.x copy and delete the files to free the space.
If you have a spare box (like a testing or staging server), you can do there first the index upgrade and even install Manticore 3 to perform several tests and if everything is ok copy the new index files to the production server. If you have multiple servers which can be pulled out from production, do it one by one and perform the upgrade on each. For distributed setups, 2.x searchd can work as a master with 3.x nodes, so you can do upgrading on the data nodes first and at the end the master node.
There have been no changes made on how clients should connect to the engine or any change in querying mode or queries behavior.
Kill-lists have been redesigned in Manticore Search 3. In previous versions kill-lists were applied on the result set provided by each previous searched index on query time.
Thus In 2.x the index order at query time mattered. For example if a delta index had a kill-list in order to apply it against the main index the order had to be main, delta (either in a distributed index or in the FROM clause).
In Manticore 3 kill-lists are applied to an index when it's loaded during searchd startup or gets rotated. New directive killlist_target in index configuration specifies target indexes and defines which doc ids from the source index should be used for suppression. These can be ids from the defined kill-list, actual doc ids of the index or the both.
Documents from the kill-lists are deleted from the target indexes, they are not returned in results even if the search doesn't include the index that provided the kill-lists. Because of that the order of indexes for searching does not matter any more. Now delta,main and main,delta will provide the same results.
In previous versions indexes were rotated following the order from the configuration file. In Manticore 3 index rotation order is much smarter and works in accordance with killlist targets. Before starting to rotate indexes the server looks for chains of indexes by killlist_target definitions. It will then first rotate indexes not referenced anywhere as kill-lists targets. Next it will rotate indexes targeted by already rotated indexes and so on. For example if we do indexer --all and we have 3 indexes : main, delta_big (which targets at the main) and delta_small (with target at delta_big), first the delta_small is rotated, then delta_big and finally the main. This is to ensure that when a dependent index is rotated it gets the most actual kill-list from other indexes.
docinfo- everything is now externinplace_docinfo_gap- not needed anymoremva_updates_pool- MVAs don’t have anymore a dedicated pool for updates, as now they can be updated directly in the blob (see below).
String, JSON and MVA attributes can be updated in Manticore 3.x using UPDATE statement.
In 2.x string attributes required REPLACE, for JSON it was only possible to update scalar properties (as they were fixed-width) and MVAs could be updated using the MVA pool. Now updates are performed directly on the blob component. One setting that may require tuning is attr_update_reserve which allows changing the allocated extra space at the end of the blob used to avoid frequent resizes in case the new values are bigger than the existing values in the blob.
Doc ids used to be UNSIGNED 64-bit integers. Now they are POSITIVE SIGNED 64-bit integers.
Read here about the RT mode⛔
Manticore 3.x recognizes and parses special suffixes which makes easier to use numeric values with special meaning. Common form for them is integer number + literal, like 10k or 100d, but not 40.3s (since 40.3 is not integer), or not 2d 4h (since there are two, not one value). Literals are case-insensitive, so 10W is the same as 10w. There are 2 types of such suffixes currently supported:
- Size suffixes - can be used in parameters that define size of something (memory buffer, disk file, limit of RAM, etc. ) in bytes. "Naked" numbers in that places mean literally size in bytes (octets). Size values take suffix
kfor kilobytes (1k=1024),mfor megabytes (1m=1024k),gfor gigabytes (1g=1024m) andtfor terabytes (1t=1024g). - Time suffixes - can be used in parameters defining some time interval values like delays, timeouts, etc. "Naked" values for those parameters usually have documented scale, and you must know if their numbers, say, 100, means '100 seconds' or '100 milliseconds'. However instead of guessing you just can write suffixed value and it will be fully determined by it's suffix. Time values take suffix
usfor useconds (microseconds),msfor milliseconds,sfor seconds,mfor minutes,hfor hours,dfor days andwfor weeks.
index_converter is a tool for converting indexes created with Sphinx/Manticore Search 2.x to Manticore Search 3.x index format. The tool can be used in several different ways:
$ index_converter --config /home/myuser/manticore.conf --index indexname$ index_converter --config /home/myuser/manticore.conf --all$ index_converter --path /var/lib/manticoresearch/data --allNew version of the index is written by default in the same folder. Previous version files are saved with .old extension in their name. An exception is .spp (hitlists) file which is the only index component that didn’t have any change in the new format.
You can save the new index version to a different folder using –output-dir option
$ index_converter --config /home/myuser/manticore.conf --all --output-dir /new/pathA special case is for indexes containing kill-lists. As the behaviour of how kill-lists works has changed (see killlist_target), the delta index should know which are the target indexes for applying the kill-lists. There are 3 ways to have a converted index ready for setting targeted indexes for applying kill-lists:
- Use
-–killlist-targetwhen converting an index$ index_converter --config /home/myuser/manticore.conf --index deltaindex --killlist-target mainindex:kl - Add killlist_target in the configuration before doing the conversion
- use ALTER ... KILLIST_TARGET command after conversion
Here's the complete list of index_converter options:
--config <file>(-c <file>for short) tells index_converter to use the given file as its configuration. Normally, it will look for manticore.conf in the installation directory (e.g./usr/local/sphinx/etc/manticore.confif installed into/usr/local/sphinx), followed by the current directory you are in when calling index_converter from the shell.--indexspecifies which index should be converted--path- instead of using a config file, a path containing index(es) can be used--strip-path- strips path from filenames referenced by index: stopwords, exceptions and wordforms--large-docid- allows to convert documents with ids larger than 2^63 and display a warning, otherwise it will just exit on the large id with an error. This option was added as in Manticore 3.x doc ids are signed bigint, while previously they were unsigned--output-dir <dir>- writes the new files in a chosen folder rather than the same location as with the existing index files. When this option set, existing index files will remain untouched at their location.--all- converts all indexes from the config--killlist-target <targets>- sets the target indexes for which kill-lists will be applied. This option should be used only in conjunction with--indexoption
You can install and start Manticore easily in Ubuntu, Centos, Debian, Windows and MacOS or use Manticore as a docker container.
- Ubuntu
- Debian
- Centos
- Windows
- MacOS
- Docker
wget https://repo.manticoresearch.com/manticore-repo.noarch.deb
sudo dpkg -i manticore-repo.noarch.deb
sudo apt update
sudo apt install manticore
sudo systemctl start manticorewget https://repo.manticoresearch.com/manticore-repo.noarch.deb
sudo dpkg -i manticore-repo.noarch.deb
sudo apt update
sudo apt install manticore
sudo systemctl start manticoresudo yum install https://repo.manticoresearch.com/manticore-repo.noarch.rpm
sudo yum install manticore
sudo systemctl start manticore- Download Windows archive from https://manticoresearch.com/downloads/
- Extract all from the archive to C:\Manticore
-
C:\Manticore\bin\searchd --install --config C:\Manticore\sphinx.conf.in --servicename Manticore - Start Manticore from the Services snap-in of the Microsoft Management Console
brew install manticoresearch
brew services start manticoresearchdocker pull manticoresearch/manticore
docker run --name manticore -p9306:9306 -p9308:9308 -p9312:9312 -d manticoresearch/manticoreFor persisting your data directory read how to use Manticore docker in production
By default Manticore is waiting for your connections on:
- port 9306 for MySQL clients
- port 9308 for HTTP/HTTPS connections
- port 9312 for connections from other Manticore nodes and clients based on Manticore binary API
- SQL
- HTTP
- PHP
- Python
- Javascript
- Java
mysql -h0 -P9306HTTP is a stateless protocol so it doesn't require any special connection phase:
curl -s "http://localhost:9308/search"// https://github.com/manticoresoftware/manticoresearch-php
require_once __DIR__ . '/vendor/autoload.php';
$config = ['host'=>'127.0.0.1','port'=>9308];
$client = new \Manticoresearch\Client($config);// https://github.com/manticoresoftware/manticoresearch-python
import manticoresearch
config = manticoresearch.Configuration(
host = "http://127.0.0.1:9308"
)
client = manticoresearch.ApiClient(config)
indexApi = manticoresearch.IndexApi(client)
searchApi = manticoresearch.searchApi(client)
utilsApi = manticoresearch.UtilsApi(client)// https://github.com/manticoresoftware/manticoresearch-javascript
var Manticoresearch = require('manticoresearch');
var client= new Manticoresearch.ApiClient()
client.basePath="http://127.0.0.1:9308";
indexApi = new Manticoresearch.IndexApi(client);
searchApi = new Manticoresearch.SearchApi(client);
utilsApi = new Manticoresearch.UtilsApi(client);// https://github.com/manticoresoftware/manticoresearch-java
import com.manticoresearch.client.ApiClient;
import com.manticoresearch.client.ApiException;
import com.manticoresearch.client.Configuration;
import com.manticoresearch.client.model.*;
import com.manticoresearch.client.api.IndexApi;
import com.manticoresearch.client.api.UtilsApi;
import com.manticoresearch.client.api.SearchApi;
...
ApiClient client = Configuration.getDefaultApiClient();
client.setBasePath("http://127.0.0.1:9308");
...
IndexApi indexApi = new IndexApi(client);
SearchApi searchApi = new UtilsApi(client);
UtilsApi utilsApi = new UtilsApi(client);Let's now create an index called "products" with 2 fields:
- title - full-text field which will contain our product's title
- price - of type "float"
- SQL
- HTTP
- PHP
- Python
- Javascript
- Java
create table products(title text, price float) morphology='stem_en';POST /sql -d "mode=raw&query=create table products(title text, price float) morphology='stem_en'"$index = new \Manticoresearch\Index($client);
$index->setName('products');
$index->create([
'title'=>['type'=>'text'],
'price'=>['type'=>'float'],
]);utilsApi.sql('mode=raw&query=create table forum(title text, price float)')res = await utilsApi.sql('mode=raw&query=create table forum(title text, price float)');utilsApi.sql("mode=raw&query=create table forum(title text, price float)");Query OK, 0 rows affected (0.02 sec){
"total":0,
"error":"",
"warning":""
}- SQL
- HTTP
- PHP
- Python
- Javascript
- Java
insert into products(title,price) values ('Crossbody Bag with Tassel', 19.85), ('microfiber sheet set', 19.99), ('Pet Hair Remover Glove', 7.99);"id":0 forces automatic ID generation.
POST /insert
{
"index":"products",
"id":0,
"doc":
{
"title" : "Crossbody Bag with Tassel",
"price" : 19.85
}
}
POST /insert
{
"index":"products",
"id":0,
"doc":
{
"title" : "microfiber sheet set",
"price" : 19.99
}
}
POST /insert
{
"index":"products",
"id":0,
"doc":
{
"title" : "Pet Hair Remover Glove",
"price" : 7.99
}
}'id' => 0 forces automatic ID generation.
$index->addDocuments([
['id' => 0, 'title' => 'Crossbody Bag with Tassel', 'price' => 19.85],
['id' => 0, 'title' => 'microfiber sheet set', 'price' => 19.99],
['id' => 0, 'title' => 'Pet Hair Remover Glove', 'price' => 7.99]
]);indexApi.insert({"index" : "test", "id" : 1, "doc" : {"title" : "Crossbody Bag with Tassel", "price" : 19.85}})
indexApi.insert({"index" : "test", "id" : 2, "doc" : {"title" : "microfiber sheet set", "price" : 19.99}})
indexApi.insert({"index" : "test", "id" : 0, "doc" : {{"title" : "Pet Hair Remover Glove", "price" : 7.99}})res = await indexApi.insert({"index" : "test", "id" : 1, "doc" : {"title" : "Crossbody Bag with Tassel", "price" : 19.85}});
res = await indexApi.insert({"index" : "test", "id" : 2, "doc" : {"title" : "microfiber sheet set", "price" : 19.99}});
res = await indexApi.insert({"index" : "test", "id" : 0, "doc" : {{"title" : "Pet Hair Remover Glove", "price" : 7.99}});InsertDocumentRequest newdoc = new InsertDocumentRequest();
HashMap<String,Object> doc = new HashMap<String,Object>(){{
put("title","Crossbody Bag with Tassel");
put("price",19.85);
}};
newdoc.index("products").id(1L).setDoc(doc);
sqlresult = indexApi.insert(newdoc);
newdoc = new InsertDocumentRequest();
doc = new HashMap<String,Object>(){{
put("title","microfiber sheet set");
put("price",19.99);
}};
newdoc.index("products").id(2L).setDoc(doc);
sqlresult = indexApi.insert(newdoc);
newdoc = new InsertDocumentRequest();
doc = new HashMap<String,Object>(){{
put("title","Pet Hair Remover Glove");
put("price",7.99);
}};
newdoc.index("products").id(0L).setDoc(doc);
indexApi.insert(newdoc);Query OK, 3 rows affected (0.01 sec){
"_index": "products",
"_id": 0,
"created": true,
"result": "created",
"status": 201
}
{
"_index": "products",
"_id": 0,
"created": true,
"result": "created",
"status": 201
}
{
"_index": "products",
"_id": 0,
"created": true,
"result": "created",
"status": 201
}Let's find one of the documents. The query we will use is 'remove hair'. As you can see it finds document with title 'Pet Hair Remover Glove' and highlights 'Hair remover' in it even though the query has "remove", not "remover". This is because when we created the index we turned on using English stemming (morphology "stem_en").
- SQL
- HTTP
- PHP
- Python
- javascript
- Java
select id, highlight(), price from products where match('remove hair');POST /search
{
"index": "products",
"query": { "match": { "title": "remove hair" } },
"highlight":
{
"fields": ["title"]
}
}$result = $index->search('@title remove hair')->highlight(['title'])->get();
foreach($result as $doc)
{
echo "Doc ID: ".$doc->getId()."\n";
echo "Doc Score: ".$doc->getScore()."\n";
echo "Document fields:\n";
print_r($doc->getData());
echo "Highlights: \n";
print_r($doc->getHighlight());
}searchApi.search({"index":"myindex","query":{"query_string":"@title remove hair"},"highlight":{"fields":["title"]}})res = await searchApi.search({"index":"myindex","query":{"query_string":"@title remove hair"}"highlight":{"fields":["title"]}});query = new HashMap<String,Object>();
query.put("query_string","@title remove hair");
searchRequest = new SearchRequest();
searchRequest.setIndex("forum");
searchRequest.setQuery(query);
HashMap<String,Object> highlight = new HashMap<String,Object>(){{
put("fields",new String[] {"title"});
}};
searchRequest.setHighlight(highlight);
searchResponse = searchApi.search(searchRequest);+---------------------+-------------------------------+----------+
| id | highlight() | price |
+---------------------+-------------------------------+----------+
| 1513686608316989452 | Pet <strong>Hair Remover</strong> Glove | 7.990000 |
+---------------------+-------------------------------+----------+
1 row in set (0.00 sec){
"took": 0,
"timed_out": false,
"hits": {
"total": 1,
"hits": [
{
"_id": "1513686608316989452",
"_score": 1680,
"_source": {
"price": 7.99,
"title": "Pet Hair Remover Glove"
},
"highlight": {
"title": [
"Pet <strong>Hair Remover</strong> Glove"
]
}
}
]
}
}Doc ID: 1513686608316989452
Doc Score: 1680
Document fields:
Array
(
[price] => 7.99
[title] => Pet Hair Remover Glove
)
Highlights:
Array
(
[title] => Array
(
[0] => Pet <strong>Hair Remover</strong> Glove
)
)`
{'hits': {'hits': [{u'_id': u'1513686608316989452',
u'_score': 1680,
u'_source': {u'title': u'Pet Hair Remover Glove', u'price':7.99},
u'highlight':{u'title':[u'Pet <strong>Hair Remover</strong> Glove']}}}],
'total': 1},
'profile': None,
'timed_out': False,
'took': 0}{"hits": {"hits": [{"_id": "1513686608316989452",
"_score": 1680,
"_source": {"title": "Pet Hair Remover Glove", "price":7.99},
"highlight":{"title":["Pet <strong>Hair Remover</strong> Glove"]}}],
"total": 1},
"profile": None,
"timed_out": False,
"took": 0}class SearchResponse {
took: 84
timedOut: false
hits: class SearchResponseHits {
total: 1
maxScore: null
hits: [{_id=1, _score=1, _source={price=7.99, title=Pet Hair Remover Glove}, highlight={title=[Pet <strong>Hair Remover</strong> Glove]}}]
aggregations: null
}
profile: null
}Let's assume we now want to update the document - change the price to 18.5. This can be done by filtering by any field, but normally you know the document id and update something based on that.
- SQL
- HTTP
- PHP
- Python
- javascript
- Java
update products set price=18.5 where id = 1513686608316989453;POST /update
{
"index": "products",
"id": 1513686608316989453,
"doc":
{
"price": 18.5
}
}$doc = [
'body' => [
'index' => 'products',
'id' => 2,
'doc' => [
'price' => 18.5
]
]
];
$response = $client->update($doc);indexApi = api = manticoresearch.IndexApi(client)
indexApi.update({"index" : "products", "id" : 2, "doc" : {"price":18.5}})res = await indexApi.update({"index" : "products", "id" : 2, "doc" : {"price":18.5}});UpdateDocumentRequest updateRequest = new UpdateDocumentRequest();
doc = new HashMap<String,Object >(){{
put("price",18.5);
}};
updateRequest.index("products").id(2L).setDoc(doc);
indexApi.update(updateRequest);Query OK, 1 row affected (0.00 sec){
"_index": "products",
"_id": 1513686608316989400,
"result": "updated"
}- SQL
- HTTP
- PHP
- Python
- javascript
- Java
delete from products where price < 10;POST /delete
{
"index": "products",
"query":
{
"range":
{
"price":
{
"lte": 10
}
}
}
}$result = $index->deleteDocuments(new \Manticoresearch\Query\Range('price',['lte'=>10]));indexApi.delete({"index" : "products", "query": {"range":{"price":{"lte":10}}}})res = await indexApi.delete({"index" : "products", "query": {"range":{"price":{"lte":10}}}});DeleteDocumentRequest deleteRequest = new DeleteDocumentRequest();
query = new HashMap<String,Object>();
query.put("range",new HashMap<String,Object>(){{
put("range",new HashMap<String,Object>(){{
put("lte",10);
}});
}});
deleteRequest.index("products").setQuery(query);
indexApi.delete(deleteRequest);
Query OK, 1 row affected (0.00 sec){
"_index": "products",
"deleted": 1
}Array
(
[_index] => products
[deleted] => 1
)When Manticore Search is installed using DEB or RPM packages, the searchd process can be run and managed by operating system's init system. Most Linux versions now use systemd, while older releases use SysV init.
If you are not sure about the type of the init system your platform use, run:
ps --no-headers -o comm 1After the installation the Manticore Search service is not started automatically. To start Manticore run the following command:
sudo systemctl start manticoreTo stop Manticore run the following command:
sudo systemctl stop manticoreThe Manticore service is set to run at boot. You can check it by running:
sudo systemctl is-active manticoreIf you want to disable Manticore starting at boot time run:
sudo systemctl disable manticoreTo enable Manticore to start at boot, run:
sudo systemctl enable manticoresearchd process logs startup information in systemd journal. If systemd logging is enabled you can view the logged information with the following command:
sudo journalctl --unit manticoreManticore can be started and stopped using service commands:
sudo service manticore start
sudo service manticore stopTo enable the sysV service at boot on RedHat systems run:
chkconfig manticore onTo enable the sysV service at boot on Debian systems (including Ubuntu) run:
update-rc.d manticore defaultsPlease note that searchd is started by the init system under manticore user and all files created by the server will be owned by this user. If searchd is started under ,for example, root user, the permissions of files will be changed which may lead to issues when running again searchd as service.