Merging two existing plain indexes can be more efficient than indexing the data from scratch and desired in some cases (such as merging 'main' and 'delta' indexes instead of simply reindexing 'main' in the 'main+delta' partitioning scheme). So indexer has an option to do that. Merging indexes is normally faster than reindexing, but still not instant on huge indexes. Basically, it will need to read the contents of the both indexes once and write the result once. Merging 100 GB and 1 GB index, for example, will result in 202 GB of I/O (but that's still likely less than the indexing from scratch requires).

The basic command syntax is as follows:

sudo -u manticore indexer --merge DSTINDEX SRCINDEX [--rotate] [--drop-src]

Unless --drop-src is specified only the DSTINDEX index will be affected: the contents of SRCINDEX will be merged into it.

--rotate switch will be required if DSTINDEX is already being served by searchd.

The typical usage pattern is to merge a smaller update from SRCINDEX into DSTINDEX. Thus, when merging attributes the values from SRCINDEX will win if duplicate document IDs are encountered. Note, however, that the "old" keywords will not be automatically removed in such cases. For example, if there's a keyword "old" associated with document 123 in DSTINDEX, and a keyword "new" associated with it in SRCINDEX, document 123 will be found by both keywords after the merge. You can supply an explicit condition to remove documents from DSTINDEX to mitigate that; the relevant switch is --merge-dst-range:

sudo -u manticore indexer --merge main delta --merge-dst-range deleted 0 0

This switch lets you apply filters to the destination index along with merging. There can be several filters; all of their conditions must be met in order to include the document in the resulting merged index. In the example above, the filter passes only those records where 'deleted' is 0, eliminating all records that were flagged as deleted.

--drop-src allows dropping SRCINDEX after the merge and before rotating the indexes, which is important in case you specify DSTINDEX in killlist_target of DSTINDEX, otherwise when rotating the indexes the documents that have been merged into DSTINDEX may be suppressed by SRCINDEX.

When using plain indexes there is a problem generated by the need of having the data in the index as fresh as possible.

In this case one or more secondary (also know as delta) indexes are used to capture the modified data between the time the main index was created and and current time. The modified data can mean new, updated or deleted documents. The search becomes a search over the main index and the delta index. This works with no obstacle when you just add new documents to the delta index, but when it comes to updated or deleted documents there remains the following issue.

If a document is present in both main and delta indexes it can cause issues at searching as the engine will see two versions of a document and won't know how to pick the right one. So the delta needs to tell somehow to the search that there are deleted documents in the main index that should be forgotten. Here comes kill lists.

Index can maintain a list of document ids that can be used to suppress records in other indexes. This feature is available for plain indexes using database sources or plain indexes using XML sources. In case of database sources, the source needs to provide an additional query defined by sql_query_killlist. It will store in the index a list of documents that can be used by the server to remove documents from other plain indexes.

This query is expected to return a number of 1-column rows, each containing just the document ID.

In many cases the query is a union between a query that gets a list of updated documents and a list of deleted documents, e.g.:

    sql_query_killlist = \
        SELECT id FROM documents WHERE updated_ts>=@last_reindex UNION \
        SELECT id FROM documents_deleted WHERE deleted_ts>=@last_reindex

A plain index can contain a directive called killlist_target that will tell the server it can provide a list of document ids that should be removed from certain existing indexes. The index can use either it's document ids as the source for this list or provide a separate list.

index products {
  killlist_target = main:kl

  path = products
  source = src_base
}

When killlist_target is specified, kill-list is applied to all the indexes listed in it on searchd startup. If any of the indexes from killlist_target are rotated, kill-list is reapplied to these indexes. When kill-list is applied, indexes that were affected save these changes to disk.

killlist_target has 3 modes of operation:

1. killlist_target = main:kl. Document ids from the kill-list of the delta index are suppressed in the main index (see sql_query_killlist).
2. killlist_target = main:id. All document ids from delta index are suppressed in the main index. Kill-list is ignored.
3. killlist_target = main. Both document ids from delta index and its kill-list are suppressed in the main index.

Multiple targets can be specified separated by comma like

killlist_target = index_one:kl,index_two:kl

ALTER TABLE delta KILLLIST_TARGET='new_main_index:kl'

POST /sql -d "mode=raw&query=
ALTER TABLE delta KILLLIST_TARGET='new_main_index:kl'"