Isolation during flushing and merging

Manticore provides isolation during the flushing and merging process of a real-time table to prevent any changes from affecting running queries.

For example, during table compaction, a pair of disk chunks are merged and a new chunk is produced. At one point, a new version of the table is created with the new chunk replacing the original pair. This is done seamlessly so that a long-running query using the original chunks will continue to see the old version of the table, while a new query will see the new version with the resulting merged chunk.

The same applies to flushing a RAM chunk, where suitable RAM segments are merged into a new disk chunk and the participated RAM chunk segments are abandoned. During this operation, Manticore provides isolation for queries that started before the operation began.

Furthermore, these operations are transparent for replaces and updates. If you update an attribute in a document that belongs to a disk chunk being merged with another one, the update will be applied to both that chunk and the resulting merged chunk. If you delete a document during a merge, it will be deleted in the original chunk and also in the resulting merged chunk, which will either have the document marked as deleted or have no such document at all if the deletion happened early in the merging process.

Freezing a table

FREEZE tbl1[, tbl2, ...]

FREEZE readies a real-time/plain table for a secure backup. Specifically, it:

  1. Deactivates table compaction. If the table is currently being compacted, FREEZE will gracefully interrupt it.
  2. Transfers the current RAM chunk to a disk chunk.
  3. Flushes attributes.
  4. Disables implicit operations that could modify the disk files.
  5. Increments the table's locked counter.
  6. Shows the actual file list associated with the table.

If the table is already frozen (locked), FREEZE will:

  1. Increment the table's locked counter.
  2. Shows the actual file list associated with the table.

The built-in tool manticore-backup uses FREEZE to ensure data consistency. You can do the same if you want to create your own backup solution or need to freeze tables for other reasons. Just follow these steps:

  1. FREEZE a table (or a few).
  2. Capture the output of the FREEZE command and back up the specified files.
  3. UNFREEZE the table(s) once finished.
‹›
  • Example
Example
📋
FREEZE t;
‹›
Response
+-------------------+---------------------------------+
| file              | normalized                      |
+-------------------+---------------------------------+
| data/t/t.0.spa    | /work/anytest/data/t/t.0.spa    |
| data/t/t.0.spd    | /work/anytest/data/t/t.0.spd    |
| data/t/t.0.spds   | /work/anytest/data/t/t.0.spds   |
| data/t/t.0.spe    | /work/anytest/data/t/t.0.spe    |
| data/t/t.0.sph    | /work/anytest/data/t/t.0.sph    |
| data/t/t.0.sphi   | /work/anytest/data/t/t.0.sphi   |
| data/t/t.0.spi    | /work/anytest/data/t/t.0.spi    |
| data/t/t.0.spm    | /work/anytest/data/t/t.0.spm    |
| data/t/t.0.spp    | /work/anytest/data/t/t.0.spp    |
| data/t/t.0.spt    | /work/anytest/data/t/t.0.spt    |
| data/t/t.meta     | /work/anytest/data/t/t.meta     |
| data/t/t.ram      | /work/anytest/data/t/t.ram      |
| data/t/t.settings | /work/anytest/data/t/t.settings |
+-------------------+---------------------------------+
13 rows in set (0.01 sec)

The file column indicates the table's file paths within the data_dir of the running instance. The normalized column displays the absolute paths for the same files. To back up a table, simply copy the provided files without additional preparation.

When a table is frozen, you cannot execute UPDATE queries; they will wait until the table is unfrozen.

Also, DELETE and REPLACE queries have some restrictions while the table is frozen:

  • If DELETE affects a document in the current RAM chunk - it is permitted.
  • If DELETE impacts a document in a disk chunk but was previously deleted - it is allowed.
  • If DELETE would alter an actual disk chunk - it will wait until the table is unfrozen.

Manually FLUSHing a RAM chunk of a frozen table will report 'success', but no real saving will occur.

DROP/TRUNCATE of a frozen table is allowed since these operations are not implicit. We assume that if you truncate or drop a table, you don't need it backed up; therefore, it should not have been frozen initially.

INSERTing into a frozen table is supported but limited: new data will be stored in RAM (as usual) until rt_mem_limit is reached; then, new insertions will wait until the table is unfrozen.

If you shut down the daemon with a frozen table, it will act as if it experienced a dirty shutdown (e.g., kill -9): newly inserted data will not be saved in the RAM-chunk on disk, and upon restart, it will be restored from a binary log (if any) or lost (if binary logging is disabled).

Unfreezing a table

UNFREEZE tbl1[, tbl2, ...]

UNFREEZE command decreases the table's locked counter, and if it reaches zero, reactivates previously blocked operations and resumes the internal compaction service. Any operations that were waiting for the table to unfreeze will also resume and complete normally.

‹›
  • Example
Example
📋
UNFREEZE tbl;

Inspecting the lock state of a table

You can use SHOW table_name STATUS to check if a table is frozen or not.

The locked counter is displayed in the table's status under the locked column. A value of zero indicates that the table is not frozen, while a non-zero value reflects the number of active locks. Each explicit FREEZE command and implicit locking (such as when the table is part of a cluster and a replication routine selects it as a donor for a replica) increases the counter. Each UNFREEZE command decreases the counter, eventually down to zero.

‹›
  • Example
Example
📋
SHOW TABLE `foo` STATUS LIKE 'locked';
‹›
Response
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| locked        | 2     |
+---------------+-------+
1 row in set (0,00 sec)

SHOW LOCKS

Locked tables are also displayed using the SHOW LOCKS command. The lock counters are shown in the Additional Info column.

‹›
  • Example
Example
📋
SHOW LOCKS;
‹›
Response
+-----------+------+-----------+-----------------+
| Type      | Name | Lock Type | Additional Info |
+-----------+------+-----------+-----------------+
| rt        | a    | freeze    | Count: 1        |
| percolate | bar  | freeze    | Count: 3        |
| rt        | foo  | freeze    | Count: 2        |
+-----------+------+-----------+-----------------+
3 rows in set (0,01 sec)

FLUSH ATTRIBUTES

FLUSH ATTRIBUTES

The FLUSH ATTRIBUTES command flushes all in-memory attribute updates in all the active disk tables to disk. It returns a tag that identifies the result on-disk state (which is basically a number of actual disk attribute saves performed since the server startup).

mysql> UPDATE testindex SET channel_id=1107025 WHERE id=1;
Query OK, 1 row affected (0.04 sec)

mysql> FLUSH ATTRIBUTES;
+------+
| tag  |
+------+
|    1 |
+------+
1 row in set (0.19 sec)