Real-time table

A real-time table is a main type of table in Manticore. It lets you add, update, and delete documents, and you can see these changes right away. You can set up a real-time Table in a configuration file or use commands like CREATE, UPDATE, DELETE, or ALTER.

Internally a real-time table consists of one or more plain tables called chunks. There are two kinds of chunks:

  • multiple disk chunks - these are saved on a disk and are structured like a plain table.
  • single ram chunk - this is kept in memory and collects all changes.

The size of the RAM chunk is controlled by the rt_mem_limit setting. Once this limit is reached, the RAM chunk is transferred to disk as a disk chunk. If there are too many disk chunks, Manticore combines some of them to improve performance.

Creating a real-time table:

You can create a new real-time table in two ways: by using the CREATE TABLE command or through the _mapping endpoint of the HTTP JSON API.

CREATE TABLE command:

You can use this command via both SQL and HTTP protocols:

‹›
  • SQL
  • JSON
  • PHP
  • Python
  • Python-asyncio
  • Javascript
  • Java
  • C#
  • Rust
  • Typescript
  • Go
  • CONFIG
📋
CREATE TABLE products(title text, price float) morphology='stem_en';
‹›
Response
Query OK, 0 rows affected (0.00 sec)

_mapping API:

NOTE: The _mapping API requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Alternatively, you can create a new table via the _mapping endpoint. This endpoint allows you to define an Elasticsearch-like table structure to be converted to a Manticore table.

The body of your request must have the following structure:

"properties"
{
  "FIELD_NAME_1":
  {
    "type": "FIELD_TYPE_1"
  },
  "FIELD_NAME_2":
  {
    "type": "FIELD_TYPE_2"
  },

  ...

  "FIELD_NAME_N":
  {
    "type": "FIELD_TYPE_M"
  }
}

When creating a table, Elasticsearch data types will be mapped to Manticore types according to the following rules:

  • aggregate_metric => json
  • binary => string
  • boolean => bool
  • byte => int
  • completion => string
  • date => timestamp
  • date_nanos => bigint
  • date_range => json
  • dense_vector => json
  • flattened => json
  • flat_object => json
  • float => float
  • float_range => json
  • geo_point => json
  • geo_shape => json
  • half_float => float
  • histogram => json
  • integer => int
  • integer_range => json
  • ip => string
  • ip_range => json
  • keyword => string
  • knn_vector => float_vector
  • long => bigint
  • long_range => json
  • match_only_text => text
  • object => json
  • point => json
  • scaled_float => float
  • search_as_you_type => text
  • shape => json
  • short => int
  • text => text
  • unsigned_long => int
  • version => string
‹›
  • JSON
JSON
📋
⚙
POST /your_table_name/_mapping -d '
{
  "properties": {
    "price": {
        "type": "float"
    },
    "title": {
        "type": "text"
    }
  }
}
'
‹›
Response
{
"total":0,
"error":"",
"warning":""
}

CREATE TABLE LIKE:

You can create a copy of a real-time table, with or without its data. Please note that if the table is large, copying it with data may take some time. Copying works in synchronous mode, but if the connection is dropped, it will continue in the background.

CREATE TABLE [IF NOT EXISTS] table_name LIKE old_table_name [WITH DATA]

NOTE: Copying a table requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

‹›
  • SQL
  • Example (WITH DATA)
📋
create table products LIKE old_products;

👍 What you can do with a real-time table:

⛔ What you cannot do with a real-time table:

  • Ingest data using the indexer feature.
  • Connect it to sources for easy indexing from external storage.
  • Update the killlist_target, as it is automatically managed by the real-time table.

Real-time table files structure

The following table outlines the different file extensions and their respective descriptions in a real-time table:

Extension Description
.lock A lock file that ensures that only one process can access the table at a time.
.ram The RAM chunk of the table, stored in memory and used as an accumulator of changes.
.meta The headers of the real-time table that define its structure and settings.
.*.sp* Disk chunks that are stored on disk with the same format as plain tables. They are created when the RAM chunk size exceeds the rt_mem_limit.

For more information on the structure of disk chunks, refer to the plain table files structure.

Creating a local table Plain table