OpenSearch Dashboards is a visual interface that allows you to explore, visualize, and create dashboards for your log data. It connects to the backend over an HTTP API. Manticore exposes that API on its HTTP listener, so you can point OpenSearch Dashboards at Manticore and use Discover, Visualize, and Dashboards in a similar way to Kibana. With this integration, you can build charts and dashboards, run ad hoc searches in Discover, and keep working with familiar ingestion tools such as Logstash and Filebeat to load log and event data for analysis.

1. Download OpenSearch Dashboards: Ensure you download an OpenSearch Dashboards version compatible with Manticore. Currently, version 3.4.0 is tested and recommended. Other versions may work but could introduce issues. Set kibana_version_string in Manticore to match your OpenSearch Dashboards version (see Configuration below).
2. Verify Manticore: Ensure your Manticore instance is running and its HTTP API is reachable (default: http://localhost:9308).
3. Manticore Buddy: Ensure Manticore Buddy is installed and running. OpenSearch Dashboards integration is implemented by Buddy's EmulateElastic plugin, which emulates the Elasticsearch-compatible HTTP API that Dashboards expects (the same code path as Kibana). Buddy starts automatically with searchd unless you disable it via buddy_path.

1. Open the OpenSearch Dashboards configuration file (opensearch_dashboards.yml; common paths include config/opensearch_dashboards.yml in the tarball layout or /etc/opensearch-dashboards/opensearch_dashboards.yml on some packages).

2. Set the URL of your Manticore instance:

   opensearch.hosts: ["http://localhost:9308"]

   When running OpenSearch Dashboards in Docker, you can set the same value via the OPENSEARCH_HOSTS environment variable.

   Since Manticore does not provide the OpenSearch Security plugin, you must disable the security dashboards plugin in OpenSearch Dashboards as well:

   • Docker: set DISABLE_SECURITY_DASHBOARDS_PLUGIN=true in the container environment.
   • Tarball install: stop OpenSearch Dashboards, then run: ./bin/opensearch-dashboards-plugin remove securityDashboards. After that, start OpenSearch Dashboards again.

   See Disabling and enabling the Security plugin for details.

3. Start OpenSearch Dashboards and open it in your browser at http://localhost:5601. Replace localhost with your server's IP or hostname if necessary.

Note: Manticore must work in real-time mode to be integrated with OpenSearch Dashboards.

Set kibana_version_string to the same version as your OpenSearch Dashboards install. OpenSearch Dashboards checks the backend version reported by Manticore and may show warnings or fail to start if they do not match.

searchd {
    listen = 127.0.0.1:9308:http
    pid_file = /var/run/manticore/searchd.pid
    data_dir = /var/lib/manticore
    kibana_version_string = 3.4.0
}

• Use the Discover tab to search and filter documents in Manticore tables interactively.

• Navigate to Visualizations to create custom visualizations:
  • Create an index pattern that matches a Manticore table name.
  • Choose a visualization type (e.g., bar chart, line chart, or pie chart) backed by supported aggregations: terms, histogram, date_histogram, range, date_range, and metric aggregations max, min, sum, avg.
  • Configure your visualization, execute it, and explore your data.
  • Save visualizations for future use.

• Access Dashboards to create or view interactive dashboards:
  • Add saved visualizations, filters, or controls for a personalized experience.
  • Interact with your data directly from the dashboard.
  • Save dashboards for future use.

• Use Management > Dashboards Management for index patterns and saved objects (visualizations, dashboards).
• Stack-level requests needed for Dashboards startup are emulated (node version, cluster settings, config objects, index listing). Full OpenSearch cluster administration is not available against Manticore.

• Currently, OpenSearch Dashboards version 3.4.0 is tested and recommended. Other versions may work but could cause issues. You must set kibana_version_string in Manticore to match the OpenSearch Dashboards version you use.
• The following OpenSearch field types are not supported:
  • Geographic field types (geo_point, geo_shape)
  • Cartesian field types (xy_point, xy_shape)
  • Range field types (integer_range, long_range, double_range, float_range, ip_range, date_range)
  • Specialized search field types (semantic, rank_feature, rank_features, percolator, star_tree, derived)
  • Vector field types (knn_vector, sparse_vector)
  • Autocomplete field types and advanced string field types (completion, search_as_you_type, match_only_text, and similar types; plain text and keyword are supported)
  • Relational field types (nested, join)
• Metric aggregation functions are limited to those supported by Manticore.
• Nested aggregations (aggs inside aggs) are not supported.
• Advanced Dashboards Query Language (DQL) features (nested field search, regular expressions, fuzzy search, proximity search, term boosting, and similar) may not work against Manticore.
• The following OpenSearch Dashboards tools and plugins are not supported:
  • Geospatial (map visualizations that depend on geographic field types)
  • Observability – Apps for monitoring infrastructure metrics, exploring logs, and trace analytics
  • Alerting – Monitors, triggers, and notifications against OpenSearch APIs
  • Anomaly Detection – Detectors for outliers in time-series data
  • Security Analytics – SIEM workflows for threat detection and investigation
  • Monitoring:
    • Index State Management – Automated index lifecycle policies
    • Performance Analyzer – Cluster performance metrics and root cause analysis
  • Index Management – A UI for managing OpenSearch indexes, templates, aliases, and lifecycle settings
  • Security plugin flows – not available on Manticore

Integrate Manticore with tools like Logstash, Filebeat, Fluentbit, or Vector.dev to ingest data from sources like web logs. Once the data is loaded into Manticore, you can explore and visualize it in OpenSearch Dashboards.

NOTE: this functionality requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Manticore supports integration with Apache Kafka real-time data ingestion through Kafka sources and materialized views, allowing for real-time data indexing and search. Currently, apache/kafka versions 3.7.0-4.1.0 are tested and supported.

To get started, you need to:

1. Define the source: Specify the Kafka topic from which Manticore Search will read messages. This setup includes details like the broker’s host, port, and topic name.
2. Set up the destination table: Choose a Manticore real-time table to store the incoming Kafka data.
3. Create a materialized view: Set up a materialized view (mv) to handle data transformation and mapping from Kafka to the destination table in Manticore Search. Here, you’ll define field mappings, data transformations, and any filters or conditions for the incoming data stream.

CREATE SOURCE <source name> [(column type, ...)] [source_options]

allowed_field_name 'original JSON key name with special symbols' type

price_field '$price' float    -- maps JSON key '$price' to field 'price_field'
field_123 '123field' text     -- maps JSON key '123field' to field 'field_123'

CREATE SOURCE kafka
(id bigint, term text, abbrev '$abbrev' text, GlossDef json)
type='kafka'
broker_list='kafka:9092'
topic_list='my-data'
consumer_group='manticore'
num_consumers='2'
batch=50

POST /sql?mode=raw -d "CREATE SOURCE kafka (id bigint, term text, abbrev '$abbrev' text, GlossDef json) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore' num_consumers='2' batch=50"

Query OK, 2 rows affected (0.02 sec)

[
  {
    "total": 2,
    "error": "",
    "warning": ""
  }
]

CREATE TABLE destination_kafka
(id bigint, name text, short_name text, received_at text, size multi);

POST /sql?mode=raw -d "CREATE TABLE destination_kafka (id bigint, name text, short_name text, received_at text, size multi)"

Query OK, 0 rows affected (0.02 sec)

[
  {
    "total": 0,
    "error": "",
    "warning": ""
  }
]

CREATE MATERIALIZED VIEW <materialized view name>
TO <destination table name> AS
SELECT [column|function [as <new name>], ...] FROM <source name>

CREATE MATERIALIZED VIEW view_table
TO destination_kafka AS
SELECT id, term as name, abbrev as short_name,
       UTC_TIMESTAMP() as received_at, GlossDef.size as size FROM kafka

Query OK, 2 rows affected (0.02 sec)

Data is transferred from Kafka to Manticore Search in batches, which are cleared after each run. For calculations across batches, such as AVG, use caution, as these may not work as expected due to batch-by-batch processing.

Here's a mapping table based on the examples above:

SHOW SOURCES

POST /sql?mode=raw -d "SHOW SOURCES"

+-------+
| name  |
+-------+
| kafka |
+-------+

[
  {
    "total": 1,
    "error": "",
    "warning": "",
    "columns": [
      {
        "name": {
          "type": "string"
        }
      }
    ],
    "data": [
      {
        "name": "kafka"
      }
    ]
  }
]

SHOW SOURCE kafka;

POST /sql?mode=raw -d "SHOW SOURCE kafka"

+--------+-------------------------------------------------------------------+
| Source | Create Table                                                      |
+--------+-------------------------------------------------------------------+
| kafka  | CREATE SOURCE kafka                                               |
|        | (id bigint, term text, abbrev '$abbrev' text, GlossDef json)      |
|        | type='kafka'                                                      |
|        | broker_list='kafka:9092'                                          |
|        | topic_list='my-data'                                              |
|        | consumer_group='manticore'                                        |
|        | num_consumers='2'                                                 |
|        | batch=50                                                          |
+--------+-------------------------------------------------------------------+

[
  {
    "total": 1,
    "error": "",
    "warning": "",
    "columns": [
      {
        "Source": {
          "type": "string"
        }
      },
      {
        "Create Table": {
          "type": "string"
        }
      }
    ],
    "data": [
      {
        "Source": "kafka",
        "Create Table": "CREATE SOURCE kafka \n(id bigint, term text, abbrev '' text, GlossDef json)\ntype='kafka'\nbroker_list='kafka:9092'\ntopic_list='my-data'\nconsumer_group='manticore'\nnum_consumers='2'\n batch=50"
      }
    ]
  }
]

SHOW MVS

POST /sql?mode=raw -d "SHOW MVS"

+------------+
| name       |
+------------+
| view_table |
+------------+

[
  {
    "total": 1,
    "error": "",
    "warning": "",
    "columns": [
      {
        "name": {
          "type": "string"
        }
      }
    ],
    "data": [
      {
        "name": "view_table"
      }
    ]
  }
]

SHOW MV view_table

POST /sql?mode=raw -d "SHOW MV view_table"

+------------+--------------------------------------------------------------------------------------------------------+-----------+
| View       | Create Table                                                                                           | suspended |
+------------+--------------------------------------------------------------------------------------------------------+-----------+
| view_table | CREATE MATERIALIZED VIEW view_table TO destination_kafka AS                                            | 0         |
|            | SELECT id, term as name, abbrev as short_name, UTC_TIMESTAMP() as received_at, GlossDef.size as size   |           |
|            | FROM kafka                                                                                             |           |
+------------+--------------------------------------------------------------------------------------------------------+-----------+

[
  {
    "total": 1,
    "error": "",
    "warning": "",
    "columns": [
      {
        "View": {
          "type": "string"
        }
      },
      {
        "Create Table": {
          "type": "string"
        }
      },
      {
        "suspended": {
          "type": "string"
        }
      }
    ],
    "data": [
      {
        "View": "view_table",
        "Create Table": "CREATE MATERIALIZED VIEW view_table TO destination_kafka AS SELECT id, term as name, abbrev as short_name, UTC_TIMESTAMP() as received_at, GlossDef.size as size FROM kafka",
        "suspended": 0
      }
    ]
  }
]

ALTER MATERIALIZED VIEW view_table suspended=1

POST /sql?mode=raw -d "ALTER MATERIALIZED VIEW view_table suspended=1"

Query OK (0.02 sec)

[
  {
    "total": 2,
    "error": "",
    "warning": ""
  }
]

You can also specify a partition_list for each Kafka topic. One of the main benefits of this approach is the ability to implement sharding for your table via Kafka. To achieve this, you should create a separate chain of source → materialized view → destination table for each shard:

Sources:

CREATE SOURCE kafka_p1 (id bigint, term text)
  type='kafka' broker_list='kafka:9092' topic_list='my-data'
  consumer_group='manticore' num_consumers='1' partition_list='0' batch=50;
CREATE SOURCE kafka_p2 (id bigint, term text)
  type='kafka' broker_list='kafka:9092' topic_list='my-data'
  consumer_group='manticore' num_consumers='1' partition_list='1' batch=50;

Destination Tables:

CREATE TABLE destination_shard_1 (id bigint, name text);
CREATE TABLE destination_shard_2 (id bigint, name text);

Materialized Views:

CREATE MATERIALIZED VIEW mv_1 TO destination_shard_1 AS SELECT id, term AS name FROM kafka_p1;
CREATE MATERIALIZED VIEW mv_2 TO destination_shard_2 AS SELECT id, term AS name FROM kafka_p2;

• In this setup, rebalancing must be managed manually.
• Kafka does not distribute messages using a round-robin strategy by default.
• To achieve round-robin-like distribution when sending data, make sure your Kafka producer is configured with:
  • parse.key=true
  • key.separator={your_delimiter}

Otherwise, Kafka will distribute messages based on its own internal rules, which may lead to uneven partitioning.

Kafka offsets commit after each batch or when processing times out. If the process stops unexpectedly during a materialized view query, you may see duplicate entries. To avoid this, include an id field in your schema, allowing Manticore Search to prevent duplicates in the table.

• Worker initialization: After configuring a source and materialized view, Manticore Search sets up a dedicated worker to handle data ingestion from Kafka.
• Message mapping: Messages are mapped according to the source configuration schema, transforming them into a structured format.
• Batching: Messages are grouped into batches for efficient processing. Batch size can be adjusted to suit your performance and latency needs.
• Buffering: Mapped data batches are stored in a buffer table for efficient bulk operations.
• Materialized view processing: The view logic is applied to data in the buffer table, performing any transformations or filtering.
• Data transfer: Processed data is then transferred to the destination real-time table.
• Cleanup: The buffer table is cleared after each batch, ensuring it’s ready for the next set of data.

NOTE: The integration with DBeaver requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

DBeaver is a SQL client software application and a database administration tool. For MySQL databases, it applies the JDBC application programming interface to interact with them via a JDBC driver.

Manticore allows you to use DBeaver for working with data stored in Manticore tables the same way as if it was stored in a MySQL database. Currently, version 25.2.0 is tested and recommended. Other versions may work but could introduce issues.

To start working with Manticore in DBeaver, follow these steps:

• Choose the New database connection option in DBeaver's UI
• Choose SQL -> MySQL as DBeaver's database driver
• Set the Server host and Port options corresponding to the host and port of your Manticore instance (keep the database field empty)
• Set root/<empty password> as authentication credentials

Since Manticore does not fully support MySQL, only a part of DBeaver's functionality is available when working with Manticore.

You will be able to:

• View, create, delete, and rename tables
• Add and drop table columns
• Insert, delete, and update column data

You will not be able to:

• Use database integrity check mechanisms (MyISAM will be set as the only storage engine available)
• Use MySQL procedures, triggers, events, etc.
• Manage database users
• Set other database administration options

Some MySQL data types are not currently supported by Manticore and, therefore, cannot be used when creating a new table with DBeaver. Also, a few of the supported data types are converted to the most similar Manticore types with type precision being ignored in such conversion. Below is the list of supported MySQL data types as well as the Manticore types they are mapped to:

• BIGINT UNSIGNED => bigint
• BOOL => boolean
• DATE, DATETIME, TIMESTAMP => timestamp
• FLOAT => float
• INT => int
• INT UNSIGNED, SMALLINT UNSIGNED, TINYINT UNSIGNED, BIT => uint
• JSON => json
• TEXT, LONGTEXT, MEDIUMTEXT, TINYTEXT, BLOB, LONGBLOB, MEDIUMBLOB, TINYBLOB => text
• VARCHAR, LONG VARCHAR, BINARY, CHAR, VARBINARY, LONG VARBINARY => string

You can find more details about Manticore data types here.

Manticore is able to handle the DATE, DATETIME and TIMESTAMP data types, however, this reqiures Manticore's Buddy enabled. Otherwise, an attempt to operate with one of these types will result in an error.

Note that the TIME type is not supported.

• DBeaver's Preferences -> Connections -> Client identification option must not be turned off or overridden. To work correctly with DBeaver, Manticore needs to distinguish its requests from others. For this, it uses client notification info sent by DBeaver in request headers. Disabling client notification will break that detection and, therefore, Manticore's correct functionality.

• When trying to update data in your table for the first time, you'll see the No unique key popup message and will be asked to define a custom unique key. When you get this message, perform the following steps:

  • Choose the Custom Unique Key option
  • Choose only the id column in the columns list
  • Press Ok

  After that, you'll be able to update your data safely.

NOTE: The integration with Apache Superset requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Apache Superset is a modern data exploration and visualization platform. Manticore Search can be integrated with Apache Superset, allowing users to harness the full potential of Manticore Search for data analysis and visualization.

Note: This integration works only with Apache Superset 3.x branch and is not yet available for version 4 or newer.

Before integrating Manticore Search with Apache Superset, ensure that:

1. Manticore Search (version 6.2.0 or later) is properly installed and configured on your server. Refer to the official Manticore Search installation guide for assistance.
2. Apache Superset is set up on your system. Follow the official Apache Superset installation guide for installation instructions.

To connect Manticore Search to Apache Superset:

1. Log in to your Apache Superset dashboard.
2. Click on "+" in the top navigation bar and choose "Database Connections".
3. Click "+ DATABASE" to add a new database.
4. Select "MySQL" as the database type.
5. In the configuration page, provide the following details:
   • Database Name: A name for the database (e.g., "Manticore")
   • Host: Your Manticore Search server host
   • Port: The port on which Manticore Search is running
   • Username: Your Manticore Search username
   • Display Name: A display name for the connection
6. Click "Connect" to verify the connection and save.

After connecting Manticore Search to Apache Superset, you can create charts and dashboards using your Manticore data:

1. In the Apache Superset dashboard, click on "Dashboards" in the top navigation bar and select "New dashboard".
2. Open the dashboard and click on the "+" icon to add a new chart.
3. Choose the dataset connected to Manticore Search.
4. Select the type of chart you want to create (e.g., bar chart, line chart, pie chart).
5. Use Apache Superset's query builder or write an SQL query to fetch data from your Manticore Search database.
6. Customize the chart's appearance, labels, and other settings as needed.
7. Click "UPDATE CHART" to update the chart.
8. Add a name for the chart and click "Save" to permanently save it in the Dashboard.

When working with Manticore Search through Apache Superset, you can:

• View and query data from Manticore Search tables
• Create various types of visualizations based on your Manticore data
• Build and customize dashboards using these visualizations

Manticore Search data types are mapped to MySQL data types when working with Apache Superset. Refer to the Manticore Search documentation for more details on supported data types and their mappings.

• Some advanced MySQL features may not be available when working with Manticore Search through Apache Superset.
• Certain data manipulation operations might be limited compared to working directly with MySQL databases.

By following these steps and guidelines, you can effectively integrate Manticore Search with Apache Superset for powerful data exploration and visualization capabilities.

Please refer to the following resources for more information on Apache Superset and Manticore Search:

• Apache Superset Integration