NOTE: The integration with Fluent Bit requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Fluent Bit is an open source, cross-platform log processor that can aggregate data from many sources and ship it to multiple destinations. You can feed Fluent Bit outputs directly into Manticore to make the collected data searchable in real time.

The following walkthrough indexes Debian's dpkg.log with Fluent Bit and Manticore.

2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1
2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>
2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>

Create a configuration file such as fluent-bit.conf:

[SERVICE]
    flush        1
    daemon       On
    log_level    info
[INPUT]
    name tail
    path /var/log/dpkg.log
    inotify_watcher false
    read_from_head true
[OUTPUT]
    name es
    match *
    host 127.0.0.1
    port 9308
    index dpkg_log

• The [SERVICE] block starts Fluent Bit in daemon mode, which is convenient for Docker-based setups. Disable the daemon flag if you want to run it in the foreground.
• inotify_watcher is turned off to avoid file notification limitations inside containers.
• The output plugin (name es) can talk to Manticore's HTTP endpoint on port 9308.
• index defines the automatic table name Manticore creates when the first batch arrives.

Run Fluent Bit with this configuration and it will tail dpkg.log, then forward each line to Manticore.

Store the configuration as fluent-bit.conf, then launch Fluent Bit:

fluent-bit -c fluent-bit.conf

To run it in Docker, mount the log file (read-only) and the configuration:

docker run --rm -v /var/log/dpkg.log:/var/log/dpkg.log:ro \
  -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro \
  fluent/fluent-bit:latest -c /fluent-bit/etc/fluent-bit.conf

After ingestion begins, Manticore automatically creates the dpkg_log table. Here is the table definition and sample data:

mysql> DESCRIBE dpkg_log;
+-------------+--------+----------------+
| Field       | Type   | Properties     |
+-------------+--------+----------------+
| id          | bigint |                |
| @timestamp  | text   | indexed stored |
| log         | text   | indexed stored |
+-------------+--------+----------------+
mysql> SELECT * FROM dpkg_log LIMIT 3\G
*************************** 1. row ***************************
id: 7856533729353662465
@timestamp: 2023-08-04T15:09:21.191Z
log: 2023-06-05 14:03:04 startup archives install
*************************** 2. row ***************************
id: 7856533729353662466
@timestamp: 2023-08-04T15:09:21.191Z
log: 2023-06-05 14:03:04 install base-passwd:amd64 <none> 3.5.47
*************************** 3. row ***************************
id: 7856533729353662467
@timestamp: 2023-08-04T15:09:21.191Z
log: 2023-06-05 14:03:04 status half-installed base-passwd:amd64 3.5.47

With this lightweight pipeline, Fluent Bit handles log collection and delivery while Manticore indexes the events for fast search and analytics. The approach works equally well for other log sources—just add more inputs and reuse the same Elasticsearch-compatible output pointed at your Manticore cluster.

NOTE: The integration with Fluent Bit requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Vector by Datadog is an open source observability data pipeline that can collect, transform, and route logs or metrics. While Vector can aggregate the data by itself, pairing it with Manticore provides a purpose-built storage and search layer.

The example below shows how to forward Debian's dpkg.log through Vector.dev and index it in Manticore.

2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1
2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>
2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>

Create a vector.toml similar to the following:

[sources.test_file]
type = "file"
include = [ "/var/log/dpkg.log" ]
[transforms.modify_test_file]
type = "remap"
inputs = [ "test_file" ]
source = """
.vec_timestamp = del(.timestamp)"""
[sinks.manticore]
type = "elasticsearch"
inputs = [ "modify_test_file" ]
endpoints = ["http://127.0.0.1:9308"]
bulk.index = "dpkg_log"

• endpoints points to Manticore's HTTP interface (port 9308 by default). Adjust it if your instance listens elsewhere.
• The remap transform moves Vector's default timestamp field to vec_timestamp, because timestamp is reserved in Manticore.
• bulk.index defines the table that will be created automatically when Vector starts sending data.

Start Vector.dev with this configuration and it will tail the log file, transform each event, and forward it directly to Manticore.

Save the configuration as vector.toml, then start the agent:

vector --config vector.toml

If you run Vector.dev in Docker, mount both the config file and the log directory, for example:

docker run --rm -v /var/log/dpkg.log:/var/log/dpkg.log:ro \
  -v $(pwd)/vector.toml:/etc/vector/vector.toml:ro \
  timberio/vector:latest --config /etc/vector/vector.toml

When the pipeline runs, Manticore creates the dpkg_log table automatically. Its schema and example documents look like this:

mysql> DESCRIBE dpkg_log;
+-----------------+---------+--------------------+
| Field           | Type    | Properties         |
+-----------------+---------+--------------------+
| id              | bigint  |                    |
| file            | text    | indexed stored     |
| host            | text    | indexed stored     |
| message         | text    | indexed stored     |
| source_type     | text    | indexed stored     |
| vec_timestamp   | text    | indexed stored     |
+-----------------+---------+--------------------+
mysql> SELECT * FROM dpkg_log LIMIT 3\G
*************************** 1. row ***************************
id: 7856533729353672195
file: /var/log/dpkg.log
host: logstash-787f68f6f-nhdd2
message: 2023-06-05 14:03:04 startup archives install
source_type: file
vec_timestamp: 2023-08-04T15:32:50.203091741Z
*************************** 2. row ***************************
id: 7856533729353672196
file: /var/log/dpkg.log
host: logstash-787f68f6f-nhdd2
message: 2023-06-05 14:03:04 install base-passwd:amd64 <none> 3.5.47
source_type: file
vec_timestamp: 2023-08-04T15:32:50.203808861Z
*************************** 3. row ***************************
id: 7856533729353672197
file: /var/log/dpkg.log
host: logstash-787f68f6f-nhdd2
message: 2023-06-05 14:03:04 status half-installed base-passwd:amd64 3.5.47
source_type: file
vec_timestamp: 2023-08-04T15:32:50.203814031Z

Using Vector.dev with Manticore lets you collect logs from virtually any source, enrich or sanitize them in-flight, and store the results in a search-ready database. This workflow keeps observability pipelines simple while still enabling advanced transformations when needed.

NOTE: Some dashboard panels and metrics require Manticore Buddy. If some panels appear empty, ensure that Buddy is installed.

Grafana is an open-source platform for data visualization and monitoring that enables the creation of interactive dashboards and charts. Manticore Search can integrate with Grafana in two primary ways:

1. Monitoring Manticore Performance: Using Prometheus to collect metrics from Manticore and visualize them in Grafana dashboards. This approach focuses on system health, performance monitoring, and alerting for the search engine itself.
2. Visualizing Search Data: Connecting via the MySQL connector to query and display data stored in Manticore tables, similar to how Kibana is used for Elasticsearch data visualization. This is ideal for analyzing trends, aggregations, and custom visualizations based on your indexed data.

Currently, Grafana versions 10.0-12.4 are tested and supported.

Manticore provides a prebuilt Grafana dashboard and Prometheus alerting rules in the dedicated repository manticoresoftware/grafana-dashboard. The repository also includes an all-in-one Docker image with Grafana and Prometheus for a quick start. These resources can be used independently with your existing Grafana and Prometheus setup, allowing you to monitor Manticore's operational metrics—such as latency, resource usage, and errors—without needing to run a separate instance.

• Additional screenshots: https://github.com/manticoresoftware/grafana-dashboard
• Dashboard JSON: grafana/dashboards/manticore-dashboard.json
• Alert rules (Prometheus/Alertmanager): prometheus/rules/manticore-alerts.yml

For a fully preconfigured setup, run the all-in-one Docker image with Grafana and Prometheus tailored for Manticore monitoring. This is ideal if you don't have an existing setup or want a quick test environment:

docker run -e MANTICORE_TARGETS=localhost:9308 -p 127.0.0.1:3000:3000 manticoresearch/dashboard:latest

MANTICORE_TARGETS is a comma-separated list of Manticore http endpoints in the host:port format (for example: localhost:9308,10.0.0.5:9308).

If you already have your own Grafana and Prometheus running, you can skip the Docker image and directly import the dashboard and alert rules as described below.

To add the Manticore dashboard to your existing Grafana instance:

1. In Grafana, navigate to Dashboards → New → Import.
2. Upload the manticore-dashboard.json file (or paste its JSON content).
3. Select your Prometheus data source when prompted.
4. Verify and adjust the job dashboard variable if needed (the default expected label value is manticoresearch).

To integrate the Manticore alert rules into your existing Prometheus setup:

1. Download manticore-alerts.yml and add it to your Prometheus rule_files configuration.
2. If your scrape job name differs from manticoresearch, update the {job="manticoresearch"} matchers in the rules accordingly.
3. Reload or restart Prometheus to apply the changes.

The manticore-alerts.yml file is designed to focus on key signals related to availability, overload, and resource exhaustion—issues that often impact users directly. Thresholds are set as safe defaults; adjust them based on your specific workload.

• ManticoreTargetDown (critical): Prometheus fails to scrape Manticore (up == 0 for 2 minutes). This typically indicates the metrics endpoint is down, unreachable, or the scrape configuration is incorrect.
• ManticoreRecentlyRestarted (warning): Uptime remains below 5 minutes for 5 minutes. This often signals instability, such as crash loops, OOM kills, or orchestrator restarts.
• ManticoreMaxedOutErrors (warning): manticore_maxed_out_error_count increases over the last 5 minutes. Manticore is rejecting requests due to hitting capacity, concurrency, or resource limits.
• ManticoreSearchLatencyP95High (warning): P95 search latency exceeds 500 ms for 10 minutes. Many users may perceive searches as slow.
• ManticoreSearchLatencyP99High (critical): P99 search latency exceeds 1,000 ms for 10 minutes. Tail latency is severely elevated, indicating very slow worst-case requests.
• ManticoreWorkQueueBacklog (warning): Work queue length exceeds 100 for 5 minutes. Requests are accumulating, which often leads to increased latency.
• ManticoreWorkersSaturated (warning): More than 90% of workers are active for 10 minutes. Worker threads are nearing full capacity, likely causing higher latency and queueing.
• ManticoreQueryCacheNearLimit (warning): Query cache usage exceeds 90% of its configured maximum for 10 minutes. This increases the risk of cache churn and evictions, potentially slowing queries; consider expanding the cache or optimizing query patterns.
• ManticoreAgentRetryCountHigh (warning): manticore_agent_retry_count increases by more than 10 in 5 minutes. This usually points to connectivity issues with remote agents (e.g., network problems, timeouts, or failures in distributed queries).
• ManticoreCurrentConnectionsHigh (warning): Current connections exceed 500 for 10 minutes. This could indicate a traffic surge, connection leaks, or slow clients.
• ManticoreSlowestThreadHigh (warning): manticore_slowest_thread_seconds exceeds 30 seconds for 10 minutes. Long-running or stuck queries are present; investigate slow queries and resource contention.
• ManticoreConnectTimeHigh (warning): manticore_connect_time_seconds exceeds 0.2 seconds for 5 minutes. Connection establishment is delayed (e.g., due to network issues, TLS overhead, or server overload).
• ManticoreSearchdCrashesRecently (critical): manticore_searchd_crashes_total increases in the last 10 minutes. A crash has occurred; review logs, core dumps, and OOM events.
• ManticoreBinlogFilesHigh (warning): Binlog file count exceeds 1,000 for 10 minutes. This suggests binlogs are not rotating or cleaning properly; check binlog settings and disk usage.
• ManticoreSearchdFdCountHigh (warning): searchd file descriptor count exceeds 4,096 for 10 minutes. This risks hitting OS limits; verify ulimit -n, connections, open files, and potential leaks.
• ManticoreBuddyFdCountHigh (warning): Buddy file descriptor count exceeds 4,096 for 10 minutes. Similar to the above, but for the Buddy process.
• ManticoreSearchdAnonRssHigh (warning): searchd anonymous RSS exceeds 8 GiB for 10 minutes. High non-file-backed memory usage; examine memory growth, query/index patterns, and container limits.
• ManticoreBuddyAnonRssHigh (warning): Buddy anonymous RSS exceeds 8 GiB for 10 minutes. Elevated Buddy memory usage; review Buddy logs and workload.
• ManticoreNonServedTables (warning): One or more tables listed in manticore.json are missing from SHOW TABLES for 10 minutes. This typically means tables failed to load or were dropped/moved; check startup logs and table paths.
• ManticoreDiskMappedCacheLow (warning): Disk mapped cache ratio falls below 50% for 15 minutes. Increased disk I/O is likely, potentially slowing searches if the working set exceeds RAM.
• ManticoreDiskMappedCacheVeryLow (critical): Disk mapped cache ratio falls below 20% for 15 minutes. Severe cache misses; anticipate heavy disk I/O and high latency.

The dashboard organizes panels by topic. Key highlights include:

• Health / Uptime: Time since the last restart; frequent drops indicate restarts or crashes.
• Performance / Search Latency (P95/P99): Tail latency metrics; persistent increases often signal overload or slow queries.
• Performance / Work Queue Length and Load / Queue Pressure: Backlog and queueing indicators; growing queues suggest the system is lagging.
• Workers Utilization % and Load / Workers Active/Total: Worker saturation levels; values near 100% typically correlate with higher latency.
• Performance / Query Cache Usage and Performance / Query Cache Hit Rate: Cache pressure and effectiveness; high usage with low hit rates may indicate inefficiency.
• Tables / Disk Mapped Cache Ratio (Worst 10): Low ratios imply more disk reads and potentially slower queries.

This approach uses Grafana's MySQL connector to directly query Manticore tables, enabling visualizations of your search data, such as time-series trends or aggregations.

Before setting up data visualization:

1. Ensure Manticore Search (version 6.2.0 or later) is installed and configured. See the official Manticore Search installation guide for details.
2. Install Grafana on your system. Refer to the official Grafana installation guide for instructions.

To connect Manticore Search to Grafana:

1. Log in to your Grafana dashboard and click on "Configuration" (the gear icon) in the left sidebar.
2. Select "Data Sources" and click on "Add data source."
3. Choose "MySQL" from the list of available data sources.
4. In the settings page, provide the following details:
   • Name: A name for the data source (e.g., "Manticore Search")
   • Host: The hostname or IP address of your Manticore Search server (with MySQL port, default: localhost:9306)
   • Database: Leave empty or specify your database name
   • User: The username with access to Manticore Search (default: root)
   • Password: The password for the specified user (default: empty)
5. Click on "Save & Test" to verify the connection.

After connecting Manticore Search to Grafana, you can create dashboards and visualizations:

1. In the Grafana dashboard, click on the "+" icon in the left sidebar and select "New dashboard."
2. Click on the "Add visualization" button to start configuring your chart.
3. Choose the Manticore Search data source that you connected via the MySQL connector.
4. Select the type of chart you want to create (e.g., Time Series, Bar chart, Candlestick, Pie chart).
5. Use Grafana's query builder or write an SQL query to fetch data from your Manticore Search tables.
6. Customize the chart's appearance, labels, and other settings as needed.
7. Click "Apply" to save your visualization to the dashboard.

Here's a simple example using time-series data. First, create a table and load sample data:

CREATE TABLE btc_usd_trading (
  id bigint,
  time timestamp,
  open float,
  high float,
  low float,
  close float
);

Load the data:

curl -sSL https://gist.githubusercontent.com/donhardman/df109ba6c5e690f73198b95f3768e73f/raw/0fab3aee69d7007fad012f4e97f38901a64831fb/btc_usd_trading.sql | mysql -h0 -P9306

In Grafana, you can create:

• Time Series chart: Visualize price changes over time
• Candlestick chart: Display open, high, low, close values for financial data
• Aggregation charts: Use COUNT, AVG, MAX, MIN functions

Example queries:

-- Time series query
SELECT time, close FROM btc_usd_trading ORDER BY time;
-- Aggregation query
SELECT DATE(time) as date, AVG(close) as avg_price
FROM btc_usd_trading
GROUP BY date
ORDER BY date;

When working with Manticore Search through Grafana, you can:

• Execute SQL queries against Manticore Search tables
• Use aggregation functions: COUNT, AVG, MAX, MIN
• Apply GROUP BY and ORDER BY operations
• Filter data with WHERE clauses
• Access table metadata via information_schema.tables
• Create various visualization types supported by Grafana

• Some advanced MySQL features may not be available when working with Manticore Search through Grafana.
• Only features supported by Manticore Search are available. Refer to the SQL reference for details.

For more information and detailed tutorials:

• Grafana integration blog post
• Official Grafana documentation

Kibana is a visual interface that allows you to explore, visualize, and create dashboards for your log data. Integrating Kibana with Manticore Search can speed up the loading of Kibana visualizations by up to 3 times compared to Elasticsearch, as demonstrated in this demo. This integration enables users to seamlessly analyze their data using interactive dashboards, custom visualizations, and real-time search capabilities. It also simplifies handling diverse data sources by supporting tools like Logstash and Filebeat for streamlined data ingestion, making it a great choice for log analysis workflows.

1. Download Kibana: Ensure you download a Kibana version compatible with Manticore. Currently, version 7.6.0 is tested and recommended. Other 7.x versions may work but could introduce issues. Version 8.x is not supported.
2. Verify Manticore: Ensure your Manticore instance is running and its HTTP API is reachable (default: http://localhost:9308).

1. Open the Kibana configuration file (kibana.yml).
2. Set the URL of your Manticore instance:

   elasticsearch.hosts: ["http://localhost:9308"]

3. Start Kibana and open it in your browser at http://localhost:5601. Replace localhost with your server's IP or hostname if necessary.

Note: Manticore does not require authentication setup when working with Kibana. Also note that Manticore must work in real-time mode to be integrated with Kibana.

    listen = 127.0.0.1:9308:http
    pid_file = /var/run/manticore/searchd.pid
    data_dir = /var/lib/manticore
 }

## Supported Features
### Discover
- Use the **Discover** tab in Kibana to search and filter your data interactively.
- Refine your searches using the query bar with simple queries in the %Kibana query language [https://www.elastic.co/guide/en/kibana/current/kuery-query.html] %.

### Visualizations
- Navigate to **Visualizations** to create custom visualizations:
  - Create a table pattern (it’s called an 'index pattern' in Kibana) if one doesn’t already exist to define your data source.
  - Choose a visualization type (e.g., bar chart, line chart, or pie chart).
  - Configure your visualization, execute it, and explore your data.
  - Save your visualizations for future use.

### Dashboards
- Access **Dashboards** to create or view interactive dashboards:
  - Add visualizations, filters, or controls for a personalized experience.
  - Interact with your data directly from the dashboard.
  - Save dashboards for future use.

### Management
- Go to **Management > Kibana** to customize settings like default time zones and visualization preferences.

## Limitations
- Currently, Kibana version 7.6.0 is tested and recommended. Other 7.x versions may work but could cause issues. Versions 8.x are not supported.
- The following Elasticsearch-specific field types are not supported:
  - %Spatial data types [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html#spatial_datatypes] %
  - %Structured data types [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html#structured-data-types] %
  - %Document ranking types [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html#document-ranking-types] %
  - %Text search types [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html#text-search-types] % (except for plain 'text')
  - %Relational data types [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html#object-types] %
- Metric aggregation functions are limited to %those supported by Manticore [../Searching/Grouping.md#Aggregation-functions] %.
- The following Kibana tools are not supported:
  - %Canvas [https://www.elastic.co/guide/en/kibana/7.6/canvas.html] % – A visualization and presentation tool for combining data with colors and images.
  - %Elastic Maps [https://www.elastic.co/guide/en/kibana/7.6/maps.html] % – A tool for analyzing geographical data.
  - %Metrics [https://www.elastic.co/guide/en/kibana/7.6/xpack-infra.html] % – An app for monitoring infrastructure metrics.
  - %Logs [https://www.elastic.co/guide/en/kibana/7.6/xpack-logs.html] % – A console-like display for exploring logs from common services.
  - Monitoring:
    - %Uptime [https://www.elastic.co/guide/en/kibana/7.6/xpack-uptime.html] % – Monitors the status of network endpoints via HTTP/S, TCP, and ICMP.
    - %APM (Application Performance Monitoring) [https://www.elastic.co/guide/en/kibana/7.6/xpack-apm.html] % – Collects in-depth performance metrics from applications.
    - %SIEM (Security Information and Event Management) [https://www.elastic.co/guide/en/kibana/7.6/xpack-siem.html] % – An interactive workspace for security teams to triage events and conduct initial investigations.
    - %ILM (Index lifecycle management) [https://www.elastic.co/guide/en/elasticsearch/reference/7.6/index-lifecycle-management.html] % - Automatically manage indices according to performance, resiliency, and retention requirements.
    - %Stack Monitoring [https://www.elastic.co/guide/en/kibana/7.6/xpack-monitoring.html] % – Provides visualizations of monitoring data across the Elastic Stack.
  - %Elasticsearch Management [https://www.elastic.co/guide/en/kibana/7.6/management.html] % – A UI for managing Elastic Stack objects, including ILM (Index Lifecycle Management), etc.

## Data Ingestion and Exploration
Integrate Manticore with tools like %Logstash [../Integration/Logstash.md] %, %Filebeat [../Integration/Filebeat.md] %, %Fluentbit [https://manticoresearch.com/blog/integration-of-manticore-with-fluentbit/] %, or %Vector.dev [https://manticoresearch.com/blog/integration-of-manticore-with-vectordev/] % to ingest data from sources like web logs. Once the data is loaded into Manticore, you can explore and visualize it in Kibana.