NOTE: The integration with Logstash requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Logstash is a log management tool that collects data from a variety of sources, transforms it on the fly, and sends it to your desired destination. It is often used as a data pipeline for Elasticsearch, an open-source analytics and search engine.

Now, Manticore supports the use of Logstash as a processing pipeline. This allows the collected and transformed data to be sent to Manticore just like to Elasticsearch. Currently, the versions 7.6-7.15 are supported.

Let’s examine a simple example of a Logstash config file used for indexing dpkg.log, a standard log file of the Debian package manager. The log itself has a simple structure, as shown below:

2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1
2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>
2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>

Here is an example Logstash configuration:

input {
  file {
    path => ["/var/log/dpkg.log"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
    exit_after_read => "true"
   file_completed_action => "log"
   file_completed_log_path => "/dev/null"
  }
}

output {
  elasticsearch {
   index => " dpkg_log"
   hosts => ["http://localhost:9308"]
   ilm_enabled => false
   manage_template => false
  }
}

Note that, before proceeding further, one crucial caveat needs to be addressed: Manticore does not support Log Template Management and the Index Lifecycle Management features of Elasticsearch. As these features are enabled by default in Logstash, they need to be explicitly disabled in the config. Additionally, the hosts option in the output config section must correspond to Manticore’s HTTP listen port (default is localhost:9308).

After adjusting the config as described, you can run Logstash, and the data from the dpkg log will be passed to Manticore and properly indexed.

Here is the resulting schema of the created table and an example of the inserted document:

mysql> DESCRIBE dpkg_log;
+------------------+--------+---------------------+
| Field            | Type   | Properties          |
+------------------+--------+---------------------+
| id               | bigint |                     |
| message          | text   | indexed stored      |
| @version         | text   | indexed stored      |
| @timestamp       | text   | indexed stored      |
| path             | text   | indexed stored      |
| host             | text   | indexed stored      |
+------------------+--------+---------------------+

mysql> SELECT * FROM dpkg_log LIMIT 1\G

*************************** 1. row ***************************
id: 7280000849080746110
host: logstash-db848f65f-lnlf9
message: 2023-04-12 02:03:21 status unpacked libc-bin:amd64 2.31-0ubuntu9
path: /var/log/dpkg.log
@timestamp: 2023-06-16T09:23:57.405Z
@version: 1

NOTE: The integration with Filebeat requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Filebeat is a lightweight shipper for forwarding and centralizing log data. Once installed as an agent, it monitors the log files or locations you specify, collects log events, and forwards them for indexing, usually to Elasticsearch or Logstash.

Now, Manticore also supports the use of Filebeat as processing pipelines. This allows the collected and transformed data to be sent to Manticore just like to Elasticsearch. Currently, all the versions >= 7.10 are supported.

Below is a Filebeat config to work with our example dpkg log:

filebeat.inputs:
- type: filestream
  id: example
  paths:
    - /var/log/dpkg.log

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index:  "dpkg_log"
  allow_older_versions: true

setup.ilm:
  enabled: false

setup.template:
  name: "dpkg_log"
  pattern: "dpkg_log"

Note that Filebeat versions higher than 8.10 have the output compression feature enabled by default. That is why the compression_level: 0 option must be added to the configuration file to provide compatibility with Manticore:

filebeat.inputs:
- type: filestream
  id: example
  paths:
    - /var/log/dpkg.log

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index:  "dpkg_log"
  allow_older_versions: true
  compression_level: 0

setup.ilm:
  enabled: false

setup.template:
  name: "dpkg_log"
  pattern: "dpkg_log"

Once you run Filebeat with this configuration, log data will be sent to Manticore and properly indexed. Here is the resulting schema of the table created by Manticore and an example of the inserted document:

mysql> DESCRIBE dpkg_log;
+------------------+--------+--------------------+
| Field            | Type   | Properties         |
+------------------+--------+--------------------+
| id               | bigint |                    |
| @timestamp       | text   | indexed stored     |
| message          | text   | indexed stored     |
| log              | json   |                    |
| input            | json   |                    |
| ecs              | json   |                    |
| host             | json   |                    |
| agent            | json   |                    |
+------------------+--------+--------------------+

mysql> SELECT * FROM dpkg_log LIMIT 1\G
*************************** 1. row ***************************
id: 7280000849080753116
@timestamp: 2023-06-16T09:27:38.792Z
message: 2023-04-12 02:06:08 status half-installed libhogweed5:amd64 3.5.1+really3.5.1-2
input: {"type":"filestream"}
ecs: {"version":"1.6.0"}
host: {"name":"logstash-db848f65f-lnlf9"}
agent: {"ephemeral_id":"587c2ebc-e7e2-4e27-b772-19c611115996","id":"2e3d985b-3610-4b8b-aa3b-2e45804edd2c","name":"logstash-db848f65f-lnlf9","type":"filebeat","version":"7.10.0","hostname":"logstash-db848f65f-lnlf9"}
log: {"offset":80,"file":{"path":"/var/log/dpkg.log"}}

NOTE: The integration with Apache Superset requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Apache Superset is a modern data exploration and visualization platform. Manticore Search can be integrated with Apache Superset, allowing users to harness the full potential of Manticore Search for data analysis and visualization.

Note: This integration works only with Apache Superset 3.x branch and is not yet available for version 4 or newer.

Before integrating Manticore Search with Apache Superset, ensure that:

1. Manticore Search (version 6.2.0 or later) is properly installed and configured on your server. Refer to the official Manticore Search installation guide for assistance.
2. Apache Superset is set up on your system. Follow the official Apache Superset installation guide for installation instructions.

To connect Manticore Search to Apache Superset:

1. Log in to your Apache Superset dashboard.
2. Click on "+" in the top navigation bar and choose "Database Connections".
3. Click "+ DATABASE" to add a new database.
4. Select "MySQL" as the database type.
5. In the configuration page, provide the following details:
   • Database Name: A name for the database (e.g., "Manticore")
   • Host: Your Manticore Search server host
   • Port: The port on which Manticore Search is running
   • Username: Your Manticore Search username
   • Display Name: A display name for the connection
6. Click "Connect" to verify the connection and save.

After connecting Manticore Search to Apache Superset, you can create charts and dashboards using your Manticore data:

1. In the Apache Superset dashboard, click on "Dashboards" in the top navigation bar and select "New dashboard".
2. Open the dashboard and click on the "+" icon to add a new chart.
3. Choose the dataset connected to Manticore Search.
4. Select the type of chart you want to create (e.g., bar chart, line chart, pie chart).
5. Use Apache Superset's query builder or write an SQL query to fetch data from your Manticore Search database.
6. Customize the chart's appearance, labels, and other settings as needed.
7. Click "UPDATE CHART" to update the chart.
8. Add a name for the chart and click "Save" to permanently save it in the Dashboard.

When working with Manticore Search through Apache Superset, you can:

• View and query data from Manticore Search tables
• Create various types of visualizations based on your Manticore data
• Build and customize dashboards using these visualizations

Manticore Search data types are mapped to MySQL data types when working with Apache Superset. Refer to the Manticore Search documentation for more details on supported data types and their mappings.

• Some advanced MySQL features may not be available when working with Manticore Search through Apache Superset.
• Certain data manipulation operations might be limited compared to working directly with MySQL databases.

By following these steps and guidelines, you can effectively integrate Manticore Search with Apache Superset for powerful data exploration and visualization capabilities.

Please refer to the following resources for more information on Apache Superset and Manticore Search:

• Apache Superset Integration