NOTE: The integration with Logstash requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Logstash is a log management tool that collects data from a variety of sources, transforms it on the fly, and sends it to your desired destination. It is often used as a data pipeline for Elasticsearch, an open-source analytics and search engine.

Now, Manticore supports the use of Logstash as a processing pipeline. This allows the collected and transformed data to be sent to Manticore just like to Elasticsearch. Currently, versions 7.6+ are supported.

Let’s examine a simple example of a Logstash config file used for indexing dpkg.log, a standard log file of the Debian package manager. The log itself has a simple structure, as shown below:

2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1
2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>
2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>

Here is an example Logstash configuration:

input {
  file {
    path => ["/var/log/dpkg.log"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
    exit_after_read => "true"
   file_completed_action => "log"
   file_completed_log_path => "/dev/null"
  }
}
output {
  elasticsearch {
   index => " dpkg_log"
   hosts => ["http://localhost:9308"]
   ilm_enabled => false
   manage_template => false
  }
}

Note that, before proceeding further, one crucial caveat needs to be addressed: Manticore does not support Log Template Management and the Index Lifecycle Management features of Elasticsearch. As these features are enabled by default in Logstash, they need to be explicitly disabled in the config. Additionally, the hosts option in the output config section must correspond to Manticore’s HTTP listen port (default is localhost:9308).

After adjusting the config as described, you can run Logstash, and the data from the dpkg log will be passed to Manticore and properly indexed.

Here is the resulting schema of the created table and an example of the inserted document:

mysql> DESCRIBE dpkg_log;
+------------------+--------+---------------------+
| Field            | Type   | Properties          |
+------------------+--------+---------------------+
| id               | bigint |                     |
| message          | text   | indexed stored      |
| @version         | text   | indexed stored      |
| @timestamp       | text   | indexed stored      |
| path             | text   | indexed stored      |
| host             | text   | indexed stored      |
+------------------+--------+---------------------+

mysql> SELECT * FROM dpkg_log LIMIT 1\G
*************************** 1. row ***************************
id: 7280000849080746110
host: logstash-db848f65f-lnlf9
message: 2023-04-12 02:03:21 status unpacked libc-bin:amd64 2.31-0ubuntu9
path: /var/log/dpkg.log
@timestamp: 2023-06-16T09:23:57.405Z
@version: 1

NOTE: The integration with Filebeat requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Filebeat is a lightweight shipper for forwarding and centralizing log data. Once installed as an agent, it monitors the log files or locations you specify, collects log events, and forwards them for indexing, usually to Elasticsearch or Logstash.

Now, Manticore also supports the use of Filebeat as processing pipelines. This allows the collected and transformed data to be sent to Manticore just like to Elasticsearch. Currently, All versions to 9.0 are fully supported.

Configuration varies depending on which version of Filebeat you're using.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/dpkg.log
  close_eof: true
  scan_frequency: 1s
output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"

For versions 8.1 through 8.10, you need to add the allow_older_versions option:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/dpkg.log
  close_eof: true
  scan_frequency: 1s
output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"

From version 8.11, output compression is enabled by default, so you must explicitly set compression_level: 0 for compatibility with Manticore:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/dpkg.log
  close_eof: true
  scan_frequency: 1s
output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"

Filebeat 9.0 introduces a major architecture change, replacing the log input type with filestream. Starting from version 9.0, the default file identification method also changed to fingerprint, which requires files to be at least 1024 bytes (see issue #44780). For Manticore compatibility with files of any size, you must disable fingerprinting.

Here's the required configuration for Filebeat 9.0 and all later versions:

filebeat.inputs:
- type: filestream
  id: dpkg-log-input
  enabled: true
  paths:
    - /var/log/dpkg.log
  prospector.scanner.check_interval: 1s
  prospector.scanner.fingerprint.enabled: false
output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"

Important notes for Filebeat 9.0+:

• The type: filestream input replaces the older type: log
• The prospector.scanner.fingerprint.enabled: false setting is required to disable fingerprint-based file identification, ensuring reliable processing of files smaller than 1024 bytes
• The id field is required for filestream inputs and must be unique

Once you run Filebeat with this configuration, log data will be sent to Manticore and properly indexed. Here is the resulting schema of the table created by Manticore and an example of the inserted document:

mysql> DESCRIBE dpkg_log;
+------------------+--------+--------------------+
| Field            | Type   | Properties         |
+------------------+--------+--------------------+
| id               | bigint |                    |
| @timestamp       | text   | indexed stored     |
| message          | text   | indexed stored     |
| log              | json   |                    |
| input            | json   |                    |
| ecs              | json   |                    |
| host             | json   |                    |
| agent            | json   |                    |
+------------------+--------+--------------------+

mysql> SELECT * FROM dpkg_log LIMIT 1\G
*************************** 1. row ***************************
id: 7280000849080753116
@timestamp: 2023-06-16T09:27:38.792Z
message: 2023-04-12 02:06:08 status half-installed libhogweed5:amd64 3.5.1+really3.5.1-2
input: {"type":"filestream"}
ecs: {"version":"1.6.0"}
host: {"name":"logstash-db848f65f-lnlf9"}
agent: {"ephemeral_id":"587c2ebc-e7e2-4e27-b772-19c611115996","id":"2e3d985b-3610-4b8b-aa3b-2e45804edd2c","name":"logstash-db848f65f-lnlf9","type":"filebeat","version":"7.10.0","hostname":"logstash-db848f65f-lnlf9"}
log: {"offset":80,"file":{"path":"/var/log/dpkg.log"}}

Kibana is a visual interface that allows you to explore, visualize, and create dashboards for your log data. Integrating Kibana with Manticore Search can speed up the loading of Kibana visualizations by up to 3 times compared to Elasticsearch, as demonstrated in this demo. This integration enables users to seamlessly analyze their data using interactive dashboards, custom visualizations, and real-time search capabilities. It also simplifies handling diverse data sources by supporting tools like Logstash and Filebeat for streamlined data ingestion, making it a great choice for log analysis workflows.

1. Download Kibana: Ensure you download a Kibana version compatible with Manticore. Currently, version 7.6.0 is tested and recommended. Other 7.x versions may work but could introduce issues. Version 8.x is not supported.
2. Verify Manticore: Ensure your Manticore instance is running and its HTTP API is reachable (default: http://localhost:9308).

1. Open the Kibana configuration file (kibana.yml).
2. Set the URL of your Manticore instance:

   elasticsearch.hosts: ["http://localhost:9308"]

3. Start Kibana and open it in your browser at http://localhost:5601. Replace localhost with your server's IP or hostname if necessary.

Note: Manticore does not require authentication setup when working with Kibana.

• Use the Discover tab in Kibana to search and filter your data interactively.
• Refine your searches using the query bar with simple queries in the Kibana query language.

• Navigate to Visualizations to create custom visualizations:
  • Create a table pattern (it’s called an 'index pattern' in Kibana) if one doesn’t already exist to define your data source.
  • Choose a visualization type (e.g., bar chart, line chart, or pie chart).
  • Configure your visualization, execute it, and explore your data.
  • Save your visualizations for future use.

• Access Dashboards to create or view interactive dashboards:
  • Add visualizations, filters, or controls for a personalized experience.
  • Interact with your data directly from the dashboard.
  • Save dashboards for future use.

• Go to Management > Kibana to customize settings like default time zones and visualization preferences.

• Currently, Kibana version 7.6.0 is tested and recommended. Other 7.x versions may work but could cause issues. Versions 8.x are not supported.
• The following Elasticsearch-specific field types are not supported:
  • Spatial data types
  • Structured data types
  • Document ranking types
  • Text search types (except for plain 'text')
  • Relational data types
• Metric aggregation functions are limited to those supported by Manticore.
• The following Kibana tools are not supported:
  • Canvas – A visualization and presentation tool for combining data with colors and images.
  • Elastic Maps – A tool for analyzing geographical data.
  • Metrics – An app for monitoring infrastructure metrics.
  • Logs – A console-like display for exploring logs from common services.
  • Monitoring:
    • Uptime – Monitors the status of network endpoints via HTTP/S, TCP, and ICMP.
    • APM (Application Performance Monitoring) – Collects in-depth performance metrics from applications.
    • SIEM (Security Information and Event Management) – An interactive workspace for security teams to triage events and conduct initial investigations.
    • ILM (Index lifecycle management) - Automatically manage indices according to performance, resiliency, and retention requirements.
    • Stack Monitoring – Provides visualizations of monitoring data across the Elastic Stack.
  • Elasticsearch Management – A UI for managing Elastic Stack objects, including ILM (Index Lifecycle Management), etc.

Integrate Manticore with tools like Logstash, Filebeat, Fluentbit, or Vector.dev to ingest data from sources like web logs. Once the data is loaded into Manticore, you can explore and visualize it in Kibana.