NOTE: The integration with Logstash requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Logstash is a log management tool that collects data from a variety of sources, transforms it on the fly, and sends it to your desired destination. It is often used as a data pipeline for Elasticsearch, an open-source analytics and search engine.

Now, Manticore supports the use of Logstash as a processing pipeline. This allows the collected and transformed data to be sent to Manticore just like to Elasticsearch. Currently, versions 7.6-9.2 are supported.

Let’s examine a simple example of a Logstash config file used for indexing dpkg.log, a standard log file of the Debian package manager. The log itself has a simple structure, as shown below:

2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1
2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>
2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>

Here is an example Logstash configuration:

input {
  file {
    path => ["/var/log/dpkg.log"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
    exit_after_read => "true"
   file_completed_action => "log"
   file_completed_log_path => "/dev/null"
  }
}
output {
  elasticsearch {
   index => " dpkg_log"
   hosts => ["http://localhost:9308"]
   ilm_enabled => false
   manage_template => false
  }
}

Note that, before proceeding further, one crucial caveat needs to be addressed: Manticore does not support Log Template Management and the Index Lifecycle Management features of Elasticsearch. As these features are enabled by default in Logstash, they need to be explicitly disabled in the config. Additionally, the hosts option in the output config section must correspond to Manticore’s HTTP listen port (default is localhost:9308).

Configuration varies depending on which version of Logstash you're using.

For Logstash 7.17, the basic configuration is straightforward and doesn't require additional ILM settings:

input {
  file {
    path => ["/var/log/dpkg.log"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
    exit_after_read => "true"
    file_completed_action => "log"
    file_completed_log_path => "/dev/null"
  }
}
output {
  elasticsearch {
    index => "dpkg_log"
    hosts => ["http://localhost:9308"]
  }
}

Run with:

logstash -f logstash.conf

Starting from version 8.0, ILM (Index Lifecycle Management) and template management are enabled by default and must be explicitly disabled for compatibility with Manticore:

input {
  file {
    path => ["/var/log/dpkg.log"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
    exit_after_read => "true"
    file_completed_action => "log"
    file_completed_log_path => "/dev/null"
  }
}
output {
  elasticsearch {
    index => "dpkg_log"
    hosts => ["http://localhost:9308"]
    ilm_enabled => false
    manage_template => false
  }
}

For versions 9.0 and 9.1, Logstash requires running as a superuser. Set the environment variable before starting:

export ALLOW_SUPERUSER=1
logstash -f logstash.conf

From version 9.2, the recommended approach is to configure the superuser setting via a configuration file instead of using environment variables. This provides a more permanent and manageable solution.

Configuration file (e.g., logstash.conf):

input {
  file {
    path => ["/var/log/dpkg.log"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
    exit_after_read => "true"
    file_completed_action => "log"
    file_completed_log_path => "/dev/null"
  }
}
output {
  elasticsearch {
    index => "dpkg_log"
    hosts => ["http://localhost:9308"]
    ilm_enabled => false
    manage_template => false
  }
}

Create /etc/logstash/logstash.yml:

allow_superuser: true

Run with:

logstash --path.settings=/etc/logstash -f logstash.conf

After adjusting the config as described, you can run Logstash, and the data from the dpkg log will be passed to Manticore and properly indexed.

Here is the resulting schema of the created table and an example of the inserted document:

mysql> DESCRIBE dpkg_log;
+------------------+--------+---------------------+
| Field            | Type   | Properties          |
+------------------+--------+---------------------+
| id               | bigint |                     |
| message          | text   | indexed stored      |
| @version         | text   | indexed stored      |
| @timestamp       | text   | indexed stored      |
| path             | text   | indexed stored      |
| host             | text   | indexed stored      |
+------------------+--------+---------------------+

mysql> SELECT * FROM dpkg_log LIMIT 1\G
*************************** 1. row ***************************
id: 7280000849080746110
host: logstash-db848f65f-lnlf9
message: 2023-04-12 02:03:21 status unpacked libc-bin:amd64 2.31-0ubuntu9
path: /var/log/dpkg.log
@timestamp: 2023-06-16T09:23:57.405Z
@version: 1

NOTE: The integration with Filebeat requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Filebeat is a lightweight shipper for forwarding and centralizing log data. Once installed as an agent, it monitors the log files or locations you specify, collects log events, and forwards them for indexing, usually to Elasticsearch or Logstash.

Now, Manticore also supports the use of Filebeat as processing pipelines. This allows the collected and transformed data to be sent to Manticore just like to Elasticsearch. Currently, versions 7.17-9.2 are supported.

Configuration varies depending on which version of Filebeat you're using.

Important: Filebeat versions 7.17.0, 8.0.0, and 8.1.0 have a known issue with glibc 2.35+ (used in Ubuntu 22.04 and newer distributions). These versions may crash with "Fatal glibc error: rseq registration failed". To fix this, add the seccomp configuration as shown below.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/dpkg.log
  close_eof: true
  scan_frequency: 1s
output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true  # Required for 8.1
# Fix for glibc 2.35+ compatibility (Ubuntu 22.04+)
seccomp:
  default_action: allow
  syscalls:
    - action: allow
      names:
        - rseq
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"

References: Issue #30576, PR #30620

For versions 8.1 through 8.10, you need to add the allow_older_versions option:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/dpkg.log
  close_eof: true
  scan_frequency: 1s
output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"

From version 8.11, output compression is enabled by default, so you must explicitly set compression_level: 0 for compatibility with Manticore:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/dpkg.log
  close_eof: true
  scan_frequency: 1s
output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"

Filebeat 9.0 introduces a major architecture change, replacing the log input type with filestream. Starting from version 9.0, the default file identification method also changed to fingerprint, which requires files to be at least 1024 bytes (see issue #44780). For Manticore compatibility with files of any size, you must disable fingerprinting.

Here's the required configuration for Filebeat 9.0 and all later versions:

filebeat.inputs:
- type: filestream
  id: dpkg-log-input
  enabled: true
  paths:
    - /var/log/dpkg.log
  prospector.scanner.check_interval: 1s
  prospector.scanner.fingerprint.enabled: false
output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"

Important notes for Filebeat 9.0+:

• The type: filestream input replaces the older type: log
• The prospector.scanner.fingerprint.enabled: false setting is required to disable fingerprint-based file identification, ensuring reliable processing of files smaller than 1024 bytes
• The id field is required for filestream inputs and must be unique

Once you run Filebeat with this configuration, log data will be sent to Manticore and properly indexed. Here is the resulting schema of the table created by Manticore and an example of the inserted document:

mysql> DESCRIBE dpkg_log;
+------------------+--------+--------------------+
| Field            | Type   | Properties         |
+------------------+--------+--------------------+
| id               | bigint |                    |
| @timestamp       | text   | indexed stored     |
| message          | text   | indexed stored     |
| log              | json   |                    |
| input            | json   |                    |
| ecs              | json   |                    |
| host             | json   |                    |
| agent            | json   |                    |
+------------------+--------+--------------------+

mysql> SELECT * FROM dpkg_log LIMIT 1\G
*************************** 1. row ***************************
id: 7280000849080753116
@timestamp: 2023-06-16T09:27:38.792Z
message: 2023-04-12 02:06:08 status half-installed libhogweed5:amd64 3.5.1+really3.5.1-2
input: {"type":"filestream"}
ecs: {"version":"1.6.0"}
host: {"name":"logstash-db848f65f-lnlf9"}
agent: {"ephemeral_id":"587c2ebc-e7e2-4e27-b772-19c611115996","id":"2e3d985b-3610-4b8b-aa3b-2e45804edd2c","name":"logstash-db848f65f-lnlf9","type":"filebeat","version":"7.10.0","hostname":"logstash-db848f65f-lnlf9"}
log: {"offset":80,"file":{"path":"/var/log/dpkg.log"}}

NOTE: The integration with Fluent Bit requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Fluent Bit is an open source, cross-platform log processor that can aggregate data from many sources and ship it to multiple destinations. You can feed Fluent Bit outputs directly into Manticore to make the collected data searchable in real time.

The following walkthrough indexes Debian's dpkg.log with Fluent Bit and Manticore.

2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1
2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>
2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>

Create a configuration file such as fluent-bit.conf:

[SERVICE]
    flush        1
    daemon       On
    log_level    info
[INPUT]
    name tail
    path /var/log/dpkg.log
    inotify_watcher false
    read_from_head true
[OUTPUT]
    name es
    match *
    host 127.0.0.1
    port 9308
    index dpkg_log

• The [SERVICE] block starts Fluent Bit in daemon mode, which is convenient for Docker-based setups. Disable the daemon flag if you want to run it in the foreground.
• inotify_watcher is turned off to avoid file notification limitations inside containers.
• The output plugin (name es) can talk to Manticore's HTTP endpoint on port 9308.
• index defines the automatic table name Manticore creates when the first batch arrives.

Run Fluent Bit with this configuration and it will tail dpkg.log, then forward each line to Manticore.

Store the configuration as fluent-bit.conf, then launch Fluent Bit:

fluent-bit -c fluent-bit.conf

To run it in Docker, mount the log file (read-only) and the configuration:

docker run --rm -v /var/log/dpkg.log:/var/log/dpkg.log:ro \
  -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro \
  fluent/fluent-bit:latest -c /fluent-bit/etc/fluent-bit.conf

After ingestion begins, Manticore automatically creates the dpkg_log table. Here is the table definition and sample data:

mysql> DESCRIBE dpkg_log;
+-------------+--------+----------------+
| Field       | Type   | Properties     |
+-------------+--------+----------------+
| id          | bigint |                |
| @timestamp  | text   | indexed stored |
| log         | text   | indexed stored |
+-------------+--------+----------------+
mysql> SELECT * FROM dpkg_log LIMIT 3\G
*************************** 1. row ***************************
id: 7856533729353662465
@timestamp: 2023-08-04T15:09:21.191Z
log: 2023-06-05 14:03:04 startup archives install
*************************** 2. row ***************************
id: 7856533729353662466
@timestamp: 2023-08-04T15:09:21.191Z
log: 2023-06-05 14:03:04 install base-passwd:amd64 <none> 3.5.47
*************************** 3. row ***************************
id: 7856533729353662467
@timestamp: 2023-08-04T15:09:21.191Z
log: 2023-06-05 14:03:04 status half-installed base-passwd:amd64 3.5.47

With this lightweight pipeline, Fluent Bit handles log collection and delivery while Manticore indexes the events for fast search and analytics. The approach works equally well for other log sources—just add more inputs and reuse the same Elasticsearch-compatible output pointed at your Manticore cluster.

NOTE: The integration with Fluent Bit requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Vector by Datadog is an open source observability data pipeline that can collect, transform, and route logs or metrics. While Vector can aggregate the data by itself, pairing it with Manticore provides a purpose-built storage and search layer.

The example below shows how to forward Debian's dpkg.log through Vector.dev and index it in Manticore.

2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1
2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>
2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>

Create a vector.toml similar to the following:

[sources.test_file]
type = "file"
include = [ "/var/log/dpkg.log" ]
[transforms.modify_test_file]
type = "remap"
inputs = [ "test_file" ]
source = """
.vec_timestamp = del(.timestamp)"""
[sinks.manticore]
type = "elasticsearch"
inputs = [ "modify_test_file" ]
endpoints = ["http://127.0.0.1:9308"]
bulk.index = "dpkg_log"

• endpoints points to Manticore's HTTP interface (port 9308 by default). Adjust it if your instance listens elsewhere.
• The remap transform moves Vector's default timestamp field to vec_timestamp, because timestamp is reserved in Manticore.
• bulk.index defines the table that will be created automatically when Vector starts sending data.

Start Vector.dev with this configuration and it will tail the log file, transform each event, and forward it directly to Manticore.

Save the configuration as vector.toml, then start the agent:

vector --config vector.toml

If you run Vector.dev in Docker, mount both the config file and the log directory, for example:

docker run --rm -v /var/log/dpkg.log:/var/log/dpkg.log:ro \
  -v $(pwd)/vector.toml:/etc/vector/vector.toml:ro \
  timberio/vector:latest --config /etc/vector/vector.toml

When the pipeline runs, Manticore creates the dpkg_log table automatically. Its schema and example documents look like this:

mysql> DESCRIBE dpkg_log;
+-----------------+---------+--------------------+
| Field           | Type    | Properties         |
+-----------------+---------+--------------------+
| id              | bigint  |                    |
| file            | text    | indexed stored     |
| host            | text    | indexed stored     |
| message         | text    | indexed stored     |
| source_type     | text    | indexed stored     |
| vec_timestamp   | text    | indexed stored     |
+-----------------+---------+--------------------+
mysql> SELECT * FROM dpkg_log LIMIT 3\G
*************************** 1. row ***************************
id: 7856533729353672195
file: /var/log/dpkg.log
host: logstash-787f68f6f-nhdd2
message: 2023-06-05 14:03:04 startup archives install
source_type: file
vec_timestamp: 2023-08-04T15:32:50.203091741Z
*************************** 2. row ***************************
id: 7856533729353672196
file: /var/log/dpkg.log
host: logstash-787f68f6f-nhdd2
message: 2023-06-05 14:03:04 install base-passwd:amd64 <none> 3.5.47
source_type: file
vec_timestamp: 2023-08-04T15:32:50.203808861Z
*************************** 3. row ***************************
id: 7856533729353672197
file: /var/log/dpkg.log
host: logstash-787f68f6f-nhdd2
message: 2023-06-05 14:03:04 status half-installed base-passwd:amd64 3.5.47
source_type: file
vec_timestamp: 2023-08-04T15:32:50.203814031Z

Using Vector.dev with Manticore lets you collect logs from virtually any source, enrich or sanitize them in-flight, and store the results in a search-ready database. This workflow keeps observability pipelines simple while still enabling advanced transformations when needed.