Configuring Logstash

To paint a better picture, Below is a config file of Logstash, which describes the Input-Transform-Output pipeline:

#Extract
input {
  beats {
    port => 5044
  }
}
#TRANSFORM
filter {
    grok {
        break_on_match => false
        match => ["message","%{NUMBER:value},%{TIMESTAMP_ISO8601:date}"]
    }
}
#LOAD
 output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "sensor_data"
  }
}

The above refers:

Input

Not given enough credit is given to Filebeat, Part of the Beats application suite. Filebeat monitors files, be it log files, or any row column data file store. Filebeat monitors files, and stream across the changes sequentially into the output chosen in the Filebeat config, This will be shown in detail later on. So in this example, I am telling Logstash to listen on port 5044, which is where Filebeat is operating from while monitoring our log files.

Filter

In this section, the input data is transformed and manipulated as required. Very commonly, This filter block is used to parse data and separate them into explicit fields, for the format in which it needs to be stored/ sent across in the outputs.

Output

Populating the Output section details, determines the output of the pipeline. Due to the large number of output plugins available, the list is endless.

Now that Elasticsearch and Logstash is out of the way, I like to touch on the final element of the ELK Stack — Kibana.