Filebeat and Elasticsearch - Adding custom fields so ingested logs are more easily searchable
This will walk through adding custom fields in filebeat.yml for one log file type in order to make those logs ingested by Elasticsearch more easily searchable.
- Elasticsearch: the search and analytics engine at the heart of the stack - store, search and analyse data, in our case logs
- Kibana: the visualisation layer and user interface for the Elastic Stack - visualise, navigate and share data, in our case logs
- Filebeat: single purpose data shippers, in our case used to send logs from the application servers to Elasticsearch
The log files that we will be ingesting
We will be ingesting the following log files from my laptop in order to simplify this blog post.
The steps are applicable for logs from a test or production environment for one particular version of your app.
The custom fields that will be added
The fields added will indicate
- The version of the app eg version 2.1.0
- What app the logs originated from, in this case “fromAnitaLaptop” Once these fields are added to the index they can be used to search and aggregate data based on these properties.
This simplifies searching for logs and creating charts aggregated by a particular version of the app on a particular server, for example if you wish to view logs “fromAnitaLaptop” running version 2.1.0
Step-by-step simple proof of concept example of adding one field to filebeat.yml
The example uses generic logs generated by my laptop
- Pre-condition: Filebeat is installed on my laptop
- Edit filebeat.yml to add the custom field for the log file
- Save the file and restart Filebeat if it was already running
- Verify the new field is showing as expected in Kibana > Discover
- Refresh the index pattern so the new field is picked up
- Verify the new field is easily searchable in Kibana > Discover
Edit filebeat.yml to add the custom field for the log file
In this example, the field app.log.origin with the value fromAnitaLaptop will be added to every indexed document in Elasticsearch coming from /var/log/.log*
# filebeat.yml --- SNIP --- # Add the following fields and fields_under_root underneath the path - type: log enabled: true paths: - /var/log/*.log fields: app.log.origin: fromAnitaLaptop fields_under_root: true
The reason I used this field name was because I am sure that there are no existing fields starting with the root name “app”.
This makes it easy for me to know which fields have been custom added, and to know that the name I chose is not overwriting an existing field and its value.
In order to simplify this exercise, it assumes there is only one log type with one path name in the filebeat.yml file.
In other words, there should be only one input type with one path in the entire filebeat.yml file.
.yml files are extremely picky with indentation
Be sure to use spaces for indentation to avoid errors resulting in Filebeat being unable to restart
Tip: Test your config file first
filebeat test config -c filebeat.yml
Save the file and restart Filebeat if it was already running
Verify the new field is showing as expected in Kibana > Discover
Notice there is a warning there is no cached mapping for this field
Refresh the index pattern by navigating to Management: Stack Management > Kibana: Index Patterns > select the index pattern, Refresh and Confirm
so the new field has a cached mapping
It should now be easy to search on logs with this field.
To do so, ensure there are some new logs entries that have been generated since updating and restarting Filebeat, then Refresh Kibana > Discover to view the latest logs.
The new field should now be easily searchable in Kibana > Discover
One way is to type in the new field name in “Search field names” under the index pattern name
An option will appear to Add the field
Add the field, which will show this field as a column in the logs view area.
Another way is to type the field name in the Discover search bar at the top…
… and select “equals” and the value should show as a suggestion.
Add a searchable version field
Add the following fields if the version number is explicitly known for a log file.
Note: app.version.patch would have been more correct as per semver.org
These custom fields make it simple to search and aggregate data for
- an explicit full version number, eg 2.1.0
- a combination of major and minor, eg major version 2 and minor version 1
- a combination of logs “fromAnitaLaptop” with a particular version combination because we are adding the version fields in addition to the existing custom field app.log.origin
# filebeat.yml --- SNIP --- # Add the following fields and fields_under_root underneath the path - type: log enabled: true paths: - /var/log/*.log fields: app.log.origin: fromAnitaLaptop app.version.display_name: "2.1.0" app.version.major: 2 app.version.minor: 1 app.version.bugfix: 0 fields_under_root: true
The newly added version fields are shown
Repeat the steps of restarting Filebeat and refreshing the Index Pattern to remove the warnings
Note that it is now easy to search upon specific combinations of versions, as well as less and greater than specific versions
It is also easy to include the app.log.origin value
Example: It is easy to filter only logs coming from major version 2 and minor version 1, where the bugfix version does not matter, and origin fromAnitaLaptop
The custom fields added to the index in Elasticsearch contain useful meta information that give the logs context, and thus the logs can be more easily searchable in that context.
When logs are more easily searchable within a context, visual charts and graphs can easily be made for these different contexts, bringing visibility into your application, enabling stakeholders to understand and act upon the application behaviour.