When running multiple applications in a Kubernetes cluster, a centralized, cluster-level logging stack can help to quickly sort and analyze the data logs generated by the pods. A popular solution is the Elasticsearch, Fluentbit, and Kibana stack (EFK).
Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. It is used for indexing and searching through large volumes of log data.
Kibana is a proprietary data visualization frontend and dashboard software for Elasticsearch. Kibana helps in exploring the log data through a web interface.
Fluent Bit is a lightweight log processor and forwarder that allows you to collect data and logs from different sources, unify them, and send them to multiple destinations.
Make sure the Ingress Controller and Log Aggregation are deselected in the default blueprint and that the update default blueprint is applied to the cluster. This removes the FluentD setup, which will be replaced by Fluent Bit.
For any customization of Fluent Bit to be deployed, a custom values YAML file is needed. The YAML file allows managing the index name and which namespace logs to view.
In the values.yaml file, the logs are fetched from two namespaces, kong and kube-system, and from two different indexes. Change the Input section as needed. Add or remove namespaces as needed.
Note
Adding or removing from the Input section requires making changes to the Filter and Output sections accordingly.
In the Output section, the Logstash_Prefix is used to create the index name. The value used will display in Kibana.
values.yaml
config:
service: |
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
Health_Check On
## https://docs.fluentbit.io/manual/pipeline/inputs
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*_kong_*.log
multiline.parser docker, cri
Tag kong.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[INPUT]
Name tail
Path /var/log/containers/*_kube-system_*.log
multiline.parser docker, cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
## https://docs.fluentbit.io/manual/pipeline/filters
filters: |
[FILTER]
Name kubernetes
Match kong.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name Kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
## https://docs.fluentbit.io/manual/pipeline/outputs
outputs: |
[OUTPUT]
Name es
Match kong.*
Host elasticsearch-master
Port 9243
HTTP_User admin
HTTP_Passwd Admin@123
Logstash_Format On
Logstash_Prefix kong
Retry_Limit False
[OUTPUT]
Name es
Match kube.*
Host elasticsearch-master
Port 9243
HTTP_User admin
HTTP_Passwd Admin@123
Logstash_Format On
Logstash_Prefix kong
Retry_Limit False
## https://docs.fluentbit.io/manual/pipeline/parsers
customParsers: |
[PARSER]
Name docker_no_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
There may be times where it is easier to exclude some namespaces when gathering the logs for all of the other namespaces. In this scenario, add the Exclude_path option in the Fluent Bit configuration file.
Exclude multiple namespaces, separating the namespace names by a comma. In the example below, the rafay-infra and rafay-system namespaces are excluded.
[INPUT]
Name tail
Path /var/log/containers/*.log
Exclude_Path /var/log/containers/*_rafay-infra_*.log,/var/log/containers/*_rafay-system_*.log
multiline.parser docker, cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On