- Published on
Exporting Kubernetes Logs to BetterStack using FluentBit
- Authors
Following on from my previous post on Exporting Kubernetes Logs to Linode S3 using FluentBit, we found that the logs were being exported to S3, but we had no way to query them or set up alerts.
To keep costs down we wanted to minimise the cluster overhead of managing a full logging stack such as Grafana Loki. As we already had BetterStack set up for uptime alerting, we explored the possibility of extending this to logs and discovered it has native support for FluentBit. The decision was made to continue to use Linode S3 for long-term storage of logs, but we will now forward error logs to BetterStack for real-time monitoring and alerting.
Overview
The architecture is shown below. Kubernetes by default rotates logs based on file size. Our aim is to export these logs now to BetterStack. BetterStack's free tier allows us to store logs for 3 days, up to 3GB in size. Because of this, we decided to only forward error logs to BetterStack.
Kubernetes Logging
Kubernetes default behaviour is defined here. Kubernetes logs and application logs are stored via the underlying container runtime on each Node. The kubelet is responsible for rotating container logs and managing the logging directory structure. The kubelet sends this information to the container runtime (using CRI), and the runtime writes the container logs to the given location.
These logs are rotated based on two environment variables:
- containerLogMaxSize (default 10Mi)
- containerLogMaxFiles (default 5)
These settings let you configure the maximum size for each log file and the maximum number of files allowed for each container respectively. Crucially, using the command kubectl logs will only ever return the latest file. The others would need to be accessed directly.
Our requirement is to extend this functionality to make these logs available in BetterStack for error reporting, and for 30 days in S3 for archive purposes.
FluentBit
FluentBit is a logging and metrics processor and forwarder. It can connect to multiple sources, allows you to enrich and filter incoming data and forward to a range of destinations. It's focus is on performance and efficiency which suits situations where resources are limited.
Comparison to FluentD
My understanding is FluentBit is the next-generation version of FluentD, heavily optimised. FluentD supports a wider range of connectors. A full comparison is provided here.
Tail Plugin
The tail plugin allows you to monitor one or several text files. We define an input block to leverage this plugin and point to log files generated by Kubernetes or Applications:
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Filter Plugins
The kubernetes plugin allows you to enrich logs with Kubernetes metadata. This is useful for identifying which pod, namespace, or container the logs originated from.
[INPUT]
Name kubernetes
Tag kube.*
Merge_Log On
Kube_Tag_Prefix kube.var.log.containers.
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Filtering for Errors:
To filter logs for errors and exclude static file requests, we can use the grep plugin.
[FILTER]
Name grep
Match kube.*
Regex log (?i)(error|failed|exception)
[FILTER]
Name grep
Match container.*
Exclude log (?i)\.(png|jpg|jpeg|gif|svg|ico|css|js|woff|woff2|ttf|eot|otf|php)\b
[FILTER]
Name grep
Match container.*
Exclude code ^(2|3)\d\d$
HTTP Output Plugin
The http plugin allows you to forward logs to an HTTP endpoint. We will use this to forward logs to the BetterStack ingestion API.
[OUTPUT]
name http
match *
tls On
host xxxx
port 443
uri /fluentbit
header Authorization Bearer xxxx
header Content-Type application/msgpack
format msgpack
retry_limit 5
Deployment
Kubernetes logs for an application are stored on it's underlying node. This means we need to deploy FluentBit as a DaemonSet, ensuring an instance of FluentBit runs once on each node.
Here is an example deployment manifest:
aapiVersion: apps/v1
kind: DaemonSet
metadata:
name: better-stack-logger
namespace: logging
labels:
k8s-app: better-stack-logger
spec:
selector:
matchLabels:
name: better-stack-logger
template:
metadata:
labels:
name: better-stack-logger
spec:
serviceAccountName: fluent-bit
containers:
- name: fluent-bit
image: fluent/fluent-bit:3.0
imagePullPolicy: Always
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlogpods
mountPath: /var/log/pods
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
- name: config
mountPath: /fluent-bit/etc/
terminationGracePeriodSeconds: 10
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlogpods
hostPath:
path: /var/log/pods
type: ""
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
type: ""
- name: config
configMap:
name: better-stack-fluent-bit-config
tolerations:
- effect: NoSchedule
operator: Exists
BetterStack
BetterStack is a monitoring and alerting platform that provides a free tier for log management. It allows you to collect, search, and analyze logs in real-time, making it suitable for monitoring applications and infrastructure. BetterStack provides a simple HTTP endpoint to which you can send logs. It supports various formats, including JSON and MsgPack.
Betterstack supports FluentBit as a data source, allowing you to forward logs directly from FluentBit to BetterStack. Their documentation provides a guide on how to set up FluentBit as a source.
