Simplified Log Aggregation on AWS EKS with Flux, ELK Stack, and Fluentd

In today’s world many of us are working on Microservices, ETL Data Pipelines and distributed Infrastructure Environments. Which basically means that efficient log management and analysis are critical for maintaining system reliability. In this article, we’ll explore how to leverage the ELK Stack, along with the Fluentd log collector, to simplify log aggregation and analysis on our AWS EKS cluster using the FluxCD.

To streamline log aggregation and analysis in our AWS Kubernetes cluster, we’ll be harnessing the power of the ELK Stack, Filebeat, and Fluentd. Let’s take a closer look at each component:

• ELK Stack: Which comprises of Elasticsearch, Logstash, and Kibana. It’s going to provide us with scalable search capabilities, as well as processing and enriching the logs, and last but not least a UI for log visualization and analysis.
• Filebeat: is basically a lightweight log shipper that collects, filters, and sends logs to our ELK Stack, capturing and forwarding all of our logs.
• Fluentd: in this case Fluentd will act as our log collector, gathering logs from various sources and forwarding them to our ELK Stack.

Elastic Cloud on Kubernetes

The Elastic Operator will ensures that all of the core components listed above are configured correctly across our AWS EKS cluster. Not to mention, it seamlessly integrates with FluxCD & will allow us to efficiently visualize and analyze our log data. We’re going to be using the eck-operator Helm Chart, so let’s start by creating a directory structure that looks like this:

├── operators
│   └── base
│       ├── elastic-operator
│       │   ├── kustomization.yaml
│       │   ├── release.yaml
│       │   └── namespace.yaml

We’re going to create a separate namespace for the Elastic Operator, optimizing the organization and management of our logging infrastructure. By isolating it in its dedicated namespace, we gain control and visibility over the associated resources, enhancing overall operational efficiency.

apiVersion: v1
kind: Namespace
metadata:
  name: elastic-operator

Ok, now the release.yaml is where we will declare

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: elastic-operator
  namespace: elastic-operator
spec:
  interval: 5m
  chart:
    spec:
      chart: eck-operator
      version: '2.8.0'
      sourceRef:
        kind: HelmRepository
        name: elastic
        namespace: flux-system
      interval: 1m