Apache Fluent Bit Setup on EKS with OpenSearch and S3 Output

Namespace and Secrets

Create the namespace if it doesn't exist:

kubectl create namespace logging

Create the secret with your OpenSearch credentials:

kubectl create secret generic opensearch-credentials \
  --from-literal=user=admin \
  --from-literal=password='Admin@123' \
  --namespace logging

Add Helm Repository and Install Fluent Bit

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update
 
helm upgrade --install fluent-bit fluent/fluent-bit \
    --namespace logging \
    -f fluent-bit-values.yaml

RBAC Configuration (`fluent-bit-rbac.yaml`)

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  - pods/logs
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: logging

Apply the RBAC configuration:

kubectl apply -f fluent-bit-rbac.yaml
kubectl rollout restart daemonset fluent-bit -n logging

Fluent Bit Values (`fluent-bit-values.yaml`)

# Set log level back to info, we don't need debug anymore
logLevel: info
 
serviceAccount:
  create: true
  name: fluent-bit
 
env:
  - name: OPENSEARCH_USER
    valueFrom:
      secretKeyRef:
        name: opensearch-credentials
        key: user
  - name: OPENSEARCH_PASSWD
    valueFrom:
      secretKeyRef:
        name: opensearch-credentials
        key: password
 
config:
  # ======
  # DEFINE A PARSER for JSON log entries
  # ======
  parsers: |
    [PARSER]
        Name          json
        Format        json
        Time_Key      time
        Time_Format   %Y-%m-%dT%H:%M:%S.%L
 
  # ======
  # INPUT: Tail all container logs
  # ======
  inputs: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Parser            docker
        Refresh_Interval  10
 
  # =======
  # FILTERS: Process the logs before sending
  # =======
  filters: |
    # This first filter adds Kubernetes metadata like pod_name, namespace, etc.
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Merge_Log           On
        Merge_Log_Key       log_processed
 
    # This filter excludes Fluent Bit's own logs from the pipeline
    [FILTER]
        Name                grep
        Match               kube.*
        Exclude             kubernetes['pod_name'] fluent-bit.*
 
    # This filter parses the main log field if it's a JSON string (if logs are not in json format then it will not send to opensearch) so better to comment it 
  #  [FILTER]
  #      Name                parser
  #      Match               kube.*
  #      Key_Name            log
  #      Parser              json
  #      Reserve_Data        On
 
  # =======
  # OUTPUT: Send to OpenSearch
  # =======
  outputs: |
    [OUTPUT]
        Name                opensearch
        Match               *
        Host                search-opensearchlab-dmfapfqlmuif3frbyf6talxbqm.aos.eu-north-1.on.aws
        Port                443
        TLS                 On
        Suppress_Type_Name  On
        HTTP_User           ${OPENSEARCH_USER}
        HTTP_Passwd         ${OPENSEARCH_PASSWD}
        Retry_Limit         5
        Logstash_Format     On
        Logstash_Prefix     eks-test-logs
 
    [OUTPUT]
        Name                s3
        Match               *
        bucket              eks-logs-archive-production-497836541334
        region              ap-south-1
        total_file_size     50M
        upload_timeout      5m
        compression         gzip
        s3_key_format       /prod-logs/%Y/%m/%d/%H/logs-$UUID.json.gz
        # ADDED FOR RELIABILITY: Buffer logs to disk before uploading
        store_dir           /var/log/flb-s3-buffers

Deploy Fluent Bit with the Values File

helm upgrade --install fluent-bit fluent/fluent-bit \
    --namespace logging \
    -f fluent-bit-values.yaml

Cleanup OpenSearch Index (Optional)

DELETE /eks-test-logs-*

Fluent Bit Logs

kubectl logs -n logging -l app.kubernetes.io/instance=fluent-bit --all-containers=true -f

Note: If the log format changes, you must delete the old index from OpenSearch.

S3 Output Specific Notes

Create IAM Policy (`fluent-bit-s3-policy.json`)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::<YOUR_BUCKET_NAME>/*"
        }
    ]
}

Create the IAM policy:

aws iam create-policy \
    --policy-name fluent-bit-s3-policy \
    --policy-document file://fluent-bit-s3-policy.json

Create IAM service account for EKS:

eksctl create iamserviceaccount \
    --name fluent-bit \
    --namespace logging \
    --cluster <your-cluster-name> \
    --attach-policy-arn <your-s3-policy-arn> \
    --approve \
    --override-existing-serviceaccounts --region=ap-south-1
 
eksctl create iamserviceaccount \
  --name fluent-bit \
  --namespace logging \
  --cluster tracemypods-auto \
  --attach-policy-arn arn:aws:iam::497836541334:policy/fluent-bit-s3-policy \
  --approve \
  --override-existing-serviceaccounts --region=ap-south-1

Upgrade Fluent Bit Values with S3 Output

# --- Output 2: S3 ---
[OUTPUT]
    Name                s3
    Match               *
    bucket              <YOUR_BUCKET_NAME>
    region              <YOUR_BUCKET_REGION>
    total_file_size     50M
    upload_timeout      5m
    compression         gzip
    s3_key_format       /prod-logs/namespace=${kubernetes['namespace_name']}/%Y/%m/%d/%H/logs-$UUID.json.gz
    # ADDED FOR RELIABILITY: Buffer logs to disk before uploading
    store_dir           /var/log/flb-s3-buffers
 
[OUTPUT]
    Name                s3
    Match               *
    bucket              eks-logs-archive-production-497836541334
    region              ap-south-1
    total_file_size     50M
    upload_timeout      5m
    compression         gzip
    s3_key_format       /prod-logs/%Y/%m/%d/%H/logs-$UUID.json.gz
    # ADDED FOR RELIABILITY: Buffer logs to disk before uploading
    store_dir           /var/log/flb-s3-buffers

Deploy after updating values:

helm upgrade --install fluent-bit fluent/fluent-bit \
    --namespace logging \
    -f fluent-bit-values.yaml

Athena Grafana.json Eks Glue