​

EKS Log Management (EKS Log MGMT)

​

1. Architecture Overview

   +--------------------+
   |   EKS Cluster      |
   |  Nodes & Pods      |
   +--------------------+
             |
             | Fluent Bit (DaemonSet)
             v
   +--------------------+
   |   Log Processing   |
   |   & Forwarding     |
   +--------------------+
        |           |
        |           |
        v           v
+---------------+   +-----------------+
| OpenSearch    |   | S3 Bucket       |
| (Real-time    |   | (Long-term      |
| querying &    |   | retention &     |
| dashboards)   |   | analytics)      |
+---------------+   +-----------------+
                         |
                         v
                  +-----------------+
                  | AWS Glue /      |
                  | Athena          |
                  | (Log Analysis)  |
                  +-----------------+

​

2. Log Collection with Fluent Bit

• Deployment: Fluent Bit is deployed as a DaemonSet on all EKS nodes.
• Sources:
  • Container logs: /var/log/containers/*.log
  • Node system logs: /var/log/messages, /var/log/syslog
• Processing:
  • Parse Kubernetes metadata (namespace, pod name, container)
  • Apply filters (JSON formatting, timestamp correction)
• Destinations:
  • OpenSearch for real-time dashboards & alerts
  • S3 for long-term retention

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            docker
    Tag               kube.*

[FILTER]
    Name              kubernetes
    Match             kube.*
    Kube_Tag_Prefix   kube.var.log.containers.

[OUTPUT]
    Name  es
    Match *
    Host  <OpenSearchDomainEndpoint>
    Port  443
    TLS   On

[OUTPUT]
    Name  s3
    Match *
    Bucket  eks-logs-retention
    Region  ap-south-1
    Total_File_Size 5M
    Upload_Timeout  10m

​

3. OpenSearch – Real-Time Log Analysis

• Purpose: Immediate insights and monitoring dashboards.
• Setup:
  • Create an OpenSearch cluster (multi-AZ recommended for production)
  • Configure index policies and IAM roles for Fluent Bit access
• Use Cases:
  • Monitor application errors and latency
  • Alerting with OpenSearch/Kibana dashboards
  • Query by pod, namespace, or node

​

4. S3 – Long-Term Retention

• Purpose: Store logs for compliance, audits, and historic analysis
• Structure: Organize by year/month/day/hour or by cluster/namespace
• Example Key Structure:

s3://eks-logs-retention/cluster-name/namespace/pod-name/YYYY/MM/DD/HH/logfile.json

• Retention: Lifecycle policies can archive old logs to Glacier or delete after N days

​

5. AWS Athena & Glue – Log Analytics

• AWS Glue: Crawls the S3 bucket to create a metadata catalog
  • Defines table schema (columns: timestamp, pod, namespace, message, log level)
• AWS Athena: SQL-like queries on logs in S3 without moving data
• Benefits:
  • Historical trend analysis
  • Identify anomalies over long periods
  • Cost-efficient analytics using S3 storage

SELECT pod_name, COUNT(*) AS error_count
FROM eks_logs
WHERE log_level='ERROR' AND year='2025' AND month='10'
GROUP BY pod_name
ORDER BY error_count DESC;

​

6. Data Flow Summary

1. EKS Nodes/Pods generate logs
2. Fluent Bit collects and processes logs
3. Logs forwarded:
   • OpenSearch: real-time queries and dashboards
   • S3: long-term retention (organized by cluster/namespace/date)
4. AWS Glue catalogs S3 logs
5. AWS Athena queries S3 logs for historical analytics

• Use IAM roles for Fluent Bit to access OpenSearch and S3
• Enable S3 encryption (SSE-S3 or SSE-KMS)
• Configure OpenSearch retention policies to prevent storage bloat
• Consider Fluent Bit buffering to handle high log volumes