AWS Log Forwarding to Azure Loki via Nginx Proxy

Overview

This guide explains how to use an Nginx-based log forwarder in AWS to centralize log forwarding to Loki hosted on an Azure VM. This method eliminates the need to manually update security groups when AWS auto-scaling occurs.

Step 1: Deploy a Log Forwarder Instance in AWS

Launch an EC2 instance to act as the log forwarder.

1.1. Launch an EC2 Instance

• Use Amazon Linux 2 or Ubuntu.
• Assign a static Elastic IP.
• Security group rules:
  • Allow incoming logs from all AWS instances.
  • Allow outgoing logs to Azure Loki.

Step 2: Install and Configure Nginx as a Log Forwarder

2.1. Install Nginx

sudo apt update && sudo apt install -y nginx

2.2. Configure Nginx as a Reverse Proxy

Edit the Nginx config file:

sudo nano /etc/nginx/nginx.conf

Add the following configuration:

http {
    server {
        listen 3100;
        location / {
            proxy_pass http://<LOKI_AZURE_VM_IP>:3100;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

Save and restart Nginx:

sudo systemctl restart nginx

Step 3: Configure Promtail on AWS Instances

Modify Promtail config on each AWS instance:

server:
  http_listen_port: 9080
  grpc_listen_port: 0
 
clients:
  - url: http://<FORWARDER_IP>:3100/loki/api/v1/push
 
positions:
  filename: /tmp/positions.yaml
 
scrape_configs:
  - job_name: "system"
    static_configs:
      - targets:
          - localhost
        labels:
          job: "varlogs"
          host: "my-aws-server"
          __path__: /var/log/*.log

Restart Promtail:

sudo systemctl restart promtail

Step 4: Update Azure Security Group

• Allow only the Elastic IP of the AWS proxy in the Azure security group.
• Open port 3100 for incoming logs.

Step 5: Testing the Setup

5.1. Check Nginx Log Forwarding

sudo tail -f /var/log/nginx/access.log

5.2. Check Loki Log Reception in Azure

curl -X GET "http://<LOKI_AZURE_VM_IP>:3100/loki/api/v1/query?query={job='varlogs'}"

Step 6: Automate Deployment (Optional)

To automate this setup:

• Terraform: Deploy EC2 instance and configure security groups.
• Ansible: Install Nginx and set up configurations.
• AWS Auto Scaling User Data: Update Promtail configurations dynamically.

Benefits of This Setup

✅ No manual updates needed for security groups. ✅ Scalable: New AWS instances automatically send logs via the proxy. ✅ Secure: Restricts direct access to Loki, reducing exposure to attacks.