Grafana + Loki + Promtail Setup with NGINX Proxy

This guide provides instructions to set up Grafana Loki for log aggregation, Promtail as a log forwarder on multiple servers, and configure NGINX to host Grafana, pushing logs from Promtail to Loki and visualizing them in Grafana.

Prerequisites

Ubuntu servers for Loki, Promtail, and Grafana.
Multiple servers (at least 5) to install Promtail and push logs to Loki.

Step 1: Install Grafana Loki

Create Loki directory:
```
sudo mkdir -p /etc/loki /var/lib/loki
```

Download and install Loki:

wget https://github.com/grafana/loki/releases/download/v2.8.1/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
sudo mv loki-linux-amd64 /usr/local/bin/loki
sudo chmod +x /usr/local/bin/loki

Create Loki configuration file:

sudo nano /etc/loki/loki-config.yml

Add the following content to loki-config.yml: Note:Below is the example configuration for 24 hours log retention

 auth_enabled: false
 
 server:
 http_listen_port: 3100
 grpc_listen_port: 9096
 log_level: debug
 grpc_server_max_concurrent_streams: 1000
 
 common:
 instance_addr: 127.0.0.1
 path_prefix: /tmp/loki
 storage:
     filesystem:
     chunks_directory: /tmp/loki/chunks
     rules_directory: /tmp/loki/rules
 replication_factor: 1
 ring:
     kvstore:
     store: inmemory
 
 ingester_rf1:
 enabled: false
 
 query_range:
 results_cache:
     cache:
     embedded_cache:
         enabled: true
         max_size_mb: 100
 
 schema_config:
 configs:
     - from: 2020-10-24
     store: tsdb
     object_store: filesystem
     schema: v13
     index:
         prefix: index_
         period: 24h
 
 pattern_ingester:
 enabled: true
 metric_aggregation:
     enabled: true
     loki_address: localhost:3100
 
 ruler:
 alertmanager_url: http://localhost:9093
 
 frontend:
 encoding: protobuf
 
 # By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
 # analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
 #
 # Statistics help us better understand how Loki is used, and they show us performance
 # levels for most users. This helps us prioritize features and documentation.
 # For more information on what's sent, look at
 # https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
 # Refer to the buildReport method to see what goes into a report.
 #
 # If you would like to disable reporting, uncomment the following lines:
 #analytics:
 #  reporting_enabled: false
 
 
 # Compactor configuration for retention
 compactor:
 working_directory: /opt/loki-logs-retention
 compaction_interval: 10m
 retention_enabled: true
 retention_delete_delay: 2h
 retention_delete_worker_count: 150
 delete_request_store: filesystem
 
 # Set global retention to 48 hours
 limits_config:
 retention_period: 25h

Create Loki systemd service:

sudo nano /etc/systemd/system/loki.service

Add the following content:

[Unit]
Description=Loki service
After=network.target
 
[Service]
ExecStart=/usr/local/bin/loki -config.file=/etc/loki/loki-config.yml
Restart=always
User=root
 
[Install]
WantedBy=multi-user.target

Start and enable Loki service:

sudo systemctl daemon-reload
sudo systemctl enable loki
sudo systemctl start loki

Verify Loki is running:
```
sudo journalctl -u loki -f
```

Step 2: Install and Configure Promtail on Multiple Servers

Add Grafana APT repository:

sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor > /etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

Install Promtail:

sudo apt-get update
sudo apt-get install promtail

Configure Promtail:

sudo nano /etc/promtail/config.yml

Example Promtail configuration to forward logs from NGINX, SSH, cron, and application logs:

clients:
  - url: http://<LOKI_SERVER_IP>:3100/loki/api/v1/push
 
scrape_configs:
  - job_name: nginx-logs
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx
          __path__: /var/log/nginx/*log
  - job_name: ssh-logs
    static_configs:
      - targets:
          - localhost
        labels:
          job: ssh
          __path__: /var/log/auth.log
  - job_name: cron-logs
    static_configs:
      - targets:
          - localhost
        labels:
          job: cron
          __path__: /var/log/syslog
  - job_name: app-logs
    static_configs:
      - targets:
          - localhost
        labels:
          job: application
          __path__: /var/log/myapp/*.log

Allow Promtail access to log files:

sudo usermod -aG adm promtail
sudo chmod o+r /var/log/nginx/

Create Promtail systemd service:

sudo nano /etc/systemd/system/promtail.service

Add the following content:

[Unit]
Description=Promtail service
After=network.target
 
[Service]
ExecStart=/usr/bin/promtail -config.file=/etc/promtail/config.yml
Restart=always
User=promtail
 
[Install]
WantedBy=multi-user.target

Start and enable Promtail service:

sudo systemctl daemon-reload
sudo systemctl enable promtail
sudo systemctl start promtail

Verify Promtail logs:
```
journalctl -u promtail -f
```

Repeat the Promtail installation on each server (at least 5 servers).

Step 3: Install Grafana and Configure NGINX

Install Grafana:
```
sudo apt-get install grafana
```

Start and enable Grafana:

sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Set up NGINX reverse proxy for Grafana: Install NGINX if it isn’t installed:
```
sudo apt-get install nginx
```

Configure NGINX for Grafana:

sudo nano /etc/nginx/sites-available/grafana.conf

Add the following configuration:

server {
    listen 80;
    server_name grafana.example.com;
 
    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Enable and restart NGINX:

sudo ln -s /etc/nginx/sites-available/grafana.conf /etc/nginx/sites-enabled/
sudo systemctl restart nginx

Step 4: Configure Grafana to Use Loki as a Data Source

Access Grafana: Open a browser and navigate to http://grafana.example.com.
Add Loki Data Source:
- Go to Configuration > Data Sources > Add data source.
- Select Loki and enter the following URL: http://localhost:3100.
- Click Save & Test to verify the connection.

Step 5: View Logs in Grafana

After setting up Promtail to push logs to Loki and configuring Grafana to use Loki, you can now visualize logs from your servers in Grafana.

Create a new dashboard in Grafana.
Add Loki-based panels to query and visualize logs from your NGINX, SSH, cron, and application logs.

Troubleshooting

Verify Promtail logs:
```
journalctl -u promtail -f
```
Check Loki service logs:
```
journalctl -u loki -f
```
Check NGINX configuration syntax:
```
sudo nginx -t
```

This completes the setup of Grafana Loki, Promtail on multiple servers, and Grafana with NGINX reverse proxy.

Labels Monitoring