Setting Up Loki and Promtail with Self-Signed Certificates for Secure Communication

This guide explains how to set up Loki and Promtail using HTTPS with self-signed certificates for secure communication. Follow the steps below for certificate generation, configuration, and integration.

1. Generate Self-Signed Certificates

Generate a Private Key:
```
openssl genrsa -out loki.key 2048
```

Generate a Certificate Signing Request (CSR):

openssl req -new -key loki.key -out loki.csr -subj "/C=US/ST=State/L=City/O=Organization/OU=Unit/CN=localhost"

Generate a Self-Signed Certificate:

openssl x509 -req -days 3650 -in loki.csr -signkey loki.key -out loki.crt

This certificate is valid for 10 years.

Place the Certificates in Secure Locations:
- Private Key: /etc/ssl/private/loki.key
- Certificate: /etc/ssl/certs/loki.crt

2. Configure Loki

Edit the Loki Configuration File: Update the Loki configuration (loki-config.yaml) with the following:

server:
  http_listen_port: 3100
  http_tls_config:
    cert_file: /etc/ssl/certs/loki.crt
    key_file: /etc/ssl/private/loki.key

Start Loki:

./loki-linux-amd64 --config.file=loki-config.yaml

3. Configure Promtail

Prepare the Certificate for Promtail: Copy the loki.crt to the Promtail server and place it in a directory, e.g., /etc/ssl/certs/loki.crt.

Edit the Promtail Configuration File: Update the Promtail configuration (promtail-config.yaml) with the following:

clients:
  - url: https://localhost:3100/loki/api/v1/push
    tls_config:
      ca_file: /etc/ssl/certs/loki.crt
      insecure_skip_verify: false
positions:
  filename: /tmp/positions.yaml
scrape_configs:
  - job_name: nginx-logs
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx
          __path__: /var/log/nginx/*.log

Start Promtail:

./promtail-linux-amd64 --config.file=promtail-config.yaml

4. Verify Secure Communication

Test Loki with HTTPS: Run the following curl command to verify Loki is serving HTTPS traffic:
```
curl -v --cacert /etc/ssl/certs/loki.crt https://localhost:3100/ready
```
Check Promtail Logs: Ensure Promtail is successfully sending logs to Loki. Look for logs like:
```
level=info msg="Batch sent successfully" tenant=""
```

5. Integrate Loki with Grafana

Access Grafana: Open Grafana in a browser (e.g., http://<grafana-ip>:3000).
Add Loki as a Data Source:
- Navigate to Configuration > Data Sources > Add Data Source.
- Select Loki.
- Configure the following:
  - URL: https://<loki-server>:3100
  - TLS Settings:
    - CA Certificate: Paste the contents of loki.crt.
    - Skip TLS Certificate Validation: Leave unchecked.
Save & Test: Ensure Grafana successfully connects to Loki.

6. Automate Certificate Renewal (Optional)

For production use, automate the certificate renewal process and update the certificates without downtime. Use tools like cron to schedule renewal and reload services.

7. Troubleshooting

Common Issues:

Certificate Verification Failed: Ensure the ca_file paths are correct in Promtail and Grafana.
Connection Refused: Verify Loki is running on HTTPS and accessible on the specified port.
Logs Not Pushing: Check Promtail logs for connectivity or configuration issues.

For more visit my blog: docs.ahmadraza.in (opens in a new tab)

Monitoring Logforwarder