Skip to main content

​
Basic Server Setup Guide

Purpose: Prepare a Linux server (Amazon Linux or Ubuntu) with hostname, system packages, timezone, utilities, database clients, AWS CLI, and logging setup for applications.

​
1. Set Hostname

Amazon Linux / Ubuntu: 
sudo hostnamectl set-hostname <your-hostname>

​
2. Update System Packages

Amazon Linux: 
sudo yum update -y
Ubuntu: 
sudo apt update && sudo apt upgrade -y

​
3. Set Timezone to IST

Amazon Linux / Ubuntu: 
sudo timedatectl set-timezone Asia/Kolkata

​
4. Install Basic Utilities

Amazon Linux: 
sudo yum install -y wget curl git unzip zip tar net-tools bind-utils htop tree vim
Ubuntu: 
sudo apt install -y wget curl git unzip zip tar net-tools dnsutils htop tree vim

​
5. Enable SSH Key Authentication (for Jenkins or root access)

mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "<your-public-key>" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

​
6. Install Database Clients

Amazon Linux: 
# MySQL client
sudo yum install -y mariadb

# PostgreSQL client
sudo amazon-linux-extras enable postgresql14
sudo yum install -y postgresql

# Verify
mysql --version
psql --version
Ubuntu: 
# MySQL client
sudo apt install -y mysql-client

# PostgreSQL client
sudo apt install -y postgresql-client

# Verify
mysql --version
psql --version
Amazon Linux / Ubuntu: 
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version

​
8. Set Up Systemd Service for Applications

Purpose: Run Python or Go apps as systemd services with persistent logging and automatic startup.

​
8.1 Create Service Unit File

Example: Python (FastAPI with uvicorn) 
[Unit]
Description=<app-name> FastAPI Application
After=network.target

[Service]
User=root
Group=root
WorkingDirectory=/home/<app-name>
ExecStart=/home/<app-name>/venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8000
Restart=always
RestartSec=5
StandardOutput=append:/var/log/<app-name>/app.log
StandardError=append:/var/log/<app-name>/app.log

[Install]
WantedBy=multi-user.target
Example: Go Application 
[Unit]
Description=<app-name> Go Application
After=network.target

[Service]
User=root
Group=root
WorkingDirectory=/home/<app-name>
ExecStart=/home/<app-name>/<app-name>-service
Restart=always
RestartSec=5
StandardOutput=append:/var/log/<app-name>/app.log
StandardError=append:/var/log/<app-name>/app.log

[Install]
WantedBy=multi-user.target

​
8.2 Register & Start Service

sudo systemctl daemon-reload
sudo systemctl enable <app-name>
sudo systemctl start <app-name>
sudo systemctl status <app-name>

​
8.3 Set Up Logging Directory

sudo mkdir -p /var/log/<app-name>
sudo chown root:root /var/log/<app-name>

​
8.4 Configure Log Rotation

Create /etc/logrotate.d/<app-name>: 
/var/log/<app-name>/app.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
    dateext
    dateformat -%Y-%m-%d
}
Note on copytruncate: Ensures Fluent Bit (or any tailing agent) continues reading logs without interruption.

​
8.5 Force Log Rotation (Testing)

sudo logrotate -f /etc/logrotate.d/<app-name>
ls -lh /var/log/<app-name>/
  • app.log → fresh, current log
  • app.log-YYYY-MM-DD.gz → rotated & compressed

​
8.6 View Logs

tail -f /var/log/<app-name>/app.log
✅ Logs are written to /var/log/<app-name>/app.log, rotated daily, and services start automatically on reboot.

​
9. Set Up Fluent Bit for Log Collection (to Kibana / S3)

Install Fluent Bit: 
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
Configure /etc/fluent-bit/fluent-bit.conf (example): 
[SERVICE]
    Flush        60
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf
    storage.path /var/lib/fluent-bit/storage
    storage.sync normal
    storage.backlog.mem_limit 128M

[INPUT]
    Name         systemd
    Tag          <app-name>
    Systemd_Filter
    Read_From_Tail On
    DB           /var/lib/fluent-bit/systemd-<app-name>.db
    _SYSTEMD_UNIT=<app-name>.service

[FILTER]
    Name   modify
    Match  <app-name>
    Rename MESSAGE log
    Remove_regex ^(?!log$).*

[OUTPUT]  # S3
    Name        s3
    Match       <app-name>
    bucket      example-co-in-aws-application-logs
    region      ap-south-1
    total_file_size 50M
    upload_timeout 1m
    store_dir   /var/lib/fluent-bit/s3
    s3_key_format /application-logs/<app-name>-prod/$TAG/%Y/%m/%d/%H/<app-name>-$UUID.json
    upload_chunk_size 5M
    use_put_object On
    content_type application/json

[OUTPUT]  # Logstash
    Name   forward
    Match  <app-name>
    Host   <logstash-hostname-or-ip>
    Port   5044
    # tls settings if needed
    # tls On
    # tls.verify Off
    # tls.ca_file /etc/ssl/certs/ca-bundle.crt

​
10. Set Up Logstash to Receive Logs

Example /etc/logstash/conf.d/<app-name>.conf: 
input {
    tcp {
        port 5001 # Use available port
        codec json
        tags ["app-name"]
    }
}

output {
    if "app-name" in [tags] {
        elasticsearch {
            hosts => ["https://localhost:9200"]
            user => "elastic"
            password => "your-password"
            ssl_certificate_verification => false
            index => "app-name-%{+YYYY.MM.dd}"
        }
    }
}
Important Notes:
  • Logstash domain usually works internally in VPC (logstash.internal.example.co.in).
  • If Kibana is in another VPC, allow ports (e.g., 5003) via VPC peering.
✅ Outcome:
  • Server is fully configured with hostname, packages, timezone, utilities, database clients, AWS CLI.
  • Applications run as systemd services with persistent logging.
  • Fluent Bit collects logs and forwards them to S3 and Logstash/Kibana reliably.