Production Linux Hardening: Defense-in-Depth with CrowdSec, Private Networks, and Rootless Containers

Many engineering teams assume cloud security simply boils down to configuring an AWS security group or relying on their VPS provider’s default edge firewall.

The reality on the public internet is relentless: the minute a server binds to a public IPv4 address, automated port scanners, SSH brute-force bots, and web exploit probes begin within minutes. If management ports listen openly or containers run with default root privileges, your infrastructure sits one vulnerability away from total host compromise.

This guide outlines the foundational defense-in-depth principles we apply across every production dedicated server and virtual machine we operate.

Principle 1: Zero Management Ports Exposed to the Public Web

The most frequent security antipattern is leaving port 22 (SSH) or administrative dashboards (Portainer, database interfaces, telemetry) listening on 0.0.0.0.

The baseline rule is simple: no internal management service should expose a public network socket.

To achieve this without degrading operational ergonomics:

1. Private overlay mesh with Tailscale: SSH daemons and administrative panels bind exclusively to the private overlay interface. If an external adversary scans the public IP, those ports simply do not exist.
2. Strict SSH key-only authentication: Passwords disabled in sshd_config, direct root logins prohibited, and idle timeout intervals enforced.

# /etc/ssh/sshd_config.d/hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
X11Forwarding no
MaxAuthTries 3
ClientAliveInterval 300
ClientAliveCountMax 2

Principle 2: Kernel sysctl Tuning for Network Attack Mitigation

Default Linux network parameters prioritize maximum interoperability over resilience against targeted scans and packet floods. Through /etc/sysctl.d/99-security.conf, we harden kernel network behavior against IP spoofing, SYN floods, and malicious route poisoning:

# Ignore ICMP broadcast requests
net.ipv4.icmp_echo_ignore_broadcasts = 1

# SYN flood protection
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2

# Reject ICMP redirect packets
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0

# Strict reverse path filtering (blocks IP spoofing)
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable source routing
net.ipv4.conf.all.accept_source_route = 0

Principle 3: Collective Defense with CrowdSec

Fail2ban has served for years as the default tool to block IPs after repeated authorization failures. However, its architecture is isolated and reactive: every server only learns from attacks it personally absorbs.

CrowdSec modernizes host defense with collective threat intelligence:

• Analyzes Nginx access logs, system journals, and application logs in real time via declarative YAML scenarios.
• When an IP scans sensitive files or attempts credential stuffing, CrowdSec drops traffic at the kernel firewall layer (nftables or iptables) with negligible CPU overhead.
• Dropped IPs are cross-referenced against a global consensus blocklist crowdsourced from thousands of servers worldwide. Known malicious actors are blocked before they ever touch your services.

Principle 4: Rootless Containers and Least Privilege

By default, docker run executes container processes with UID 0 (root). If a container breakout vulnerability (such as a kernel privilege escalation or runc bug) is exploited, the attacker obtains root access over the host operating system.

To constrain container blast radius:

1. User namespaces (userns): Processes inside containers map to an unprivileged user on the host (UID 10001), ensuring a breakout does not yield host root privileges.
2. Read-only root filesystems: Mount the container root as read_only: true, providing ephemeral writable storage only where strictly required via tmpfs mounts (/tmp, /run).
3. Dropping all Linux capabilities: Strip all kernel capabilities and re-enable only what is functionally essential (such as NET_BIND_SERVICE if listening on low ports).

# Hardened docker-compose.yml
services:
  api:
    image: fesa-api:production
    read_only: true
    user: "10001:10001"
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp:rw,noexec,nosuid

Engineering Takeaway

Infrastructure resilience does not require expensive, opaque proprietary software. It relies on ruthlessly minimizing the attack surface:

• If a port is not publicly bound, it cannot be probed.
• If the kernel validates route paths, packets cannot be spoofed.
• If containers run rootless, an exploit cannot compromise the host.

Embedding these patterns into baseline deployment scripts prevents security breaches, preserves customer data integrity, and keeps production stable 24/7.