Skip to content

Instantly share code, notes, and snippets.

@muhammadardie
Created October 16, 2025 05:12
Show Gist options
  • Save muhammadardie/547763d7b3dd8ee14bd3b006c04d053e to your computer and use it in GitHub Desktop.
Save muhammadardie/547763d7b3dd8ee14bd3b006c04d053e to your computer and use it in GitHub Desktop.
How to Solve "Failed to create session: Maximum number of sessions (xxx) reached, refusing further sessions"

How to Solve "Failed to create session: Maximum number of sessions (8192) reached, refusing further sessions"

Problem Description

When you see this error in your system logs:

pam_systemd(crond:session): Failed to create session: Maximum number of sessions (8192) reached, refusing further sessions.
pam_systemd(sshd:session): Failed to create session: Maximum number of sessions (8192) reached, refusing further sessions.

This means your system has reached the maximum number of systemd sessions (default: 8192) and is refusing to create new sessions. This prevents:

  • New SSH logins
  • Cron jobs from running
  • Services from starting properly
  • Users from logging in

Common Causes

  1. SSH Brute Force Attacks - Each failed/successful login attempt creates a session that may not be cleaned up properly
  2. Lingering Sessions - Sessions that aren't terminated when users disconnect
  3. Misconfigured Services - Services or cron jobs that create sessions without cleaning them up
  4. PAM Configuration Issues - systemd-logind not properly cleaning up sessions

Diagnosis

Check Current Session Count

# Count active sessions
loginctl list-sessions --no-legend | wc -l

# List all sessions
loginctl list-sessions --no-pager

# Show sessions by user
loginctl list-sessions --no-pager | awk '{print $3}' | sort | uniq -c | sort -rn

# Show all users with sessions
loginctl list-users --no-pager

Identify the Source

# Check for SSH brute force attempts
grep "Failed password\|Accepted password" /var/log/secure | tail -100

# Count login attempts by IP
grep "Failed password" /var/log/secure | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn

# Check systemd-logind logs
journalctl -u systemd-logind -n 1000 | grep "New session"

Immediate Solution

Option 1: Restart systemd-logind (Most Effective)

This will clear ALL sessions instantly. This is the fastest way to recover.

systemctl restart systemd-logind

Result: This typically reduces sessions from 8000+ to ~200 immediately.

Warning: This will disconnect all active user sessions except the current terminal.

Option 2: Terminate Sessions by User

# Terminate all sessions for a specific user
loginctl terminate-user root
loginctl terminate-user <username>

# Terminate sessions for all users
for user in $(loginctl list-users --no-legend | awk '{print $2}'); do
    loginctl terminate-user "$user"
done

Option 3: Terminate Individual Sessions

# List sessions
loginctl list-sessions --no-pager

# Terminate specific session
loginctl terminate-session <session-id>

# Bulk terminate all sessions
loginctl list-sessions --no-legend | awk '{print $1}' | xargs -I {} loginctl terminate-session {}

Option 4: Disable Lingering Sessions

# Disable lingering for specific user
loginctl disable-linger <username>

# Disable for all users
for user in $(loginctl list-users --no-legend | awk '{print $2}'); do
    loginctl disable-linger "$user" 2>/dev/null
done

Permanent Prevention

1. Configure systemd-logind for Better Cleanup

Edit /etc/systemd/logind.conf:

[Login]
# Enable automatic process cleanup
KillUserProcesses=yes

# Remove IPC objects when sessions end
RemoveIPC=yes

# Reduce delay before killing processes
InhibitDelayMaxSec=5

# Increase limit if legitimately needed
SessionsMax=8192

# Limit tasks per user
UserTasksMax=12288

Apply changes:

systemctl restart systemd-logind

2. Fix PAM Configuration

For Cron Jobs - Prevent cron from creating systemd sessions:

Edit /etc/pam.d/crond:

# Comment out this line:
# -session   optional   pam_systemd.so

For SSH - Only create sessions on successful authentication:

Edit /etc/pam.d/sshd:

# Change from:
session    optional     pam_systemd.so

# To:
session [success=ok default=ignore] pam_systemd.so

Or disable it completely if you don't need systemd session management:

# Comment out:
# session    optional     pam_systemd.so

Restart SSH after changes:

systemctl restart sshd

3. Install and Configure Fail2Ban (Prevent Brute Force)

# Install Fail2Ban
yum install fail2ban -y  # RHEL/CentOS
# or
apt install fail2ban -y  # Debian/Ubuntu

# Enable and start
systemctl enable fail2ban
systemctl start fail2ban

Create /etc/fail2ban/jail.local:

[DEFAULT]
# Ban for 24 hours
bantime = 86400
# Check for failures in last 10 minutes
findtime = 600
# Allow 3 retries before ban
maxretry = 3
# Email notifications (optional)
destemail = [email protected]
sendername = Fail2Ban
action = %(action_mwl)s

[sshd]
enabled = true
port = ssh
logpath = /var/log/secure  # or /var/log/auth.log on Debian
maxretry = 3
bantime = 86400
findtime = 600

Restart Fail2Ban:

systemctl restart fail2ban

Check banned IPs:

fail2ban-client status sshd

4. Harden SSH Configuration

Edit /etc/ssh/sshd_config:

# Disable root login
PermitRootLogin no

# Use key-based authentication only
PasswordAuthentication no
PubkeyAuthentication yes

# Change default port (security through obscurity)
Port 2222

# Limit authentication attempts
MaxAuthTries 3
MaxSessions 10

# Reduce login grace time
LoginGraceTime 30

# Enable timeout for idle sessions
ClientAliveInterval 300
ClientAliveCountMax 2

# Limit which users can SSH
AllowUsers your_username

# Or limit by group
AllowGroups sshusers

Restart SSH:

systemctl restart sshd

5. Set Up Firewall Rules

Using iptables:

# Rate limit SSH connections
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP

# Block specific attacking IP
iptables -A INPUT -s 104.28.213.127 -j DROP

# Save rules
service iptables save  # RHEL/CentOS
# or
iptables-save > /etc/iptables/rules.v4  # Debian/Ubuntu

Using firewalld:

# Block specific IP
firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='104.28.213.127' reject"

# Limit SSH connections
firewall-cmd --permanent --add-rich-rule='rule service name="ssh" limit value="3/m" accept'

# Reload
firewall-cmd --reload

6. Create Automatic Session Cleanup Script

Create /usr/local/bin/cleanup-sessions.sh:

#!/bin/bash
# Automatic session cleanup script

# Set threshold (start cleanup at 7000 sessions)
THRESHOLD=7000
SESSION_COUNT=$(loginctl list-sessions --no-legend | wc -l)

# Log to syslog
logger "Session cleanup check: Current sessions = $SESSION_COUNT"

if [ $SESSION_COUNT -gt $THRESHOLD ]; then
    logger "WARNING: High session count detected: $SESSION_COUNT. Starting cleanup..."
    
    # Terminate all sessions (aggressive)
    for user in $(loginctl list-users --no-legend | awk '{print $2}'); do
        loginctl terminate-user "$user" 2>/dev/null
    done
    
    # Alternative: Restart systemd-logind
    # systemctl restart systemd-logind
    
    NEW_COUNT=$(loginctl list-sessions --no-legend | wc -l)
    logger "Session cleanup completed. Sessions reduced from $SESSION_COUNT to $NEW_COUNT"
fi

Make it executable:

chmod +x /usr/local/bin/cleanup-sessions.sh

Add to crontab:

crontab -e

Add this line:

# Run cleanup every 30 minutes
*/30 * * * * /usr/local/bin/cleanup-sessions.sh

7. Monitor Session Count

Create a monitoring script /usr/local/bin/monitor-sessions.sh:

#!/bin/bash
# Monitor session count

SESSION_COUNT=$(loginctl list-sessions --no-legend | wc -l)
THRESHOLD=7000

if [ $SESSION_COUNT -gt $THRESHOLD ]; then
    echo "WARNING: High session count: $SESSION_COUNT / 8192"
    echo "Top users by session count:"
    loginctl list-sessions --no-pager | awk '{print $3}' | sort | uniq -c | sort -rn | head -10
fi

Run it periodically or add to monitoring system.

Verification

After applying fixes:

# Check current session count
echo "Active sessions: $(loginctl list-sessions --no-legend | wc -l)"

# Monitor in real-time
watch -n 2 'loginctl list-sessions --no-legend | wc -l'

# Check Fail2Ban status
fail2ban-client status sshd

# Test SSH connection
ssh localhost -p 22

# Review recent authentication attempts
tail -f /var/log/secure  # or /var/log/auth.log

Security Audit After Attack

If you were under brute force attack and root was compromised:

# Check successful logins
grep "Accepted password" /var/log/secure

# Check login history
last -a | head -20
lastlog

# Check active connections
w
who

# Look for suspicious processes
ps auxf | less
top

# Check for unauthorized users
cat /etc/passwd

# Review cron jobs
crontab -l
ls -la /etc/cron.*

# Check for suspicious files
find / -type f -mtime -1  # Files modified in last 24 hours

# Check network connections
netstat -tulpn
ss -tulpn

# Install and run security scanners
yum install rkhunter chkrootkit -y
rkhunter --check
chkrootkit

Additional Security Measures

  1. Change passwords for all users, especially root
  2. Rotate SSH keys - Generate new keys and update authorized_keys
  3. Enable SELinux or AppArmor for additional protection
  4. Set up VPN access - Restrict SSH to VPN IPs only
  5. Use port knocking - Hide SSH port until specific sequence
  6. Enable two-factor authentication - Use Google Authenticator for SSH
  7. Monitor logs - Set up log aggregation and alerting
  8. Regular updates - Keep system packages up to date

Quick Reference Commands

# Check session count
loginctl list-sessions --no-legend | wc -l

# Immediate fix (restart logind)
systemctl restart systemd-logind

# Terminate user sessions
loginctl terminate-user root

# Check Fail2Ban
fail2ban-client status sshd

# Unban IP
fail2ban-client set sshd unbanip 1.2.3.4

# Monitor authentication attempts
tail -f /var/log/secure

Conclusion

The session limit issue is often caused by brute force attacks or misconfigured services. The immediate fix is to restart systemd-logind, which typically reduces sessions from 8000+ to ~200 instantly. For long-term prevention, implement Fail2Ban, harden SSH configuration, and configure proper session cleanup in systemd-logind.


Last Updated: October 16, 2025
Tested On: RHEL/CentOS 7/8/9, Ubuntu 20.04/22.04, Debian 10/11

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment