Integration of Fail2Ban as An Artificial Intelligence Based Cyber Security System with Random Forest Algorithm for Adaptive Detection of SSH Brute Force Attacks - Sugiarto, Bambang; Nursetyo, Arif; Taufiq Subagio, Ridho; Kusnadi; Sokibi, Petrus (Cirebon Annual Multidiciplinary International Conference (CAMIC), 2026)
Brute force attacks targeting Secure Shell (SSH) services remain one of the most prevalent threats to Linux-based servers, particularly when traditional security mechanisms rely solely on static threshold rules. This study proposes and evaluates the integration of Fail2Ban with a machine learning approach using the Random Forest algorithm to enhance adaptive detection of SSH brute force attacks. The experimental setup was implemented in a controlled virtual environment consisting of an Ubuntu Server as the target system and Kali Linux as the attacker. Fail2Ban was configured using the jail.local policy with parameters maxretry = 5, findtime = 3 minutes, and bantime = 1 hour. Authentication logs generated from repeated failed SSH login attempts were collected and processed as input features for the Random Forest classifier, including failed login frequency per IP address, inter-arrival time of login attempts, targeted usernames, destination ports, and connection status. Experimental results demonstrate that Fail2Ban successfully blocked malicious IP addresses after 15 failed login attempts, while the Random Forest model significantly improved detection performance by reducing false positives and enabling adaptive recognition of evolving attack patterns. The findings indicate that combining rule-based intrusion prevention with machine learning-based log analysis provides a more intelligent, efficient, and adaptive cyber defense mechanism compared to conventional static approaches. This research contributes both practically and academically to the development of artificial intelligence-assisted log monitoring systems for strengthening Linux server security against brute force SSH attacks.