Integration of Fail2Ban as An Artificial Intelligence Based Cyber Security System with Random Forest Algorithm for Adaptive Detection of SSH Brute Force Attacks
Keywords:
Fail2Ban, Random Forest, SSH Security, Brute Force Attack, Cyber SecurityAbstract
Brute force attacks targeting Secure Shell (SSH) services remain one of the most prevalent threats to Linux-based servers, particularly when traditional security mechanisms rely solely on static threshold rules. This study proposes and evaluates the integration of Fail2Ban with a machine learning approach using the Random Forest algorithm to enhance adaptive detection of SSH brute force attacks. The experimental setup was implemented in a controlled virtual environment consisting of an Ubuntu Server as the target system and Kali Linux as the attacker. Fail2Ban was configured using the jail.local policy with parameters maxretry = 5, findtime = 3 minutes, and bantime = 1 hour. Authentication logs generated from repeated failed SSH login attempts were collected and processed as input features for the Random Forest classifier, including failed login frequency per IP address, inter-arrival time of login attempts, targeted usernames, destination ports, and connection status. Experimental results demonstrate that Fail2Ban successfully blocked malicious IP addresses after 15 failed login attempts, while the Random Forest model significantly improved detection performance by reducing false positives and enabling adaptive recognition of evolving attack patterns. The findings indicate that combining rule-based intrusion prevention with machine learning-based log analysis provides a more intelligent, efficient, and adaptive cyber defense mechanism compared to conventional static approaches. This research contributes both practically and academically to the development of artificial intelligence-assisted log monitoring systems for strengthening Linux server security against brute force SSH attacks.
References
A. Aldweesh, A. Derhab, and A. Z. Emam, “Deep learning approaches for anomaly-based intrusion detection systems: A survey, taxonomy, and open issues,” Knowledge-Based Systems, vol. 189, pp. 105124, 2020.
M. Alshamrani, A. Chowdhary, S. Pisharody, D. Huang, and A. Sabur, “A defense system for defeating SSH brute force attacks,” IEEE Access, vol. 9, pp. 70753–70765, 2021.
S. Hosseini and M. Azizi, “The hybrid anomaly detection model using machine learning algorithms for intrusion detection,” Journal of Information Security and Applications, vol. 62, pp. 102987, 2021.
A. K. Shukla, P. Singh, and M. Vardhan, “Machine learning-based intrusion detection system for SSH attacks,” International Journal of Information Security, vol. 21, no. 4, pp. 891–904, 2022.
R. Vinayakumar et al., “Deep learning approach for intelligent intrusion detection system,” IEEE Access, vol. 7, pp. 41525–41550, 2019.
A. Kurniawan, D. S. Nugroho, and R. Wardoyo, “Network intrusion detection using Random Forest and feature selection,” Journal of Big Data, vol. 8, no. 1, pp. 1–18, 2021.
M. Ring, D. Landes, and A. Hotho, “Detection of slow brute force attacks using log-based machine learning,” Computers & Security, vol. 100, pp. 102086, 2021.
M. Almseidin et al., “Evaluation of machine learning algorithms for intrusion detection system,” Journal of Network and Computer Applications, vol. 110, pp. 112–123, 2020.
H. H. Pajouh, R. Javidan, R. Khayami, D. Ali, and K. Choo, “A two-layer dimension reduction and two-tier classification model for anomaly-based intrusion detection in IoT backbone networks,” IEEE Transactions on Emerging Topics in Computing, vol. 7, no. 2, pp. 314–323, 2019.
A. Aljawarneh, M. B. Yassein, and M. Aljundi, “An enhanced J48 classification algorithm for the anomaly intrusion detection systems,” Cluster Computing, vol. 22, pp. 10549–10565, 2019.
M. Conti, Q. Qiu, A. P. Mathur, and C. Lal, “Secure and resilient cyber–physical systems,” IEEE Design & Test, vol. 37, no. 2, pp. 78–87, 2020.
Y. Xin et al., “Machine learning and deep learning methods for cybersecurity,” IEEE Access, vol. 6, pp. 35365–35381, 2018.
T. A. Tang, L. Mhamdi, D. McLernon, S. A. Raza Zaidi, and M. Ghogho, “Deep recurrent neural network for intrusion detection in SDN-based networks,” IEEE Transactions on Network and Service Management, vol. 15, no. 1, pp. 1–14, 2018.
A. Ferrag, L. Maglaras, A. Argyriou, D. Kosmanos, and H. Janicke, “Security for 5G and IoT networks: A survey,” Computer Networks, vol. 178, pp. 107–122, 2020.
S. Behl and A. Behl, “Cyberwar, cyberterrorism and cybercrime: A review,” Journal of Strategic Security, vol. 10, no. 4, pp. 1–18, 2017.
