Machine Learning Algorithms for IT Project Risk Level Identification: A Multi Classifier Framework with Systematic Preprocessing and Per-Class Evaluation

Abstract

The persistently high failure rate of Information Technology (IT) projects, driven by inadequate risk identification, remains a critical organizational challenge, and conventional risk assessment based on subjective expert judgment struggles to model the non-linear interdependencies in large-scale project data. This research proposes a multi-classifier machine learning (ML) framework for automated IT project risk level identification (Low, Medium, High), built on two contributions: a systematic, auditable preprocessing protocol (explicit target-leakage screening, a controlled before/after comparison of the Synthetic Minority Over-sampling Technique (SMOTE), and hyperparameter optimization) and a per-class ordinal-consistency evaluation that decomposes misclassifications by their severity relative to the ordinal risk structure. Five classifiers were evaluated on 10,919 historical IT project records: Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF), Decision Tree (DT), and K-Nearest Neighbors (KNN). The leakage screening identified and removed a variable that deterministically encoded the target label. After SMOTE and hyperparameter tuning, SVM achieved the best single-model performance (accuracy 83.65%, macro F1-score 0.759), followed by RF (82.14%, 0.731), LR (80.82%, 0.744), DT (73.72%, 0.669), and KNN (69.64%, 0.596); a five-model soft-voting ensemble reached 83.42% accuracy and a macro F1-score of 0.757. Across all sixteen model/stage configurations (7,165 aggregate misclassifications), the non-adjacent error rate, a High-risk project predicted as Low-risk or vice versa, was exactly zero, a reliability property invisible to aggregate accuracy or F1-score.