CS512 Machine Learning | Course project | Sabancı University

Model Selection and Explainability in Traffic Accident Severity Prediction

Oct 2025 – Jan 2026 Sima Adleyba, Başak Çarhacıoğlu, and Ahmet İhsan Gül

This project uses data from 101,904 U.S. police-reported crashes (CRSS, 2016–2023) to compare a class-weighted Linear SVM and an ordinal XGBoost model for four-class injury-severity prediction. We use SHAP, LIME, and DiCE to examine which features matter most to each model.

Overview Dataset Model selection Methodology Explainability Results Gallery Materials
01 |

Overview

Problem

We inspect what the models consider important when classifying injury severity. The four classes are no injury, minor, severe, and fatal. Fatal crashes make up only ~2.4% of cases.

Dataset

101,904 police-reported U.S. crashes from the Crash Report Sampling System (CRSS, 2016–2023).

Models

We compared a class-weighted Linear SVM with an ordinal XGBoost model that divides the four-class task into three binary thresholds.

Explainability

SHAP for global feature attribution, LIME for local instance-level explanations, and DiCE for counterfactuals.

Takeaway

Both models reach ≈ 0.59 accuracy and 0.53–0.54 macro-F1. Oversampling lifts fatal-class F1 from 0.26 to 0.59. Feature reduction does not improve performance.

Dataset & task

The Crash Report Sampling System (CRSS) contains U.S. police-reported crash records. The raw data are organized in 28 relational tables.

101,904 crashes in the full dataset
163 crash-level features
28 relational source tables
40.9% of records contain an unknown indicator

We combined information about vehicles, drivers, and occupants into one record per crash. Counts, proportions, summary statistics, and indexed fields such as TRAV_SP_1 and TRAV_SP_2 make up the features. We used MAX_SEV, the highest injury severity in the crash, as the target. Its four classes are 0 for no injury, 1 for minor injury, 2 for severe injury, and 3 for fatal injury. Possible injuries are included in the minor class.

We kept separate indicators for missing values (_UNKNOWN) and fields that did not apply (_NOT_APPLICABLE). These indicators can affect the predictions. The models use a feature set designed to avoid leakage. LIME and DiCE use a separate imputed version of the data for explanations.

We applied SMOTE only to the training split. It increased the share of fatal crashes from 2.38% to 5.38%, without fully balancing the classes. We then compared models trained with and without oversampling.

Class distribution with held-out test split

0: no injury 48.87%
1: minor 37.69%
2: severe 11.08%
3: fatal 2.37%

The splits include about 71K training, 15K validation, and 15K test crashes.

Model selection

The first round of model selection was part of the project's progress report. We compared five model families before choosing the final models.

This initial comparison used a 20,000-crash subset during the progress report stage.

Selected for the final study Screened out
Logistic regression 0.378
Linear SVM 0.384
Linear SVM + SMOTE 0.443
Nyström SVM (kernel approx.) 0.419
Random Forest (best config) 0.396
XGBoost (cumulative + SMOTE) 0.450

The bars show the best macro-F1 for each model family in the progress report.

Random Forest SHAP summary for the fatal class
Fig. S1: Random Forest SHAP, fatal class (screening stage; 20k subset)

Methodology

We used fixed train, validation, and test splits and compared models with and without oversampling. We aggregated explanations over a fixed evaluation subset, then retrained models with fewer features and evaluated them on the held-out test set.

28 CRSS tablesCrash, vehicle, person
→
Crash-level record163 features from indexed fields and aggregates
→
Train / val / test71k / 15k / 15k
→
± SMOTEtrain split only
→
Linear SVM ∥ Ordinal XGBoost
→
SHAP, LIME, DiCE

05 | Models

Linear SVM

  • We used balanced class weights because fatal crashes account for about 2.4% of records.
  • We scaled numeric features, encoded categorical features, and tuned C on the validation set.
  • The linear SVM had the highest macro-F1 among the linear models in the initial comparison.
  • We produced separate SHAP and LIME explanations for each severity class.

Ordinal XGBoost

  • We trained three binary classifiers for P(y≥1), P(y≥2), and P(y≥3).
  • Each classifier used early stopping on the validation set and scale_pos_weight for its class distribution.
  • We calculated four-class probabilities from the cumulative predictions: P(y=0)=1−p₁, P(y=1)=p₁−p₂, P(y=2)=p₂−p₃, and P(y=3)=p₃. We selected the class with the highest probability.
  • This setup accounts for the order of the severity classes.

Explainability

We used SHAP to examine feature importance across the models, LIME to inspect individual predictions, and DiCE to see which feature changes alter predicted severity.

We used mean absolute SHAP values to compare feature importance. For XGBoost, we calculated them separately for each ordinal threshold. Features with high importance include NMACTION_NA, VEH_COUNT, GVWR_AVG_1, ALCOHOL, DRDISTRACT_1, and TRAV_SP_*. For the SVM's fatal class, TOTAL_COUNT and NMOTHPRE_UNKNOWN increase the prediction, while REGION and the 45–60 age share decrease it.

XGBoost SHAP summary for P(y≥1)
Fig. 2a: XGBoost Variant A, SHAP for P(y≥1): any injury
XGBoost SHAP summary for P(y≥2)
Fig. 2b: SHAP for P(y≥2): severe or fatal
XGBoost SHAP summary for P(y≥3)
Fig. 2c: SHAP for P(y≥3): the fatal threshold
SVM SHAP top features for class 0
Fig. 5a: SVM SHAP, class 0 (no injury)
SVM SHAP top features for class 3
Fig. 5b: SVM SHAP, class 3 (fatal)

This comparison uses the 20,000-crash subset from the progress report.

We removed the *_UNKNOWN indicators from the XGBoost model to see how the explanations changed. With those indicators, the fatal-threshold model relies heavily on fields such as NMHELMET_UNKNOWN and NMACTION_UNKNOWN. Without them, ALCOHOL becomes more prominent. Macro-F1 falls from 0.45 to 0.44 on this subset.

SHAP summary for P(y≥3) including unknown indicators SHAP summary for P(y≥3) after removing unknown indicators

XGBoost SHAP for the fatal threshold P(y≥3), before / after removing *_UNKNOWN features: progress report, Model 8 vs. Model 8′ (20k subset).

We used an imputed version of the data for LIME explanations. We aggregated XGBoost LIME weights over 89 fixed evaluation cases. VTRAFWAY_4, PEDESTRIAN_COUNT, and P_CRASH1_4 appeared most often. For the SVM, we inspected explanations for individual cases by severity class.

SVM LIME explanation for class 3
Fig. 7: SVM LIME, class 3 (fatal)

We generated three DiCE counterfactuals for each of 20 fatal test crashes to see which feature changes moved their predictions away from fatal. For XGBoost, P_CRASH1_2, MANEUVER_1, and DRIVER_DRUGS_PCT are changed most often; for the SVM, PEDESTRIAN_COUNT, DRIVERRF_4, and VEHICLECC_4 are changed most often. These changes show how the models respond to inputs. They do not establish causal effects.

SVM DiCE most important features, source class 0
Fig. 8a: DiCE, source class 0 → fatal
SVM DiCE most important features, source class 1
Fig. 8b: DiCE, source class 1 → fatal
SVM DiCE most important features, source class 2
Fig. 8c: DiCE, source class 2 → fatal

Results & findings

These are the results from the final full-dataset report. Values are from validation unless labeled as test results.

≈0.59 accuracy, both models (oversampled eval)
0.53–0.54 macro-F1, both models (oversampled eval)
0.26 → 0.59 fatal-class F1, XGBoost without → with SMOTE
163 → 154 features after explanation-driven reduction (XGB)
Validation confusion matrices for XGBoost Variant A, Variant B, and SVM
Fig. 1: Validation confusion matrices: XGBoost Var A, Var B, SVM

Oversampling, class by class: XGBoost per-class F1

Variant A: oversampled Variant B: non-oversampled
Class 0, no injury (A) 0.734
Class 0, no injury (B) 0.734
Class 1, minor (A) 0.460
Class 1, minor (B) 0.447
Class 2, severe (A) 0.315
Class 2, severe (B) 0.289
Class 3, fatal (A) 0.595
Class 3, fatal (B) 0.264

Validation, XGBoost Variant A vs Variant B: final report, Table 8

ModelMacro-F1Weighted-F1
XGBoost Var A (oversampled)0.52350.5849
XGBoost Var B (non-oversampled)0.43210.5616
Linear SVM0.53560.5678

Final report, Table 2

MetricFull (163 feat.)Reduced (154 feat.)
F1: C0 no injury0.7263340.725452
F1: C1 minor0.4222370.422542
F1: C2 severe0.2742830.274321
F1: C3 fatal0.5142200.507269
Macro F10.4842690.482396
Weighted F10.5553340.554659

Final report, Table 9: reduction removes features consistently negligible across SHAP/LIME/DiCE

MetricXGBoost FullXGBoost ReducedSVM Full
F1: C0 no injury0.7263340.7254520.700000
F1: C1 minor0.4222370.4225420.450000
F1: C2 severe0.2742830.2743210.220000
F1: C3 fatal0.5142200.5072690.400000
Macro F10.4842690.4823960.445644
Weighted F10.5553340.5546590.540000
# Features163154163

Final report, Table 11: evaluated on the test set

Findings & limitations

  • Oversampling mainly helps the fatal class. Its F1 rises from 0.264 to 0.595, while the majority classes change little.
  • Both models often confuse minor and severe injuries, as the confusion matrices show.
  • Removing features based on the explanations did not improve performance. XGBoost test macro-F1 fell from 0.4843 to 0.4824. SVM validation macro-F1 fell from 0.5356 to 0.5321 after reducing the feature count from 163 to 55.
  • DiCE shows sensitivity near a model's decision boundary, not causes of injury severity. Missingness indicators such as *_UNKNOWN also affect predictions.
  • Future work could use more explicitly ordinal or cost-sensitive training, improve calibration for minority-class decisions, and add contextual features.

Materials

The full report as submitted for CS512 (Machine Learning), Sabancı University, January 2026. Authors: Sima Adleyba, Başak Çarhacıoğlu, and Ahmet İhsan Gül.

Model Selection and Explainability in Traffic Accident Severity Prediction

Final report, 11 pages, PDF, January 2026

Figures and numbers come from the final report unless a section says it uses the 20,000-crash subset from the progress report. There is no public code repository for this project.