Student Dropout Prediction: LightGBM vs MLP

# 🎓 Student Dropout Prediction — UCI #697 **Task:** Predict whether a student **drops out**, stays **enrolled**, or **graduates**, using enrollment data plus 1st/2nd-semester performance (UCI *Predict Students' Dropout and Academic Success* — 4,424 students, 35 features, 3-class imbalanced target). **Pipeline:** 1. **Load** — download `data.csv` from UCI (Kaggle mirror as fallback) 2. **Preprocess** — clean column names, encode target, stratified 70/15/15 train/val/test split, standardize features (for the NN arm) 3. **Parallel experiment** — two approaches, each tuned on validation: - **A: LightGBM** (gradient boosting) — randomized hyperparameter search - **B: MLP** (neural network) — randomized hyperparameter search + early stopping 4. **Converge** — winner picked by **dropout-class F1 on validation** (the class of interest); macro-F1 & balanced accuracy reported alongside 5. **Evaluate the winner** on the held-out test set, then feature importance + conclusions *References: Martins et al. 2021 (dataset paper); Grinsztajn et al., NeurIPS 2022 (tree models vs deep learning on tabular data); NN-vs-GBM comparison for dropout prediction (2024).*

"""Download the UCI 'Predict Students' Dropout and Academic Success' dataset (data.csv) and sanity-check it."""
import io
import urllib.request
import zipfile

import numpy as np
import pandas as pd

# Two known UCI endpoints for dataset 697 (static path + CDN download link).
UCI_URLS = [
    "https://archive.ics.uci.edu/static/public/697/predict+students+dropout+and+academic+success.zip",
    "https://cdn.uci-ics-mlr-prod.aws.uci.edu/697/predict+students+dropout+and+academic+success.zip",
]

df = None
for url in UCI_URLS:
    try:
        with urllib.request.urlopen(url, timeout=90) as resp:
…

Loaded 'data.csv' (sep=';') from:
  https://archive.ics.uci.edu/static/public/697/predict+students+dropout+and+academic+success.zip

Shape: 4424 students x 37 columns
Missing values: 0

Target distribution:
Target
Graduate    0.499
Dropout     0.321
Enrolled    0.179

First rows:

"""Preprocess: clean names, encode target, stratified 70/15/15 split, scale for NN, shared eval helpers."""
import json

import numpy as np
import pandas as pd
from sklearn.metrics import (balanced_accuracy_score, classification_report,
                             confusion_matrix, f1_score)
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

df = pd.read_csv("dropout_data.csv")
df.columns = df.columns.str.strip()

# Encode the 3-class target as defined by the dataset authors.
target_map = {"Dropout": 0, "Enrolled": 1, "Graduate": 2}
y = df["Target"].map(target_map).values
X = df.drop(columns=["Target"])
CLASSES = ["Dropout", "Enrolled", "Graduate"]
…

Features: 36  |  X: 4424 rows
Split sizes -> train: 3096 | val: 664 | test: 664

Helpers ready: evaluate(y_true, y_pred, prefix) -> {balanced_acc, macro_f1, dropout_f1}
Data ready: X_tr/X_val/X_te (raw), X_tr_s/X_val_s/X_te_s (scaled), y_tr/y_val/y_te, CLASSES

"""Converge: compare both approaches and pick the winner by validation dropout-F1."""
import pandas as pd

# Validation metrics reported by each variant branch (captured from their cell outputs).
results = [
    {"approach": "A — LightGBM (GBDT)", "balanced_acc": 0.7010, "macro_f1": 0.6976, "dropout_f1": 0.7376},
    {"approach": "B — MLP (NN)",        "balanced_acc": 0.6497, "macro_f1": 0.6520, "dropout_f1": 0.6983},
]
print(pd.DataFrame(results).set_index("approach").round(4).to_string())

winner = "gbdt" if results[0]["dropout_f1"] >= results[1]["dropout_f1"] else "mlp"
print(f"\n>>> WINNER (max validation dropout-F1): {winner.upper()}")

# Tuned hyperparameters from the winning branch run (reused for the final refit).
WINNER_PARAMS = {
    "gbdt": {"subsample": 0.8, "reg_lambda": 0.1, "reg_alpha": 1.0, "num_leaves": 127,
             "n_estimators": 600, "min_child_samples": 40, "learning_rate": 0.05,
             "colsample_bytree": 0.7, "class_weight": "balanced"},
…

balanced_acc  macro_f1  dropout_f1
approach                                               
A — LightGBM (GBDT)        0.7010    0.6976      0.7376
B — MLP (NN)               0.6497    0.6520      0.6983

>>> WINNER (max validation dropout-F1): GBDT
Winner params: {'subsample': 0.8, 'reg_lambda': 0.1, 'reg_alpha': 1.0, 'num_leaves': 127, 'n_estimators': 600, 'min_child_samples': 40, 'learning_rate': 0.05, 'colsample_bytree': 0.7, 'class_weight': 'balanced'}

"""Test evaluation of the winner: refit on train+val, evaluate on the untouched test split."""
import joblib

import numpy as np
from lightgbm import LGBMClassifier

X_full = np.vstack([X_tr, X_val])
y_full = np.concatenate([y_tr, y_val])
print(f"Refit on train+val: {X_full.shape[0]} rows")

model = LGBMClassifier(random_state=42, n_jobs=1, verbosity=-1, **WINNER_PARAMS)
model.fit(X_full, y_full)
y_te_pred = model.predict(X_te)

test_metrics = evaluate(y_te, y_te_pred, prefix="TEST (GBDT winner)")
joblib.dump(model, "gbdt_winner.joblib")
print("\nSaved model -> gbdt_winner.joblib")

Refit on train+val: 3760 rows

"""Feature importance (gain) of the winning LightGBM + normalized confusion matrix on test."""
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix

# Gain-based importances (the sklearn API reports 'split' by default — use gain explicitly).
imp = pd.Series(model.booster_.feature_importance(importance_type="gain"), index=X.columns)
imp = imp.sort_values().tail(15)

fig, ax = plt.subplots(figsize=(8, 5.5))
imp.plot.barh(ax=ax, color="#4C72B0")
ax.set_title("LightGBM feature importance (gain) — top 15")
ax.set_xlabel("Total gain")
plt.tight_layout()
plt.show()

# Normalized confusion matrix on the test set.
…

Top-5 features by gain: Admission grade, Curricular units 1st sem (grade), Curricular units 1st sem (approved), Curricular units 2nd sem (grade), Curricular units 2nd sem (approved)

## Results & Conclusion **Winner: LightGBM (gradient boosting)** — picked by validation dropout-F1 (0.738 vs 0.698 for the MLP). | Approach (tuned on validation) | Val balanced acc | Val macro-F1 | **Val dropout-F1** | Test dropout-F1 (winner) | |---|---|---|---|---| | **A — LightGBM (GBDT)** | 0.701 | 0.698 | **0.738** | **0.809** | | B — MLP (neural network) | 0.650 | 0.652 | 0.698 | — | **Held-out test — winner refit on train+val:** accuracy **0.795** · balanced accuracy 0.744 · macro-F1 0.746 · **dropout-F1 0.809** (precision 0.862 / recall 0.762). In practical terms: ~76% of real dropouts are flagged, and ~86% of dropout flags are correct. **Key dropout drivers** (LightGBM gain importance): admission grade, 1st-semester grade, 1st-semester approved units, 2nd-semester grade, 2nd-semester approved units — i.e., first-year academic performance dominates, consistent with the dataset's own literature. **Notes** - The MLP (runner-up) stays available on its branch — keep it if you ever need a probability-smooth model or lower-latency serving; it lost on the headline metric but is a close second on macro-F1. - Compute: LightGBM ~40 s vs MLP ~14 min of tuning on this tiny dataset — trees won on quality *and* cost, matching the tabular-data literature. - Pipeline detail: the first GBDT attempt timed out from nested parallelism (`n_jobs=-1` in both the search and the estimator); the fix (single-threaded estimator + joblib-parallel search) cut it to ~40 s. The trained winner i…