Hyperparameter Tuning with Bayesian Optimization: Beyond Grid & Random Search

The Inefficiency of Grid & Random Searches Grid Search suffers from the **curse of dimensionality**; as the number of hyperparameters grows, the combinatorial space explodes exponentially. Random Search improves upon this by not wasting trials on uninformative dimensions, but it fails to learn from past evaluations.

Bayesian Optimization & Tree-Structured Parzen Estimators (TPE) Bayesian optimization models the probability distribution of hyperparameter configurations conditioned on target loss. **Optuna** uses TPE to sample new configurations from regions of the parameter space that maximize the **Expected Improvement (EI)**.

import optuna
from sklearn.ensemble import HistGradientBoostingClassifier

def objective(trial): params = { 'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3, log=True), 'max_iter': trial.suggest_int('max_iter', 100, 1000, step=100), 'max_depth': trial.suggest_int('max_depth', 3, 10), 'l2_regularization': trial.suggest_float('l2_regularization', 1e-6, 10.0, log=True), 'min_samples_leaf': trial.suggest_int('min_samples_leaf', 10, 100) }

model = HistGradientBoostingClassifier(**params, random_state=42) score = cross_val_score(model, X_train, y_train, cv=5, scoring='f1_weighted').mean() return score

# Initialize Bayesian Study study = optuna.create_study(direction='maximize', sampler=optuna.samplers.TPESampler(seed=42)) study.optimize(objective, n_trials=60, timeout=1200)

print(f"Optimal F1 Score: {study.best_value:.4f}") print("Best Hyperparameters:", study.best_params) ```

Production Rules - **Pruning**: Enable Optuna's `MedianPruner` to terminate unpromising training trials after initial epochs, doubling optimization speed. - **Logarithmic Scaling**: Always use `log=True` for learning rates and regularization penalties to explore across orders of magnitude evenly.