# Optuna and the Bayes Rule

### Hyperparameter Optimization

#### (Without Burning the Planet)

<br>

**Hobson Lane**

Python User Group

*July 24, 2026*

---

![](2026-06-25/covid_test_rate.png)

---

## Deriving Bayes' Rule

<table style="border:none;margin:auto;font-size:0.85em;border-spacing:0.2em 0.8em">
<tr style="vertical-align:middle">
  <td style="border:none;text-align:right;padding:0 0.4em">$P(A|B) \cdot P(B)$</td>
  <td style="border:none;text-align:center;padding:0 0.3em">$=$</td>
  <td style="border:none;text-align:left;padding:0 0.4em">$P(B|A) \cdot P(A)$</td>
</tr>
<tr style="vertical-align:middle">
  <td style="border:none;text-align:right;padding:0 0.4em">$P(A|B)$</td>
  <td style="border:none;text-align:center;padding:0 0.3em">$=$</td>
  <td style="border:none;text-align:left;padding:0 0.4em">$\displaystyle\frac{P(B|A) \cdot P(A)}{P(B)}$</td>
</tr>
<tr style="vertical-align:middle">
  <td style="border:none;text-align:right;padding:0 0.4em">$P(A|B)$</td>
  <td style="border:none;text-align:center;padding:0 0.3em">$=$</td>
  <td style="border:none;text-align:left;padding:0 0.4em">$\displaystyle\frac{P(B|A) \cdot P(A)}{P(B|A) \cdot P(A) + P(B|\lnot A) \cdot P(\lnot A)}$</td>
</tr>
</table>

---

## Bayes Rule

| Prediction of Interest |     |  Stats about the world                                                                      |
|-----------------------:|-----|:--------------------------------------------------------------------------------------------|
| $P(A|B) \cdot P(B)$    | $=$ | $P(B|A) \cdot P(A)$                                                                         |
|                        |     |                                                                                             |
|            $P(A|B)$    | $=$ | $\displaystyle\frac{P(B|A) \cdot P(A)}{P(B)}$                                               |
|                        |     |                                                                                             |
|            $P(A|B)$    | $=$ | $\displaystyle\frac{P(B|A) \cdot P(A)}{P(B|A) \cdot P(A) + P(B|\lnot A) \cdot P(\lnot A)}$  |

|        Disease        |     | Symptoms |
|----------------------:|-----|:---------|
| $P(A|B) \cdot P(B)$   | =   | 0        |
| $P(A|B)$              | =   | 99,989   |
| $P(A|B)$              | =   | 99,989   |

---

## Cancer Screening Example

| Cancer \ Symptom | Yes | No     | Total   |
|------------------|----:|-------:|--------:|
| **Yes**          | 1   | 0      | 1       |
| **No**           | 10  | 99,989 | 99,999  |
| **Total**        | 11  | 99,989 | 100,000 |

---

## The Problem

You trained a model.

It's... fine.

But is it *optimal*?

---

## Hyperparameters

Not learned from data — chosen by *you*

- Learning rate
- Number of layers, neurons
- Dropout rate
- Batch size
- Regularization strength

---

## Hyperparameter Search Results

<table class="st"><thead><tr>
  <th>Study</th><th>Tr</th><th>Ep</th><th>Val Acc ↑</th><th>State</th>
  <th>Heads*</th><th>Layers*</th><th>h_dim</th><th>lr*</th>
</tr></thead><tbody>
<tr><td>23-layer</td><td>3</td><td>85</td><td><b>0.4187</b></td><td class="ok">COMPLETE</td><td>3</td><td>27</td><td>256</td><td>3.70e-04</td></tr>
<tr><td>24-26-layer</td><td>2</td><td>129</td><td>0.4139</td><td class="ok">COMPLETE</td><td>3</td><td>24</td><td>256</td><td>2.50e-04</td></tr>
<tr><td>23-layer</td><td>0</td><td>110</td><td>0.3837</td><td class="ok">COMPLETE</td><td>3</td><td>19</td><td>256</td><td>2.01e-04</td></tr>
<tr><td>emma-24-26</td><td>2</td><td>121</td><td>0.3729</td><td class="ok">COMPLETE</td><td>—</td><td>24</td><td>256</td><td>5.47e-04</td></tr>
<tr><td>23-layer</td><td>4</td><td>85</td><td>0.3701</td><td class="ok">COMPLETE</td><td>4</td><td>27</td><td>256</td><td>1.13e-03</td></tr>
<tr><td>24-26-layer</td><td>0</td><td>110</td><td>0.3552</td><td class="ok">COMPLETE</td><td>2</td><td>24</td><td>256</td><td>5.53e-04</td></tr>
<tr><td>emma-24-26</td><td>0</td><td>25</td><td>0.3444</td><td class="run">RUNNING</td><td>—</td><td>24</td><td>256</td><td>5.99e-04</td></tr>
<tr><td>23-layer</td><td>5</td><td>70</td><td>0.3387</td><td class="fail">FAIL</td><td>5</td><td>27</td><td>256</td><td>1.06e-03</td></tr>
<tr><td>nonuniform-12</td><td>0</td><td>100</td><td>0.3384</td><td class="ok">COMPLETE</td><td>—</td><td>—</td><td>128</td><td>1.74e-03</td></tr>
<tr><td>nonuniform-12</td><td>1</td><td>100</td><td>0.3370</td><td class="ok">COMPLETE</td><td>—</td><td>—</td><td>128</td><td>1.85e-03</td></tr>
<tr><td>24-26-layer</td><td>1</td><td>154</td><td>0.3314</td><td class="ok">COMPLETE</td><td>2</td><td>24</td><td>256</td><td>1.67e-04</td></tr>
<tr><td>24-26-layer</td><td>3</td><td>49</td><td>0.3257</td><td class="ok">COMPLETE</td><td>3</td><td>24</td><td>256</td><td>1.96e-03</td></tr>
<tr><td>emma-24-26</td><td>1</td><td>34</td><td>0.3252</td><td class="fail">FAIL</td><td>—</td><td>26</td><td>256</td><td>6.04e-04</td></tr>
<tr><td>23-layer</td><td>1</td><td>35</td><td>0.3356</td><td class="ok">COMPLETE</td><td>4</td><td>19</td><td>256</td><td>2.56e-04</td></tr>
<tr><td>23-layer</td><td>2</td><td>85</td><td>0.3337</td><td class="ok">COMPLETE</td><td>5</td><td>19</td><td>256</td><td>3.87e-04</td></tr>
</tbody></table>

---

## Learning Curves

<a href="2026-06-25/large_learning_curves.html" style="display:none">_</a>

---

## Naive Approach: Grid Search

```python
from sklearn.model_selection import GridSearchCV

param_grid = {
    'learning_rate': [0.001, 0.01, 0.1],
    'n_estimators': [100, 500, 1000],
    'max_depth': [3, 5, 10],
}
```

3 × 3 × 3 = **27 trials**

Add one more param with 3 values → **81 trials**

*Curse of dimensionality*

---

## Random Search Is Better Than Grid

- Random covers the space more evenly
- Often finds good regions with fewer trials
- But still ignores what it has already learned

---

## Bayes' Rule

$$P(H | E) = \frac{P(E | H) \cdot P(H)}{P(E)}$$

- $P(H)$ — prior: what we believed before
- $P(E | H)$ — likelihood: how well $H$ explains evidence
- $P(H | E)$ — posterior: updated belief after seeing data

---

## Bayesian Optimization

*Use past trial results to choose the next trial*

1. Build a **surrogate model** of the objective function
2. Use an **acquisition function** to pick next point
3. Evaluate the real objective
4. Update the surrogate
5. Repeat

---

## Surrogate Model

A cheap approximation of your expensive function

- Gaussian Process (classic)
- Random Forest
- **TPE** — Tree-structured Parzen Estimator

Optuna uses **TPE** by default

---

## TPE: The Intuition

Split past trials into two groups:

- **Good** — top 25% of results
- **Bad** — bottom 75% of results

Model $p(x | \text{good})$ and $p(x | \text{bad})$

Next trial: maximize $\frac{p(x | \text{good})}{p(x | \text{bad})}$

---

## This IS Bayes' Rule

$$P(\text{good} | x) \propto \frac{p(x | \text{good})}{p(x)}$$

We want hyperparameters that are *more likely* in the good region than the bad region

---

## Optuna in 10 Lines

```python
import optuna

def objective(trial):
    lr = trial.suggest_float('lr', 1e-5, 1e-1, log=True)
    n_layers = trial.suggest_int('n_layers', 1, 5)
    return train_and_evaluate(lr, n_layers)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

print(study.best_params)
```

---

## suggest_* Methods

```python
# Continuous (log scale great for learning rates)
trial.suggest_float('lr', 1e-5, 1e-1, log=True)

# Integer
trial.suggest_int('n_layers', 1, 10)

# Categorical
trial.suggest_categorical('optimizer', ['adam', 'sgd', 'rmsprop'])
```

---

## Pruning: Stop Bad Trials Early

```python
for epoch in range(n_epochs):
    accuracy = train_epoch(model)

trial.report(accuracy, epoch)
    if trial.should_prune():
        raise optuna.exceptions.TrialPruned()
```

Optuna's **MedianPruner** kills trials performing below median at each step

---

## Visualization

```python
optuna.visualization.plot_optimization_history(study)
optuna.visualization.plot_param_importances(study)
optuna.visualization.plot_contour(study)
```

Which hyperparameters matter most?

---

## Parallel Trials

```python
study = optuna.create_study(
    storage='sqlite:///my_study.db',
    study_name='xgboost_tuning',
    direction='maximize',
    load_if_exists=True,
)

# Run in multiple terminals / processes
study.optimize(objective, n_trials=50)
```

---

## Samplers

| Sampler | Best For |
|---------|----------|
| `TPESampler` (default) | Most cases |
| `CmaEsSampler` | Continuous, low-dim |
| `NSGAIISampler` | Multi-objective |
| `RandomSampler` | Baseline / debugging |

---

## Multi-Objective Optimization

```python
def objective(trial):
    # minimize error AND inference time
    return accuracy, latency_ms

study = optuna.create_study(
    directions=['maximize', 'minimize']
)
```

Returns a **Pareto front** of non-dominated solutions

---

## Bayes Rule in the Wild

Bayesian thinking shows up everywhere:

- Spam filters (Naive Bayes)
- Medical diagnosis
- A/B testing (Bayesian inference)
- Hyperparameter tuning ← *we are here*
- Reinforcement learning (model-based)

---

## Compare: Random vs TPE

100 trials on XGBoost with 6 hyperparameters:

| Method | Best Accuracy |
|--------|--------------|
| Grid Search | 0.847 |
| Random Search | 0.861 |
| Optuna (TPE) | **0.883** |

TPE finds good regions and *exploits* them

---

## Exploration vs Exploitation

The acquisition function balances:

- **Explore** — try uncertain regions
- **Exploit** — refine known good regions

*Same tradeoff as RL, bandit algorithms, science*

---

## iPython tip

```python
>>> %store study
>>> %store -r study
```

Persist your study across sessions without a database

---

## When NOT to Use Optuna

- You have < 5 hyperparameters → grid search is fine
- Each trial takes < 1 second → random search is fine
- You have domain knowledge → just set them manually

Bayesian optimization pays off with **expensive evaluations**

---

## Links

- Optuna docs: https://optuna.readthedocs.io
- TPE paper: Bergstra et al. 2011 "Algorithms for Hyper-Parameter Optimization"
- Optuna paper: Akiba et al. 2019 (NeurIPS)
- This talk: https://slides.nlpia.org

---

## Try It

```bash
pip install optuna optuna-dashboard

optuna-dashboard sqlite:///my_study.db
```

Live dashboard at `http://localhost:8080`

---

## Questions?

**Hobson Lane**

hobson@corethink.ai

Python User Group — July 2026