benchmark_for_optuna_pr_4631.py

import math
import time
import optuna
import numpy as np


def objective(trial: optuna.Trial, n_params: int) -> float:
    return sum([
        math.sin(trial.suggest_float('param-{}'.format(i), 0, math.pi * 2))
        for i in range(n_params)
    ])


def main():
    storage_url = "mysql+pymysql://optuna:[email protected]:3306/optuna"
    sampler = optuna.samplers.TPESampler(seed=1)
    optuna.logging.set_verbosity(optuna.logging.ERROR)

    for n_params in [5, 30]:
        elapsed = []
        for i in range(5):
            study = optuna.create_study(storage=storage_url, sampler=sampler)

            start = time.time()
            study.optimize(lambda t: objective(t, n_params=n_params), n_trials=100)
            elapsed.append(time.time() - start)

            optuna.delete_study(study_name=study.study_name, storage=storage_url)
        print(f"Elapsed {n_params=}: {np.mean(elapsed):.4f} (+/- {np.std(elapsed):.4f})s", elapsed)


if __name__ == '__main__':
    main()

Setup MySQL database

$ docker run -d --rm -p 3306:3306 -e MYSQL_USER=optuna -e MYSQL_DATABASE=optuna -e MYSQL_PASSWORD=password -e MYSQL_ALLOW_EMPTY_PASSWORD=yes --name optuna-mysql mysql:8.0

Benchmark result

PR 4631

$ python tmp/benchmark_mysql.py
Elapsed n_params=5: 10.1026 (+/- 0.1046)s [9.970459938049316, 10.248955249786377, 10.20033884048462, 10.051590919494629, 10.041691064834595]
Elapsed n_params=30: 35.8279 (+/- 0.1306)s [35.60630702972412, 35.980170011520386, 35.935057163238525, 35.82879400253296, 35.789116859436035]

cache-set-trial-state-values

$ python tmp/benchmark_mysql.py
Elapsed n_params=5: 9.6567 (+/- 0.1974)s [9.326624870300293, 9.673264980316162, 9.870671033859253, 9.574559926986694, 9.838436841964722]
Elapsed n_params=30: 35.8301 (+/- 0.1761)s [35.60757279396057, 35.91744899749756, 35.63586711883545, 36.04937410354614, 35.94012999534607]

I rejected this change due to similar speed as PR 4631 branch.

main

$ python tmp/benchmark_mysql.py
Elapsed n_params=5: 7.6167 (+/- 0.0959)s [7.439380168914795, 7.693116188049316, 7.603490829467773, 7.705552101135254, 7.642206192016602]
Elapsed n_params=30: 32.8131 (+/- 0.1010)s [32.76653695106506, 32.937047719955444, 32.92544412612915, 32.67996597290039, 32.756728172302246]

v3.1.1

$ python tmp/benchmark_mysql.py
Elapsed n_params=5: 9.0216 (+/- 0.1590)s [8.70728611946106, 9.113286256790161, 9.066829919815063, 9.08311128616333, 9.137273073196411]
Elapsed n_params=30: 42.7193 (+/- 0.1940)s [42.79313778877258, 42.459089040756226, 42.88873600959778, 42.52017593383789, 42.93545913696289]

Branch	n_params=5 (Average Time ± Std Deviation) (s)	n_params=30 (Average Time ± Std Deviation) (s)
PR 4631	10.1026 ± 0.1046	35.8279 ± 0.1306
cache-set-trial-state-values	9.6567 ± 0.1974	35.8301 ± 0.1761
main	7.6167 ± 0.0959	32.8131 ± 0.1010
v3.1.1	9.0216 ± 0.1590	42.7193 ± 0.1940

PR 4672

$ python tmp/benchmark_mysql.py
Elapsed n_params=5: 10.6553 (+/- 0.1029)s [10.456530094146729, 10.68004822731018, 10.673655033111572, 10.745635986328125, 10.720831871032715]
Elapsed n_params=30: 38.3893 (+/- 1.0469)s [40.44225525856018, 38.048758029937744, 38.034186124801636, 37.936782121658325, 37.484426975250244]

master ee15fcc95

$ python tmp/benchmark_mysql.py
Elapsed n_params=5: 10.7897 (+/- 0.0556)s [10.832643032073975, 10.73856782913208, 10.873303890228271, 10.728083848953247, 10.77579402923584]
Elapsed n_params=30: 38.0671 (+/- 0.5129)s [37.734352111816406, 39.08030891418457, 37.885014057159424, 37.919699907302856, 37.71600008010864

Branch	n_params=5 (Average Time ± Std Deviation) (s)	n_params=30 (Average Time ± Std Deviation) (s)
master ee15fcc95	10.7897 (+/- 0.0556)	38.0671 (+/- 0.5129)
PR 4672	10.6553 (+/- 0.1029)	38.3893 (+/- 1.0469)