Benchmark Result Report

Report ID: ecb7024f149c2653a906f600d048eac2658f94bc2cfc8e50bfd7356c0fb1e29b
Kurobako Version: 0.1.3
Number of Solvers: 4
Number of Problems: 2
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Overall Results
Individual Results
Solvers
Problems
Studies

Overall Results

Solver	Borda	Firsts
RandomSampler	0	0
TPESampler	2	1
TPESampler with MedianPruner	2	1
TPESampler with SuccessiveHalvingPruner	4	2

Individual Results

(1) Problem: HPO-Bench-Parkinson

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	TPESampler with MedianPruner (study)	0.010070 +- 0.004299	0.803 +- 0.299	132.749 +- 15.810
1	TPESampler (study)	0.009882 +- 0.003529	0.945 +- 0.354	2.775 +- 0.349
1	TPESampler with SuccessiveHalvingPruner (study)	0.011867 +- 0.003252	0.879 +- 0.279	26.767 +- 10.526
4	RandomSampler (study)	0.015558 +- 0.005880	1.304 +- 0.444	0.614 +- 0.083

(2) Problem: HPO-Bench-Protein

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	TPESampler with SuccessiveHalvingPruner (study)	0.231011 +- 0.009908	17.172 +- 0.744	21.536 +- 8.320
2	TPESampler with MedianPruner (study)	0.232737 +- 0.010008	17.874 +- 0.884	131.829 +- 12.683
2	TPESampler (study)	0.236102 +- 0.014100	18.529 +- 1.390	2.781 +- 0.370
4	RandomSampler (study)	0.255230 +- 0.016135	19.817 +- 1.112	0.624 +- 0.080

Solvers

ID: fba765577d406565f676869bca11c65117476c35f2dbf3d0fc5d9025ffda9a43

recipe:

{
  "name": "RandomSampler",
  "optuna": {
    "sampler": "random",
    "pruner": "nop"
  }
}

specification:

{
  "name": "RandomSampler",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.0.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 870a5088fd88d2fb78e09b23b5a123f76e562a194c038eccd5b5e51bd95931cf

recipe:

{
  "name": "TPESampler",
  "optuna": {
    "pruner": "nop"
  }
}

specification:

{
  "name": "TPESampler",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.0.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 679fa4abbe034c1371f96080e14c0796e5257021868dc8e7c666cc25929bca75

recipe:

{
  "name": "TPESampler with MedianPruner",
  "optuna": {
    "median_warmup_steps": 4
  }
}

specification:

{
  "name": "TPESampler with MedianPruner",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.0.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: cc04a6c5ae0a5d76bbfd007aa9f3ca55b672b949760b3f92a4b390bfc52195c0

recipe:

{
  "name": "TPESampler with SuccessiveHalvingPruner",
  "optuna": {
    "pruner": "asha",
    "asha_min_resource": 4
  }
}

specification:

{
  "name": "TPESampler with SuccessiveHalvingPruner",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.0.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: 4f3ab413a790160a528bbc3790c3a332868e4bd2268005108644e09e2e3968b8

recipe:

{
  "hpobench": {
    "dataset": "fcnet_parkinsons_telemonitoring_data.hdf5"
  }
}

specification:

{
  "name": "HPO-Bench-Parkinson",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.3"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}

ID: 81bde99ec2ab16438bf4c7145468df6993c53913a034bd3ee4ee65fdbf1a3bcf

recipe:

{
  "hpobench": {
    "dataset": "fcnet_protein_structure_data.hdf5"
  }
}

specification:

{
  "name": "HPO-Bench-Protein",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.3"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}

Studies

ID: 12e94aa3b91bc181e091a67a1725adbc21d6f6fc838f8eaaa1a4fb1d3b3cbadf

problem: HPO-Bench-Parkinson
solver: RandomSampler
budget: 80
repeats: 30
concurrency: 1

ID: 431414b3e295d2647ebca78d922daa39b5091de27f73d648986e5589a8c4a222

problem: HPO-Bench-Parkinson
solver: TPESampler
budget: 80
repeats: 30
concurrency: 1

ID: 09968fca2def713da7c876682aa0fee4ab350fdec71388d2890bb8ac4106168e

problem: HPO-Bench-Parkinson
solver: TPESampler with MedianPruner
budget: 80
repeats: 30
concurrency: 1

ID: 4ddb19c7790a16c70fa771b63ff831577b3db6aef7c12fdb75d319b8a6457922

problem: HPO-Bench-Parkinson
solver: TPESampler with SuccessiveHalvingPruner
budget: 80
repeats: 30
concurrency: 1

ID: 28a919018ae3856bd6ba62cb273fabd521827b078b2b482ccaf2db1acda6b977

problem: HPO-Bench-Protein
solver: RandomSampler
budget: 80
repeats: 30
concurrency: 1

ID: d8e15e76a9cd34a3ca46921edc1d6afdda0c3f1a0dc3b3060ec0bf445576aebe

problem: HPO-Bench-Protein
solver: TPESampler
budget: 80
repeats: 30
concurrency: 1

ID: feb526ce52a481282f4074285bc3c6ddcecccb6b2902325c162a9d750697bec8

problem: HPO-Bench-Protein
solver: TPESampler with MedianPruner
budget: 80
repeats: 30
concurrency: 1

ID: ec52b06a2a3e5529ceeab40e6e7ac97cda7afd3621ee62901b9d29c105348298

problem: HPO-Bench-Protein
solver: TPESampler with SuccessiveHalvingPruner
budget: 80
repeats: 30
concurrency: 1

sile/report.md

Benchmark Result Report

Table of Contents

Overall Results

Individual Results

(1) Problem: HPO-Bench-Parkinson

(2) Problem: HPO-Bench-Protein

Solvers

ID: fba765577d406565f676869bca11c65117476c35f2dbf3d0fc5d9025ffda9a43

ID: 870a5088fd88d2fb78e09b23b5a123f76e562a194c038eccd5b5e51bd95931cf

ID: 679fa4abbe034c1371f96080e14c0796e5257021868dc8e7c666cc25929bca75

ID: cc04a6c5ae0a5d76bbfd007aa9f3ca55b672b949760b3f92a4b390bfc52195c0

Problems

ID: 4f3ab413a790160a528bbc3790c3a332868e4bd2268005108644e09e2e3968b8

ID: 81bde99ec2ab16438bf4c7145468df6993c53913a034bd3ee4ee65fdbf1a3bcf

Studies

ID: 12e94aa3b91bc181e091a67a1725adbc21d6f6fc838f8eaaa1a4fb1d3b3cbadf

ID: 431414b3e295d2647ebca78d922daa39b5091de27f73d648986e5589a8c4a222

ID: 09968fca2def713da7c876682aa0fee4ab350fdec71388d2890bb8ac4106168e

ID: 4ddb19c7790a16c70fa771b63ff831577b3db6aef7c12fdb75d319b8a6457922

ID: 28a919018ae3856bd6ba62cb273fabd521827b078b2b482ccaf2db1acda6b977

ID: d8e15e76a9cd34a3ca46921edc1d6afdda0c3f1a0dc3b3060ec0bf445576aebe

ID: feb526ce52a481282f4074285bc3c6ddcecccb6b2902325c162a9d750697bec8

ID: ec52b06a2a3e5529ceeab40e6e7ac97cda7afd3621ee62901b9d29c105348298