Benchmark Result Report

Report ID: 4ba0c3ff34e08ddaac04387c13628701e21fd683a713b496b3f78c3d541beac9
Kurobako Version: 0.1.3
Number of Solvers: 3
Number of Problems: 4
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Overall Results
Individual Results
Solvers
Problems
Studies

Overall Results

Solver	Borda	Firsts
Hyperband (n_brackets=3)	1	4
Hyperband (n_brackets=4)	1	4
Hyperband (n_brackets=5)	0	3

Individual Results

(1) Problem: HPO-Bench-Parkinson

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	Hyperband (n_brackets=4) (study)	0.010274 +- 0.002866	0.593 +- 0.168	77.627 +- 4.032
1	Hyperband (n_brackets=3) (study)	0.010998 +- 0.004389	0.610 +- 0.216	78.957 +- 4.106
1	Hyperband (n_brackets=5) (study)	0.010452 +- 0.003867	0.683 +- 0.214	76.188 +- 3.679

(2) Problem: HPO-Bench-Naval

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	Hyperband (n_brackets=4) (study)	0.000088 +- 0.000083	0.015 +- 0.016	78.439 +- 3.745
1	Hyperband (n_brackets=3) (study)	0.000113 +- 0.000151	0.023 +- 0.034	81.952 +- 5.304
1	Hyperband (n_brackets=5) (study)	0.000095 +- 0.000095	0.023 +- 0.026	76.634 +- 3.675

(3) Problem: HPO-Bench-Protein

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	Hyperband (n_brackets=4) (study)	0.229968 +- 0.008488	10.355 +- 0.517	77.086 +- 4.335
1	Hyperband (n_brackets=3) (study)	0.229168 +- 0.006514	10.320 +- 0.377	78.119 +- 4.569
3	Hyperband (n_brackets=5) (study)	0.235814 +- 0.011201	10.631 +- 0.555	75.984 +- 3.760

(4) Problem: HPO-Bench-Slice

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	Hyperband (n_brackets=4) (study)	0.000360 +- 0.000145	0.021 +- 0.008	77.230 +- 5.050
1	Hyperband (n_brackets=3) (study)	0.000315 +- 0.000083	0.018 +- 0.006	79.189 +- 5.027
1	Hyperband (n_brackets=5) (study)	0.000310 +- 0.000119	0.022 +- 0.011	75.585 +- 5.596

Solvers

ID: e7f3a145b144a9a3b2bc99b714e4e9f698e8a67f82ece410de77e2abd496c4de

recipe:

{
  "name": "Hyperband (n_brackets=3)",
  "command": {
    "path": "python",
    "args": [
      "hyperband-solver.py",
      "--n-brackets",
      "3"
    ]
  }
}

specification:

{
  "name": "Hyperband (n_brackets=3)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=0.19.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: a04abee366e2d572f34e07fbf80e1796ab42e10a0872755487a1de10a9946a5b

recipe:

{
  "name": "Hyperband (n_brackets=4)",
  "command": {
    "path": "python",
    "args": [
      "hyperband-solver.py",
      "--n-brackets",
      "4"
    ]
  }
}

specification:

{
  "name": "Hyperband (n_brackets=4)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=0.19.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: f94c12441e4b8915be38df76527f83b9094787327a83115a835e8a20e97385e3

recipe:

{
  "name": "Hyperband (n_brackets=5)",
  "command": {
    "path": "python",
    "args": [
      "hyperband-solver.py",
      "--n-brackets",
      "5"
    ]
  }
}

specification:

{
  "name": "Hyperband (n_brackets=5)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=0.19.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: 880b007e3c61f4aecb7e9b0aa2f9be5fea9f491a076853f68f402769aa254034

recipe:

{
  "hpobench": {
    "dataset": "./fcnet_naval_propulsion_data.hdf5"
  }
}

specification:

{
  "name": "HPO-Bench-Naval",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.3"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}

ID: 445bfa45fdbb8ec6ae6d4dba1909114f3948fa67b47209258db9291480b405b5

recipe:

{
  "hpobench": {
    "dataset": "./fcnet_parkinsons_telemonitoring_data.hdf5"
  }
}

specification:

{
  "name": "HPO-Bench-Parkinson",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.3"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}

ID: add73d4788d7900b34988a8b91cde43e820cac99f9e354e1e71b0ea0be3ef4a6

recipe:

{
  "hpobench": {
    "dataset": "./fcnet_protein_structure_data.hdf5"
  }
}

specification:

{
  "name": "HPO-Bench-Protein",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.3"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}

ID: d88e1704447639bde17f236f7af47f93274d1f02bc8ec66733146ff9cdf50196

recipe:

{
  "hpobench": {
    "dataset": "./fcnet_slice_localization_data.hdf5"
  }
}

specification:

{
  "name": "HPO-Bench-Slice",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.3"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}

Studies

ID: 1ec54571e58d20e8e37e8f6278873dfbd3e2d394a5bbc6836ac2b46475ea2efc

problem: HPO-Bench-Naval
solver: Hyperband (n_brackets=3)
budget: 50
repeats: 30
concurrency: 1

ID: db7879f140bd6bbd39e5dc3da4d2359977b2590a0b25b1de6ba18538b54bf423

problem: HPO-Bench-Naval
solver: Hyperband (n_brackets=4)
budget: 50
repeats: 30
concurrency: 1

ID: 7e08a69a7d165bf1eb3343c72d6aae203c3a42997161a528a0b7414699aa8ba2

problem: HPO-Bench-Naval
solver: Hyperband (n_brackets=5)
budget: 50
repeats: 30
concurrency: 1

ID: 2d2379ebcdd6c97581709feca19edcf633f0199e4f162bb5d05a5774c5485d75

problem: HPO-Bench-Parkinson
solver: Hyperband (n_brackets=3)
budget: 50
repeats: 30
concurrency: 1

ID: f2017cf186fd4f7977b8027fda44d7f345ac36bdcb03fd3d8c0420af66c5eb2f

problem: HPO-Bench-Parkinson
solver: Hyperband (n_brackets=4)
budget: 50
repeats: 29
concurrency: 1

ID: ccf2040fe63eb90c4cdac1ec7f502d24cf602878ad1bc7ace3d9912510397f5d

problem: HPO-Bench-Parkinson
solver: Hyperband (n_brackets=5)
budget: 50
repeats: 30
concurrency: 1

ID: bf18cc1fdd3930db2a02e8cedd6e01ef9a884f2a0b6db768b0825464eb5a1d29

problem: HPO-Bench-Protein
solver: Hyperband (n_brackets=3)
budget: 50
repeats: 30
concurrency: 1

ID: c3d7e3ac7ef91c2f119ec2b985e5bff193012a5c51d2695f38f8afa2c483de88

problem: HPO-Bench-Protein
solver: Hyperband (n_brackets=4)
budget: 50
repeats: 30
concurrency: 1

ID: 5a1f46e2d15ebf39196563225d9fc43d76756a127e5f4d0815e0ff7db30025f2

problem: HPO-Bench-Protein
solver: Hyperband (n_brackets=5)
budget: 50
repeats: 30
concurrency: 1

ID: 51e26ad115a26c2b7fc086be41a7137036b22b4b5775ed318e6643332db7b3ce

problem: HPO-Bench-Slice
solver: Hyperband (n_brackets=3)
budget: 50
repeats: 30
concurrency: 1

ID: 8610e788ca37c18a6b098df57b392a63fa8da7d9e18a7d173abd209040d044d1

problem: HPO-Bench-Slice
solver: Hyperband (n_brackets=4)
budget: 50
repeats: 30
concurrency: 1

ID: 1f2c8bfe46db290a36a96b630310be009198e2d9305396d21376c1cc21c6c4e8

problem: HPO-Bench-Slice
solver: Hyperband (n_brackets=5)
budget: 50
repeats: 30
concurrency: 1

sile/_main.md

Benchmark Result Report

Table of Contents

Overall Results

Individual Results

(1) Problem: HPO-Bench-Parkinson

(2) Problem: HPO-Bench-Naval

(3) Problem: HPO-Bench-Protein

(4) Problem: HPO-Bench-Slice

Solvers

ID: e7f3a145b144a9a3b2bc99b714e4e9f698e8a67f82ece410de77e2abd496c4de

ID: a04abee366e2d572f34e07fbf80e1796ab42e10a0872755487a1de10a9946a5b

ID: f94c12441e4b8915be38df76527f83b9094787327a83115a835e8a20e97385e3

Problems

ID: 880b007e3c61f4aecb7e9b0aa2f9be5fea9f491a076853f68f402769aa254034

ID: 445bfa45fdbb8ec6ae6d4dba1909114f3948fa67b47209258db9291480b405b5

ID: add73d4788d7900b34988a8b91cde43e820cac99f9e354e1e71b0ea0be3ef4a6

ID: d88e1704447639bde17f236f7af47f93274d1f02bc8ec66733146ff9cdf50196

Studies

ID: 1ec54571e58d20e8e37e8f6278873dfbd3e2d394a5bbc6836ac2b46475ea2efc

ID: db7879f140bd6bbd39e5dc3da4d2359977b2590a0b25b1de6ba18538b54bf423

ID: 7e08a69a7d165bf1eb3343c72d6aae203c3a42997161a528a0b7414699aa8ba2

ID: 2d2379ebcdd6c97581709feca19edcf633f0199e4f162bb5d05a5774c5485d75

ID: f2017cf186fd4f7977b8027fda44d7f345ac36bdcb03fd3d8c0420af66c5eb2f

ID: ccf2040fe63eb90c4cdac1ec7f502d24cf602878ad1bc7ace3d9912510397f5d

ID: bf18cc1fdd3930db2a02e8cedd6e01ef9a884f2a0b6db768b0825464eb5a1d29

ID: c3d7e3ac7ef91c2f119ec2b985e5bff193012a5c51d2695f38f8afa2c483de88

ID: 5a1f46e2d15ebf39196563225d9fc43d76756a127e5f4d0815e0ff7db30025f2

ID: 51e26ad115a26c2b7fc086be41a7137036b22b4b5775ed318e6643332db7b3ce

ID: 8610e788ca37c18a6b098df57b392a63fa8da7d9e18a7d173abd209040d044d1

ID: 1f2c8bfe46db290a36a96b630310be009198e2d9305396d21376c1cc21c6c4e8