Last active
February 26, 2021 20:32
-
-
Save lucmos/534c14a3ed7fd6ed651962a41680ccc7 to your computer and use it in GitHub Desktop.
wandb + hydra sweeps
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import logging | |
| import hydra | |
| import wandb | |
| from omegaconf import DictConfig, OmegaConf | |
| log = logging.getLogger(__name__) # Hydra configured | |
| @hydra.main(config_name="testconf") | |
| def run_train(cfg: DictConfig) -> None: | |
| wandb_run = wandb.init( | |
| config=OmegaConf.to_container(cfg, resolve=True), | |
| reinit=True, | |
| group="debuga", | |
| project="foo", | |
| entity="lucmos", | |
| ) | |
| print(f"Started Run {cfg}") | |
| wandb_run.log({"lr": cfg.learning_rate}, step=0) | |
| wandb_run.finish() | |
| print("Finished Run") | |
| if __name__ == "__main__": | |
| run_train() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| learning_rate: 0.2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ❯ PYTHONPATH=. python src/mwe.py -m learning_rate=1,2,3,4 | |
| [2021-02-26 20:41:13,957][HYDRA] Launching 4 jobs locally | |
| [2021-02-26 20:41:13,957][HYDRA] #0 : learning_rate=1 | |
| wandb: Currently logged in as: gladia (use `wandb login --relogin` to force relogin) | |
| wandb: wandb version 0.10.20 is available! To upgrade, please run: | |
| wandb: $ pip install wandb --upgrade | |
| wandb: Tracking run with wandb version 0.10.18 | |
| wandb: Syncing run firm-donkey-15 | |
| wandb: ⭐️ View project at https://wandb.ai/lucmos/foo | |
| wandb: 🚀 View run at https://wandb.ai/lucmos/foo/runs/3dzuonvc | |
| wandb: Run data is saved locally in /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-41-13/0/wandb/run-20210226_204114-3dzuonvc | |
| wandb: Run `wandb offline` to turn off syncing. | |
| Started Run {'learning_rate': 1} | |
| wandb: Waiting for W&B process to finish, PID 814198 | |
| wandb: Program ended successfully. | |
| wandb: | |
| wandb: Find user logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-41-13/0/wandb/run-20210226_204114-3dzuonvc/logs/debug.log | |
| wandb: Find internal logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-41-13/0/wandb/run-20210226_204114-3dzuonvc/logs/debug-internal.log | |
| wandb: Run summary: | |
| wandb: lr 1 | |
| wandb: _runtime 1 | |
| wandb: _timestamp 1614368475 | |
| wandb: _step 0 | |
| wandb: Run history: | |
| wandb: lr ▁ | |
| wandb: _runtime ▁ | |
| wandb: _timestamp ▁ | |
| wandb: _step ▁ | |
| wandb: | |
| wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) | |
| wandb: | |
| wandb: Synced firm-donkey-15: https://wandb.ai/lucmos/foo/runs/3dzuonvc | |
| Finished Run | |
| [2021-02-26 20:41:18,626][HYDRA] #1 : learning_rate=2 | |
| wandb: wandb version 0.10.20 is available! To upgrade, please run: | |
| wandb: $ pip install wandb --upgrade | |
| wandb: Tracking run with wandb version 0.10.18 | |
| wandb: Syncing run young-planet-16 | |
| wandb: ⭐️ View project at https://wandb.ai/lucmos/foo | |
| wandb: 🚀 View run at https://wandb.ai/lucmos/foo/runs/wfg36iwz | |
| wandb: Run data is saved locally in /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-41-13/1/wandb/run-20210226_204118-wfg36iwz | |
| wandb: Run `wandb offline` to turn off syncing. | |
| Started Run {'learning_rate': 2} | |
| wandb: Waiting for W&B process to finish, PID 814317 | |
| wandb: Program ended successfully. | |
| wandb: | |
| wandb: Find user logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-41-13/1/wandb/run-20210226_204118-wfg36iwz/logs/debug.log | |
| wandb: Find internal logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-41-13/1/wandb/run-20210226_204118-wfg36iwz/logs/debug-internal.log | |
| wandb: Run summary: | |
| wandb: lr 2 | |
| wandb: _runtime 1 | |
| wandb: _timestamp 1614368479 | |
| wandb: _step 0 | |
| wandb: Run history: | |
| wandb: lr ▁ | |
| wandb: _runtime ▁ | |
| wandb: _timestamp ▁ | |
| wandb: _step ▁ | |
| wandb: | |
| wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) | |
| wandb: | |
| wandb: Synced young-planet-16: https://wandb.ai/lucmos/foo/runs/wfg36iwz | |
| Finished Run | |
| --- Logging error --- | |
| Traceback (most recent call last): | |
| File "/home/luca/.pyenv/versions/3.8.6/lib/python3.8/logging/__init__.py", line 1084, in emit | |
| stream.write(msg + self.terminator) | |
| OSError: [Errno 9] Bad file descriptor | |
| Call stack: | |
| File "src/mwe.py", line 28, in <module> | |
| run_train() | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/main.py", line 32, in decorated_main | |
| _run_hydra( | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 354, in _run_hydra | |
| run_and_report( | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 198, in run_and_report | |
| return func() | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 355, in <lambda> | |
| lambda: hydra.multirun( | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 136, in multirun | |
| return sweeper.sweep(arguments=task_overrides) | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/_internal/core_plugins/basic_sweeper.py", line 154, in sweep | |
| results = self.launcher.launch(batch, initial_job_idx=initial_job_idx) | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/_internal/core_plugins/basic_launcher.py", line 68, in launch | |
| log.info(f"\t#{idx} : {lst}") | |
| Message: '\t#2 : learning_rate=3' | |
| Arguments: () | |
| wandb: wandb version 0.10.20 is available! To upgrade, please run: | |
| wandb: $ pip install wandb --upgrade | |
| wandb: Tracking run with wandb version 0.10.18 | |
| wandb: Syncing run vocal-shape-17 | |
| wandb: ⭐️ View project at https://wandb.ai/lucmos/foo | |
| wandb: 🚀 View run at https://wandb.ai/lucmos/foo/runs/3ogbq4a7 | |
| wandb: Run data is saved locally in /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-41-13/2/wandb/run-20210226_204123-3ogbq4a7 | |
| wandb: Run `wandb offline` to turn off syncing. | |
| wandb: Waiting for W&B process to finish, PID 814451 | |
| wandb: Program ended successfully. | |
| wandb: | |
| wandb: Find user logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-41-13/2/wandb/run-20210226_204123-3ogbq4a7/logs/debug.log | |
| wandb: Find internal logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-41-13/2/wandb/run-20210226_204123-3ogbq4a7/logs/debug-internal.log | |
| wandb: Run summary: | |
| wandb: lr 3 | |
| wandb: _runtime 0 | |
| wandb: _timestamp 1614368483 | |
| wandb: _step 0 | |
| wandb: Run history: | |
| wandb: lr ▁ | |
| wandb: _runtime ▁ | |
| wandb: _timestamp ▁ | |
| wandb: _step ▁ | |
| wandb: | |
| wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) | |
| wandb: | |
| wandb: Synced vocal-shape-17: https://wandb.ai/lucmos/foo/runs/3ogbq4a7 | |
| --- Logging error --- | |
| Traceback (most recent call last): | |
| File "/home/luca/.pyenv/versions/3.8.6/lib/python3.8/logging/__init__.py", line 1085, in emit | |
| self.flush() | |
| File "/home/luca/.pyenv/versions/3.8.6/lib/python3.8/logging/__init__.py", line 1065, in flush | |
| self.stream.flush() | |
| OSError: [Errno 9] Bad file descriptor | |
| Call stack: | |
| File "src/mwe.py", line 28, in <module> | |
| run_train() | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/main.py", line 32, in decorated_main | |
| _run_hydra( | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 354, in _run_hydra | |
| run_and_report( | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 198, in run_and_report | |
| return func() | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 355, in <lambda> | |
| lambda: hydra.multirun( | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 136, in multirun | |
| return sweeper.sweep(arguments=task_overrides) | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/_internal/core_plugins/basic_sweeper.py", line 154, in sweep | |
| results = self.launcher.launch(batch, initial_job_idx=initial_job_idx) | |
| File "/home/luca/Repositories/line-graph-learning/venv/lib/python3.8/site-packages/hydra/_internal/core_plugins/basic_launcher.py", line 68, in launch | |
| log.info(f"\t#{idx} : {lst}") | |
| Message: '\t#3 : learning_rate=4' | |
| Arguments: () | |
| wandb: wandb version 0.10.20 is available! To upgrade, please run: | |
| wandb: $ pip install wandb --upgrade | |
| wandb: Tracking run with wandb version 0.10.18 | |
| wandb: Syncing run hopeful-eon-18 | |
| wandb: ⭐️ View project at https://wandb.ai/lucmos/foo | |
| wandb: 🚀 View run at https://wandb.ai/lucmos/foo/runs/v7qtg93k | |
| wandb: Run data is saved locally in /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-41-13/3/wandb/run-20210226_204127-v7qtg93k | |
| wandb: Run `wandb offline` to turn off syncing. | |
| wandb: Waiting for W&B process to finish, PID 814571 | |
| wandb: Program ended successfully. | |
| wandb: | |
| wandb: Find user logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-41-13/3/wandb/run-20210226_204127-v7qtg93k/logs/debug.log | |
| wandb: Find internal logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-41-13/3/wandb/run-20210226_204127-v7qtg93k/logs/debug-internal.log | |
| wandb: Run summary: | |
| wandb: lr 4 | |
| wandb: _runtime 1 | |
| wandb: _timestamp 1614368488 | |
| wandb: _step 0 | |
| wandb: Run history: | |
| wandb: lr ▁ | |
| wandb: _runtime ▁ | |
| wandb: _timestamp ▁ | |
| wandb: _step ▁ | |
| wandb: | |
| wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) | |
| wandb: | |
| wandb: Synced hopeful-eon-18: https://wandb.ai/lucmos/foo/runs/v7qtg93k | |
| Exception ignored in: <_io.TextIOWrapper name=1 mode='w' encoding='UTF-8'> | |
| OSError: [Errno 9] Bad file descriptor | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ❯ PYTHONPATH=. WANDB_START_METHOD=thread python src/mwe.py -m learning_rate=1,2,3,4 | |
| [2021-02-26 21:27:30,535][HYDRA] Launching 4 jobs locally | |
| [2021-02-26 21:27:30,535][HYDRA] #0 : learning_rate=1 | |
| wandb: Currently logged in as: gladia (use `wandb login --relogin` to force relogin) | |
| wandb: Tracking run with wandb version 0.10.20 | |
| wandb: Syncing run firm-microwave-23 | |
| wandb: ⭐️ View project at https://wandb.ai/lucmos/foo | |
| wandb: 🚀 View run at https://wandb.ai/lucmos/foo/runs/1fip1ewb | |
| wandb: Run data is saved locally in /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/21-27-30/0/wandb/run-20210226_212730-1fip1ewb | |
| wandb: Run `wandb offline` to turn off syncing. | |
| Started Run {'learning_rate': 1} | |
| wandb: Waiting for W&B process to finish, PID 0 | |
| wandb: Program ended successfully. | |
| wandb: | |
| wandb: Find user logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/21-27-30/0/wandb/run-20210226_212730-1fip1ewb/logs/debug.log | |
| wandb: Find internal logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/21-27-30/0/wandb/run-20210226_212730-1fip1ewb/logs/debug-internal.log | |
| wandb: Run summary: | |
| wandb: lr 1 | |
| wandb: _runtime 1 | |
| wandb: _timestamp 1614371251 | |
| wandb: _step 0 | |
| wandb: Run history: | |
| wandb: lr ▁ | |
| wandb: _runtime ▁ | |
| wandb: _timestamp ▁ | |
| wandb: _step ▁ | |
| wandb: | |
| wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) | |
| wandb: | |
| wandb: Synced firm-microwave-23: https://wandb.ai/lucmos/foo/runs/1fip1ewb | |
| Finished Run | |
| [2021-02-26 21:27:34,678][HYDRA] #1 : learning_rate=2 | |
| wandb: Tracking run with wandb version 0.10.20 | |
| wandb: Syncing run earthy-fire-24 | |
| wandb: ⭐️ View project at https://wandb.ai/lucmos/foo | |
| wandb: 🚀 View run at https://wandb.ai/lucmos/foo/runs/irdqg3iz | |
| wandb: Run data is saved locally in /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/21-27-30/1/wandb/run-20210226_212734-irdqg3iz | |
| wandb: Run `wandb offline` to turn off syncing. | |
| Started Run {'learning_rate': 2} | |
| wandb: Waiting for W&B process to finish, PID 0 | |
| wandb: Program ended successfully. | |
| wandb: | |
| wandb: Find user logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/21-27-30/1/wandb/run-20210226_212734-irdqg3iz/logs/debug.log | |
| wandb: Find internal logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/21-27-30/1/wandb/run-20210226_212734-irdqg3iz/logs/debug-internal.log | |
| wandb: Run summary: | |
| wandb: lr 2 | |
| wandb: _runtime 3 | |
| wandb: _timestamp 1614371257 | |
| wandb: _step 0 | |
| wandb: Run history: | |
| wandb: lr ▁ | |
| wandb: _runtime ▁ | |
| wandb: _timestamp ▁ | |
| wandb: _step ▁ | |
| wandb: | |
| wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) | |
| wandb: | |
| wandb: Synced earthy-fire-24: https://wandb.ai/lucmos/foo/runs/irdqg3iz | |
| Finished Run | |
| [2021-02-26 21:27:41,112][HYDRA] #2 : learning_rate=3 | |
| wandb: Tracking run with wandb version 0.10.20 | |
| wandb: Syncing run dazzling-feather-25 | |
| wandb: ⭐️ View project at https://wandb.ai/lucmos/foo | |
| wandb: 🚀 View run at https://wandb.ai/lucmos/foo/runs/1lbxu0wu | |
| wandb: Run data is saved locally in /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/21-27-30/2/wandb/run-20210226_212741-1lbxu0wu | |
| wandb: Run `wandb offline` to turn off syncing. | |
| Started Run {'learning_rate': 3} | |
| wandb: Waiting for W&B process to finish, PID 0 | |
| wandb: Program ended successfully. | |
| wandb: | |
| wandb: Find user logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/21-27-30/2/wandb/run-20210226_212741-1lbxu0wu/logs/debug.log | |
| wandb: Find internal logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/21-27-30/2/wandb/run-20210226_212741-1lbxu0wu/logs/debug-internal.log | |
| wandb: Run summary: | |
| wandb: lr 3 | |
| wandb: _runtime 1 | |
| wandb: _timestamp 1614371262 | |
| wandb: _step 0 | |
| wandb: Run history: | |
| wandb: lr ▁ | |
| wandb: _runtime ▁ | |
| wandb: _timestamp ▁ | |
| wandb: _step ▁ | |
| wandb: | |
| wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) | |
| wandb: | |
| wandb: Synced dazzling-feather-25: https://wandb.ai/lucmos/foo/runs/1lbxu0wu | |
| Finished Run | |
| [2021-02-26 21:27:45,635][HYDRA] #3 : learning_rate=4 | |
| wandb: Tracking run with wandb version 0.10.20 | |
| wandb: Syncing run winter-wave-26 | |
| wandb: ⭐️ View project at https://wandb.ai/lucmos/foo | |
| wandb: 🚀 View run at https://wandb.ai/lucmos/foo/runs/38e0xl4r | |
| wandb: Run data is saved locally in /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/21-27-30/3/wandb/run-20210226_212745-38e0xl4r | |
| wandb: Run `wandb offline` to turn off syncing. | |
| Started Run {'learning_rate': 4} | |
| wandb: Waiting for W&B process to finish, PID 0 | |
| wandb: Program ended successfully. | |
| wandb: | |
| wandb: Find user logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/21-27-30/3/wandb/run-20210226_212745-38e0xl4r/logs/debug.log | |
| wandb: Find internal logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/21-27-30/3/wandb/run-20210226_212745-38e0xl4r/logs/debug-internal.log | |
| wandb: Run summary: | |
| wandb: lr 4 | |
| wandb: _runtime 1 | |
| wandb: _timestamp 1614371266 | |
| wandb: _step 0 | |
| wandb: Run history: | |
| wandb: lr ▁ | |
| wandb: _runtime ▁ | |
| wandb: _timestamp ▁ | |
| wandb: _step ▁ | |
| wandb: | |
| wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) | |
| wandb: | |
| wandb: Synced winter-wave-26: https://wandb.ai/lucmos/foo/runs/38e0xl4r | |
| Finished Run |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ❯ PYTHONPATH=. python src/mwe.py -m learning_rate=42 | |
| [2021-02-26 20:43:56,351][HYDRA] Launching 1 jobs locally | |
| [2021-02-26 20:43:56,351][HYDRA] #0 : learning_rate=42 | |
| wandb: Currently logged in as: gladia (use `wandb login --relogin` to force relogin) | |
| wandb: wandb version 0.10.20 is available! To upgrade, please run: | |
| wandb: $ pip install wandb --upgrade | |
| wandb: Tracking run with wandb version 0.10.18 | |
| wandb: Syncing run electric-plasma-19 | |
| wandb: ⭐️ View project at https://wandb.ai/lucmos/foo | |
| wandb: 🚀 View run at https://wandb.ai/lucmos/foo/runs/2gkrsggh | |
| wandb: Run data is saved locally in /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-43-56/0/wandb/run-20210226_204356-2gkrsggh | |
| wandb: Run `wandb offline` to turn off syncing. | |
| wandb: Waiting for W&B process to finish, PID 818063 | |
| wandb: Program ended successfully. | |
| wandb: | |
| wandb: Find user logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-43-56/0/wandb/run-20210226_204356-2gkrsggh/logs/debug.log | |
| wandb: Find internal logs for this run at: /home/luca/Repositories/line-graph-learning/multirun/2021-02-26/20-43-56/0/wandb/run-20210226_204356-2gkrsggh/logs/debug-internal.log | |
| wandb: Run summary: | |
| wandb: lr 42 | |
| wandb: _runtime 1 | |
| wandb: _timestamp 1614368637 | |
| wandb: _step 0 | |
| wandb: Run history: | |
| wandb: lr ▁ | |
| wandb: _runtime ▁ | |
| wandb: _timestamp ▁ | |
| wandb: _step ▁ | |
| wandb: | |
| wandb: Synced 4 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) | |
| wandb: | |
| wandb: Synced electric-plasma-19: https://wandb.ai/lucmos/foo/runs/2gkrsggh | |
| Finished Run | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment