Skip to content

Instantly share code, notes, and snippets.

@bityob

bityob/torch-multi.ipynb

Created December 25, 2023 23:47
Show Gist options
  • Save bityob/cab6fa78ef6382d1f55082a60dd6fc62 to your computer and use it in GitHub Desktop.
Save bityob/cab6fa78ef6382d1f55082a60dd6fc62 to your computer and use it in GitHub Desktop.
Download ZIP
Run torch with multiprocessing on Jupyter Notebook (aka IPython, Jupyter Lab etc.)
Display the source blob
Display the rendered blob
Raw
torch-multi.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@bityob
Copy link
Author

bityob commented Dec 25, 2023

References:

GitHub Issues

pytorch/pytorch#17680
pytorch/pytorch#62943
ipython/ipython#10894

StackOverFlow

https://stackoverflow.com/questions/41385708/multiprocessing-example-giving-attributeerror

Demo Repository regarding issues of using multiprocessing and IPython/Jupyter

https://github.com/mszhanyi/pymultiprocessdemo

Also liked the https://pypi.org/project/multiprocess/ fork, but can't use it with pytorch since they have their own multiprocesses fork.

The %%save2file tmp.py and %run tmp.py magic usages, are the most practical solutions I have found for this.

It works just fine now from inside my notebook

@IliasAarab
Copy link

Smart! What about rerunning the last cell without restarting the kernel, is this possible?

@bityob
Copy link
Author

bityob commented Jun 17, 2024

@IliasAarab

Not sure I understand your question.

The last cell doesn't need kernel restart anyway, since it's run on seperate python process with the %run magic.

So you can update the second cell (with the %%save2file tmp.py in top) and then run the third (last) cell and it will run the script on fresh python.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment