Skip to content

Instantly share code, notes, and snippets.

@dasayan05

dasayan05/sync_gradients.py

Created February 27, 2019 10:20
Show Gist options
  • Save dasayan05/9b9cdc4a272500ecbfa648dd54d5fbde to your computer and use it in GitHub Desktop.
Save dasayan05/9b9cdc4a272500ecbfa648dd54d5fbde to your computer and use it in GitHub Desktop.
Download ZIP
Synchronize gradients
Raw
sync_gradients.py
def sync_gradients(model, rank, world_size):
for param in model.parameters():
dist.all_reduce(param.grad.data, op=dist.reduce_op.SUM)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment