All-reduce
AIAll-reduce: A collective operation that combines values, such as gradients, from every device and then shares the result back to all of them.
All-reduce: A collective operation that combines values, such as gradients, from every device and then shares the result back to all of them.