Distributed Training Theory

4 selectedDifficulty 4-54 unseenView topic

IntermediateNew

0 answered

4 intermediateAdapts to your performance

Question 1 of 4

120sintermediate (4/10)compute

A model has 1 0^{9} parameters in FP16. In a ring AllReduce, what is the right order-of-magnitude bandwidth payload per worker, ignoring latency?