The Intrinsic Relationships between Distributed Computing and Iterative Methods

You can never exchange information in a group of task instances. The only proper way is use a reducer to exchange information.

If your problem is in a divide and conquer manner, you can divide the problem into several partitions and process it with a group of task instance.

But if unfortunately, your problem can not be solved with a one-shoot method in the divide and conquer manner, iteration methods may help you to make the problem in a dc mode.

Iteration methods make you able to solve a problem bit by bit. That's because, of course, solve a bit of a problem is usually much easier than solving the problem completely in a single step.