We probably only need to merge this or #7839, but not both. This approach makes stronger guarantees about synchronicity of message processing.

CCing @jdfreder, @minrk for comments

This may be a better approach for 4.0 (or later), but I think for 3.x the simplicity and minimal impact of #7839 is preferred. @minrk opinion?

I think this is better than #7839 for the following reasons:

1. in python, there really is an assumption of synchronous execution. This is conceptually simpler than the Handle CommManager messages in order #7839, as it just insists that all message handling be synchronous. We already have messages deserialized synchronously; this makes the entire execution synchronous.
2. It's simpler to reason about the payload handlers if they are also executing synchronously, and this also guarantees that if they return a promise, they are executed before the next messages are processed. That's a nice thing to have.

A message handler doesn't have to be synchronous, of course. If it doesn't return a promise, we won't wait for it. This just makes the default to wait for handling before trying to handle more messages.

After a day of thinking about it, I'm inclined to agree. @jdfreder this PR is smaller than it looks, since ~half of it is actually #7780. The actual changes are here. It's mostly adding returns statements.

Since 7780 was merged, this is probably the best solution.

@jasongrout Can you rebase this one now that #7780 is merged so this accurately represents the diff?

@minrk: rebased.

@jasongrout thanks!