Reading an article on VoIP audio quality, it seems that one can increase the quality of VoIP audio by increasing buffering, essentially giving the connection more time to deliver the packets correctly. The tradeoff is that it adds a delay, somewhat like talking to someone in a far-flung country, or via satellite. In the article, the author contends that delays are tolerable but low audio quality is not.
People converse in a fairly simple fashion: A talks, then B talks, then A talks. We do not all talk at once, so synchronisation of speed between A and B is not important.
The idea, then, is that we work out when B starts talking, and send a burst of packets with low latency. The receiver then starts playing these immediately (no delay) but plays the buffer at a lower speed (pitch-adjusted) until it has built up a reasonable buffer. Thus there is a delay between the B talking and A hearing it (ignoring the transmission time), but not an audible silent delay between B starting to talk and A hearing it.
It is the usual game of introducing complexity to handle to competing goals. Is the complexity worth it?