Although +Jason Paul McCartan and I just recently kicked off our Tenkar & The Badger Podcast, I've been a podcast consumer for the last three years or so, and I have some thoughts on the number of hosts that are idea for a podcast.
First, there is no hard number. Two works well for the Ken and Robin Talk About Stuff. I feel that Happy Jacks often suffers under the weight of "too many voices, too little substance" curse that follows when listeners can't figure out who is talking (or no longer care). Well that and just far too much belching.
When Jason and I finalized plans for the podcast, we decided we wanted a rotating third chair. While we feel we have the chops to have an entertaining show with just the two of us, the third voice adds a dimension we would be lacking otherwise. At the same time, we also like the idea of frequent guests, and our fear was that a regular three plus a guest per episode would leave someone with limited airtime.
Recently, the Save or Die Podcast went from four hosts to three. While I'm not sure if that's a temporary change or a permanent adjustment, the show does seem to flow better. I suspect that it's because the three remaining hosts have more time to make their own points and observations and not because of any particular change in the line up.
Spellburn recently went from three to four hosts, and while it works, it tends to suffer a bit when they add a guest. Not because the guests they have aren't awesome, but now you have a conversation with five folks, and that can get unnecessarily busy and again, runs the risk of leaving someone on the sidelines.
If I do have one complaint, it's the excessive reading of letters on the air. This goes for just about every podcast out there. I find letter reading to be a common crutch on podcasts in the RPG Community. There is no need to read every one, let alone every sentence of each of them. Edit them to the kernel that will drive the discussion - the rest is just annoying filler usually.