Jason Snell has some podcasting advice:
If you’re just starting out, don’t allow yourself to be intimidated by all this audio talk. If you have something to say, say it.
I don’t deny that I’ve heard some pretty awful sounding podcasts in my day. Audio quality does matter. I’d just argue that beyond a certain point, it only matters to audio snobs.
When I was the first guest on the new version of The Talk Show, my audio wasn’t horrible, but it could have been better. Someone left a one-star iTunes comment that read something like “It’s hard to take this new version of the show seriously when the guest can’t afford a decent microphone.” (It was actually a fine microphone that I just had it on the wrong setting, but thanks for implying that only people who can afford high-end audio equipment have voices that matter.)
Try to sound good. Be open to constructive criticism and suggestions. But don’t bust your ass trying to please jerks.