What a great topic! So few companies take the time to look at these things...especially before getting ready to launch a new marketing initiative. Good job.
I'll be happy to share my line of thinking with you, hopefully other folks will too as this is one of those areas where there is so much variation.
- Establishing an initial benchmark data set What's important and where do I find it?
My primary concern when I'm getting ready to market is:
"what is my profit margin"
You need to find out what your current costs are per lead/sale/product, etc... You should ideally know this broken down by different mediums (yellow pages, tv, newspaper, email, etc...)
You'll also want to know what your current conversion rate is. In other words, what percentage of people that visit your site end up becoming customers.
To get even more in-depth...it's great if you can find out your conversion rate by keyword or traffic source.
Does a link from site X convert 45% of the time? If so, you need to put effort toward finding more sites like X.
Do people that find you by typing in blue widget buy 20% of the time? Great! Can you find other similar keywords.
Also, knowing what your amount of profit it. If Site X converts 45% of the time, but only results in $2 profit, then it's not near as valuable as the blue widget keyword that converts have as much but results in say $20 profit.
(I hope I'm not losing you...)
- Evaluating competition ads and site performance for benchmark inclusion
Are they in my ad market, and how is their site doing?
It's hard to do much beyond find out what ads your competitors are purchasing, and maybe, on Overture, finding out how much they are willing to spend. There are some tools for this (Googspy is a great one for finding out what people are bidding on on AdWords) but a lot of it is just time and effort on your part running searches.
As for how their site does, it's unlikely that you can learn about conversion rates, profit levels, etc... most of that is guess work.
- Evaluating competition pay-per-click costs for benchmark inclusion
Am I paying too much for the wrong terms, compared to my competition?
It doesn't matter what your competition is spending. Really, it doesn't. You don't have any way to know what your competitors margin is. You don't know what the cost of their rent is, or their warehouse space, or even what they buy their product for or pay their employees. So what they can afford to pay doesn't mean anything.
You are paying too much if you are paying more than you can afford. It's as simple as that. Your job is to figure out what you can afford to pay based on your conversion rate and your profit margins. Then, your job is to test changes to your campaign until you figure out how to get the highest possible conversion for the lowest possible cost.
- Evaluating competition site traffic
How many visitors are they getting and (ideally) how many "conversions"?
That's a toughie. Very very hard. There are some traffic estimating programs out there and some competitive research programs like HitWise that can give you this, depending on the size competitor you are talking about. But...I've never seen any site or service that could tell you the conversion rate of your competitors.