I would also recommend looking around at local credit unions. One of the things I see with our bookkeeping clients a lot is they end up spending a lot of money in ATM fees and card fees and overdraw fees and account usage fees and interest fees. It's a never ending list.
But a credit union is owned and run by the members (ie; you) so the fees tend to be much more reasonable. Plus, the credit unions don't seem to have been hit by the credit crunch nearly as hard as the banks.
The other thing I would recommend you do is look at what services you might be needing in the future for your business - will you be accepting credit cards? If so, you might ask about the bank/credit union's merchant services program and compare those rates with other banks.
Also, if you think you might be needing a loan at some point, then it may be worth checking to see if they have a Small Business Loan department. If not, you may be forced to get a loan through a different bank (which means making payments is less convenient) or you may need to take a personal loan out for the business.
Good luck hunting!