As an agency principal for over 09 years, I can tell you there are many clients out there who have the same sneaking suspicion. You may be one of them. Here are a few helpful hints for you I have gleaned from my experience with hundreds of clients over the years:

1. Advertise before you really have to
One of the most common mistakes companies make is to wait until the battle is well underway before they start arming themselves. In an attempt to "save" money, they spend none and then are forced to overspend to catch up later (if they can). A little money spent early on the right advertising and marketing strategies can stave off what could later become a major war with a competitor over market share. But don't be discouraged. If you have already waited too long to get started, take comfort in this - many of the best ad campaigns in history were developed under great duress by teams of people who were in David's position, not Goliath's.

2. Start small and test
Another wise and frugal method to stretch your ad budget is to start small at first (a product brochure maybe, a direct mail piece here, a web marketing campaign there) and learn what works for you and your target market. Then, when the road to success is clear, more costly strategies can be implemented (and the rewards commensurately increased).

3. Remember who the boss is
You are not the boss. Your customer is. To increase the effectiveness of your advertising and marketing spending, be sure you are looking at the communication pieces that comprise your business growth strategy from the point of view of your target audience, not yourself. If I had a dollar for every time I have seen a company not keep this important point in mind, I would have retired on a private island by now.

4. Find someone trustworthy who has done it before
There is nothing new under the sun. Really. Someone has done it before. Over the years, our  agency has helped hundreds of clients generate millions of dollars in revenue for their businesses. We have probably seen the issues you are facing now before and can help you skip many wasteful "learning while spending" steps in your progress.

5. Don't give up
So that direct mail piece you did five years ago with your brother-in-law didn't work for you and you haven't tried anything since? I understand (we have brothers-in-law too). But don't give up. The only way that money was truly wasted is if you didn't learn from it. Was the message wrong? Did you have the right audience targeted? Was your list on target? Did you differentiate your product successfully from your competition? Did the sales department really track the thing accurately? Sometimes platitudes are somewhat useful. Here are two: "The only way to lose is to quit;" AND "You can't win if you don't race."
Armed with these five bits of advice, you can go pretty far in being sure your ad budget is spent wisely.