Testing and your Direct Mail-Out

Because community groups have limited money, time and resources, groups want to maximise the success of any direct marketing campaign. That means that you don't want to send out thousands of letters when it's obvious that the first 200 that go out have not elicited the response you want.

The best campaigns are those where groups test the market and tweak their campaign until they come up with a letter or direct mail package that works better than the original.


What is Testing?

Testing is a four-part concept which involves your group or organisation:
  1. Firstly making a change to an element of your direct mail-out approach. Some examples of this could be slight changes to the wording of your letter, or an alteration to the content of your mail-out pack. The next step is
  2. Trying it out on a small section or segment of those on your database – maybe 5 or 10 per cent of them - before
  3. Measuring any impact the change may have, in comparison to the unaltered portion – maybe increase response rates, membership renewals or donations, and
  4. Finally considering incorporating the change into your normal direct mail-out approach.

The four parts can be remembered easily - change, analyse, measure, incorporate.

Testing needs not only a sample which has been changed in some way (known as the "test group"), but a portion, usually larger, that is unchanged (this is known as the "control group") to compare it against.

For example, if you have 200 donor contacts on your group's database, you could look to change your direct mail-out fundraising campaign to maybe 10% or even 25% of your list (20 or 50 donors).

If, after the donations have arrived, you find the test group has donated, on average, 5% more money per donor than those in the control group, your organisation could then consider incorporating the change across your whole donor list for your next mail-out.


Why Should Community Groups Test?

The main reason to test is to increase the response to your direct mail outs.
  • Testing can also streamline your mail-out efforts, and help your group save its hard-earned money through better targeted mail-outs.
  • Testing can help your direct mail campaigns keep up with changing trends, preferences or altered list characteristics.
  • Testing gives a more accurate reflection of people's behaviour – it shows how people actually do behave rather than what they say they will do.
  • It can result in better relationships with contacts through tested techniques that have been shown to work. Alternatively, it also helps eliminate campaigns or ideas that have been shown not to work in testing before they damage or impact on your entire list of contacts.
  • Testing helps plan and define future mail-out campaigns.
  • It literally tests out any previous findings or beliefs – validating them or debunking them.
  • Finally, it can help your group find new, creative, dynamic or attractive ways of getting its message out there.



What to Test?

What your group or organisation can test is only limited by its usefulness to your organisation, your ability to test it, and your imagination. There are literally hundreds of mail-out elements you can test – some are trivial (and should be avoided) and some more major.

Some of the elements of your direct mail-out that could be worth testing include:
  • Creative elements: The appearance of the elements in your direct mail-out – everything from the lay-out of your main letter to colour or mono printing to the headline on your envelope.
  • Subject/contents of mail-out: It could involve testing the format, for example, checking whether your response rate is higher with a brochure plus letter, or with a letter alone.
  • Attractiveness of offer: Simply put, does what you are offering (possibly in terms of a function or event) seem interesting and attractive enough for people to attend?
  • Effectiveness of incentives: Can involve looking at prizes offered in a raffle or benefits received for signing up another member.
  • Timing of mail-out: Is your campaign going out at the right time of year to get the best possible response?

The main point to remember is to be very clear on what your group is actually testing, why it is testing this element and what it is testing against.

Once you know these things, stay focussed on them and don't be tempted to deviate and test something else as well – you could end up with too many different elements between test and control groups, leaving your group unable to pin down what element in particular has made a difference.


Top Testing Tips

Following are some tips and things to remember when it comes to executing a successful testing exercise.
  • Test one thing at a time to ensure you get a clean result.
    • For example, if you want to know whether one headline is better than another, create two different mail packs whose only difference is the headline.
  • The sample size you choose for your test needs to be representative of your list as a whole, so use a decent sample or segment size.
    • As a rule of thumb, use a sample size of 10% of your database.
    • Trust your own judgement: if 10% is too small or too large, adjust the percentage.
    • At the same time, only expose a small segment of your database to the test. That way, if the test is a failure, your market is unaffected.   
  • Make sure your test segment and control segment share a similar profile.
    • If they are too different, the test is not a level one and will be invalid.
  • Benchmark.
    • Be clear on what you control sample is, or what you are benchmarking your test against.
  • Test the big things, and test one element at a time.
    • Stay away from trivial tests. Testing minor changes is a waste of time, effort and money that could be spent on testing larger or more significant changes.
    • A test becomes invalid if the test mail-out and control mail-out are too different from one another, because you won't be able to work out which particular element affected the response rate.
  • Plan a range of tests.
    • Develop a few tests your group can carry out, and prioritise them over a period of time.
  • You have to analyse and measure.
    • Testing is a useless process if your group does not then analyse the results and measure their impacts.
    • Define your results well, and be precise about what you have learned.
  • Act on it.
    • When your test shows you that your group has, without a doubt, found a successful change it can make to its mail-out move quickly to roll it out to the rest of those on your database.
  • Testing is a continuous process.
    • Things evolve over time, so your testing should be continuous to reflect that.

For more information on how to successfully test, including analysing your testing, refer to the How to Test help sheet, available at the Media & Marketing Centre on the Our Community website.