Marketing & Communications Centre > Marketing Essentials > Help Sheets > How to Test

How to Test

The help sheet Relating to Contacts – Testing looked at what testing was, and how community groups could use it to improve the responses of their direct mail-out campaigns.

Testing involves making a change to some part of your direct mail out – be it the wording of your letter or the contents of your mail pack - and then trying (or testing) it on a small pre-defined segment of your database or contact list.

Following that, your group measures any impact this change may have in comparison to the remainder of your mail-out who received your group's "normal" mail-out.

If there is some positive change – an increased response rate to your direct mail-out, or a larger number of donations, for example – your group can then consider integrating that change permanently into your future mail-outs.

Testing is another tool that is aimed at increasing the response to your group's direct mail-outs, as well as being a way to improve the content and style of your mail-out campaigns to make them more efficient, personal and attractive to recipients.

Testing is a process that is somewhat scientific, with rules and steps that need to be followed to ensure its effectiveness. This help sheet will provide your community group with a run-down of the mechanics of testing, as well as outlining a basic guide on how to pull off a successful testing exercise.


The Mechanics of Testing

Next time you are going to carry out a direct mail-out to those on your database:
  • Extract 10% of the names on the list - ensuring their profile is representative of that of the wider group. This group of names can be called the "test sample" or "test group".
  • The remaining 90% of your database or contact list should receive a mail-out in a format that has worked well in the past. This group can be called the "control group" or "control sample".
  • The 10% of people in your "test group" should receive the same communication piece as the "control group", but with one element changed.
  • When the responses come in, analyse and measure them to determine whether it was the test or control group that pulled in the higher level of response.
  • The aim is for your test piece to "beat" the control piece in responses – be they inquiries, replies, registrations, donations, memberships, raffle ticket sales, etc.

If your group wishes to continue testing, what it can then do is:
  • Re-designate your successful "test piece" as your new "control piece".
  • Decide on another possible change to your direct mail campaign you wish to test.
  • Choose another 10% of your database, again ensuring their profile is representative of that of the wider group. This is your new "test sample", while the remaining 90% of your list is your new "control sample".
  • The "control group" receives your new "control piece", while your "test group" receives your new "test piece" – which is, again, the same communication as the "control group" receive but with one change.
  • Again, when the responses come in, analyse and measure them to find out if "test" has beaten "control". If so, you have found another change you can make to your direct mail-out that has received positive responses through testing.



What to Remember Before you Test

The single most important thing to remember before you embark on a testing exercise is to be clear on what you are doing.

Preparation is a key – make sure your group is clear on:
  • What you are testing.
  • Why you are testing it.
  • How it is being tested.
  • How you are going to analyse the results of your testing, and.
  • What benchmark you are going to use to decide if any changes can be or need to be made.
Also, have your test sample/segment selected, and make sure its characteristics match those of your wider database or contact list – this ensures your testing is comparing "apples with apples".


Analysing your Testing and its Results

Some questions your group may wish to ask itself when analysing its testing and the results it has brought about are:
  • Was the test executed as planned?    
  • Which samples or groups performed better than others?    
  • Were these samples or groups contacted through direct mail at the same time?
  • Were there any other variables that need to be taken into account that could have affected results – a hold up in the post, for example, or rain damaging mail-out packs which had been delivered to letterboxes?



Tracking Your Testing

Your group needs to make sure it can track the testing it does, and has clearly differentiated between those contacts in the "test group" and those in the "control group."

This can be achieved easily by just noting the names of those contacts on your database you have put in your "test group" before you send the direct mail, and then keeping track of their responses in comparison to those in the "control group".

Another way of doing this is actually putting a mark or asterisk next to their name in your database itself – be it stored on computer or otherwise. That way you have flagged those people as having been in the "test group" for a particular mail-out from your group.

If your group is doing multiple batches of testing across a number of direct mail campaigns, it should also differentiate between those who made up the "test group" in each separate campaign. Again, it could be as easy as marking their name with a number: "1" for the first mail out, "2" for the second, and so on.

And if one of your contacts has been in the "test group" for one mail out, it does not exempt them from being in the "test group" for any subsequent mail-outs. Exempting their name from future tests can skew your data and even invalidate a test.


A Final Word

The more testing you do, the more clearly you will understand your contacts and their needs and desires – which means your direct mail campaigns will become better targeted.

All information is valuable in direct mailing. If your campaign doesn't achieve the results you want, you will have valuable pointers on what strategies or tactics to avoid in future.



The information contained on this site is subject to change. Our Community will not be liable for any loss or damage whatsoever coming from reliance placed on all or part of its contents.