Control Groups (an overview)

 

Marketers frequently must operate in a world of limited knowledge.   In many industries there is little information about customer groups and their unique needs -- the number one factor separating master marketers from the rest is the knowledge they glean from their campaigns.  One of the best ways marketers acquire campaign knowledge is through controlled experiments.

 

Controlled experiments allow you to clearly determine when a campaign is a success.  In addition, they quantify exactly how successful your campaigns are.  They allow you to know what is working.

 

 

 

Many times control groups are sacrificed in the name of generating more sales or saving time.  We understand the pressures to maximize sales, particularly in this economy.  However, we believe that the benefits of control groups far outweigh the small delay in maximizing sales. 

 

If you can’t clearly measure a change in sales, how can you claim responsibility/ credit for the improved results?  If you don’t truly know if your campaign had a positive impact on consumer behavior, did you actually save time by rushing it out the door?  Marketers need to fight for their control groups just as they would fight for their jobs.  In challenging economic times a control group might be the only thing that proves how successful your decisions really are.  

 

Approaches to Control Groups & Testing

 

Simple:
* Control Groups
* A/B Splits with a control group

 

More complex:
* Experimental Design (includes Taguchi method and MVT testing)
* Weighted Designs
* Half-life analysis with staged rolled out

 

What is a control group?

 

A control group is simply a comparable population that does not get the offer of interest.  In theory this is simple—withhold the offer from a certain percentage of randomly selected prospects or customers.  But, in practice, there are several common obstacles:

 

Technical
* Determining the control group for many simultaneous campaigns
* Determining how large the group needs to be
* Determining a control group when campaign messaging varies across segments

 

Organizational

* Sales incentives make control groups unpopular
* Sales territories control access to customers

 

Common Mistakes

 

The most common mistake we see is marketers failing to measure their campaigns in a meaningful way. There are several approaches that can cause challenges:
1. Using opted out groups in place of a control group.  This approach can be risky when the opt-out group is composed of fundamentally different type of customers.  While it depends on the business opt-outs are frequently less engaged with the brand, receives lower level of services, or has lower levels of satisfaction than the opt-in group. 
2. Using year over year comparisons in place of a control group.  This may work in slow changing, insulated markets, but for most products and services, the results are not truly comparable.
3. Comparing to a different campaign.  Even if you ignore all the serious challenges of confounding variables (an extremely large “if”), how do you know the success of the previous campaign without a control group?
4. Using non-selected customers as the control group.  For campaigns where business rules are used to select recipients, the non-recipients are rarely comparable.  (If they were why didn’t you select them?) 
5. Using a control group that is too small.  We find this is especially true for campaigns with low response rates.  Control group size is very sensitive to response rate.  Also if results are needed for different territories, segments, or other characteristics that will also affect control group size.

 

Why it can take time
* Politics.  Many organizations include people who view control groups as a “tax on revenue.”  While we believe that control groups are essentially to refining strategies that ultimately grow revenue, it can take time to build the consensus that control groups are essential.

 

What is the M Squared seasoning?
* Our rapid analysis approach combined with tight project management allows us to help our clients get the best result to as many targets as possible.
      - We can effectively lessen control group size through half-life analysis. This technique uses a staged approach. 

         First the campaign cells are validated (to show the campaign itself drives sales). 

         Then the control cells are remarketing to, minimizing sales loss. 
* We can increase campaign response rates through our experience with a wide variety of methods to prioritize target, including our experience with a range of segmentation schemes and multiple behavioral modeling techniques