A/B Testing – Does it really matter?

A/B Testing is still a relatively new development and it has emerged as traditional ways of marketing and focusing on the opinion of the HiPPO (Highest Paid Person with an Opinon) are less effective on customers. A/B testing allows companies to compare differences to a website, email, or other digital media such as changing the color of buttons or changing the layout. The article Listen to Your Customers not your HiPPO, discussed how quite simply A/B testing is a controlled experiment that has “two variants: control (A), or treatment (B)”. After comparing the differences between the pieces of digital marketing, marketers can analyze how it affects the overall decisions or purchases their customers make. I couldn’t resist adding in this funny A/B testing joke (see cheesy joke below) because I feel like it accurately describes A/B testing and the comparison between differences. Let’s hope the guy below decided to buy her something from Tiffany & Co. to apologize instead of using marketing puns.
ab testing The Optimizely site defined A/B testing as, “a simple way to test changes to your page against the current design and determine which ones produce positive results. It is a method to validate that any new design or change to an element on your webpage is improving your conversion rate before you make that change to your site code.”
So why is A/B testing important to companies? Why does it matter whether the button is green or red or on the right or left side of the page? As illustrated below, A/B testing is important because it can create more conversions and ultimately it can help bring in more revenue for your company, an essential goal of many but not all businesses. Ultimately, visual aesthetics, appealing layouts, and colors choices that capture the attend of customers is essential in digital media and can drastically influence a company’s conversion rate.ab explainedDue to the shift of focusing on the customer, the HiPPO (Highest Paid Person’s Opinion) is no longer the most important opinion when making marketing decisions, and the customer and what the data collected from A/B testing experiments is becoming increasingly important. Various aspects of digital content can be A/B tested and there are few aspects that can’t be tested leading to testing various content on viewers. A/B testing can be performed on the following content: headlines, sub-headlines, paragraph text, testimonials, call to action text, call to action button, links, images and media models. This is not an exhaustive list and there are other forms of content that can be A/B tested, however I’m going to focus on the content mentioned above in the rest of the post.

During Obama’s re-election campaign in 2012, his team focused on digital content, emails in particular, and utilized A/B testing to determine how layout visual aesthetics, and wording influenced their audience to donate to the campaign. The campaign team also tested subject lines in the emails they sent out to see which subject lines were more effective.

Here are some examples of the A/B testing the campaign team used:


Though a mere 5% increase between the two versions may seem small, when talking about hundreds of millions of dollars it does make a difference. Something essential the campaign team used when performing A/B testing was ensuring the audience for both views were randomly selected. This is something important that businesses need to implement when using A/B testing – the individuals who are a part of their A/B testing need to be randomly selected. Also, the participants of the A/B testing should ideally be new customers/ visitors not returning customers/ visitors. They should be new visitors because they haven’t seen your previous content and they won’t observe a change. If your company used returning visitors to test on and decided to revert to their old formatting and content, they would confuse their users. As shown below, the two sites differ drastically from each other and would definitely catch the attention of returning users.


Another implementation suggestion for businesses, generate personal content compared to generic and bland content because it will draw a larger audience and will also help the maintain this as well.

The key takeaway from the Obama re-election campaign is sending out emails at the same volume as the campaign team did will not benefit your organization, but will most likely hurt it. Why? Receiving countless numbers of emails will annoy customers and could simply remind them to unsubscribe for your email list. Personally, I’ve unsubscribed from email lists because of the sheer volume of emails I received (roughly 3 emails a day) and it was simply based on the volume not on the content of the email. My advice, limit the number of emails you send out to your email list and make sure the content you’re sending them is important, engaging, and straight to the point.

Until next time,



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s