In this edition of #ThrowbackThursday, I need to jump back to my discussion about Google Website Optimizer in September 2011. In 2012, Google closed down their Website Optimizer standalone product and merged its features into Google Analytics. They now call it Experiments instead of Website Optimizer.
The old Optimizer was designed to handle A/B testing, which is when you feed two versions of the same web page to your users in order to discover which version has the higher probability of leading them to your target goal.
Sometimes that goal is a purchase, and sometimes it's just to convince people to stay on your site a little longer. The problem with the old Website Optimizer is that it was only able to test two versions of the same web page at a time. Meanwhile, other testing companies had methods of testing smaller components within a single page, perhaps small one word changes, or subtle color changes. These smaller changes are known as multivariate testing. These multi-variations might sometimes lead to dozens of variable combinations.
With the launch of Google Analytics Experiments, Google now has the ability to test up to 10 versions of the same page, effectively creating a multivariate test that allows for those subtle changes of words, colors, or other design elements.
Setting Up an Experiment
To begin the setup, you first log into your Google Analytics account then navigate to Behavior -> Experiments as shown here:
This brings you to the main Experiments page. Click on the "Create experiment" button to start the setup of your test.
You'll then see this Content Experiment setup screen:
Give the experiment a name, then click the drop down box to select a metric to test as shown here:
Testing for Bounces will be a simple way to measure the effectiveness of two different home pages. Lowering the bounce rate of your home page is a simple way to improve your website results.
Testing for Pageviews will show you which starting page variation lead to a deeper engagement in your website.
Testing for Session Duration will measure how long people stay on your website based on a different start page variation.
Side Note for Advanced Users: Any previously set up Goals will also appear in this list.
Next, you will choose the percentage of traffic that you want to see this experiment as you see here:
If your website has a lot of traffic and especially if e-commerce is involved, you might want to limit the number of people who see this alternative page. This limitation is especially useful if you are testing out large changes, like a completely new home page design that you only want 5% or 10% of your visitors to see. No sense disorienting all your users in that scenario.
However, since most jewelers have a limited amount of traffic, I do suggest leaving it at 100%; otherwise it might take a very long time before you can measure results.
You can let this experiment send you email updates during the progress of the experiment. These come in handy when you are running longer experiments that you don't want to risk forgetting about.
Next click the blue Advanced Options link to open these settings:
The option to "Distribute traffic evenly across all variations" allows the experiment to choose how people see the test variations. While it might seem logical to evenly distribute traffic, doing so will cause the experiment to take a while longer to discover results.
The next option allows you to choose between "2 weeks," "3 days," or "1 week" as the experiment minimum duration. Sites with small amounts of traffic will need the 2 week minimum duration. Sites with thousands of visitors per day could easily discover a winning variation in as little as 3 days.
The "confidence threshold" you see there allows you to select 95%, 99%, or 99.5% confidence in the winning page. The higher confidence level will require a longer testing duration.
Click the blue "Next Step" button to open the page variations screen you see here:
This is where you enter the exact URLs of the original page and the variation pages. You shouldn't leave anything to chance in this setup. Make sure to use the full URL even if you don't need to. For example, If I want to test the home page of the jwag.biz website I could enter "www.jwag.biz" into the Original Page field. My website will automatically serve the "www.jwag.biz/index.html."
However, to avoid any potential confusion during testing, and prevent hidden server issues, you should fully spell out the URL, including the understood page name.
I suggest you select the option to "Consolidate experiment for other content reports" so the rest of your Google Analytics reports are not affected by the experiment. If you don't select this option, your other reports will show an unusual drop in traffic during the experiment period.
Put the alternate version of your web page in the Variation 1 field. If you have more variations, you can click the blue "Add Variation" link to expand another option.
Once you've added the page URLs, you'll see a similar screen like this one with thumbnails of the web pages:
I should mention that you need to fully set up all the variations of your web page before you even start this experiment setup. For the purposes of writing this Nugget, I set up a temporary "index2.html" page on my site, but I didn't fill it in with real information. That's why the thumbnail screen shot above looks overly simplified compared to the original.
It should be easy to set up page variations in any content management system.
Click the blue "Next Step" button to continue to the experiment code. At this stage you probably need to select the option to "Send the code to webmaster" as shown here:
Once you've entered your webmaster's email address, you will again click the blue "Next Step" button to continue to the final review as shown here:
It will validate the code on the website before you are allowed to begin the experiment. The validation will succeed once your webmaster has installed the experiment code. Until then you can click the gray "Save for later" button.
Once you start the experiment, you'll be able to watch the results daily. Otherwise, just let it run until a winner is found.
Here's a screen shot of an experiment I ran quite some time ago:
The results showed a 21% change that the Variation 1 page would outperform the original. Once you see similar results you will want to permanently apply the changes to your page.
Conclusions
Setting up the experiments in Google Analytics is the easy part. Determining a reason to run the experiment, and setting up the website are the difficult first steps.
I consider multivariate and A/B testing to be a high level user interface design test. I don't often write about this type of high level testing, but it was important to update my outdated Nugget from a few years ago.
Anyone struggling to improve their website beyond what they have achieved so far should consider this as a next step in their SEO and usability process. The results will probably surprise you, and lead you down new paths to major website improvements.
