Usability and A/B split testing: presentation to Christchurch Web Design & Development Meetup

#1 Continuous improvement

Good websites get better with lots of small steps. And a few big steps.

The key is to define what you want your site visitors to do, measure this and improve it.

• Define
• Measure
• Improve
• Improve again

Visitor actions can include:

• Buying something
• Entering their contact details (lead generation)
• Downloading a pdf
• Calling you
• Visiting your restaurant / museum / gallery

All of these actions can be tracked - which means measured which means improved.

The jargon word for a visitor taking a desired action is converting. The percentage of visitors who take an action is the conversion rate.

#2 A/B split testing

This is the scientific way to find out if a change has really improved conversions.

You call the existing version of something "A" and create a new version called "B". The "something" can be a headline, a button, a colour scheme, a layout, a sequence of pages - in fact any part of your site.

You then use special software to randomly show version A to half your visitors and version B to the other half.

You then compare the conversion rates for A and B.

The reason for doing this instead of just trying "B" for a few weeks is to eliminate other factors from the comparison - such as seasonality, traffic sources, competitor activity, other changes on your site.

I've covered A/B testing in more detail.

The stats test is called unpaired one-tailed equal-variance t-test.

#3 Usability testing

This gives you insight into how ordinary people use your site (and the web).

You put an ordinary person in front of your site with some tasks and watch what happens.
Be ready for surprises.

Recruitment

Getting some good but ordinary people takes a lot of time.

• Avoid IT people (including web designers)
• Avoid IT people (including web designers)
• Avoid people who know your site or your business
• Avoid total web novices

You learn a lot about your site's problems from the first 2 people, then less from the 3rd, then less again.

By the time you have done 5 tests you have found most of the usability issues.

It's better to do 3 tests, then improve your site, then do 3 more tests with the new site.

In fact doing any usability test - even just getting a neighbour - is so much better than not doing this at all. Don't wait for perfection - go ahead with imperfect usability testing.

One more tip: don't do tests back to back - you may learn things during one test then re-design the rest of the tests.

Design the tasks

Design tasks like "buy a kettle just like the one at home".

The trick is to make the task specific enough - not just "buy any old kettle".

But don't put words into the user's mouth. A task like "open a 90-day bonus extra account" is steering the user towards the site's jargon and navigation. Better task would be "You want to save about $5000 with a good interest rate. You don't need the money right away."

Allow enough tasks for 45 minutes. Some people will only complete the first one or two so you must prioritise.

You may want to try the same tasks on competitor sites. Mix up the order of the sites for different users and do not say who is paying. The user must not start with any preferences.

Execution

Make sure the PC is clean - eg auto-completion and visited URLs must be deleted.

I use Morae software from TechSmith to track what happens on-screen and record the user's voice (and my voice).

Before the test I explain the recording software and emphasise that the site is being tested - not the user.

I start with a dummy task to get the user warmed up. "A friend lives in Bristol. Go to the BBC website and see what the weather is like in Bristol."

Results

The output of usability testing must be the input to some re-design work.

After the test your head is full of insights into the problems that real people have using the site. You need to capture and share this insight. The recording software is key to this - I also like to write a transcript with timings.

Read more about usability principles and testing from Jakob Nielsen who started this discipline.