Discuss: The Myth of Usability Testing
by Robert Hoekman Jr.
- Editorial Comments
2 RE: Provocative, in a good way
Luke — thanks for your comments. Yes, I do think intuition can be unreliable, and yes, I do think the evaluation methods I recommended here can be be just as flawed as traditional testing. This is why I recommended a process that uses a combination of these methods. I said: “Consider a revised workflow that begins with an expert-level heuristic evaluation used in conjunction with informal testing methods, followed by informal and formal testing. More specifically, consider using online tools and paid services to investigate hunches, then use more formal methods to test and validate revised solutions that involve a designer’s input.”
You should definitely do anything you can to validate design decisions once they’re implemented. That’s always true. Metrics analysis is one good way to do that (provided the metrics are put in context of goals, of course). I certainly didn’t mean to suggest you can spot important issues and automatically make millions of dollars as a result (though it has been done, albeit rarely and always with a little luck).
Design is an ongoing experiment. It’s never truly finished. It’s never truly perfect. Put your best ideas out there, measure them, and revise or rethink as needed.
To your question about online tools: I don’t think 5-second tests are meaningful in and of themselves; I’m not sure <em>any<em> evaluation method is meaningful all by itself. Here, I’ve recommended using them to investigate hunches. If you already identify what you think is a usability issue and results from a 5-second test support or contradict what your belief, that’s a useful insight. In other words, use these tools to bear out a hypothesis, not as a way to discover a hypothesis. The difference is important.
Thanks again for your comments. Cheers!
posted at 10:45 am on October 20, 2009 by Robert Hoekman Jr.
3 User Experience is a Verifiable Art
Great article, Robert. User experience engineering is an art just like writing and graphic design. You can follow all the rules, but it takes a subjective element to get it right for human consumption. Metrics help by making the subjective objective. They validate or invalidate subjective feelings and decisions.
Silverback is a tool worth mentioning because it aids in the observation of user experiences not just report metrics.
posted at 10:59 am on October 20, 2009 by ianlotinsky
4 Not convinced by the method
I don’t agree with your conclusion of the Molich tests. It’s not usability testing that you should blame, but the usability companies that participated. Maybe they are not that good.
Or maybe the method itself. Maybe usability as a science is not that exact as some people hope.
Imagine you have a huge garden and you invite 10 companies to maintain it. Will the result be the same?
Definitely not. Some differences will be the result of a bad knowledge of gardening. Or lazy employees. Of gardeners not knowing the business. I think most of the ‘mistakes’ made, would be the result of one of those elements.
Other differences will be the result of a different approach. And who is really capable of telling that those differences ‘are’ mistakes, things that the gardeners missed?
Gardening is not an exact science.
Neither is usability. (Whatever method you use.) There will always be differences.
posted at 02:10 pm on October 20, 2009 by KarlGilis
5 Get a real usability test
If you’re going to do usability testing, do it thoughtfully, thoroughly, and by collaborating with the team. There is a lot of art involved in user research — it isn’t science. But a usability test well done can reveal amazing insights that you can’t get any other way, even if you have observed hundreds of people using designs.
I have a lot to say about usability testing. Some of what I have to say I said in this periodical, just a couple of weeks ago: http://www.alistapart.com/articles/preview/usability-testing-demystified/.
(If that’s not enough for you, check out my blog: http://usabilitytestinghowto.blogspot.com/. If that’s not enough, come to my session at UI 14, ‘Mastering The Art of User Research’ http://www.uie.com/events/uiconf/2009/program/#chisnell)
posted at 04:45 pm on October 20, 2009 by Dana Chisnell
6 Blame, mistakes, and inconsistencies
ianlotinsky: Thanks for the kind words, and yes, I think Silverback is a wonderful tool. Thanks for mentioning it.
Dana Chisnell: I agree — usability testing can, and does, reveal amazing things you can’t get any other way. (Full disclosure: I consulted with Dana prior to writing this article to get her expert perspective on the subject.)
KarlGilis: Interestingly, I think you and I agree more than disagree here. I wonder if you’d consider reading the article a second time with your initial reaction in mind.
First, I didn’t “blame” usability testing for anything here (I’m not sure if you think I was trying to place blame or if you were only using that word for the sake of your gardening analogy). To the contrary, I frequently sing its praises. I’ve simply posited in this article that the main benefits of testing appear to be different than what is commonly proposed as its core purpose: determining what a team’s priorities should be. In 2003, when asked how teams could confidently conclude they were addressing the right issues, Molich himself said, “It’s very simple: They can’t be sure!”
Second, please note that I didn’t say anyone made “mistakes” in their evaluations, just that 17 teams in CUE-4 came up with 17 different evaluations, and 9 teams in CUE-2 came up with 9 different evaluations. Indeed, testing is not as scientific as one might hope — that was exactly my point. It is only natural multiple teams will achieve different results — perhaps even obvious — but typical public opinion is that usability testing is a scientific method for determining all the usability issues in a design. The CUE tests clearly contradict that notion.
I hope this clears up any confusion.
posted at 07:35 pm on October 20, 2009 by Robert Hoekman Jr.
7 5-second tests
i’m pretty skeptical about the 5-second test. what could you hope to discover from a test like this. i think it could only be useful as a very narrow test, ie what do users notice first. anyone found it genuinely useful?
posted at 06:50 am on October 21, 2009 by roginator
8
Thanks for the article. While agree with most of what you’ve said, but couldn’t you pick a better title? I’m getting linked to this article by people who know me (since I specialize in this field) who think that it’s against usability testing.
Anyway, results of usability testings can vary greatly depending on few factors such as participants, tasks given to participants, the experience of the moderator…etc, so of course you’ll get different results. That doesn’t mean that usability testing is ineffective. It just means that you need to be careful and understand what you’re doing.
posted at 07:50 am on October 21, 2009 by Mashhoor
9 RE 5-second tests
roginator: That’s exactly what it’s for, actually. You can use it to determine what people think is the primary purpose of a page, to see if they can tell in five seconds, or ask them how they’d perform a task and then show the screen for 5 seconds and see if they can spot what to click. It’s very useful for quick and dirt cheap feedback on a page with a single purpose. And using www.fivesecondtest.com, you can get a ton of responses really quickly.
posted at 04:45 pm on October 21, 2009 by Robert Hoekman Jr.
10 That pesky title
Mashhoor: Ha! Sorry if it’s causing you trouble. I’ve heard a few people say they think the title is inappropriate. But the article is in fact about a myth: that usability testing is good for determining what to focus on next. It also talks about why different teams produce different results, and what a better (cheaper, more effective) process might look like, so the title is indeed appropriate. That said, I knew it would probably get exactly this type of reaction, and after measuring the pros and cons, I decided the potential controversy could only draw attention to it.
Do please point out to your colleagues, though, that I never said testing was ineffective. To the contrary, I said it was very effective, just for different reasons. I wholeheartedly support running usability studies. If anything, do them more. Just don’t do them with the goal of determining what to focus on next, because they’re wholly ineffective at accomplishing that.
Cheers!
posted at 04:54 pm on October 21, 2009 by Robert Hoekman Jr.
Got something to say?
Discuss this article. We reserve the right to delete flames, trolls, and wood nymphs.
Create a new account or sign in below if you’d like to leave a comment.
Subscribe to this article's comments: RSS (what’s this?)
1 Provocative, in a good way
Thanks for a thought provoking article! I wasn’t aware of the Molich experiments, but am glad I am now, and enjoyed the tips on simpler ways to do user testing.
I’m a data-driven design guy, so I appreciate the more evidence-based approaches to what we do in design and usability, but I worry here we’re replacing one flawed system with another.
Firstly, as for the tools you recommend (most of which I would echo), aren’t they just as prone to the problems Molich found? I can’t imagine the five second test being particularly meaningful, for instance.
Secondly, hasn’t this desire for more usability testing come about after recognizing the limitations of our intuition? Sure, it’s a nice ego boost to think that I, Joe or Jane Professional, can simply know, Gladwell-style, in the blink of an eye what the problems are (though I wouldn’t cite “Blink” as evidence for anything, really), but I don’t think it holds up in reality. If it did, I would spot problems costing businesses $millions, charge them $1 million for my blink-of-an-eye fix, they could fire their internal team, make millions, and everyone wins. Well, everyone except their internal team.
What evidence is there that a well trained intuition isn’t just as faulty as the usability teams? Ask a dozen designers, get a dozen different answers.
It’s like saying an experienced stock trader can just “know” which stocks are going up. If that were true, we’d all give them our money, and we’d all be rich.
I think our intuition is brilliant for coming up with ideas and possible solutions (our design is only ever going to be as good as our best ideas, after all), but we, as a profession, still need a formal way of measuring, testing, and publishing our results, in my opinion.
I’m not for flawed usability testing as descried — pseudo-science doesn’t help anyone — but nor am I for returning to expert intuition either, as informed by testing or otherwise. But I do think we need a new way of thinking about design on the web.
posted at 09:14 am on October 20, 2009 by Luke Stevens