21 Why can't Usability testing drive priorities?

I am not sure what you mean by usability testing cannot drive team priorities? Your explanation is all about why BAD usability testing cannot drive priorities. I think I failed to see where you spoke of GOOD usability testing to drive priorities.

But I do agree with you, usability testing should be in context. If your usability test cases capture the business needs properly, then it can direct the development efforts in making that test pass (TDD).

posted at 09:49 am on October 28, 2009 by franz_see

22 Diminishing returns on Usabiilty Testing

The article ‘Why You Only Need to Test with 5 Users’ (http://www.useit.com/alertbox/20000319.html) speaks of that.

And I guess that could be an explanation as well the numbers in “User Interface Engineering”.

posted at 09:51 am on October 28, 2009 by franz_see

23 Negative feedback analysis

Usability tests are great at telling a team what direction they should not pursue, but probably not much else. Unfortunately that is perhaps the most important information regarding creative direction that “experts” may ever hope to receive that is not often appreciated strictly with this regard. Web developers are, by the way, experts at knowing what the customer wants, which is why they are so good at telling the customer what they want.

My employer subscribes to usability tests that provide incredible feedback. How scientific is that information and how wonderful are those picky details? I don’t know. The evidence does not suggest a decisive direction to pursue, but it is quick to tell you when you are wrong in comparison to nearly identical expectations from competitor websites. If you are wrong over and over… eventually a patter should emerge of what you should NOT be doing. I find that information to be of profound value, although it is commonly in directly conflict with expectations of what usability should be.

posted at 05:10 am on November 2, 2009 by austincheney

24 It Has its Uses

I agree with this article that Usability Testing isn’t the ultimate standard in catching issues and solving problems on the web. However, I’ve found in my own experience that testing can provide some insights into how your website is perceived, as well as being able to let you separate yourself from your own company jargon.

We used testing on a redesign for a university in Philadelphia, and some of the best insight we gained was from what users wanted to get to first, second, etc. We also gained insight into the application process we had built, and was able to rewrite instructions in layman’s terms, instead of the “advertising” jargon we had been using.

posted at 06:20 pm on November 5, 2009 by psuhiker

25 Usability Testing vs. Usability Listening

You can’t just sit there listening to everyone’s comments. Many users have a terrible sense of aesthetics or want the application to be tailor-made for them. Every site and application needs to undergo usability testing of some sort, but more important is having the services of a designer (and hopefully developers too) who has usability patterns and standards down to a T. A designer with the right eye can pick the true usability flaws out of the sea of personal preferences expressed in during usability testing.

posted at 12:48 am on November 10, 2009 by Thomas Allen

26

Aside from the fact that I would absolutely believe that a typical Microsoft application like hotmail has AT LEAST 300 usability problems, there are some other oddities about this study.

The reviewers of the study mention that there were reporting problems from the teams, as well as being pretty fine grained about wether ‘problems’ were in fact the same or not. It almost difficult to trust the outcomes of these studies.

Also I find it hard to believe that if you continue to expand usability studies you won’t be able to find a correlation with major problems. Most studies are able to zoom right in to 1-4 major problems right away. This study did in fact that there were 9(?) problem that were reported by more than a few teams. This makes perfect sense. You need to prioritize and fix major problems.

I agree that you need to understand what your expectations are from these kinds of studies. You looking for ‘usability’ problems, not people’s opinions. If you want opinions go have a code review. If people are unable to complete a well written task, well, then you have a problem – which is why these studies are run.

It’s also my opinion that bringing in current users of a system is a problem. Even sites that require a lot of domain knowledge are able to be tested with first time users and a well written script. Bringing people back to test again is usually not a good idea because (like you mentioned) they’ve become familiar already with your screwed up navigation.

posted at 01:56 pm on November 12, 2009 by ascotan

27 Use Proper External Usability Testing

Nice article and I do agree with many points. We find that the best method is to have a group of 10-15 external users all with different tasks to perform, e.g. buy a pair of jeans, sign up to the newsletter, find a course, etc. This way you get a good cross section and then you sit down with your creative and development teams and analyse the data. Then we would make any recommendations for design/fucntionality changes going forward.

posted at 04:57 pm on February 23, 2010 by cal1977

28 Usability testing is great. Usability labs are so

If you’re testing an interface for a product that is not strongly influenced by the user’s personal context or emotional state, then testing in a lab will yield decent results. If you’re trying to measure the persuasive power or conversion potential of your site (which should be pretty high on your list of research objectives if you’re running an ecommerce site), then lab-based testing is a complete waste of time. There are a bunch of cost-effective alternatives that can help you identify roadblocks to conversion in real-time.

posted at 10:35 am on May 9, 2010 by uxshark

29 I think Randolph's onto something

“Page views and time-spent-per-page metrics, while often foolishly considered standard measures of site effectiveness, are meaningless until they are considered in context of the goals of the pages being visited.

Is a user who visits a series of pages doing so because the task flow is effective, or because he can’t find the content he seeks? Are users spending a lot of time on a page because they’re engaged, or because they’re stuck? While NYTimes.com surely hopes readers will stay on a page long enough to read an article in full or scan all its headlines, Google’s goal is for users to find what they need and leave a search results page as quickly as possible. A lengthy time-spent metric on NYTimes.com could indicate a high-quality or high-value article. For Google’s search workflow, it could indicate a team’s utter failure.”

> You seem to be confusing analytics with usability. The whole purpose behind a think-aloud test is to uncover what’s behind this sort of thing. BTW, that’s all that Crazy Egg, Chalkmark, and five second test tell you too. User Testing is a real usability test, but without any ability to follow up. In that sense, it’s inferior to a moderated test.

“And interestingly, many of the most compelling usability test insights come not from the elements that are evaluated, but rather those not evaluated. They come from the almost unnoticeable moments when a user frowns at a button label, or obviously rates a task flow as easier than it appeared during completion, or claims to understand a concept while simultaneously misdefining it. The unintended conclusions—the peripheral insights—are often what feed a designer’s instincts most.”

> Most experienced facilitators ignore these in favor of verbalizations. And if users exhibit some sort of body language and don’t verbalize, experienced facilitators prompt them to do so (“What are you thinking?”)

“Finally, while testing alone is not a good indicator of where a team’s priorities should lie, it is most certainly part of the triangulation process. When put in context of other data, such as project goals, user goals, user feedback, and usage metrics, testing helps establish a complete picture.”

> User feedback and usage metrics cannot be used if the system hasn’t been put into production yet. Usability testing is usually done pre-release, to get feedback on something without it’s being exposed to the whole world. Also, evals and tests, if done properly consider business and user goals in the tasks they test and the things they look for.

“Without this context, however, testing can be misleading or misunderstood at best, and outright damaging at worst. This is also true for non-testing-based evaluation methods, such as heuristic reviews.”

> Actually, it’s not the context so much as the things you covered earlier – basically, inexperienced usability practitioners.

“There is a catch to all of the preceding arguments, however: They revolve around the notion that testing should be used primarily to identify problems with existing designs. This is where teams get into trouble—they assume testing is worth more than it truly is, resolve to address problems based purely on testing data, and revise strategies based entirely on comments made by test participants.”

> Usability testing doesn’t prove anything. It’s meant to inform your judgment. You still have to make a decision. Hopefully, there is now evidence (numbers, quotes, etc.) that you can use to make an informed decision.

“As we’ve seen, test results and research can point teams toward solutions that are not only ill-advised, but in direct conflict with their goals.”

“usability evaluation is unreliable”

“While usability testing fails wholly to do what many people think is its most pertinent and relevant purpose—to identify problems and point a team in the right direction.”

“Test for the right reasons and you stand a good chance of achieving a positive outcome. Test for the wrong ones, however, and you may not only produce misleading results, but also put your entire business at risk.”

> These are pretty strong claims. Given that, you really need to back them up. You cite the CUE studies, but there’s a lot to these studies. It’s actually rather complicated what he’s really saying, and trying to present it all in this way is (yes) very attention-getting, but also very wrong.

“Asked how development teams could be confident they are addressing the right problems on their websites, Molich concluded, “It’s very simple: They can’t be sure!”

> Well, here’s what Molich has also said, from the CUE site:

>> Six – or even 15 – test participants are nowhere near enough to find 80% of the usability problems. Six test participants will, however, provide sufficient information to drive a useful iterative development process.

>> The limited overlap may be a result of the large number of usability problems in [the system being tested]. It could also be due to the different approaches to usability testing that the participating teams took – in particular, the selection of different usability test scenarios.

>> Realize that there is no foolproof way to identify usability flaws. Usability testing by itself can’t develop a comprehensive list of defects. Use an appropriate mix of methods.

>> Place less focus on finding “all” problems. Realize that the number of usability problems is much larger than you can hope to find in one or even a few tests. Choose smaller sets of features to test iteratively and concentrate on the most important ones.

>> Realize that single tests aren’t comprehensive. They’re still useful, however, and any problems detected in a single professionally conducted test should be corrected.

>> Increase focus on quality and quality assurance. Prevent methodological mistakes in usability testing such as skipping high-priority features, giving hidden clues, or writing usability test reports that aren’t fully usable.

posted at 06:59 pm on October 21, 2010 by cliff a