Real User Testing

June 16, 2004 | 9 Comments

I just got through reading 90% of All Usability Testing is Useless by Lane Becker.

It’s a great read and reflects very closely my own views on user testing.

Usability doesn’t need to stay within the academic, numbers-based realm of traditional HCI. It’s not rocket science and great benefits can be achieved by bringing Web teams and stakeholders into the usability process. Lane says:

So it’s time to get your hands dirty. Usability testing for the Web doesn’t require outside firms that hold themselves separate from the design process. Understanding context means embracing this research as part of the design process. It should be performed with the full participation of the design team, so that everyone can internalize the user’s perspective. Developing and leading usability tests should be a core competency owned by someone within your team (though many folks believe it ought not to be the original designer).

Amen brother.

I’m so glad to hear I’m not the only one who feels this way. Down at the hospital it was recently brought up by one of my co-workers that we think about doing some user testing with our stakeholders.

If we can get them to do it, I think that’s an amazing idea. Not only does it educate them as to the needs of the users, but it gives them a better sense as to what makes a quality Web site. I’m all for it and will let you all know how it goes.

Usability doesn’t have to remain in the hands of the academics and the gurus. Everyone involved with the creation of a Web site should have an opportunity to be involved and make that connection with the user. I’m no guru, but I’ve had many, many opportunities to get to know and observe users.

Running a usability test isn’t really that hard, I’ve done it quite a few times, and I’m just a Web designer at heart!

It well worth the effort as it’ll improve your sites in a way quantitative research never will and in the end will make you a better designer.

Filed under: IA and Usability

Comments

1. anders said:

if you haven’t already, you should definitely read Joel Spolsky’s book “User Interface Design For Programmers”, which is also available in its entirety online. i think he would agree with you. in particular, he writes about the importance of “hallway usability testing”, which basically just means frequently grabbing people out of the hall and getting them to attempt a small task or two with your software and watching how successful they are; the key idea being that usability testing should be informal but frequent and included in every stage of design and development.

Posted on June 16, 2004 02:49 PM | #

2. Keith said:

anders – I’ve not read the book you mention, I’ll check it out. However, one thing I’d advise against when it comes to “real user testing” is relying solely on what you term “hallway usability testing” – I do think there is some value in that, but all to often that is the only testing done.

Testing with coworkers and wise fool testers cannot replace the value you get with testing with real users. Let’s make that clear. I’ve seen that kind of testing used as a crutch and/or excuse far to often.

Posted on June 16, 2004 03:07 PM | #

3. Joshua Kaufman said:

The traditional usability testing that Lane is describing is called benchmarking. It has its own very important role in usability testing, but it’s not usability testing by itself. Of course usability testing is about context and qualitative results. Isn’t that what the guru’s have been telling us for years? I’d like to know who’s been telling Lane otherwise. He makes some good points, but I think his claims are far too over-generalized. Not to mention that his title is attention hungry as a Nielsen usability claim.

Posted on June 16, 2004 05:43 PM | #

4. Steveg said:

Sometimes engaging the interest of the stakeholders can be a problem. Occasionally the response is: “Why are you bothering us? We don’t have the resources to do your job.”

Usually (although not always) the actual end users are interested but at the managerial level there can be a reluctance to commit resources. I guess that is one of the reasons why iterative development is not popular in more companies.

Posted on June 16, 2004 06:35 PM | #

5. Keith said:

Joshua – I guess I can see what you mean. I didn’t get so much that he was saying you shouldn’t do things like Benchmarking – but maybe I missed that?

To me, as someone who has no choice to do my own, I really appreciate where he is coming from, but I’m not opposed to quantitative user research as well.

Steveg – Too true.

Posted on June 16, 2004 06:44 PM | #

6. Michael Watts said:

I’ve recently completed a course that covered alot of aspects of IT. One thing that was really pushed was the importance of working closely with stakeholders throughout the Design and Development process. It’s all well and good to have something you think is great and your colleuges think is great, but it really doesn’t mean alot if you’ve misinterpreted the wants and needs of the intended users of a product.

Logically I can’t see how it would be benificial to work in any other way.

Posted on June 17, 2004 12:15 AM | #

7. Joshua Porter said:

In my day job I work for an “outside” usability company. While I don’t think we really fit into Lane’s generalization about usability testing (because we insist upon working with designers), I do think that it is important to note that it really doesn’t matter how you do it, just do it.

One of the reasons why usability testing is usually only talked about (and not done) is, I believe, because of the enormous expense. There’s the employee time, of course, but there are also many other factors like having a place to test (regularly), find the right users and scheduling them and paying them, analyzing results (which can take weeks), and putting together the results in a meaningful way. From my own experience this last one is often the toughest hurdle: to accurately present observations to the design team and to discuss and come to consensus about what went on. It certainly helps when the designers are observers, too. Even though multiple people see the same thing, it can still be difficult to crystalize the findings in a definitively way.

As Joshua K. alluded to, Lane mentioned a very specific metric in usability testing, task time completion. He generalizes by saying that this is a useless benchmark. It is, of course, for most applications. But what about a web-based call center application? Would task completion time be important there? Maybe so, maybe not. Let’s go talk to the call center manager and find out.

It’s important to remember, in my opinion, that each and every web site/app and the business that drives them will have different goals. Sometimes qualitative analysis is necessary, sometimes quantitative. Often both. Every project is different! What are your goals?

Regardless, Lane’s point is clear: test your site to see if people can’t do what they want/need to do. If they can’t, find out why. If they can, don’t change it!

Posted on June 17, 2004 06:59 AM | #

8. Keith said:

Joshua P. – Very well said. I like the bit about “it doesn’t matter how you do it. Just do it.”

There has only been on place I’ve worked where we could even afford to do the kind of testing you guys do (although the people we hired never did get with us designers and I always wondered about that) and that was Connexion by Boeing.

For many, the kind of user testing I talk about is the only option. Is it better? Not necessarily, but it’s better than nothing at all, that’s for damn sure.

Posted on June 17, 2004 08:30 AM | #

9. Ron Zeno said:

Meaningless. Perhaps the author should have only written, “90% of All Usability Testing as Done by Adaptive Path is Useless”. He may be an authority on the uselessness of his company’s practices, but I see no evidence, certainly not from this article, that he has any expertise on the effectiveness of others’ practices. It’s your basic strawman article with the controversial title.

Posted on June 21, 2004 09:47 AM | #