Do All Web Sites Need Usability Testing?

June 07, 2005 | 17 Comments

I’ve been lucky enough to have quite a few very educational and informative experiences with usability and user testing. I’ve found just about each and every one worthwhile and think I learn a bit every time.

However, recently I’ve been working on sites where I’ve done little in the way of formal user testing. From what I can tell, while not perfect (what site is?) these sites have come through rather well in the absence of formal testing.

It’s caused me to question whether, under certain circumstances, usability testing is needed for a Web site. I don’t know the answer, and in general I’d always recommend user testing if you have the ability and resources to do it, but I’d like to explore the idea a bit.

Why would you not do user testing?

First of all, we need to talk about why you wouldn’t do any user testing on your projects. I can think of a few reasons:

Don’t know how.
Don’t have time or resources.
Client won’t pay for it.
Can’t get participants.

Let’s face it, user testing, even on an informal level, is hard work. It takes knowledge, effort, time and money. Depending on the needs of your project, it might not be a viable option for you or perhaps the decision has been taken out of your hands—which is very common, unfortunately.

I always say that if you can do, you should. But we all know it’s not that simple.

User patterns

One of the things I’ve noticed in my most recent formal user testing sessions (it’s actually been awhile though) is that I was seeing the same things. Granted, I was working with sites and applications that had similar audiences and similar goals.

In essence, I wasn’t learning much that was new to me. I feel like if you’ve spent quite a bit of time observing users you tend to see patterns and are better able to recall those patterns and design (or architect) accordingly.

So a designer who has had the opportunity to participate in a bunch of user testing is better equipped to deal with a project where there isn’t any? Maybe.

The Wise Fool

One thing I have done quite a bit of on my recent projects is what is called ‘Wise Fool” testing. This is where you have someone who is “in the know” pretend they are a user, test and report any issues or problems.

This works well as an option if you’ve got no other means to test your work. I’ve found that even doing this myself, on my own sites, has uncovered some problems I’d have missed. Again, I feel that someone who has been a frequent observer of user behavior (hopefully on similar projects) would be better suited to this than one who hasn’t.

Common sense and self-education

I think part of the reason why some sites who forego user testing do well might be because:

The team has tested with users before and know what works and what doesn’t
The team is educated on common user problems and is “up-to-snuff” with heuristic usability principles
The team has a good understanding of the audience and has applied common sense and lessons learned to their design decisions

Testing is always better

So what do you think, does every Web site need user testing?

I’m going to say that, no, not every site needs it to succeed. However, I think Web professionals need it to help make better design decisions and I think that some user testing is better than no user testing if for no other reason but to help develop an affinity for the people who deal with our sites.

Filed under: Usability
Keyword Tags: user+testing usability design

Comments

1. Garrett said:

I don’t believe every site needs user testing. If you have experienced and well-read people, they can get a site 80% there. If there was any user involvement in requirements gathering, you’re probably adding another 5%. Now, if you do some hallway testing, you can comfortably get another 5%.

That last 10% is the fine-tuning in my opinion. It can definitely help if it’s possible, but an experienced person who is willing to recognize that understanding the users is important can do most of the job.

Would a car designer expect to have user testing to make sure the steering wheel is in the right place? Probably not. Most usability issues can be addressed by an experienced IA or designer.

One huge benefit that is hard to recreate is the ability to step back. Often, the designers and other team members are so close to the project, it becomes hard to step back and have an objective opinion. Users can help significantly in cases like this. However, hallway testing of individuals not directly involved can help as well. Just my experience.

Good post, Keith.

Posted on June 7, 2005 01:34 PM | #

2. Jeff Croft said:

I agree that not every site requires it, and I am am beginning to think the user-oriented design concept is going to far. To me, my job as a web designer is to give the user a clear path to two things:

What he/she wants.

What I (or my client) want them to want.

In other words, most sites have a marketing component, too. If I rely on usability test so much that I am simply giving the user what he/she wants, then I cease being a designer. I’m also doing a horrible disservice to my client if I’m not pushing the things they want me to push, because users don’t know they want them (or maybe, they actually don’t want them).

Bottom line: I’m all for user-oriented design and usability at the forefront, but let’s not take it too far. At times, I think it’s getting out of hand.

Posted on June 7, 2005 01:36 PM | #

3. Garrett said:

Jeff - User testing should never be just a matter of listening to what they ask for, but instead, it should be “hearing” what they need.

I do agree wholeheartedly that it can get out of hand though. Oftentimes, you will hear a user say they want something, but after observing them, their actual actions will contradict what they requested.

Like the old saying…people should be thankful for unanswered prayers. In most cases, practicing usability is matter of determining how to answer the user’s prayers in the best way possible rather than just doing exactly what is asked.

Posted on June 7, 2005 02:05 PM | #

4. Keith said:

Garrett – This is too true: “Oftentimes, you will hear a user say they want something, but after observing them, their actual actions will contradict what they requested.”

People tend to describe their perceived solution rather than the actual problem. You might hear:

“This page is too busy, remove X.” Or “I’d really like a button over here.”

These don’t really get at an underlying problem. One thing frequent testing and research does do is help you know what to listen for.

Posted on June 7, 2005 02:25 PM | #

5. Jeroen said:

The dillemma here is that the only way to know whether your users ‘get’ it is to look at them while they visit your site. You only know that your site does not need any usability testing after doing a usability test. Difficult… I usually do at least a minimal test with my designs by asking my wife/mother/neighbour/friend to spend 15 minutes looking at a site.

Posted on June 8, 2005 12:16 AM | #

6. Kev said:

Everyone makes assumptions. A designer can assume they know their users and a user can assume s/he knows whats wrong with an interface. To me, usability testing is about stepping away from individual opinion and getting an overall consensus - which is why the more participants you have in a testing scenario the better quality your conclusions may be. Bearing that in mind I think its dangerous to not do any usability testing at all except in situations where convention has evolved to the point of saturation i.e. clickable logo’s go top left, tabindex through form elements. Every designer (I would hope!) knows these things and hence I don’t feel its important to look closely at them.

Posted on June 8, 2005 12:35 AM | #

7. Jens Meiert said:

Yes, definitely. It just gets interesting when it comes to /how/ this is (to be) done - be it “guerilla testing” or any more professional approach. In my opinion, we’re still at a point in time where it’s not important /how/ testing is actually done, but that /something/ is done at all. (Same still applies to any accessibility efforts, of course - and unfortunately.)

Posted on June 8, 2005 02:17 AM | #

8. Coop said:

Usability testing is a necessity for all websites, IMHO. The truth is that it doesn’t have to be a huge process. A site is usable based on whether or not a user can find the information they are looking for quickly and easily. However, I find that having someone who is inexperienced with the industry the best for testing. Giving them topics to find on a site. A feel a site is successful if the “dummy” can find information on the site. As I’ve said before, content is the key to any site. Being able to write marketing material for the web is critical, and if a “dumb” user can understand the purpose of a site, then usability has been accomplished. Navigation testing in usability has gone too far. The truth is that everyone that uses the web knows the basics of navigation, but if the “dumb” user cannot figure out the message of the site, then you really have to analyze why that message is not perfectly clear.

Posted on June 8, 2005 07:36 AM | #

9. drag said:

I think it is not a matter of success of a website, rather than a moral obligation for us webguys. well moral is maybe a too strong of a word, but i see it as discrimination if you don’t do the basic things.

I think you don’t need to do it do it for your blog or something … but for bigger projects, me thinks it’s mandatory. (mandatory=required … right?)

Posted on June 8, 2005 01:33 PM | #

10. DaveMo said:

I’m a code grunt for a State agency website and ironically, although we’re supposed to be totally accessible, I have yet to see any hint of concern about it at any level above mine!

I try to sneak stuff in as I code, like “title” attributes in links and applying as much semantic markup and CSS as I can get away with, but I am frankly confounded by the fact that usability concerns are a profoundly low priority. “Just use basic HTML and it’ll be fine.” seems to be the mind set.

Soon we’ll be using a Web CMS system and the big answer to accessibility is to slap a UsableNet Transcoder on the site, while the rest of the coding is a nightmare of tables and redundent tags and JavaScript hacks.

I get the most condecending looks and attitude whenever I bring the subject up.

Just shoot me now.

Posted on June 8, 2005 02:14 PM | #

11. Donna Maurer said:

There are loads of times when it is silly to usability test a website. One of the key times usability testing doesn’t work is when your site is suited to people who don’t really know what they need and want to explore. If you give them a directed task to do in a usability test, it does not reflect real life and is a waste of time.

I have an article half written about the differences between testing informational and interactive sites - must finish and publish it ;)

Posted on June 9, 2005 04:34 AM | #

12. Kev said:

Hmm. I see what you’re saying Donna, but IMO when a user doesn’t know what they want thats not a usability issue but an Info Architecture issue.

If a site isn’t tested for usability, those who don’t know what they want will definitely need to find navigation usable and intuitive and how can you be sure it is unless you test?

Posted on June 9, 2005 05:39 AM | #

13. Jennifer Kyrnin said:

I actually feel very strongly about usability and user testing, and I was prepared to be upset at your post, but as I read along I found myself nodding “yes, yes, that’s so true” (especially the “client won’t pay for it part” *laugh*). I still think it’s important to do user testing when you can, but I’m willing to be flexible.

Posted on June 9, 2005 06:23 PM | #

14. Keith said:

Jennifer – I too feel strongly about usability and user testing. If you go back through my archives you’ll note I’ve changed my tune a bit with this post. I think part of the reason is that there are times when I’ve found testing doesn’t show me anything I didn’t already know or when I’ve gone to great lenghts to test, only to look back and realize it might not have been worth the effort.

Kev (#12) brought up IA and, I’m assuming, findability. While I’d agree you can’t be sure if an architecture supports a user needs without testing, in some cases you can make “educated assumptions” based on prior knowledge that will be proven out by testing. At that point testing is simply proving something you already know.

For example, why would I need to test for findability on say, a professional services site, that has amost the exact same audience, goals and content of a site I’d tested before? Shouldn’t my prior experience tell me something useful? I think so.

I think a take away from this is just as you say–be flexible and when you simply can’t formally test, do your best with the information and data you do have. It’s a balancing act and you need to weigh the benefits (or lack therof) against resources, cost, etc.

A year ago I’d have said user testing shoud win out and be worth it in every case, now I don’t think so.

Posted on June 9, 2005 07:02 PM | #

15. Kev said:

“For example, why would I need to test for findability on say, a professional services site, that has amost the exact same audience, goals and content of a site I’d tested before? Shouldn’t my prior experience tell me something useful? I think so.”

You’re absolutely right. I’ll shut up now ;o)

Posted on June 10, 2005 12:37 AM | #

16. goodwitch said:

I totally agree that it would be impossible to do formal hands-on usability testing on every site you build. When I’m scoping out a project, I always include a usability feedback task and we discuss what will work within the time/budget restraints of the project.

One option is to have a “usability expert” do a heuristic evaluation. At times that usability expert might have to be the same person doing the design…but if you can swing it, tap a buddy on the shoulder and get a fresh perspective!

My favorite usability moment was when we did both a formal hands-on test and had an expert do a heuristic review. The two reports were over 90% in agreement! Why in the world did we do both? Politics. We did the hands-on test (on a shoe string) so we could hear what real people had to say. We had the usability expert do a heuristic report for that 100% credibility factor.

I wouldn’t advise doing both. I just found the results interesting. Mark it down as the advantage of working at a university where you’ve got amazing talent you can convince to do things for “free”.

Posted on June 12, 2005 01:24 PM | #

17. Marc Grobman said:

Certainly all sites deserve good design. And all sites should be tested on some level. I don’t agree that all sites need to have usability testing. We work with a lot of small businesses who have some pretty standard requirements. A very good sense of the principles of design and an intuitive sense for presenting information clearly is ultimately what is important.

And budget not to be taken lightly. Our clients struggle to pay for quality design work and have many other options including overseas development and templates. I know, and you know, that quality design is worth the money, but for some reason many companies fail to see that.

We often even struggle to convince our clients to take our advice on the basic organization of websites.

Getting back to the question – yes all sites
should be usability tested. It is just not practical.

Posted on June 13, 2005 12:30 PM | #