User Testing -- Your Take
January 10, 2004 | 
11 Comments
I was reading through this post on liquid vs. fixed designs at Design by Fire and the consequent discussion and it got me to thinking about user testing.
(Don’t worry, I’m not going to resurrect the whole liquid vs. fixed thing!)
In his post Andrei says:
Even after you have been given great data on what users need, you have to remember that what they need is only one third of the problem you have to solve as a designer. The other two thirds have to do with what makes sense for the business, and what makes sense for the content and data itself. The business goals and the content itself are just as important as what the user wants to do with it when creating design solutions. Making design decisions based on only what users want creates compromised designs.
He makes a good point, if a design decisions are made on user data alone you are in for trouble.
I have a method of making design decisions that has worked very well for me that I call “The Golden Triangle.” The idea is that you have to balance business goals, organizational goals and the user’s goals when making design decisions. I’ve talked about it a bit on this site and maybe someday, when I have time, I’ll write it up in more detail.
But I digress. What struck me was when Andrei says, “Even after you have been given great data on what users need.” It got me to thinking how many Web designers have had the opportunity to watch users use their sites personally and how other designers view user testing and user research.
I’ve done a ton of “hands-on” user research and think I’m a better designer for having done it. I’ve also worked on projects where I was given reports and data from usability specialists. I’m not really saying one way is better than another, but I would strongly suggest that if you do have to opportunity to interact with your users personally you take it. It can be a real eye opener.
I know that before I sat down with users I had a rather different view of how people used Web sites. Being told or reading about how users interact with sites and actually watching them are two very different things.
I’m also very surprised how many designers I’ve known over the years say things like “users want…” and “users do…” without having any prior usability experience at all. I realize that not all designers have had the opportunity do be involved with usability and frankly I think that is a cryin’ shame.
Agree? Disagree? Have you been involved with user testing and research? If so, to what extent? Did you find it helpful? I’d really be interested in hearing from those of you how come from a graphic or visual design background.
So, what’s your take on user testing?
Filed under: IA and Usability
Comments
1. patrick h. lauke said:
it saddens me to say that i’ve never had the time to do user testing directly, only responding to user feedback (be it email, web-forms, polls, etc) indirectly.
i’d love to do more of it. i see the wisdom in it (and yes, the results still need to be interpreted, and even before and during the test you need to set yourself precise targets on what you’re actually testing…it’s no good just to make a site and then “let them at it”).
in my experience with internal customers at our university, i also find that often - as patronising as it sounds - users don’t know what they want, and only after i implement something they see the logic in it and comment positively on it at later opportunities.
so i’m in two minds about it. i’d love to do more of it, but in some cases “mother knows best” can also work. probably a fine balancing act ? perhaps…
Posted on January 10, 2004 07:57 PM | #
2. Will said:
I’ve been using the Golden Triangle for a while now, without having a name for it. I find even with a mental model it can be quite difficult to achieve an elegant balance.
One practice I’ve taken up is that as I present clients with development versions of their sites, I shut my mouth and write down all their observations on a piece of paper. After they’re done, I then break their comments down into categories: errors, could-be-done-betters and matters of taste. When it comes to the latter, I explain why I did it the way I did and evaluate their response. After the meeting, I take my notes back to the “lab” and get cracking.
Posted on January 10, 2004 10:02 PM | #
3. Scrivs said:
I believe a problem that many people encounter with user testing is that they mistake it for opinion testing. A lot of times you will hear the user say that they “wish” something was here or that this should be placed here regardless if the site is usable or not. These type of “matters of taste” (as Will refers to them) should not be ignored, but possibly put at the bottom of the pile.
User testing should involve making sure the user can accomplish the company goals setup for the site.
Posted on January 10, 2004 11:44 PM | #
4. jarrod said:
I think Patrick’s “mother knows best” approach is best applied when showing comps to clients. That’s not really user testing, that’s showing the owner what you’ve done, and if you treat it like a usability test, you’re going to end up with something geared towards what the site owner wants, rather than what the end user of the site is going to be comfortable using.
That type of testing would be better applied as paper prototyping, in my humble opinion. It’s still a usability test, but in the early stages of the design, it’s easier to make changes to the layout rather than have to go back and expend more time moving things around that the site owner insists on. Then, once you have a layout that the owner is happy with, you can put it together and do some real testing, so you get valid information to take back to the site owner to justify making any additional changes.
I will admit to not having done any formal usability testing, but I have done some focus group work in the past relating to software (why, oh why did I take marketing in college?) in the past.
Just my $0.0253839 (current Canadian exchange rate :))
Posted on January 11, 2004 10:44 AM | #
5. Chris Wood said:
Love your blog - keep up the great work!
I agree with you 100% about the need for user testing and everything. At the risk of stretching things geometrically, how about a “golden square”? The other cornerstone being technical viability and non-goals.
These elements, I find, have played many a role in the downfall of projects I’ve worked on. For example, a developer goes off on a “wouldn’t be cool if” tangent and causes project deadlines to slip.
Another distinction that I often have to deal with is determining the difference between “user needs” and “user desires”. The example that’s often used in marketing circles is the difference between “I’m thirsty” (need) and “I need a Coke” (desire). The most remarkable example I can think of personally in IT terms was when a user once wanted several new check boxes in a user interface. The need they were actually expressing was for a new invoicing system!
Posted on January 11, 2004 10:51 AM | #
6. Bernd said:
Triangle, square… forgive me if I don’t get the point, but how can doing the right thing for the customer / user be in conflict with other interests?
I will gladly admit that there may be conflicting interests between developers and customers, but almost always the customer is right. I have been guilty too many times of justifying bad design myself with the “technically inevitable” label. And know what: User testing has always put it back on the table. You just can’t hide from it when naive users point out the flaws in your design. And if it causes a problem with the design: Fix the design.
Now, that may still mean that you have to interpret your user’s feedback. That’s when you have to distinguish between goals and non-goals. But whatever they say. There is always some truth in what they say!
Posted on January 11, 2004 03:37 PM | #
7. Ian said:
(Disclaimer: traditional IT developer background, not graphic design, and more informal work with users during analysis and post-implementation follow-up, but anyway …)
Chris gives an example of “user needs” vs “user desires” (the check-boxes vs. invoicing system one).
To me, that is an example of a more subtle and dangerous trap - the “user-defined solution”, more than a “desire” on the part of the user. I think the difference between “needs” and “desires” isn’t usually a clear-cut distinction, it’s more of a continuum that can be sorted by priority. Though I suspect (and will probably know for certain soon enough) that users will have more scope to request cosmetic changes to a web-based (i.e. site or web application) system than when constrained by an 80x24 dumb terminal where you can have any colour combination you like provided it’s green on black.
It’s all too easy to solve the wrong problem when a user presents a plausible solution as the requirement. Some instances of this can be harder to spot than others - sometimes it’s too easy to focus on the “solution” and infer the (wrong) requirement from that, without either party being aware that they’re not really addressing the underlying problem.
Posted on January 11, 2004 04:18 PM | #
8. Anton said:
I agree completely.
I too have had the opportunity to be a part of first-hand usability sessions - from my own work, to work that was outsourced to other design firms (I work at a large company), because we have a performance support team that works with us and project management.
Being a good designer is more than just coming up with what the company goals or client is looking for. It’s being able to accurately forcast beyond those scopes. Come up with something that is so intuitive that it becomes nearly invisible. To me, the best interfaces are the ones that are noticed the least.
It’s funny how a properly designed navigation or content structure flows into how the user just ‘clicks through’ items, not noticing anything until it gets in the way.
Posted on January 12, 2004 06:49 AM | #
9. Jeremy Flint said:
Usabitliy for our small company usually means sending the development URL to a bunch of friends and asking their opinions. Often times we will ask specific questions. “Can you find information about X” or “Tell me where Y is located”.
A lot of times if it is a redesign, I will ask my wife or my mom or another close friend not in the design field what they would change about a site if they could. Does the menu make sense to them. Do they feel that they can get to various parts of the site easily AND get back easily.
Posted on January 12, 2004 03:39 PM | #
10. Andy Budd said:
User testing is invaluable.
When web designers say things like “The user wants” what they are generally saying is “as a user, what I want” which is quite a different thing
Web designers tend to be super users. We are used to multitasking (often having multiple windows open at once) and aren’t afraid of trying a course of action out just to see what happens. However this isn’t the norm for the “average user”. It’s only too easy for us experienced users to forget what it was like when the web was new to us and we hadn’t learnt the conventions and devised a coping strategy.
Usability testing is a great way to make sure that your system is as intuitive and as easy to use as possible. However usability testing isn’t a magic bullit. It doesn’t tell you if the product, service or even the visual design is going to be a success. The purpose of usability testing is to take the sharp edges of your system so the users don’t end up hurting themselves.
What the web design industry needs to do is take a leaf out of traditional industries like advertising, marketing and PR and start to use surveys and focus groups in order to extract statistically valid data and help focus the clients offerings. Usability testing can help make a system easy to use, but good planning and research is needed to make sure the system fulfils a need, has a market and is likely to be a viable proposition.
Posted on January 14, 2004 02:04 AM | #
11. mill said:
hi,
was wondering what’s the difference between user testing and usability testing? Am trying to make a clear distinction and would appreciate it if someone could shed light for me pls? Thanks…
Posted on July 25, 2004 06:25 AM | #
Comments are now closed