User Research Smoke & Mirrors

by Christopher Fahey (2006)

I find it so intriguing, the way Fahey approaches the subject matter, and the way he writes, of course. Seemingly casual upon first glance, he actually takes my reflections to a deeper (more intellectual) level by critiquing the real value of research in designing for users. Truly, it was as if I could feel his burden for subjectivity imposing its weight on results and therefore biased design experiences.

I suspect that a lot of user research in this industry is a sham.
That line really takes the cake. I like his style :)

There are many methods which seem to not be such an issue but turn out to be the parasites of each research:
1. small samples
absurdly crafted test cases
methods which have passing resemblences to science
misinterpretation of data
subjective studies
and subjective studies which are made to look objective

Fahey brought up one point which made me think, "Back to basics!" He wrote that people are giving "greater credence to (expensive) pseudo-scientific processes than to common-sense good design principles".

I found that this is so common-sensical, if I may abuse my limited vocabulary for description. It is so true that people start to go for all the empirical evidences and design technologies/ advancements that have appeared in the market, that they are giving the common-sense a miss. Perhaps it rises from the need to supercede the self and the constant need to prove something - and that stems from the inherent competitive nature of human beings. That's just my take.

Given the example of the Eyetracking tool which tracks the eye movement of viewers of websites, I agree that the system only tracks what people are looking at but can we really find out what they are seeing? "Seeing" takes it one level further from "looking". "Seeing" implies processing the information and applying it in a suitable way. And the analysis discovered that good conclusions seem to be the same conclusion that a good UI designer would go by just pure design instincts, whether they make use of tools or not. It's about the feeeeeel....

Such tools just scratch the surface in knowing what users really want, think, feel and need. Sure, they help, but they are just aids in giving the designer an area to pay attention to. For instance, the example giving in the article is about the Eyetracking tool showing that users look at the company logo the most. Does this mean that the logo needs to be changed? I think the inherent assumption that people have from interpreting data from such tools is that whatever element the users look at the most, there is where it is the most problematic. That is nonsense! It doesn't mean that looking at something means it is erronous and incoherent with "nature". (Men like to look at pretty girls, so is there anything wrong with pretty girls? hahaha) So is that something interesting or perplexing?

This would then be a great stepping stone to probing what is the issue behind the attention paid to certain elements. It could be a positive or negative response. And that is for the designer to find out. Don't assume! I think we still got to get to the root of the issue and that can only be elicited by going to the root of the issue - the users.

But there's always redtape and bureaucracy like in any capitalist society. And knowing creativity and design is a flair that not everyone has. Like how not everyone has scientifically-tuned brains which could possibly think up concepts like gravity.

So the emphasis is that research can be a good means to justify design decisions especially when the other party is not well-versed in design but are stakeholders (and where the money is coming from). But research just cannot do it they simply give the illusion of a level of authority and accuracy that simply does not exist (as in the Forrester case).

I like it when Fahey describes graphic designers as professional communicators whose experience and talent largely define their effectiveness and that it is a waste of time and money to request for justifications for designs that stretch the limits of credibility.

I personally have a preference for qualitative research over quantitative research. True, quantitative results can give us statistics to base our arguments on. They are facts as collected from our researches and observations. However, I question the truth of surveys because they are basically just surveying human beings and how true can the "true" results really be? It is not a general phenomenon nor can the single truth ever be formulated. I find that personal experiences and stories as narrated by people are great indications as close to the "truth" as possible. They can be subjective; that we cannot discount. However, as people give their honest (trust their integrity here) feedback and accounts, there will be instances to elicit much necessary and useful information for design processes. But is this the case and belief of everyone out there?

Fahey mentioned that it is easier to defend science than to defend opinion. This is the point exactly; the essence of scientific cold hard facts as supporting data for decisions, versus the feel and the instinct of something considered superfluous like design and creativity. It seems to be a battle between the tangible and the intangible. People find a need to explain things so they will feel more at peace with the world and themselves. This is one way people seek to gain control over their surroundings. And science comes along with the promise of solution. But we cannot ever discount that, as Fahey puts it, it'll appear to just be "psuedoscience" - science for the sake of being science.

Being able to prove something is right has its own dangers because it means that it will also be able to prove something is wrong.

Experience is experience, not fact :)