https://web.media.mit.edu/~lieber/Misc/Tyranny-Evaluation.html
Lieberman argues that Academic HCI over-emphasizes evaluation, mostly from an epistemological perspective—i.e. that the methods being used will not achieve their ostensible goals.
That’s fine, but I think this is less interesting than the deeper argument that it’s a focus on the wrong kind of knowledge. Henry turns to that at the end:
I hate to break the news to some of you, but user interface design is, in no small part, art, as much as it is science or engineering. That’s not a bad thing. It doesn’t mean that we shouldn’t ask scientific and engineering questions about our interfaces, just that they are not the whole story.
More importantly, they simply don’t seem to align with real-world successes as often as you might hope:
User interface evaluation has a lousy track record. It isn’t much good at predicting which interfaces will be accepted and appreciated by users, or which will be commercially successful. Even when there is a definitive result from testing, people often ignore it, most often wrongly. Studies show that pie menus are better than linear menus. The results make sense and have been empirically verified. Yet few commercial systems use pie menus. And, for God’s sake, why don’t we yet have two-handed pointing despite experiments showing it is superior for many tasks to a single mouse?