By Sarah Barrett
At the IA Summit in Minneapolis a few weeks ago, I noticed that “data science” had become a bit of a buzzword, and without pointing any fingers, I don’t think we as a discipline have really come to an understanding of what data science is or what it can do for us. Usually it’s a handy synecdoche for using more numbers in our work. I’d argue, though, that we need to focus less on the “data” and more on the “science.” Data science itself often makes the same mistake, so the UX/IA world doesn’t need to feel too bad, but examining the challenges data scientists are facing can potentially give us a framework for thinking about how to bring more rigor to our own work.
A recent article in Science by Jeffery T. Leek and Roger D. Peng gets at one of the simplest and most fundamental error humans make when doing analysis of any kind: “What is the Question?” Their subtitle is a simple assertion: “Mistaking the type of question being considered is the most common error in data analysis.” You could easily substitute “user research” or “analytics” for data analysis, and have it be equally true.
The flowchart below helps you figure out what kind of analysis you’re doing or are potentially set up to do: a key step in determining what your question is:
![What is the question? Jeffery T. Leek and Roger D. Peng Science 20 March 2015: 347 (6228), 1314-1315.Published online 26 February 2015 [DOI:10.1126/science.aaa6146]](https://factorfirm.com/wp-content/uploads/2015/04/leekarticleimage.png)
The vast majority of our work is going to be in that “Exploratory” stage, because making the step toward quantifying whether discoveries are likely to hold in a new sample —rather than assuming they will— is new to us.
I’ve found this way of thinking about structuring inquiry and analysis to be profoundly useful, partially because it shows how near the top of that chart our work —even at its most rigorous— remains, but also because it provides a way forward. How can we make sure we’re measuring accurately? How can we think more concretely about whether we’re describing or predicting behavior? What are we really trying to get out of our work?
Integrating data science into UX and IA work is ultimately going to take more than a greater attention to analytics or A/B testing, and focusing on those small elements of the larger scientific practice misses the real value of what that discipline can give to us. The tradition of the scientific method in the west has centuries of thinking about how to structure inquiry to answer questions, and how to know what question you’re asking. This is a good place to start.
