Not everything that can be counted counts

I think the following is a quote from Einstein:

"Not everything that can be counted counts, and not everything that counts can be counted"

I have been considering this a lot recently because as you try to improve products you take in a bunch of metrics and ideas but not all of them are useful and indeed many may be misleading. As I consider the data gathering exercise I am being made aware of the well documented strategies for systematic collection, organization and interpretation of information.

For my work they roughly bifurcate into two areas quantitative and qualitative.

The following lists compare and contrasts the key differences:

Qualitative

• Approach: Inductive (ground up)
• Goal: Depth of understanding, generate hypotheses
• Setting: Natural setting (observation)
• Sample: Purposeful (explicit, )
• Data Collection: Interview guides, observation tools, open ended
• Data Analysis: Iterative interpretation

Quantitative

• Approach: Deductive
• Goal: Breadth of understanding, test hypothesis
• Setting: Experimental
• Sample: Random
• Data Collection: surveys, administrative/clinical data
• Data Analysis: Statistical tests, modeling

I am working on an area that is fairly well understood, that is, a lot of clever people have considered these concepts for a long time. This is one of the reasons why I have narrowed my analysis to qualitative measures, my ultimate goals are to:-

1. Discover patterns and unknown insights
2. Set objectives, Create hypotheses, and reach conclusions
3. Invent possible futures

Ethnographic research

Ethnographic research is probably the most in-depth observational method that studies people in their naturally occurring environment. One of the things I have noticed, after interviewing dozens of people, is that what people say and what they do can be radically different. Occasionally with the best of intentions developers present the best versions of the themselves, or even the worst, neither are necessarily true. So rather than ask folks what they do, I observe what they do, it is always the most important version.

Even with this kind of observation it still necessitates questioning and the very best interviewers use a kind of open ended questioning. Additionally you can use what is called a probing technique, this allows you to extend each question to uncover more ideas and insights that the participant may not have originally thought of. For example a person responds with a one word answer, you might say the following:

• "Why is that?"
• "Why do you think that is?"
• "You said" (retrospective probe)
• Say nothing…
• "Can you tell me more about that?"
• "What happened after that?
• "Can you give me an example of that?"
• "What did you think/feel about that?"

Anyway I am really enjoying this process now, took me a while to get the hang of it. I am perfecting my technique by actively listening and asking revealing questions.