Wednesday, October 25, 2017

Statistical Sins: Throw Some Psychometrics At It

Around the time I started learning about social media data mining approaches, I began subscribing to a newsletter from a social media analysis group called Simply Measured. I've shared some of these resources with friends who oversee social media for different nonprofits, and I occasionally do learn something from these newsletters. I was excited when I saw the top story for their recent newsletter was using psychometrics in marketing.

And then confused when I read the one-line summary of the blog post: Learn how psychological motivations can help you build compelling content and drive more sales.

Wait, do they think psychometrics means "measuring something psychological"? I decided to read the post. Here's how they define the term:
The term refers to ways of measuring an individual’s motivations, preferences, and interests without directly asking, adding a level of authenticity to your data as we observe our audience’s natural behaviors. For this reason, psychometrics is often more accurate than direct questioning.
Yeah, not exactly. You could certain directly question people about something, then use psychometric analysis approaches to identify the best questions. Ways of measuring something refers to the type of test or measure, which could be paper/pencil, observation, etc.

Later on they say, "Psychometrics is an old method of gathering data." Again, no. Psychometrics is not a data gathering method at all. A survey would be a data gathering method, and you could use psychometrics to create and analyze your survey data.

Psychometrics is basically a set of methods and statistical analyses that are used to develop, validate, and standardize measures. Those measures could be about "motivations, preferences, and interests," but they could also be about ability, personality, presence of symptoms of a physical health condition, and so on. It could be about anything you want to measure, not just about individual people but even about organizations or teams. And psychometric analyses provide tools to assess the validity (measures what it's supposed to) and reliability (measures something consistently) of your measure. Many of these tools are general statistics, like correlation coefficients, that are simply being applied to psychometric research, but other tools, like item difficulty, internal reliability, and item fit, were specifically developed for psychometric applications.

But the shark-jumping moment of the post was when they started talking about using psychometrics for online quizzes - those quizzes that promise to tell you "what puppy you should adopt based on the color of your aura" or "which Taylor Swift song you embody when you're depressed" and so on. I mean, I love dumb quizzes as much as the next guy, but I don't ascribe any validity to these measures. And if someone told me they used psychometrics analyses to develop one of these quizzes, I would be very confused.

I mean, sure, I guess you could do that. It wouldn't really work, but hey, give it a shot. You see, part of developing a strong measure is establishing content validity. As I mentioned in a previous post on content validation studies, there are different ways to do that, but you need some method of showing the items you have on your measure really do relate to the underlying concept. And I'm going to go out on a limb and say there is no gold standard for assessing your personal Taylor Swift song. Maybe if my content validation study panel consisted of the various Taylor Swifts from the "Look What You Made Me Do" video:

The whole post felt like the author wasn't sure what to write about, found a word on a webpage somewhere, and decided to write an entire post on what he thought that thing was without doing any kind of research into what the thing actually is. Yes, you can absolutely use psychometrics in marketing research. In fact, it would be great if people could demonstrate some validity of the measures they use in marketing research. But you won't learn how to do that from that Simply Measured post. In fact, you'll walk away from the post with a flat-out incorrect conception of what psychometricians actually do.

I'm fully aware that in writing this blog post, I'm basically this guy:

Simply Measured, you done me and my field wrong.


  1. Interesting post, thank you for the context! It looks like the author is simply repeating the misuse of the term he encountered elsewhere - like in the articles he links to:

    1. Good point. I can't exactly pin this entire statistical sin on this one poor blog post author. I'll admit I didn't click on every single link, but I did glance at this one. It perpetuated some incorrect information, but still, it wasn't solely about psychometrics, unlike the Simply Measured post.