Statistical Sins: Olympic Figure Skating and Biased Judges

The 2018 Winter Olympics are almost here! And, of course, everyone is already talking about the events that have me as mesmerized as the gymnasts in the Summer Olympics - figure skating.

Full confession: I love figure skating. (BTW, if you haven't yet seen I, Tonya, you really should. If for no other reason than Margot Robbie and Allison Janney.)

In fact, it seems everyone loves figure skating, so much that the sport is full of drama and scandals. And with the Winter Olympics almost here, people are already talking about the potential for biased judges.

We've long known that ratings from people are prone to biases. Some people are more lenient while others are more strict. We recognize that even with clear instructions on ratings, there is going to be bias. This is why in research we measure things like interrater reliability, and work to improve it when there are discrepancies between raters.

And if you've peeked at the current International Skating Union (ISU) Judging System, you'll note that the instructions are quite complex. They say the complexity is designed to prevent bias, but when one has to put so much cognitive effort into understanding something so complex, they have less cognitive energy to suppress things like bias. (That's right, this is a self-regulation and thought suppression issue - you only have so many cognitive resources to go around, and anything that monopolizes them will leave an opening for bias.)

Now, bias in terms of leniency and severity is not the real issue, though. If one judge tends to be more harsh and another tends to be more lenient, those tendencies should wash out thanks to averages. (In fact, total score is a trimmed mean, meaning they throw out the highest and lowest scores. A single very lenient judge and a single very harsh judge will then have no impact on a person's score.) The problem is when the bias emerges with certain people versus others.

At the 2014 Winter Olympics, the favorite to win was Yuna Kim of South Korea, who won the gold at the 2010 Winter Olympics. She skated beautifully; you can watch here. But she didn't win the gold, she won the silver. The gold went to Adelina Sotnikova of Russia (watch her routine here). The controversy is that, after her routine, she was greeted and hugged by the Russian judge. This was viewed by others as a clear sign of bias, and South Korea complained to the ISU. (The complaints were rejected, and the medals stood as awarded. After all, a single biased judge wouldn't have gotten Sotnikova such a high score; she had to have high scores across most, if not all, judges.) A researcher interviewed for NBC news conducted some statistical analysis of judge data and found an effect of judge country-of-origin:

As a psychometrician, judge ratings are a type of measurement, and I personally would approach this issue as a measurement problem. Rasch, the measurement model I use most regularly these days, posits that an individual's response to an item (or, in the figure skating world, a part of a routine) is a product of the difficulty of the item and the ability of the individual. If you read up on the ISU judging system (and I'll be honest - I don't completely understand it but I'm working on: perhaps for a Statistics Sunday post!), they do address this issue of difficulty in terms of the elements of the program: the jumps, spins, steps, and sequences skaters execute in their routine.

There are guidelines as to which/how many of the elements must be present in the routine and they are ranked in terms of difficulty, meaning that successfully executing a difficult element results in more points awarded than successfully executing an easy element (and failing to execute an easy element results in more points deducted than failing to execute a difficult element).

But a particular approach to Rasch allows the inclusion of other factors that might influence scores, such as judge. This model, which considers judge to be a "facet," can model judge bias, and thus allow it to be corrected when computing an individual's ability level. The bias at issue here is not just overall; it's related to the concordance between judge home country and skater home country. This effect can be easily modeled with a Rasch Facets model.

Of course, part of me feels the controversy at the beginning of the NBC article and video above is a bit overblown. The video fixates on an element Sotnikova blew - a difficult combination element (triple flip-double toe-double loop) she didn't quite execute perfectly. (She did land it though; she didn't fall.)

But the video does not show the easier element, a triple Lutz, that Kim didn't perfectly execute. (Once again, she landed it.) Admittedly, I only watched the medal-winning performances, and didn't see any of the earlier performances that might have shown Kim's superior skill and/or Sotnikova's supposed immaturity, but I could see, based on the concept of element difficulty, why one might have awarded Sotnikova more points than Kim, or at least, have deducted fewer points for Sotnikova's mistake than Kim's mistake.

In a future post, I plan to demonstrate how to conduct a Rasch model, and hopefully at some point a Facets model, maybe even using some figure skating judging data. The holdup is that I'd like to demonstrate it using R, since R is open source and accessible by any of my readers, as opposed to the proprietary software I use at my job (Winsteps for Rasch and Facets for Rasch Facets). I'd also like to do some QC between Winsteps/Facets and R packages, to check for potential inaccuracies in computing results, so that the package(s) I present have been validated first.

On New Year's Resolutions

Probably because I was on the road most of yesterday, no one has asked me yet about my New Year's resolutions (which is fine, because I was still trying to figure out what they would be exactly). But Glamour thinks you shouldn't even bother:

For thousands of years (truly—going all the way back to the Roman era, when people made promises to the god Janus), we’ve endured your vicious circle, listing our perceived shortcomings, trying to improve for a few weeks, then giving up and feeling bad all over again. Consider this our breakup notice. Turns out you don’t really help anyway: “The rah-rah excitement about a resolution works for only a week or two,” says John Norcross, Ph.D., a professor of psychology at the University of Scranton in Pennsylvania and author of Changeology. (He studies the stickability of New Year’s resolutions for a living, so trust.) And as much as we might want to eat more greens or go to the gym more often, most of all we want to be happy in 2017: free of guilt and shame. So we’re pledging not to make any of these clichéd, old, tired resolutions again. Who’s with us?

It's true, New Year's resolutions are difficult to keep - here's a blog post from last year in which I cited research by Dr. Norcross quoted above. But that doesn't mean you should stop trying; you just need to be SMART about it. That is, make sure your goals are:

Specific
Measurable
Attainable
Relevant
Time-Bound

So, what are my New Year's resolutions?

1. Visit at least 1 new state - I'll be turning 35 in less than a week, and I've currently visited 34 states. So why not make it 35 states at 35 years old?
2. Finish my book - and hopefully get it submitted somewhere for publication. I know there are options like self-publishing, which I would probably more strongly consider if writing were my profession, but I don't want to deal with the hassle of promotion, etc., myself. Since this is just a hobby, I can be a little more particular about these things.
3. Average 1 blog post per day for the year - I know there will be days I write two posts and days I don't write one at all, but as long as I end up with 365+ blog posts for the year, I'll be happy
4. Make it a mission to reach 10,000 steps per day - I was doing really well until I started my new job, but this year, I'm going to try to reach 10,000 steps each day (knowing there will be days where it's just impossible, e.g., I get a migraine, I spend the whole day driving, etc.)
5. Read 24 books - I've already set my Goodreads challenge; last year, my goal was 25, and I wanted to increase it for this year, but since I want to write more, I think 2 books a month is a good goal

Oversharing in the Age of Social Media

Self-disclosure is how we build strong relationships with others. By telling someone else about your inner world, you are showing that you trust them and can build a deeper connection with them, especially when they, in turn, show you their inner world back (which people are generally inclined to do, through a concept known as reciprocity). In fact, there are many explanations for why self-disclosure improve bonds - not just the trust it displays, it also often provides information that may show you have something else in common with someone (putting them more strongly in your in-group, which you are biased to evaluate positively). Sharing also increases the perception that a person is trustworthy. In fact, research has shown that people who self-disclose are evaluated more positively than people who withhold information, even when the information disclosed is really terrible (like failing-to-tell-someone-you-have-an-STD terrible).

The advent of Facebook and other social media sites provide many opportunities to share information about oneself, but as many have observed, it also provides an avenue to overshare. Now any thought you had, any activity you did, can be broadcast to all of your friends, or an even larger audience if a post is "public." And people use social media to share things with the world that they would never dare shout in a crowded room, even though it's basically the same thing (and arguably worse unless someone in that crowded room is recording the whole thing). But why? Researchers offer some answers, and they're pretty much what you would expect (if you know about social psychology):

Social scientist and author Sherry Turkle thinks we’re losing a healthy sense of compartmentalization. Last year, researchers at Harvard found that the act of sharing our personal thoughts and feelings activates the brain’s neurochemical reward system in a bigger way than when we merely report the attitudes and opinions of others. Meanwhile, Elizabeth Bernstein of the Wall Street Journal asked around and concluded that our newfound urge to disclose is partially due to not only the erosion of private life through the proliferation of reality TV and social media, but also due to our subconscious attempts at controlling anxiety.

“This effort is known as ‘self regulation’ and here is how it works,” she writes. “When having a conversation, we can use up a lot of mental energy trying to manage the other person’s impression of us. We try to look smart, witty, and interesting, but the effort required to do this leaves less brain power to filter what we say and to whom.”

Another crucial ingredient encouraging online exhibitionism is, as stated by [Russell W.] Belk, [chair in marketing at York University in Toronto], the “tension between privacy and potential celebrity.” For some people, the longing to be popular far outweighs the longing to be respected, and their social media accounts can verify this.

I would also add that many people don't think through the gravity of what they share online. They're in a situation where they can impulsively share whatever they want, but don't think through the potential consequences. For instance, I would assume most people know not to let all their friends take a picture of their credit card, while adding, "BTW, the code on back is...", and yet, there are countless people who have posted a picture of their credit card on their social media (along with, "The code on back is... Why is everyone asking me that?").

A few years ago, my mom posted an unedited picture of her passport online, because she was happy to finally have one. Since I get notifications when she posts something, and I know her username/password (guess who set it up for her), I was able to login and delete it only a few minutes after it was posted. My mom is one of the smartest, most logical people I know, who would absolutely know better than to let a stranger take a picture of her passport. I don't think she made the connection that posting it on social media is basically the same thing. (A public post, no less! We had a convo about privacy settings after that.)

And I think the reason for that is because we perceive social media as a toy, something for entertainment value. We know better than to do some of these things in the real world, because that's our life, but Facebook is just something we do for fun. So while Sherry Turkle above may say we're losing a sense of compartmentalization, I would argue we're too compartmentalized in how we perceive social media. We're creating a division between real life with real consequences and online life for play, failing to realize that our social media activity also can have real consequences: having your credit card info "stolen" (or rather, giving it away without realizing it), losing a job, being put on a no fly list, all things that have happened.

Several years ago, I did a study on the potential negative effects of Facebook on mental health. When people asked me if the point of my study was to tell people not to use Facebook, my response was this: Facebook is a tool. A tool isn't inherently good or bad. It's a matter of how it's used. A hammer can help you build a house (good) or smash your finger (bad). The problem isn't the hammer, it's how you used it.

But when you're using a tool, it's important to know how to use it properly and safely. We might get that lesson from someone the first time we pick up a hammer. But no one gives us that lesson the first time we log on to Facebook. And maybe they should.

More About Self Theories: It's Not About Views on Success, But Views on Failure

I blogged recently about self theories, which refer to whether one believes intelligence is innate (fixed) or learned (incremental). As a result of this work, many psychologists have cautioned parents about the type of praise they give their children when they succeed. "You're so smart" implies that intelligence is innate, while "You worked really hard" implies that intelligence/ability can be learned.

However, I learned about a study yesterday that says it isn't parents' views on success, but failure, that impact their children's self theories. They conducted 4 studies, 2 with parents only and 2 with parents and their children. Across all four studies, they found that parents' self theories did not predict children's self theories, but that instead, parent's views on failure predicted children's self theories:

Our findings indeed show that parents who believe failure is a debilitating experience have children who believe they cannot develop their intelligence. The findings further suggest that this is because these parents react to their children’s failures by focusing more on their children’s ability or performance than on their learning. Taken together, our findings seem to have identified a parental belief that translates into concerns and behaviors that are visible to children and that, in turn, shape children’s own beliefs.

It is easier - although maybe not easy - to be watchful about the kind of compliments one gives to children, but when your child has failed at something and he/she is upset, it's more difficult to think rationally about the right response. In those situations, we tend to just react. The problem is, if that reaction is devastation that your child has failed, it sends a message that your child can do nothing to learn and develop his/her abilities. As a result, children may disengage and instead do something "safer" - something at which they know they will succeed. They may miss the opportunity to grow and challenge themselves - challenges which have many cognitive and emotional benefits.

What this means is, if we want to avoid having our children develop a fixed view of intelligence, we have to change our own view of intelligence - not just in what we say. How do we do that? Unsurprisingly, Carol Dweck, who developed the concept of self theories, has designed an intervention called Mindset, though it is aimed at students. But perhaps another way is to adapt an approach used in cognitive therapy and similar interventions: self-monitoring and mindfulness.

In these approaches, the individual monitors his or her thoughts for instances of negative self-talk. They are not trying to suppress those thoughts - because we know that can backfire. Instead, when they encounter those thoughts, they practice acceptance of any mistakes they made that led to that self-talk and may even come up with rebuttals to those negative thoughts. You can read a little more about mindfulness and negative self talk here. In fact, as you'll see in the article, there are many common threads between mindfulness and incremental views of intelligence. For instance, the article states:

People with good self-esteem see mistakes and failures as opportunities to learn about themselves. They take a "beginner's mind" approach - putting aside the judgements and conclusions from past behaviour and actions and, instead, thinking about what they've learned from these experiences.

And for a humorous approach to defeating negative self-talk, you could try this:

Follow-up On Self-Regulation and New Year's Resolutions

January is over and February is quickly coming to an end. But as this blog post points out, it's never too late to make a resolution and set some goals. The post also discusses some research on self-regulation, finding that people who followed a weight management program for two months also saw improvements in their ability to complete other important tasks. This just drives home the point that self-regulation is a general skill and that learning this skill, even in one specific application, can have far-reaching improvements on other aspects of life.

At the same time, it is important to remember that self-regulation uses mental resources, and regulating one aspect - especially if it is very taxing to maintain - can limit how much you can regulate in other domains. It's all about balance.

My New Year's Resolutions

New Year's Day was a week ago, but it's never too late to come up with some New Year's resolutions, right?

1. Write more - That includes not only writing more for this blog, but also getting back to writing projects I started ages ago. For the last couple of years, I've done National Novel Writing Month (or NaNoWriMo), which occurs in November. And each November I get little to nothing written. In part, it's because November is a busy time and I did a lot of writing in November 2015, just not on my novel. Instead, I worked on four grant proposals submitted in December, including one that I wrote primarily in my "free" time. November was also difficult this last year for other reasons, another reason for neglecting novel ideas. But that's okay, because there are 11 other months in the year that can be spent writing!


2. Stop stressing over things I can't change and do something to change the things I can - This sounds suspiciously like the Serenity Prayer. I prefer this one:

We can't always control the things that happen to us. And we certainly can't control the reactions and behaviors of others. The only thing we can control is our own behavior and reactions. So my resolution for this year is to stop worrying about whether people like me, or if things are going poorly, or any number of things outside of my control. Not only that, but I can always reframe those negative things outside of my control - thinking of them in a way that makes them neutral or even positive (maybe even funny!).

3. Take better care of myself - Not just physically (though I'm working on it), but also emotionally. This ties into the first two, of course. Writing is one of the activities I enjoy, that gives me peace. And worrying - something I did a lot of grad school, but had been able to stop doing for several years - has been go-to response lately. I just have to get better at telling myself to stop it, and I'm happy to report in the last couple weeks, I've succeeded pretty well.

4. Savor the good things in life - Let's face it, I'm pretty damn lucky. Even with set-backs, things work out. And fortunately I was gifted with a great brain that helps me get out of these set-backs and get back to good. So it's time to savor those good things. I've done some research on savoring before, a concept developed by one of the professors I worked with in grad school. It involves thinking about and mentally re-experiencing the good things in life to maximize enjoyment. We can savor past events (reminiscing), present events (savoring the moment), and potential future events (anticipating). Savoring increases happiness and well-being, and buffers the impact of negative life events and stress.

The tl;dr version?

Resolvedly yours,

~Sara

More on Self-Improvement: New Year's Resolutions

Happy New Year!

Congratulations, everyone - we've made it to 2016! Hopefully you had a wonderful celebration bringing in the new year and did exactly what you wanted to do - whether that's going to a party or staying home for a quiet night.

The new year is an exciting time because for many, it is seen as an opportunity to start over. To make changes from what we didn't like about the previous year and move toward self-improvement. One way we do this is in the form of New Year's resolutions.

Research (by Norcross, Mrykalo, & Blagys, 2002) suggests that almost half of Americans make a New Year's resolution, with the most frequent being: weight loss, exercise, and smoking cessation. Unfortunately, the same research suggests that we have a lot of difficulty maintaining these resolutions.

No, seriously, a lot.

Norcross and colleagues found that after one week, only 77% of people who had made a New Year's resolution had stuck with it. By the two year mark, only 19% had stuck with their resolution - mind you, these resolutions are usually lifestyle changes that people would expect (hope) to be permanent. When a person resolves to, say, quit drinking, they don't usually mean for just a few days.

Why are people who voluntarily make a New Year's resolution - I mean, it's unlikely someone put a gun to their head and forced them - so bad at sticking with it?

As a recovering behaviorist, I can tell you there are a variety of factors at play. Most importantly, behavior change isn't easy. If it were, Americans wouldn't be spending almost $10 million a year on self-help books and programs. And they certainly wouldn't be spending $90 billion on alcohol each year, nor would we have over 260 billion cigarettes purchased each year, in the US alone.

There are a variety of influences on our behavior. Some are internal, involving our feelings and thoughts, like motivational states, or cravings and physiological reactions. Others are external, which can include reactions by others, or something more concrete, like money. Successful behavior change involves targeting both types of influences. For instance, a person who wants to quit smoking not only needs to feel strong motivation and figure out ways to handle cravings and withdrawal symptoms, they may also need to make changes to their environment, even changing who they associate with and settings in which they socialize.

Changing a behavior, too, requires a combination of internal and external motivators, though internal motivators tend to be stronger. Why? Because external reinforcers of behavior, like money, food, or even praise from others, lose their effectiveness over time. Receiving $5 when you have nothing is lot more rewarding then receiving $5 when you already have $100 accumulated. Being given tasty food when you're hungry is more rewarding then being given tasty food when you're full. And so on. As the old joke goes, it may only take one psychologist to change a light bulb, but the light bulb has to want to change.

People may also set goals without really thinking about the specifics about that goal and how to get there. One thing behaviorists encourage is for people to set SMART goals: specific, measurable, attainable, realistic, and time-sensitive. I may resolve to exercise more, but lose steam because I don't have the specific details about what I mean by that. If I don't sit down and determine those details, it will be difficult, maybe even impossible, to get started.

On the other hand, if I resolve to exercise 30 minutes a day, three times a week, I have something to work with. True, there's more I need to figure out, like, "Is that goal realistic with my schedule?" or "What kind of exercise?" or "How long should I give myself to work up to that level of exercise?". But even before I figure out those issues, I already have something specific and measurable.

Whatever the goal, there are multiple programs and interventions that can help people reach it. However, another problem people run into is that some (maybe many) programs aren't based on strong science. So people may jump on a particular bandwagon without doing their homework to learn whether that bandwagon actually works. This is likely another reason for the high attrition in maintaining resolutions. Doing that homework requires a certain level of scientific literacy, but probably a good mindset to start with is, if it sounds too good to be true (Quit smoking in 3 days! Lose 10 pounds in a week!), it probably is.

When I taught Learning & Behavior at the undergraduate level, I had my students complete a behavior modification project, in which they identified a behavior they wanted to change and spent the semester applying the topics they learned in class to changing it. Students seemed to enjoy the experience, and many were successful. Those who weren't reported that they had made some positive changes, and now had the tools (and hopefully motivation) to keep working. Too bad I can't follow-up now and see how many are still successful!

What about you, dear reader? Did you make a New Year's resolution? If not, no worries. You're awesome! If so, I wish you all the luck and simply ask that you consider some of the things I've shared to help you get there.

Newly yours,
~Sara

On the Brain, Learned Helplessness, and Self-Improvement

As I was out to lunch today with a friend, I heard a TV show host once again bring up the old myth that you only use 10% of your brain. In the story, she was talking about what people will look like in 10,000 years. Spoiler alert: We have bigger eyes and darker skin. She mentioned that we'd probably also be capable of using more of our brains at that point.

The origin of the "10% myth" is debated, though the most likely source is psychologist William James, who talked about how humans only use a fraction of their potential mental energy. This statement was twisted somehow into the belief that we only use a fraction of our brain - not quite the same thing.

Why does this myth prevail despite its obvious falsehood? After all, if you really sit down and think logically about all the things your brain does (beyond conscious thought), the amount of body energy the brain uses, and the significant impact of insults to the brain (such as stroke or injury), you would have to conclude that we use much more than 10% of it.

One reason this myth may prevail is potential. Or rather, the desire of people to believe they have untapped potential. Believing that you can improve in some way is incredibly motivating. On the other hand, believing that you lack any potential can result in stagnation and inaction (even when you actually can do something).

Some of the early research in the concept of learned helplessness involved putting dogs in no-win situations. The dogs were paired (yoked) with another dog who learned a task. If the learner dog behaved incorrectly, it received a shock - but so did the helpless dog. So while the learner dog received cues that would warn of an impending shock, and could change its behavior to avoid the shock, the helpless dog did not and could not. When the helpless dog was put in the learner dog's place, it did nothing to avoid the shocks. The helplessness of its prior, yoked situation carried over into the learning situation, and prevented the dog from seeing how its behavior could affect the outcome. Later research has demonstrated humans can also exhibit learned helplessness, and this concept has been used to describe the behaviors of survivors of domestic abuse.

The desire for self-improvement, an outcome of the belief in potential, drives a great deal of our behavior. For instance, I learned that one of the most popular Christmas presents this year was the FitBit, a wearable device that tracks activity. I also received a FitBit for Christmas and have so far really enjoyed having it. Not only does it tell me what I'm already doing (number of steps, hours of sleep, and so on), it gives me data that can be used for self-improvement.

With New Year's upon us, many people will probably begin a path to self-improvement through New Year's resolutions. Look for a post about that (hopefully) tomorrow!

Potentially yours,
~Sara