Statistical Sins: Olympic Figure Skating and Biased Judges

The 2018 Winter Olympics are almost here! And, of course, everyone is already talking about the events that have me as mesmerized as the gymnasts in the Summer Olympics - figure skating.

Full confession: I love figure skating. (BTW, if you haven't yet seen I, Tonya, you really should. If for no other reason than Margot Robbie and Allison Janney.)

In fact, it seems everyone loves figure skating, so much that the sport is full of drama and scandals. And with the Winter Olympics almost here, people are already talking about the potential for biased judges.

We've long known that ratings from people are prone to biases. Some people are more lenient while others are more strict. We recognize that even with clear instructions on ratings, there is going to be bias. This is why in research we measure things like interrater reliability, and work to improve it when there are discrepancies between raters.

And if you've peeked at the current International Skating Union (ISU) Judging System, you'll note that the instructions are quite complex. They say the complexity is designed to prevent bias, but when one has to put so much cognitive effort into understanding something so complex, they have less cognitive energy to suppress things like bias. (That's right, this is a self-regulation and thought suppression issue - you only have so many cognitive resources to go around, and anything that monopolizes them will leave an opening for bias.)

Now, bias in terms of leniency and severity is not the real issue, though. If one judge tends to be more harsh and another tends to be more lenient, those tendencies should wash out thanks to averages. (In fact, total score is a trimmed mean, meaning they throw out the highest and lowest scores. A single very lenient judge and a single very harsh judge will then have no impact on a person's score.) The problem is when the bias emerges with certain people versus others.

At the 2014 Winter Olympics, the favorite to win was Yuna Kim of South Korea, who won the gold at the 2010 Winter Olympics. She skated beautifully; you can watch here. But she didn't win the gold, she won the silver. The gold went to Adelina Sotnikova of Russia (watch her routine here). The controversy is that, after her routine, she was greeted and hugged by the Russian judge. This was viewed by others as a clear sign of bias, and South Korea complained to the ISU. (The complaints were rejected, and the medals stood as awarded. After all, a single biased judge wouldn't have gotten Sotnikova such a high score; she had to have high scores across most, if not all, judges.) A researcher interviewed for NBC news conducted some statistical analysis of judge data and found an effect of judge country-of-origin:

As a psychometrician, judge ratings are a type of measurement, and I personally would approach this issue as a measurement problem. Rasch, the measurement model I use most regularly these days, posits that an individual's response to an item (or, in the figure skating world, a part of a routine) is a product of the difficulty of the item and the ability of the individual. If you read up on the ISU judging system (and I'll be honest - I don't completely understand it but I'm working on: perhaps for a Statistics Sunday post!), they do address this issue of difficulty in terms of the elements of the program: the jumps, spins, steps, and sequences skaters execute in their routine.

There are guidelines as to which/how many of the elements must be present in the routine and they are ranked in terms of difficulty, meaning that successfully executing a difficult element results in more points awarded than successfully executing an easy element (and failing to execute an easy element results in more points deducted than failing to execute a difficult element).

But a particular approach to Rasch allows the inclusion of other factors that might influence scores, such as judge. This model, which considers judge to be a "facet," can model judge bias, and thus allow it to be corrected when computing an individual's ability level. The bias at issue here is not just overall; it's related to the concordance between judge home country and skater home country. This effect can be easily modeled with a Rasch Facets model.

Of course, part of me feels the controversy at the beginning of the NBC article and video above is a bit overblown. The video fixates on an element Sotnikova blew - a difficult combination element (triple flip-double toe-double loop) she didn't quite execute perfectly. (She did land it though; she didn't fall.)

But the video does not show the easier element, a triple Lutz, that Kim didn't perfectly execute. (Once again, she landed it.) Admittedly, I only watched the medal-winning performances, and didn't see any of the earlier performances that might have shown Kim's superior skill and/or Sotnikova's supposed immaturity, but I could see, based on the concept of element difficulty, why one might have awarded Sotnikova more points than Kim, or at least, have deducted fewer points for Sotnikova's mistake than Kim's mistake.

In a future post, I plan to demonstrate how to conduct a Rasch model, and hopefully at some point a Facets model, maybe even using some figure skating judging data. The holdup is that I'd like to demonstrate it using R, since R is open source and accessible by any of my readers, as opposed to the proprietary software I use at my job (Winsteps for Rasch and Facets for Rasch Facets). I'd also like to do some QC between Winsteps/Facets and R packages, to check for potential inaccuracies in computing results, so that the package(s) I present have been validated first.

Book Review: Stories of Your Life and Others

A friend sent me a copy of this collection of stories, Stories of Your Life and Others, and I just finished reading it over the weekend. You may not be familiar with the author, Ted Chiang, but you're probably familiar with a movie based on his work: Arrival. Now that I have a bit of time on my hands, I thought I'd sit down and write a quick review. (I also hope to write reviews for the two other books I finished recently: Sleeping Beauties and Weapons of Math Destruction, so stay tuned.)

Tower of Babylon - Unlike the other stories, which take place in a similar time period as our own (perhaps in the not-too-distant future), this story takes place in the distant past: the building of the tower of Babylon. The tower has been built, and miners have been brought in to complete the goal: to mine into the vault of Heaven. The story is told from the viewpoint of one of the miners, as he and his colleagues make their way to the top of the tower. It takes 4 months to reach the top. Communities are built at different points in the tower, and gardens are built to provide food. The miner is skeptical as to whether this undertaking is a good idea, while the builders have no doubts, and those who live on the tower have no desire to go back down to earth. I really felt the creeping discomfort as they went higher in the tower (probably in part because I'm afraid of heights).

Despite taking place in the past, the story explores a more modern issue that comes up with regard to science and technology: just because you can do something doesn't mean that you should.

Understand - This is the standout of the collections, though I enjoyed all of the stories. Leon Greco fell into a frozen lake and spent an hour underwater. But he is saved from braindeath by Hormone K, an experimental treatment for people with brain damage. Not only does he recover, he comes back with more cognitive ability than he had before. He volunteers for a study of Hormone K, so that he can receive more, and becomes addicted to his increased intelligence. As his abilities increase, he is able to get control over his own meta-cognition, suddenly able to literally program his brain.

The story is a great example of speculative fiction: what would happen if we suddenly become this intelligent? He reaches a point where he doesn't sleep, just rests and hallucinates (which is what happens to us when we're deprived of sleep too long; one could argue dreams are phenomenologically similar to hallucinations). This makes a lot of sense to me: Leon reaches a point where he has control over every system of his body, essentially making his entire brain capable of executive function (a much cooler and more accurate version of the "we only use X% of our brain"), but the cost of that may be that he needs to maintain conscious control over all of his bodily functions, like heartrate and breathing.

One of many things I got out of this story is the issue with such extreme levels of intelligence (and one could probably argue, this issue is also true for people of much lower levels of intelligence): there's no persuading them. To persuade someone, they have to, at some level, think they could be wrong. If a person doesn't recognize that they're wrong, either because of inability to consider other viewpoints or because of extreme intelligence, there's no way to change their mind.

In fact, this issue of communication and trying to make another person understand your viewpoint is a recurring theme in this collection. When you are finally able to understand and see something the way the other person does, it changes you.

Division by Zero - This story follows the lives and marriage of two academics: a mathematician and a biologist, showing how our mental life can strongly impact our emotional and romantic life. The mathematician makes a discovery that rocks her entire worldview, and she tries to help her husband understand while also dealing with the aftermath of her discovery. It explores the cost of discovery, touching back on the issue of can do something versus should. Any more information about the story would probably spoil it, since it's a short read.

Story of Your Life - This is the story on which Arrival is based. Many elements from the story made their way into the movie, though there are a lot of big differences. I really enjoyed learning more about the heptopods and their language; the story answers a few of the key questions I had after watching the movie. I could also understand some of the differences, and why they changed different aspects for the movie. I'd highly recommend anyone who saw the movie, regardless of whether you liked it, read this story.

Seventy-Two Letters - This story blends science and technology with religion and fantasy. It follows Robert, who builds simple golem (clay figures brought to life with a name) as a child and in his adulthood, creates complex golem that can take on many human tasks. The key issue here seems to be that gray area between can and should. Every new technology has consequences, many of them unintended. But the key question, that doesn't have an easy answer, is: do the consequences outweigh the benefits?

I recognized at the very beginning of the story that this was about golem (full disclosure: I know what golem are and how they work from an episode of Supernatural, not because I'm some expert). While I thought it was a cool idea, I was a bit disappointed with this story. It didn't feel as cohesive as the other stories in the collection.

The Evolution of Human Science - The shortest of the collection, this story is kind of a follow-up to Understand. A drastic increase in cognitive ability has created a new species: metahumans. Humans, long the most intelligence creatures, have to now deal with a more intelligent lifeform. Metahumans go in their own direction regarding scientific inquiry, and share little with humans, because humans don't have the ability to understand. It's the proverbial question in most close encounter literature: would more intelligent  lifeforms even bother with us or would they view communication with us in the same way we would view communicating with a cockroach?

Hell is the Absence of God - A great example of world-building in a short story, an ability I greatly admire. In this alternate reality, angelic visitation, miracle cures, and death by literal "act of God" are commonplace. The story follows three people impacted by angelic visitation: one whose wife died during a visitation, one who was given a deformity then later healed of it by two angelic visitations, and one trying to find his own purpose in life following his own angelic visitation. Souls are visible, as is their destination (Heaven or Hell), which has both positive and negative consequences for many people in the story. I don't want to say too much that might spoil this story. But one of the things I noticed is that there is actually very little dialogue in the story; instead, the narrator paraphrases many conversations. I wasn't sure I'd like that, but it helped keep the story going at an even pace. I might have to explore dialogue-free writing myself.

Liking What You See: A Documentary - Another excellent example of speculative fiction, that I could see turned into an episode of Black Mirror. (Bad news, that won't happen because, good news, I just learned this story is being adapted for AMC.) Scientists have figured out how to remove value judgments based on appearance, essentially removing the emotional reaction we get when looking at things that are beautiful or ugly. What they are doing is inducing a kind of agnosia (loss of an ability, usually due to brain damage), which they call "calli." This is short for calliagnosia, which I don't think is a real thing, though a related agnosia they discuss, prosopagnosia, the loss of the ability to recognize human faces, is a real thing. The story is the text of interviews with people on different sides of the calli debate - should people do it or not? - as well as news stories covering the topic. The focus is on a single community and the vote by community members as to whether the procedure should be required.

Overall, you can see Ted Chiang's training as a computer scientist, and love of math, work its way into his fiction. But he also shows a strong appreciation for linguistics, and how words have power over our way of thinking and direction in life. This is seen in Story of Your Life, but also Understand and Seventy-Two Letters. His stories also focus heavily on how technology affects us, and the thin line between technology used for good and technology used for evil. I'm looking forward to reading more of his work!

New History of Psychology Book to Check Out

One of my favorite topics is History of Psychology. For every psychology class I teach, I often spend the first lecture giving students historical background on the field/subfield, even if that information isn't discussed in the text. I love tracing the background and showing how our current position is a product of or reaction to everything coming before it.

So I'm always excited to learn about a new History of Psychology book to check out:

William James is responsible for bringing the field of psychology to the United States, and is considered the founder of the functionalism school of thought. That is, one of the early debates in the field of psychology was structuralism vs. functionalism. Basically, structuralists focused on whether consciousness is the product of definable components (structures of the mind) and functionalists viewed consciousness as an active adaptation to one's environment, resulting from complex interactions (focused on function of the mind, rather than the individual components). So you can think of these schools as trees vs. forest focus.

You are probably also familiar with James's brother, novelist Henry James (who wrote The Portrait of a Lady and The Wings of the Dove) and possibly his sister, Alice James (who suffered from life-long mental illness and published her diaries on the topic).

James is well-known for his two-volume Principles of Psychology (which is public domain and can be found here). This new book can be a companion piece to that book and helps place James's work in its historical context.

If only I hadn't made a New Year's Resolution to purchase no books...

Afternoon Reading

I'm currently working from home, meaning I'm sitting on my couch, watching the snow falling as I write this. I'll be driving into the city later this afternoon. For now, I'm happy to be in my warm apartment.

Here's what I'm reading this afternoon:

• According to The Economist, the Berlin Wall has now been down longer than it was up
• In case you were wondering, the average football player produces about a half gallon of sweat per game
• Flipping off a state trooper is punishable with a $500 fine in Indiana, and one man's lawsuit is seeking to change that
• A New Hampshire woman who won the $560 million Powerball doesn't want to claim the prize until she knows she can do so anonymously
• Apple Music has 36 million paid subscribers, but if Apple really wants to take over this market, it should consider having a free option like Spotify

Statistics Sunday: Quantile Regression (Part 2)

Last Sunday, I wrote about quantile regression and gave the basics of how it worked. I used the example of when I first analyzed data using quantile regression, in working with data following a growth curve. Specifically, I was analyzing data from an individually-administered adaptive language survey. The important thing to know about this kind of battery is that each test is one continuous unit, with items ranging from very easy to very difficult. But no one would complete all items; the test administrator would only have the examinee take however many items were necessary to determine their ability level. (Anyone administering this battery and others like it have specific training to help them understand how to do this.)

That ability level, of course, is determined based on which specific items the examinee answers correctly. But ability level is partially correlated with age - the easiest questions at the front of the booklet would generally be administered to the very young, while the most difficult questions at the back are for much older examinees. There's obviously individual variation within age groups, such that an older individual might only receive easier items and tap out before getting to more difficult items, or a younger person might be able to breeze through difficult items.

We were using quantile regression to help determine recommended starting points for a new test added to the battery. Instead of simply using age, we decided to use score on another test in the battery, a test that would be administered prior to administering this one, and that taps a very similar skillset. So we expected to have subgroups in our data, and those groups were strongly related to the score a given individual could achieve on the test; that is, one could only get a very high score if they responded correctly to the difficult items that are beyond most examinees' levels. Factor in that the relationship between our main predictor and our outcome was not linear but quadratic as well as the heteroskedasticity issue, and we had a situation in which quantile regression was a good approach. (As others have pointed out, there are others analysis methods we could have used. More on those in future posts.)

Today, I want to show you how to conduct quantile regression in R, using the quantreg package. I'll also be using the ggplot2 package, so if you want to follow along in R, you'll want to load (install if necessary) both of those packages. Also, the last time I used the quantreg package, I sometimes got my results in scientific notation, which I find difficult to read. If that's the case for you, I recommend suppressing scientific notation.

library(quantreg)
library(ggplot2)
options(scipen=999)

To make it easy for you to follow along, let's use an example dataset that comes with the quantreg package: engel. This dataset contains two variables: household income and annual food expenditures among Belgian working class families; both variables are in Belgian francs. Let's load this data and plot our two variables.

data(engel)
ggplot(engel, aes(foodexp,income)) + geom_point() + geom_smooth(method="lm")

The scores are clustered together at the lowest levels of income and food expenditures, and spread out more as values increase. (Note: If you look at the quantreg documentation on this dataset, you'll see they recommend displaying log-transformed data, which is a useful skill in many situations. Look for a future post on that.) Because we're violating a key assumption of least-squares regression, quantile regression is a good option; it doesn't have any assumptions about heteroskedasticity.

To run the quantile regression, you first want to define what quantiles you'll be using - that is, how many subgroups exist in your outcome variable. For my own data quandary at HMH, I set my number of subgroups as the number of test blocks: groups of items that align with the potential starting points. But you can set those quantiles to be whatever you would like. For instance, let's say you were interested in looking at percentiles by tenths. You could create the following R object, which gives you these percentiles (e.g., 0.1, 0.2, 0.3), that you can call with your quantile regression code:

qs <- 1:9/10

The code for a quantile regression is very simple. You use the function rq, and identify the regression equation in the Outcome~Predictor format. Next, refer to your selected quantiles as tau. Finally, identify your dataset.

qreg <- rq(foodexp~income, tau=qs, data=engel)

View simple results by just calling for your quantile regression object, qreg, or simple results plus confidence intervals with summary(qreg). If you want significance tests (testing whether the constant and slopes are significantly different from 0), use summary(qreg, se = "nid"). This will provide you useful information on how your quantiles differ from each other, such as whether some have nonsignificant slopes or whether slopes become negative for certain groups.

For brevity, I'll only display the simple results, but I'll mention that all constants and slopes are significant:

As you can see, you have information for a regression equation at each level of tau (your quantiles). We can easily plot this information by using the plot code above, with a simple substitution for the geom_smooth:

ggplot(engel, aes(foodexp,income)) + geom_point() + geom_quantile(quantiles=qs)

The nice thing about defining your quantiles with an object, rather than an embedded list in your quantile regression code, is that it's easy to switch out if you find different values would work better. For these data, it may not make sense to use these quantiles. The relationship appears to be linear and quite consistent across quantiles. So it may make more sense to use a single regression line, with tau set to the median:

qreg_med <- rq(foodexp~income, tau=0.5, data=engel)
ggplot(engel, aes(foodexp,income)) + geom_point() + geom_quantile(quantiles=0.5)

Hopefully this is enough to get you started with quantile regression. As I've mentioned before, I'm planning to do more analysis of real-world data, so I'm certain quantile regression will show up again. And since March will be here before we know it, it won't be long before Selection Sunday and March Madness. Perhaps some statistical bracketology is in order?

Psychology for Writers: Amnesia

When I started getting ready for this blog series, I added some posts to writer groups I belong to, asking people to describe some of the tropes in fiction that drive them crazy. One person commented on how head injuries are portrayed in fiction, and many replied with similar frustrations: people are knocked unconscious for a time and wake up sometime later with no lingering issues, bumps to the head are used to recover memories, and of course, the whole "don't let someone suffering from a concussion fall asleep" nonsense.

Two jobs ago, I worked as a researcher for the Department of Veterans Affairs. Though I didn't study this topic of head injuries myself, many of my colleagues did. (I've also suffered a couple of those in my lifetime, so I can talk about some of the experience firsthand.)

Traumatic brain injury, or TBI, is the medical term for a head injury. I was originally going to write this post on TBI, but realized that would make for a very long post - or else, a post in which I gloss over a lot of detail. So instead, I'm going to talk specifically about amnesia.

TBI is usually classified into one of three categories: mild, moderate, and severe. But any insult to the brain can bring with it side effects. The person suffering the head injury will probably experience what's called "post-traumatic amnesia," where they may lose their memory of things happening just before and/or just after the event. And they probably won't remember the experience of hitting their head at all. For a mild head injury, the individual may have a more severe bout of amnesia that comes on later, but will still be short-term. In fact, the delay between the head injury and this amnesia can also be seen as an indicator of severity. For more severe head injuries, amnesia and other neurological symptoms, like seizure, may be almost immediate.

The length of this experience of amnesia is also related to severity. For extremely mild injuries, post-traumatic amnesia may last only a few minutes. For a severe injury, it can last days.

And no, another blow to the head won't bring the memory back. In fact, a new head injury shortly after an old one is very bad, because the brain can't heal itself as well. Too many injuries, and the brain may be unable to heal itself at all. This is why retired professional football players have disorders that look like dementia. Their brain has essentially become scar tissue.

For me, the amnesia happened about 3 hours after the injury - I couldn't respond to questions about who I was, who my family was, the day, or my dog's name (all questions my mom asked me). I remember this feeling odd but not as terrifying as you might think. In fiction, amnesia sufferers wander around terrified, asking who they are. But for me, it was more like the answers to these questions were on the tip of my tongue, and with just a little more time, I could answer all of them. Less "Oh my god, I don't remember!" and more "That's strange. I swore I knew that." So at some level, I knew that I had the answers to these questions somewhere in my brain. 28 years later, I still remember all of this, and all of the questions I was asked at the time. The only memory I have never recovered is the second when my head came in contact with the wall. (The experience in the ER is also a bit fuzzy, but mostly because I kept falling asleep - I was exhausted by that point. And I slept a lot in the months after, as I recovered.)

I was feeling back to normal, memory-wise, within a few hours. Now, there are longer-term side effects of a head injury - things like fatigue, insomnia, and intermittent memory issues, which comprise what's known as post-concussive syndrome. Another post for another day.

So amnesia via head injury is generally short-term unless the person is severely injured, and at that point, a little amnesia may be the least of the person's worries.

It's actually much more believable for someone to experience amnesia as a result of some kind of psychological trauma. Dissociation is a psychological term for when a person loses his or her sense of self, and encompasses a handful of related disorders. What was previously known as multiple personality disorder is a dissociative disorder, now called dissociative identity disorder. Dissociation becomes a defense mechanism to protect oneself from re-experiencing trauma.

During dissociative fugue - a much better explanation if you want a character to wander into a town with no memory - the person will often run away from their life and commitments. (In fact, the word "fugue" comes from the Latin word for flight.) While in this state, the person may forget who they are and details of their life. It isn't unusual for a person in a fugue to assume another identity, likely because they realize they need one to function. They probably won't appear to be in immediate distress. After all, the fugue was brought on to escape distress, and to most observers, the person will appear normal. The person may eventually come out of the fugue, but it's harder to predict. Unlike a head injury, which has a physical cause, and therefore a clear trajectory of healing, the block here is psychological and will exist as long as the person needs protection from whatever is causing distress.

Finally, there's a type of dissociation/amnesia many have experienced - alcohol-induced blackouts. Once again, short-term and physical in cause. The hangover will probably last longer than the amnesia itself.

Releasing an Oscar-Worthy Movie

Apparently, having an Oscar-worthy movie isn't just about the script, the cast, or the director - it's also about how the movie is released. Market Match published this great story last week, that describes the two approaches to releasing movies:

A wide release is usually pursued when a distributor believes a movie will be an immediate hit, on the basis of its existing fandom or a major marketing campaign.

The platform release, on the other hand, can be used to gauge a film’s appeal. “A platform release takes a lot of planning. The strategy usually changes on the fly through the response of public and critics,” said Paul Dergarabedian, a senior media analyst with comScore.

This year, only three of the Best Picture nominees used a wide release approach: Dunkirk, Get Out, and The Post. The rest (Three Billboards Outside Ebbing, Missouri, Darkest Hour, Lady Bird, The Shape of Water, Call Me By Your Name, and Phantom Thread), used the platform release approach.

And in a fantastic visual display that you really need to see for yourself (scroll partway down), the authors demonstrate that, over the last 8 years, 7 of the Best Picture winners used the platform release approach.