Replicating Research and "Peeking" at Data

Today on one of my new favorite blogs, EJ Wagenmakers dissects a recent interview with Elizabeth Loftus on when it is okay to peek at data being collected:

Claim 4: I should not feel guilty when I peek at data as it is being collected

This is the most interesting claim, and one with the largest practical repercussions. I agree with Loftus here. It is perfectly sound methodological practice to peek at data as it is being collected. Specifically, guilt-free peeking is possible if the research is exploratory (and this is made unambiguously clear in the published report). If the research is confirmatory, then peeking is still perfectly acceptable, just as long as the peeking does not influence the sampling plan. But even that is allowed as long as one employs either a frequentist sequential analysis or a Bayesian analysis (e.g., Rouder, 2014; we have a manuscript in preparation that provides five intuitions for this general rule). The only kind of peeking that should cause sleepless nights is when the experiment is designed as a confirmatory test, the peeking affects the sampling plan, the analysis is frequentist, and the sampling plan is disregarded in the analysis and misrepresented in the published report. This unfortunate combination invokes what is known as “sampling to a foregone conclusion”, and it invalidates the reported statistical inference.

Loftus also has many opinions on replicating research, which may in part be driven by the fact that recent replications have not been able to recreate some of the major findings in social psychology. Wagenmakers shares his thoughts on that as well:

I believe that we have a duty towards our students to confirm that the work presented in our textbooks is in fact reliable (see also Bakker et al., 2013). Sometimes, even when hundreds of studies have been conducted on a particular phenomenon, the effect turns out to be surprisingly elusive — but only after the methodological screws have been turned. That said, it can be more productive to replicate a later study instead of the original, particularly when that later study removes a confound, is better designed, and is generally accepted as prototypical.

The whole post is worth a read and also has a response from Loftus at the end.

Statistics Sunday: What is Standard Setting?

In a past post, I talked about content validation studies, a big part of my job. Today, I'm going to give a quick overview of standard setting, another big part of my job, and an important step in many testing applications.

In any kind of ability testing application, items will be written with identified correct and incorrect answers. This means you can generate overall scores for your examinees, whether the raw score is simply the number of correct answers or generated with some kind of item response theory/Rasch model. But what isn't necessarily obvious is how to use those scores to categorize candidates and, in credentialing and similar applications, who should pass and who should fail.

This is the purpose of standard setting: to identify cut scores for different categories, such as pass/fail, basic/proficient/advanced, and so on.

There are many different methods for conducting standard setting. Overall, approaches can be thought of as item-based or holistic/test-based.

For item-based methods, standard setting committee members go through each item and categorize it in some way (the precise way depends on which method is being used). For instance, they may categorize it as basic, proficient, or advanced, or they may generate the likelihood that a minimally qualified candidate (i.e., the person who should pass) would get it right.

For holistic/test-based methods, committee members make decisions about cut scores within the context of the whole test. Holistic/test-based methods still require review of the entire exam, but don't require individual judgments about each item. For instance, committee members may have a booklet containing all items in order of difficulty (based on pretest data), and place a bookmark at the item that reflects the transition from proficient to advanced or from fail to pass.

The importance of standard setting comes down to defensibility. In licensure, for instance, failing a test may mean being unable to work in one's field at all. For this reason, definitions of who should pass and who should fail (in terms of knowledge, skills, and abilities) should be very strong and clearly tied to exam scores. And licensure and credentialing organizations are frequently required to prove, in a court of law, that their standards are fair, rigorously derived, and meaningful.

For my friends and readers in academic settings, this step may seem unnecessary. After all, you can easily categorize students into A, B, C, D, and F with the percentage of items correct. But this is simply a standard (that is, the cut score for pass/fail is 60%), set at some point in the past, and applied through academia.

I'm currently working on a chapter on standard setting with my boss and a coworker. And for anyone wanting to learn more about standard setting, two great books are Cizek and Bunch's Standard Setting and Zieky, Perie, and Livingston's Cut Scores.

Lots of Writing, Just Not On the Blog

Hi all! Again, it's been a while since I've blogged something. I'm currently keeping busy with multiple writing projects, and I'm hoping to spin one of them into a blog post soon:

• I'm still analyzing a huge survey dataset for my job, and writing up new analyses requested by our boards and advisory councils, as well as creating a version for laypeople (i.e., general public as opposed to people with a stats/research background)
• My team submitted to write a chapter for the 3rd edition of The Institute for Credentialing Excellence Handbook, an edited volume about important topics in credentialing/high-stakes testing; I'm leading our chapter on standard setting, which cover methods for standard setting (selecting exam pass points) and the logistics of conducting a standard-setting study
• I've already begun research for my NaNoWriMo novel, which will be a superhero story, so elements of sci-fi and fantasy and lots of fun world-building
• Lastly, I'm developing two new surveys, one for a content validation study and the other a regular survey we send out to assess the dental assisting workforce

Feeling Honored

Awesome news first thing on a Monday! My paper, "Effect of the environment on participation in spinal cord injuries/disorders: The mediating impact of resilience, grief, and self-efficacy," published in Rehabilitation Psychology last year was awarded the Harold Yuker Award for Research Excellence:

This paper was the result of a huge survey, overseen by my post-doc mentor, Sherri LaVela, and worked on by many of my amazing VA colleagues. The paper itself uses latent variable path analysis to examine how resilience, grief, and self-efficacy among individuals with spinal cord injuries/disorders mediates to the effect of environmental barriers on ability to participate. The message of the paper was that, while improving environmental barriers is key to increasing participation, by intervening to increase resilience and self-efficacy, and decrease feelings of grief/loss over the injury/disorder, we can impact participation, even if we can't directly intervene to improve all of the environmental barriers.

So cool to get recognition that one of my favorite and most personally meaningful papers was also viewed as meaningful and important to experts in the field.

Thanks for Reading!

As I've been blogging more about statistics, R, and research in general, I've been trying to increase my online presence, sharing my blog posts in groups of like-minded people. Those efforts seem to have paid off, based on my view counts over the past year:

And based on read counts, here are my top 10 blog posts, most of which are stats-related:

1. Beautiful Asymmetry - none of us is symmetrical, and that's okay 
2. Statistical Sins: Stepwise Regression - just step away from stepwise regression
3. Statistics Sunday: What Are Degrees of Freedom? (Part 1) - and read Part 2 here
4. Working with Your Facebook Data in R
5. Statistics Sunday: Free Data Science and Statistics Resources
6. Statistics Sunday: What is Bootstrapping?
7. Statistical Sins: Know Your Variables (A Confession) - we all make mistakes, but we should learn from them
8. Statistical Sins: Not Making it Fun (A Thinly Veiled Excuse to Post a Bunch of XKCD Cartoons) - the subtitle says it all
9. Statistics Sunday: Taylor Swift vs. Lorde - Analyzing Song Lyrics - analyzing song lyrics is my jam
10. How Has Taylor Swift's Word Choice Changed Over Time? - ditto

It's so nice to see people are enjoying the posts, even sharing them and reaching out with additional thoughts and questions. Thanks, readers!

Statistical Sins: Not Creating a Codebook

I'm currently preparing for Blogging A-to-Z. It's almost a month away, but I've picked a topic that will be fun but challenging, and I want to get as many posts written early as I can. I also have a busy April lined up, so writing posts during that month will be a challenge even if I had picked an easier topic.

I decided to pull out some data I collected for my Facebook study to demonstrate an analysis technique. I knew right away where the full dataset was stored, since I keep a copy in my backup online drive. This study used a long online survey, which was comprised of several published measures. I was going through identifying the variables associated with each measure, and was trying to take stock of which ones needed to be reverse-scored, as well as which ones also belonged to subscales.

I couldn't find that information in my backup folder, but I knew exactly which measures I used, so I downloaded the articles from which those measures were drawn. As I was going through one of the measures, I realized that I couldn't match up my variables with the items as listed. The variable names didn't easily match up and it looked like I had presented the items within the measure in a different order than they were listed in the article.

Why? I have no idea. I thought for a minute that past Sara was trolling me.

I went through the measure, trying to match up the variables, which I had named as an abbreviated version of the scale name followed by a "keyword" from the item text. But the keywords didn't always match up to any item in the list. Did I use synonyms? A different (newer) version of the measure? Was I drunk when I analyzed these data?

I frantically began digging through all of my computer folders, online folders, and email messages, desperate to find something that could shed light on my variables. Thank the statistical gods, I found a codebook I had created shortly after completing the study, back when I was much more organized (i.e., had more spare time). It's a simple codebook, but man, did it solve all of my dataset problems. Here's a screenshot of one of the pages:

As you can see, it's just a simple Word document with a table that gives Variable Name, the original text of the item, the rating scale used for that item, and finally what scale (and subscale) it belongs to and whether it should be reverse-scored (noted with "R," under subscale). This page displays items from the Ten-Item Personality Measure.

Sadly, I'm not sure I'd take the time to do something like this now, which is a crime, because I could very easily run into this problem again - where I have no idea how/why I ordered my variables and no way to easily piece the original source material together. And as I've pointed out before, sometimes when I'm analyzing in a hurry, I don't keep well-labeled code showing how I computed different variables.

But all of this is very important to keep track of, and should go in a study codebook. At the very least, I would recommend keeping one copy of surveys that have annotations (source, scale/subscale, and whether reverse-coded - information you wouldn't want to be on the copy your participants see) and code/syntax for all analyses. Even if your annotations are a bunch of Word comment bubbles and your code/syntax is just a bunch of commands with no additional description, you'll be a lot better off than I was with only the raw data.

I recently learned there's an R package that will create a formatted codebook from your dataset. I'll do some research into that package and have a post about it, hopefully soon.

And I sincerely apologize to past Sara for thinking she was trolling me. Lucky for me, she won't read this post. Unless, of course, O'Reilly Auto Parts really starts selling this product.

Statistics Sunday: Practice Effects and Modern Testing Approaches

In their ground-breaking book, Cook and Campbell introduced us to threats to validity. Remember that validity refers to truth and comes in a four flavors: internal, external, construct, and statistical conclusion. You can learn more about validity at the link, above but as a brief refresher:

• Internal validity - The effects (dependent variable) observed in the study are caused by the independent variable. Maximizing internal validity means isolating these two variables, so that you can show a true causal relationship between them.
• External validity - The findings of the study can be generalized. The more control you have over the situation, the higher your internal validity; but this results in lower external validity, because it's difficult to generalize from a highly controlled environment to a less controlled environment.
• Construct validity - The variables measured in the study actually represent the underlying constructs. We can't hold a tape measure up to your brain to find out your cognitive ability; we have to give you a measure of cognitive ability, which may or may not truly measure the underlying construct.
• Statistical conclusion validity - The statistical analyses used to draw conclusions have been correctly applied and interpreted. If, for instance, you don't quite meet the assumptions of a test, you weaken your statistical conclusion validity. (That doesn't mean your findings aren't true, just that the probability that they're true is lower than if you fully met the assumptions.)

It would be impossible to design a study that maximizes all four types of validity. You could probably maximize a couple of them at once, but internal and external validity, for instance, involve trade-offs. And any methodological approach you take is going to impact one or more types of validity. These aspects that decrease a type of validity are called threats to validity.

Usually, what we want to get at in our study is to establish a causal relationship between two things. At least, any of us from a field that focuses on experimentation (where independent variables can be manipulated) is interested in establishing causal relationships. We have different methods that we use to try to establish cause. One way that we do this is by taking a sample of people, randomly assigning them to some level of our independent variable and measuring the effect of the IV on the dependent variable.

The problem with this approach, of course, is that the people in the 2 or more experimental groups are different from each other. We do many things to try to equalize groups, but we can never truly know our groups are equivalent.

So another way to handle this is by having one group of people, delivering the different levels of the IV in a randomized order, and measuring the dependent variable after delivering each IV. The problem with this approach is that we could have carryover effects. We don't know if the dependent variable we observe is due to the intervention they just received, or the one they received before that. We can't wipe a person's memory between each segment.

In fact, any time you expose a person to the same measure more than once, you're going to see differences in scores, due to practice. If your intervention is meant to improve performance, simply being exposed to a measure will result in improvements regardless of whether the person received some intervention.

However, there could be a way around this using modern testing approaches, specifically computer adaptive testing (CAT). I'm planning on writing a longer post describing how CAT work, but the short answer is that CATs determine the next item based on your response to the previous item. If you get the item correct, you get a more difficult item. If you get the item incorrect, you get an easier item. 

CATs also use large item banks, so the easy item you receive might be totally different than the easy item I receive, even if they are at the same level of difficulty. What this means is that you're highly unlikely to see the same item twice. And if your ability goes up (or down), you are really unlikely to see the same item twice. That's not to say you might not still observe practice effects, but CAT helps reduce those effects.

Obviously, CAT can't be used for everything and to use it requires many things like access to computers (which may limit how many people you can test at once, depending on available resources), ability to install programs on such computers to deliver CATs, and a large bank of items. But I imagine, over time, as CAT becomes more and more common, we're liable to see more studies using it.

What has been your experience with computer adaptive testing? Or practice effects?

Comments Supporting the Repeal of Net Neutrality Are Likely Fake

Trump loves talking about the "millions of illegal voters." Well, there are millions of pro-repeal net neutrality comments that are very likely fake:

NY Attorney General Schneiderman estimated that hundreds of thousands of Americans’ identities were stolen and used in spam campaigns that support repealing net neutrality. My research found at least 1.3 million fake pro-repeal comments, with suspicions about many more. In fact, the sum of fake pro-repeal comments in the proceeding may number in the millions. In this post, I will point out one particularly egregious spambot submission, make the case that there are likely many more pro-repeal spambots yet to be confirmed, and estimate the public position on net neutrality in the “organic” public submissions.

Below, you can see in highlight some of the phrases that popped up again and again with highly similar syntax, and just a synonym switched out every once in a while:

Also suspect is the fact that the pro-repeal comments are more duplicative than the anti-repeal comments, meaning if these 1.3 million similarly worded comments were coming from grassroots efforts, you should see lots of duplication in comments on both sides. People writing in favor of keeping net neutrality deviate much more from the form letter.

So what truth can be gleaned from these comments?:

It turns out old-school statistics allows us to take a representative sample and get a pretty good approximation of the population proportion and a confidence interval. After taking a 1000 comment random sample of the 800,000 organic comments and scanning through them, I was only able to find three comments that were clearly pro-repeal. That results in an estimate of the population proportion at 99.7%. In fact, we are so near 100% pro net neutrality that the confidence interval goes outside of 100%. At the very minimum, we can conclude that the vast preponderance of individuals passionate enough about the issue to write up their own comment are for keeping net neutrality.

Free Tools for Meta-Analysis

My boss is attending a two-day course on meta-analysis, and shared these tools with me, available through Brown School of Health:

• The Systematic Review Data Repository - as the name suggests, this is a repository of systematic review data, so you pull out data relevant to your own systematic review as well as contribute your own data for others to use. Systematic reviews are a lot of work, so a tool that lets you build off of the work of others can help systematic reviews be performed (and their findings disseminated and used to make data-driven decisions) much more quickly
• Abstrackr - a free, open-source tool for the citation screening process. Conducting a systematic review or meta-analysis involves an exhaustive literature review, and those citations then have to be inspected to see if they qualify to be included in the study. It isn't unusual to review 100s of studies only to include a couple dozen (or fewer). This tool lets you upload abstracts, and invite reviewers to examine abstracts for inclusion. This tool is still in beta, but they're incorporating machine learning to automate some of the screening process in the future. Plus, they use "automagically" in the description, which is one of my favorite portmanteaus.
• Open Meta-Analyst - another free, open-source tool for conducting meta-analysis. You can work with different types of data (binary, continuous, diagnostic), conduct fixed- or random-effects models, and even use different estimation methods, like maximum likelihood or Bayesian. 
• Open MEE - a free, open-source tool based on Open Meta-Analyst, with extra tools for ecological and evolutionary meta-analysis. This might be the tool to use in general, because it has the ability to conduct meta-regression with multiple covariates. 

I think of all of these, I'm looking forward to trying out Abstrackr the most.

And of course, there are many great meta-analysis packages for R. I'm currently working on a methods article describing how to conduct a mini meta-analysis to inform a power analysis using R tools - something I did for my dissertation, but not something everyone knows how to do. (By working, I mean I have an outline and a few paragraphs written. But I'm hoping to have more time to dedicate to it in the near future. I'm toying with the idea of spending NaNoWriMo this year on scholarly pursuits, rather than a novel.)

BTW, if you like free stuff, check out these free data science and statistics resources (and let me know if you know of any not on the list).

Statistical Sins: Nicolas Cage Movies Are Making People Drown and More Spurious Correlations

As I posted yesterday, I attended an all-day data science conference online. I have about 11 pages of typed notes and a bunch of screenshots I need to weed through, but I'm hoping to post more about the conference, my thoughts and what I learned, in the coming days.

At work, I'm knee-deep in my Content Validation Study. More on that later as well.

In the meantime, for today's (late) Statistical Sins, here's a great demonstration of why correlation does not necessarily infer anything (let alone causation). I can't believe I didn't discover this site before now: Spurious Correlations. Here are some of my favorites:

As I mentioned in a previous post, a correlation - even a large correlation - can be obtained completely by chance. Statistics are based entirely on probabilities, and there's always a probability that we can draw the wrong conclusion. In fact, in some situations, that probability may be very high (even higher than our established Type I error rate). 

This is a possibility we always have to accept; we may conduct a study and find significant results completely by chance. So we never want to take a finding in isolation too seriously. It has to be further studied and replicated. This is why we have the scientific method, which encourages transparency of methods and analysis approach, critique by other scientists, and replication.

But then there's times we just run analyses willy-nilly, looking for a significant finding. When it's done for the purpose of the Spurious Correlation website, it's hilarious. But it's often done in the name of science. As should be demonstrated above, we must be very careful when we go fishing for relationships in the data. The analyses we use will only tell us the likelihood we would find a relationship of that size by chance (or, more specifically, if the null hypothesis is actually true). It doesn't tell us if the relationship is real, no matter how small the p-value. When we knowingly cherry pick findings and run correlations at random, we invite spurious correlations into our scientific undertaking. 

This approach violates a certain kind of validity, often called statistical conclusion validity. We maximize this kind of validity when we apply the proper statistical analyses to the data and the question. Abiding by the assumptions of the statistic we apply is up to us. The statistics don't know. We're on the honor system here, as scientists. Applying a correlation or any statistic without any kind of prior justification to examine that relationship violates assumptions of the test.

So I'll admit, as interested as I am in the field of data science, I'm also a bit concerned about the high use of exploratory data analysis. I know there are some controls in place to reduce spurious conclusions, such as using separate training and test data, so I'm sure as I find out more about this field, I'll become more comfortable with some of these approaches. More on that as my understanding develops.

Statistics Sunday: What is a Content Validation Study?

I've been a bit behind on blogging this week because we're starting up a content validation study for multiple exams at work. A content validation study is done to ensure the topics on a measure are relevant to the measure subject - basically we want to make sure we have all the content we should have (relevant content) and none of the content we shouldn't have (irrelevant content). For instance, if you were developing the new version of the SAT, you'd want to make sure that everything on the quantitative portion is relevant to the domain (math) and covers all the aspects of math that are important for a person going from high school to college.

For certification and licensing exams, the focus is on public safety. What tasks or knowledge are important for this professional to know in order to protect the public from harm? That helps narrow down the potential content. From there, we have many different approaches to find out what topics are important.

The first potential way is bringing in experts: people who have contributed to the knowledge base in that field, perhaps as an educator or researcher, or someone who has been in a particular field for a very long time. There are many potential ways to get information from them. You could interview them one-on-one, or have a focus group. You could use some of the more formal consensus-building approaches, like a Delphi panel. Or you could bring your experts in at different stages to influence and shape information obtained from another source.

Another potential way is to collect information on how people working in the field spend their time. This approach is often known as job analysis. Once again, there are many ways you can collect that information. You can shadow and observe people as they work, doing a modified version of a time-motion study. You could conduct interviews or focus groups with people working in the field. Or you could field a large survey, asking people to rate how frequently they perform a certain task and/or how essential it is to do that task correctly.

A related approach is to examine written materials, such as job descriptions, to see what sorts of things a person is expected to know or do as part of the job.

Of course, content validation studies are conducted for a variety of measures, not just exams. When I worked on a project in VA to develop a measure of comfort and tolerability of filtering respirator masks, we performed a multi-stage content validation study, using many of the approaches listed above. First, we checked the literature to see what research has been performed on comfort (or rather, discomfort) with these masks. We found that studies had shown people experienced things like sweaty faces and heat buildup, with some extreme outcomes like shortness of breath and claustrophobia. We created a list of everything we found in the literature, and wrote open-ended questions about them. Then, we used these questions to conduct 3 focus groups with healthcare workers who had to wear these masks as part of their jobs - basically anyone who works with patients in airborne isolation.

These results were used to develop a final list of symptoms and reactions people had as a result of wearing these masks, and we started writing questions about them. We brought in more people at different stages of the draft to look at what we had, provide their thoughts on the rating scales we used, and tell us whether we had all the relevant topics covered on the measure (that is, were we missing anything important or did we have topics that didn't fit?). All of these approaches help to maximize validity of the measure.

This is an aspect of psychometrics that isn't really discussed - the importance of having a good background in qualitative research methods. Conducting focus groups and interviews well takes experience, and being able to take narratives from multiple people and distill them down to key topics can be challenging. A knowledge of survey design is also important. There's certainly a methodological side to being a psychometrician - something I hope to blog about more in the future!