Planned Missing Data Designs

(from a post I made on the Applied Linguistics Research Methods Facebook group page)

 

"What's on your mind, Dan?"

"Well thanks for asking, Facebook. I'm thinking about planned missing data, actually."

"Why would you plan to have missing data?"

"So I can get more people and/or items without boring anyone to death!"

I've just taken the plunge in a research design with planned missing data. The idea here is getting data on more items or people than you could reasonably expect if you required everyone to respond to all the items at every time point - instead, you as some of the people to respond to some of the items some of the times.

While the most advanced planned missing data designs are found in longitudinal, well-funded developmental studies (e.g., in health, education; see https://quantitudethepodcast.org/listen/ episode 17 for a nice discussion), it's possible to take advantage of planned missing data in simpler designs as well, like one-shot cross-sectional designs.

More concretely: What I'm collecting data on right now is L2 speech dimensions. I have speech samples from over 200 speakers. In this type of research, it is common to get at least 10 people to judge a few dimensions of each speech sample (e.g., comprehensibility, accentedness). Instead of asking/finding a small number of people to spend well over 2 hours listening and trying to consistently judge these dimensions (to be honest, the speech samples get a bit repetitive - all on the same task/topic), with a planned missing data design I can ask 40 or so people to spend 20-30 minutes judging around 30 samples. All it requires is for listeners to have a bit of overlap in the speech samples they judge. Out of this, I will get estimates of the speech dimensions for each sample, and as a bonus thanks to involving more listeners, I can also investigate potentially listener differences in judgments of these dimensions.

Has anyone else tried out planned missing designs and care to share? Or read any good applied linguistics papers featuring planned missingness? These kinds of designs are sort of common in assessment research, but may not be planned as research, per se, they arise from using operational test data with sparse rating designs (e.g., TOEFL writing, where each essay is scored by just a couple raters) or linked/equated test forms (i.e., two test forms with a handful of overlapping items).

"Why aren't you finishing up that manuscript you were working on, Dan?"

"...Facebook, what aren't you tracking? Jeez."