Strand Leaders: Ms. Joanna D’Ardenne, National Centre for Social Research and Professor Annette Jäckle, University of Essex
This strand will provide practical resources for survey researchers on 1) how to reduce mode effects through good questionnaire design, and 2) how to identify and evaluate mode effects in survey data.
The first component of this strand will update an existing valuable resource that can be used to help control mode effects and will produce associated good practice guidance. NatCen previously developed a framework for evaluating the risk of measurement effects when transitioning a questionnaire from one mode to another.
The framework was based on a review conducted by Campanelli et al (2011) and allows practitioners to assess survey questions against a checklist of criteria likely to increase the risk of non-equivalence (e.g. question sensitivity, complexity and visual presentation). The framework has been applied to Understanding Society (d’Ardenne et al, 2017), the English Longitudinal Study of Aging (d’Ardenne et al, 2019), and the English Housing Study (unpublished).
Since the framework was produced, there have been changes in online technology (notably, increased use of smartphones), advances in methodological knowledge (e.g. Endres et al, 2022; Lipps et al, 2022), and additional empirical evidence on measurement effects. We will conduct a literature review on measurement effects based on what has been published since the Campanelli et al (2011) review and will collect feedback from practitioners who have used the original framework to gain insight into how it could be improved and whether the risks identified were corroborated by subsequent analyses. Based on the findings, RS6 will update the framework and produce a guidance document on sources of measurement effects and how these can be mitigated, designed for use by survey researchers from multiple disciplines. These resources will be made freely available and will be publicised through an online event.
The second component of this strand will produce a practical guide to identifying the effects of data collection modes on measurement. The key challenge is that different modes of data collection can lead to differences in the way respondents answer survey questions (measurement effects), but can also lead to differences in the types of people who complete the survey (selection effects). These two effects are typically confounded. Methods to identify the effects of mode on measurement need to be able to distinguish the measurement effect from the selection effect. We will review the research designs that have been used to do this. These include comparing responses by randomised mode allocation, hall test experiments, record linkage studies, test retest or repeated measures studies, weighting or covariate adjustments to account for differences in sample composition, and experiments with non-compliance to treatment. For each method RS6 will review the assumptions underpinning the method and the required analysis methods. RS6 will also discuss the limitations of each method in terms of how successfully it controls for differences in selection between modes and whether it produces unbiased estimates of the effect of mode on measurement. To illustrate each of the research designs and corresponding analysis methods, we will develop a set of case studies, using examples from the literature as well as studies conducted on Understanding Society. This will be accompanied by an annotated bibliography.