Careless responding in online student learning surveys: profiles and impact on data quality
Main Article Content
Abstract
Investigations into the prevalence of careless responding (CR) in online student learning survey research are scarce, and little to no information is present about the impact on data quality. This study illustrates how an unobtrusive post hoc method of CR detection can be applied using four indices, which can be calculated in most student learning surveys. By applying latent profile analyses in an exemplary sample of 1,111 first-year university students, different responder profiles are identified, showing the presence of careful and careless responder profiles. The study further examines the impact of CR on data quality, raising concerns about reliability and validity. The study calls for more attention to the presence of CR when analysing data in online student learning survey research. CR detection is also valuable for practice, specifically when online survey-based feedback is provided to learners with more questionable responding behaviour.
Article Details
FLR adopts the Attribution-NonCommercial-NoDerivs Creative Common License (BY-NC-ND). That is, Copyright for articles published in this journal is retained by the authors with, however, first publication rights granted to the journal. By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.
References
Chauliac, M. (2021). Treading the tightrope. Towards a deeper understanding of completing self-report questionnaires on student learning and its related data quality. Academic dissertation, University of Antwerp.
Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-19. doi.org/10.1016/j.jesp.2015.07.006
Demulder, L., Lacante, M., & Donche, V. (2021). Large scale measurements to support students in their transition to higher education. The importance of including a non-cognitive perspective. In E. Braun (Ed.), Research on teaching and learning in higher education (pp. 11-20). Waxman.
Dunn, A. M., Heggestad, E. D., Shanock, L. R., & Theilgard, N. (2018). Intra-individual response variability as an indicator of insufficient effort responding: Comparison to other indicators and relationships with individual differences. Journal of Business and Psychology, 33(1), 105-121. doi.org/10.1007/s10869-016-9479-0
Gijbels, D., Donche, V., Richardson, J. T. E., & Vermunt, J. D. (Eds.). (2014). Learning patterns in higher education: dimensions and research perspectives. New York: Routledge.
Gratz, K. L., and Roemer, L. (2004). Multidimensional assessment of emotion regulation and dysregulation: Development, factor structure, and initial validation of the difficulties in emotion regulation scale. J. Psychopathol. Behav. Assess. 26, 41–54. doi: 10.1023/B:JOBA.0000007455.08539.94
Iaconelli, R., & Wolters, C. A. (2020). Insufficient effort responding in surveys assessing self-regulated learning: Nuisance or fatal flaw? Frontline Learning Research, 8(3), 104-125. doi.org/10.14786/flr.v8i3.521
Lovibond, S., and Lovibond, P. (1995). Manual for the Depression Anxiety Stress Scales. Sydney, NSW: School of Psychology. University of New South Wales. doi: 10.1037/t01004-000
Marjanovic, Z., Holden, R., Struthers, W., Cribbie, R., & Greenglass, E. (2015). The inter-item standard deviation (ISD): An index that discriminates between conscientious and random responders. Personality and Individual Differences, 84, 79-83. doi: 10.1016/j.paid.2014.08.021
Milienos, F., Rentzios, Ch., Catrysse, L., Gijbels, D., Mastrokoukou, S., Longobardi, C. & Karagiannopoulou, E. (2021). The contribution of learning and mental health variables in first-year students’ profiles. Frontiers in Psychology, 12, 627118. doi.org/10.3389/FPSYG.2021.627118
Parpala, A. & Lindblom-Ylänne, S. (2012). Using a research instrument for developing quality at the university. Quality in Higher Education, 18 (3), 313–328.
Pintrich, P. R., Smith, D. A. F., García, T., and McKeachie, W. J. (1991). A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ). Ann Arbor, MI: University of Michigan, National Center for Research to Improve Postsecondary Teaching and Learning.
Schwarz, N. (2007). Cognitive aspects of survey methodology. Applied Cognitive Psychology, 21(2), 277-287. doi.org/10.1002/acp.1340.
Solomon, L. J., and Rothblum, E. D. (1984). Academic procrastination: frequency and cognitive-behavioral correlates. J. Couns. Psychol. 31:503. doi: 10.1037/0022-0167.31.4.503
Tourangeau, R., Rips, L., & Rasinski, K. (2000). The psychology of survey response. Cambridge: Cambridge University Press.
Yentes, R., & Wilhelm, F. (2018). Careless (Version 1.1.3) [Computer software]. https://cran.rproject.org/web/packages/careless
Vermunt, J. D. (1998). The regulation of constructive learning processes. British Journal of Educational Psychology, 68, 149–171. doi: 10.1111/j.2044-8279.1998.tb01281.x
Vermunt, J. D., & Donche, V. (2017). A learning patterns perspective on student learning in higher education: state of the art and moving forward. Educational Psychology Review, 29, 269–299. doi: 10.1007/s10648-017-9414-6