Treffer: Measuring executive functions online: Interactive effects of experimenter presence, instruction feedback, session order, and task difficulty.
Title:
Measuring executive functions online: Interactive effects of experimenter presence, instruction feedback, session order, and task difficulty.
Authors:
Dumo J; Department of Psychology, University of Northern British Columbia, Prince George, British Columbia, Canada. jihanne.dumo@alumni.unbc.ca.; Department of Psychology, Simon Fraser University, Burnaby, BC, Canada. jihanne.dumo@alumni.unbc.ca., White N; Department of Psychology, University of Northern British Columbia, Prince George, British Columbia, Canada., Jhajj K; Department of Psychology, Memorial University of Newfoundland, St. John's, Newfoundland and Labrador, Canada., Duchesne A; Department of Psychology, University of Northern British Columbia, Prince George, British Columbia, Canada. annie.duchesne@unbc.ca.; Department of Psychology, Université du Québec à Trois-Rivières, Trois- Rivières, Québec, Canada. annie.duchesne@unbc.ca.
Source:
Psychological research [Psychol Res] 2025 Dec 26; Vol. 90 (1), pp. 10. Date of Electronic Publication: 2025 Dec 26.
Publication Type:
Journal Article
Language:
English
Journal Info:
Publisher: Springer-Verlag Country of Publication: Germany NLM ID: 0435062 Publication Model: Electronic Cited Medium: Internet ISSN: 1430-2772 (Electronic) Linking ISSN: 03400727 NLM ISO Abbreviation: Psychol Res Subsets: MEDLINE
Imprint Name(s):
Original Publication: Berlin, New York, Springer-Verlag.
MeSH Terms:
References:
Adam, K. C. S., & Vogel, E. K. (2016). Reducing failures of working memory with performance feedback. Psychonomic Bulletin & Review, 23(5), 1520–1527. https://doi.org/10.3758/s13423-016-1019-4. (PMID: 10.3758/s13423-016-1019-4)
Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. K. (2020). Gorilla in our midst: An online behavioral experiment builder. Behavior Research Methods, 52(1), 388–407. https://doi.org/10.3758/s13428-019-01237-x. (PMID: 10.3758/s13428-019-01237-x31016684)
Anwyl-Irvine, A., Dalmaijer, E. S., Hodges, N., & Evershed, J. K. (2021). Realistic precision and accuracy of online experiment platforms, web browsers, and devices. Behavior Research Methods, 53(4), 1407–1425. https://doi.org/10.3758/s13428-020-01501-5. (PMID: 10.3758/s13428-020-01501-533140376)
Archibald, M. M., Ambagtsheer, R. C., Casey, M. G., & Lawless, M. (2019). Using zoom video conferencing for qualitative data collection: Perceptions and experiences of researchers and participants. International Journal of Qualitative Methods, 18, 160940691987459. https://doi.org/10.1177/1609406919874596. (PMID: 10.1177/1609406919874596)
Arechar, A. A., & Rand, D. G. (2021). Turking in the time of COVID. Behavior Research Methods, 53(6), 2591–2595. https://doi.org/10.3758/s13428-021-01588-4. (PMID: 10.3758/s13428-021-01588-4339634958103881)
Backx, R., Skirrow, C., Dente, P., Barnett, J. H., & Cormack, F. K. (2020). Comparing web-based and lab-based cognitive assessment using the Cambridge neuropsychological test automated battery: A within-subjects counterbalanced study. Journal of Medical Internet Research, 22(8), e16792. https://doi.org/10.2196/16792. (PMID: 10.2196/16792327499997435628)
Bar-Hillel, M., Noah, T., & Frederick, S. (2019). Solving stumpers, CRT and CRAT: Are the abilities related? Judgment and Decision Making, 14(5), 620–623. https://doi.org/10.1017/S1930297500004927. (PMID: 10.1017/S1930297500004927)
Barda, G., Mizrachi, Y., Borokchovich, I., Yair, L., Kertesz, D. P., & Dabby, R. (2021). The effect of pregnancy on maternal cognition. Scientific Reports, 11(1), 12187. https://doi.org/10.1038/s41598-021-91504-9. (PMID: 10.1038/s41598-021-91504-9341085328190150)
Baron, R. S. (1986). Distraction-Conflict Theory: Progress and problems. In L. Berkowitz (Ed.), Advances in Experimental Social Psychology (pp. 1–40). Academic Press. https://doi.org/10.1016/S0065-2601(08)60211-7.
Bartels, C., Wegrzyn, M., Wiedl, A., Ackermann, V., & Ehrenreich, H. (2010). Practice effects in healthy adults: A longitudinal study on frequent repetitive cognitive testing. BMC Neuroscience, 11(1), 118. https://doi.org/10.1186/1471-2202-11-118. (PMID: 10.1186/1471-2202-11-118208464442955045)
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear Mixed-Effects models using lme4. Journal of Statistical Software, 67(1). https://doi.org/10.18637/jss.v067.i01.
Belletier, C., & Camos, V. (2018). Does the experimenter presence affect working memory? Annals of the New York Academy of Sciences, 1424(1), 212–220. https://doi.org/10.1111/nyas.13627. (PMID: 10.1111/nyas.1362729524358)
Belletier, C., Davranche, K., Tellier, I. S., Dumas, F., Vidal, F., Hasbroucq, T., & Huguet, P. (2015). Choking under monitoring pressure: Being watched by the experimenter reduces executive attention. Psychonomic Bulletin & Review, 22(5), 1410–1416. https://doi.org/10.3758/s13423-015-0804-9. (PMID: 10.3758/s13423-015-0804-9)
Belletier, C., Normand, A., & Huguet, P. (2019). Social-facilitation-and-impairment effects: From motivation to cognition and the social brain. Current Directions in Psychological Science, 28(3), 260–265. https://doi.org/10.1177/0963721419829699.
Belleville, S., LaPlume, A. A., & Purkart, R. (2023). Web-based cognitive assessment in older adults: Where do we stand? Current Opinion in Neurology, 36(5), 491–497. https://doi.org/10.1097/WCO.0000000000001192. (PMID: 10.1097/WCO.00000000000011923759130310487375)
Benton, A. L., Hamsher, D. S. K., & Sivan, A. B. (1983). Controlled Oral Word Association Test [dataset]. https://doi.org/10.1037/t10132-000.
Bond, C. F., & Titus, L. J. (1983). Social facilitation: A meta-analysis of 241 studies. Psychological Bulletin, 94(2), 265–292. https://doi.org/10.1037/0033-2909.94.2.265. (PMID: 10.1037/0033-2909.94.2.2656356198)
Bowden, E. M., & Jung-Beeman, M. (2003). Normative data for 144 compound remote associate problems. Behavior Research Methods Instruments & Computers, 35(4), 634–639. https://doi.org/10.3758/BF03195543. (PMID: 10.3758/BF03195543)
Bridges, D., Pitiot, A., MacAskill, M. R., & Peirce, J. W. (2020). The timing mega-study: Comparing a range of experiment generators, both lab-based and online. PeerJ, 8, e9414. https://doi.org/10.7717/peerj.9414. (PMID: 10.7717/peerj.9414330054827512138)
Buso, I. M., Di Cagno, D., Ferrari, L., Larocca, V., Lorè, L., Marazzi, F., Panaccione, L., & Spadoni, L. (2021). Lab-like findings from online experiments. Journal of the Economic Science Association, 7(2), 184–193. https://doi.org/10.1007/s40881-021-00114-8. (PMID: 10.1007/s40881-021-00114-88683827)
Calamia, M., Markon, K., & Tranel, D. (2012). Scoring higher the second time around: Meta-analyses of practice effects in neuropsychological assessment. The Clinical Neuropsychologist, 26(4), 543–570. https://doi.org/10.1080/13854046.2012.680913. (PMID: 10.1080/13854046.2012.68091322540222)
Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29(6), 2156–2160. https://doi.org/10.1016/j.chb.2013.05.009. (PMID: 10.1016/j.chb.2013.05.009)
Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1), 112–130. https://doi.org/10.3758/s13428-013-0365-7. (PMID: 10.3758/s13428-013-0365-723835650)
Clifford, S., & Jerit, J. (2014). Is there a cost to convenience? An experimental comparison of data quality in laboratory and online studies. Journal of Experimental Political Science, 1(2), 120–131. https://doi.org/10.1017/xps.2014.5. (PMID: 10.1017/xps.2014.5)
Collins, C. L., Pina, A., Carrillo, A., Ghil, E., Smith-Peirce, R. N., Gomez, M., Okolo, P., Chen, Y., Pahor, A., Jaeggi, S. M., & Seitz, A. R. (2022). Video-based remote administration of cognitive assessments and interventions: A comparison with in-lab administration. Journal of Cognitive Enhancement, 6(3), 316–326. https://doi.org/10.1007/s41465-022-00240-z. (PMID: 10.1007/s41465-022-00240-z352619618891736)
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS ONE, 8(3), Article e57410. https://doi.org/10.1371/journal.pone.0057410. (PMID: 10.1371/journal.pone.0057410235164063596391)
Cuttler, C., LeFrance, E. M., & Stueber, A. (2021). Acute effects of high-potency cannabis flower and cannabis concentrates on everyday life memory and decision making. Scientific Reports, 11(1), Article 13784. https://doi.org/10.1038/s41598-021-93198-5. (PMID: 10.1038/s41598-021-93198-5342157848253757)
Dahm, S. F., Ort, E., Büsel, C., Sachse, P., & Mathot, S. (2023). Implementing multi-session learning studies out of the lab: Tips and tricks using OpenSesame. The Quantitative Methods for Psychology, 19(2), 156–164. https://doi.org/10.20982/tqmp.19.2.p156. (PMID: 10.20982/tqmp.19.2.p156)
de Gregorio, F., & Windels, K. (2021). Are advertising agency creatives more creative than anyone else? An exploratory test of competing predictions. Journal of Advertising, 50(2), 207–216. https://doi.org/10.1080/00913367.2020.1799268. (PMID: 10.1080/00913367.2020.1799268)
Diamond, A. (2013). Executive functions. Annual Review of Psychology, 64(1), 135–168. https://doi.org/10.1146/annurev-psych-113011-143750. (PMID: 10.1146/annurev-psych-113011-14375023020641)
Eagle, D. E., Rash, J. A., Tice, L., & Proeschold-Bell, R. J. (2021). Evaluation of a remote, internet-delivered version of the Trier social stress test. International Journal of Psychophysiology, 165, 137–144. https://doi.org/10.1016/j.ijpsycho.2021.03.009. (PMID: 10.1016/j.ijpsycho.2021.03.00933865901)
Feenstra, H. E. M., Vermeulen, I. E., Murre, J. M. J., & Schagen, S. B. (2017). Online cognition: Factors facilitating reliable online neuropsychological test results. The Clinical Neuropsychologist, 31(1), 59–84. https://doi.org/10.1080/13854046.2016.1190405. (PMID: 10.1080/13854046.2016.119040527266677)
Friedman, N. P., & Robbins, T. W. (2022). The role of prefrontal cortex in cognitive control and executive function. Neuropsychopharmacology, 47(1), 72–89. https://doi.org/10.1038/s41386-021-01132-0. (PMID: 10.1038/s41386-021-01132-034408280)
Fyfe, E. R., DeCaro, M. S., & Rittle-Johnson, B. (2015). When feedback is cognitively-demanding: The importance of working memory capacity. Instructional Science, 43(1), 73–91. https://doi.org/10.1007/s11251-014-9323-8. (PMID: 10.1007/s11251-014-9323-8)
Gabrys, R. L., Tabri, N., Anisman, H., & Matheson, K. (2018). Cognitive control and flexibility in the context of stress and depressive symptoms: The cognitive control and flexibility questionnaire. Frontiers in Psychology, 9, 2219. https://doi.org/10.3389/fpsyg.2018.02219. (PMID: 10.3389/fpsyg.2018.02219305105306252356)
Gagné, N., & Franzen, L. (2023). How to run behavioural experiments online: Best practice suggestions for cognitive psychology and neuroscience. Swiss Psychology Open, 3(1), 1. https://doi.org/10.5334/spo.34. (PMID: 10.5334/spo.34)
Garcia-Marques, T., & Fernandex, A. C. (2024). Meta-analysis of social presence effects on Stroop task performance. Psychological Reports. https://doi.org/10.1177/00332941241227150. (PMID: 10.1177/003329412412271503829160712717298)
Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012). Is the web as good as the lab? Comparable performance from web and lab in cognitive/perceptual experiments. Psychonomic Bulletin & Review, 19(5), 847–857. https://doi.org/10.3758/s13423-012-0296-9. (PMID: 10.3758/s13423-012-0296-9)
Gill, J. (1973). Current status of multiple comparisons of means in designed experiments. Journal of Dairy Science, 56(8), 973–977. (PMID: 10.3168/jds.S0022-0302(73)85291-9)
Gimmig, D., Huguet, P., Caverni, J. P., & Cury, F. (2006). Choking under pressure and working memory capacity: When performance pressure reduces fluid intelligence. Psychonomic Bulletin & Review, 13(6), 1005–1010. https://doi.org/10.3758/BF03213916. (PMID: 10.3758/BF03213916)
Grootswagers, T. (2020). A primer on running human behavioural experiments online. Behavior Research Methods, 52(6), 2283–2286. https://doi.org/10.3758/s13428-020-01395-3. (PMID: 10.3758/s13428-020-01395-332291730)
Gunnar, M. R., Reid, B. M., Donzella, B., Miller, Z. R., Gardow, S., Tsakonas, N. C., Thomas, K. M., DeJoseph, M., & Bendezú, J. J. (2021). Validation of an online version of the Trier social stress test in a study of adolescents. Psychoneuroendocrinology, 125, 105111. https://doi.org/10.1016/j.psyneuen.2020.105111. (PMID: 10.1016/j.psyneuen.2020.10511133341502)
Guo, X., Wang, Y., Kan, Y., Zhang, J., Ball, L. J., & Duan, H. (2024). How does stress shape creativity? The mediating effect of stress hormones and cognitive flexibility. Thinking Skills and Creativity, 52, 101521. https://doi.org/10.1016/j.tsc.2024.101521. (PMID: 10.1016/j.tsc.2024.101521)
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487. (PMID: 10.3102/003465430298487)
Hausknecht, J. P., Halpert, J. A., Di Paolo, N. T., & Moriarty Gerrard, M. O. (2007). Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology, 92(2), 373–385. https://doi.org/10.1037/0021-9010.92.2.373. (PMID: 10.1037/0021-9010.92.2.37317371085)
Heinzen, E., Sinwell, J., Atkinson, E., Gunderson, T., & Dougherty, G. (2021). arsenal: An Arsenal of R functions for large-scale statistical summaries (R package version 3.6.3) [Computer software]. https://CRAN.R-project.org/package=arsenal.
Hicks, K. L., Foster, J. L., & Engle, R. W. (2016). Measuring working memory capacity on the web with the online working memory lab (the OWL). Journal of Applied Research in Memory and Cognition, 5(4), 478–489. https://doi.org/10.1016/j.jarmac.2016.07.010. (PMID: 10.1016/j.jarmac.2016.07.010)
Hilbig, B. E. (2016). Reaction time effects in lab- versus web-based research: Experimental evidence. Behavior Research Methods, 48(4), 1718–1724. https://doi.org/10.3758/s13428-015-0678-9. (PMID: 10.3758/s13428-015-0678-926542972)
Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425. https://doi.org/10.1007/s10683-011-9273-9. (PMID: 10.1007/s10683-011-9273-9)
Howlett, M. (2022). Looking at the ‘field’ through a zoom lens: Methodological reflections on conducting online research during a global pandemic. Qualitative Research, 22(3), 387–402. https://doi.org/10.1177/1468794120985691. (PMID: 10.1177/1468794120985691356630979095994)
James, E., Gaskell, M. G., Pearce, R., Korell, C., Dean, C., & Henderson, L. M. (2021). The role of prior lexical knowledge in children’s and adults’ incidental word learning from illustrated stories. Journal of Experimental Psychology: Learning, Memory, and Cognition, 47(11), 1856–1869. https://doi.org/10.1037/xlm0001080. (PMID: 10.1037/xlm000108034570545)
Keilp, J. G., Sackeim, H. A., & Mann, J. J. (2005). Correlates of trait impulsiveness in performance measures and neuropsychological tests. Psychiatry Research, 135(3), 191–201. https://doi.org/10.1016/j.psychres.2005.03.006. (PMID: 10.1016/j.psychres.2005.03.00615996748)
Kelley, C. M., & McLaughlin, A. C. (2012). Individual differences in the benefits of feedback for learning. Human Factors: The Journal of the Human Factors and Ergonomics Society, 54(1), 26–35. https://doi.org/10.1177/0018720811423919. (PMID: 10.1177/0018720811423919)
Kirschner, P., Kirschner, F., & Paas, F. (2009). Cognitive load theory. In E. M. Anderman & L. H. Anderman (Eds.). Psychology of classroom learning: An encyclopedia (Vol. 1, pp. 205–209). Macmillan Reference.
Kulikowski, K., & Potasz-Kulikowska, K. (2016). Can we measure working memory via the internet? The reliability and factorial validity of an online n-back task. Polish Psychological Bulletin, 47(1), 51–61. https://doi.org/10.1515/ppb-2016-0006. (PMID: 10.1515/ppb-2016-0006)
Kumle, L., Võ, M., & Draschkow, D. (2021). Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R. Behavior Research Methods, 53, 2528–2543. https://doi.org/10.3758/s13428-021-01546-0. (PMID: 10.3758/s13428-021-01546-0339549148613146)
Lenth, R. (2023). emmeans: Estimated Marginal Means, aka Least-Squares Means (R package version 1.8.8) [Computer software]. https://CRAN.R-project.org/package=emmeans.
Leong, V., Raheel, K., Sim, J. Y., Kacker, K., Karlaftis, V. M., Vassiliu, C., Kalaivanan, K., Chen, S. H. A., Robbins, T. W., Sahakian, B. J., & Kourtzi, Z. (2022). A new remote guided method for supervised web-based cognitive testing to ensure high-quality data: Development and usability study. Journal of Medical Internet Research, 24(1), e28368. https://doi.org/10.2196/28368. (PMID: 10.2196/28368349896918778570)
Lourenco, S. F., & Tasimi, A. (2020). No participant left behind: Conducting science during COVID-19. Trends in Cognitive Sciences, 24(8), 583–584. https://doi.org/10.1016/j.tics.2020.05.003. (PMID: 10.1016/j.tics.2020.05.003324512397211671)
Lukasik, K. M., Waris, O., Soveri, A., Lehtonen, M., & Laine, M. (2019). The relationship of anxiety and stress with working memory performance in a large non-depressed sample. Frontiers in Psychology, 10, 4. https://doi.org/10.3389/fpsyg.2019.00004. (PMID: 10.3389/fpsyg.2019.00004307287906351483)
Maresh, E. L., Teachman, B. A., & Coan, J. A. (2017). Are you watching me? Interacting effects of fear of negative evaluation and social context on cognitive performance. Journal of Experimental Psychopathology, 8(3), 303–319. https://doi.org/10.5127/jep.059516. (PMID: 10.5127/jep.059516)
McLaughlin, A. C., Rogers, W. A., & Fisk, A. D. (2008). Feedback support for training: Accounting for learner and task. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 52(26), 2057–2061. https://doi.org/10.1177/154193120805202605. (PMID: 10.1177/15419312080520260525574118)
Ollesch, H., Heineken, E., & Schulte, F. (2006). Physical or virtual presence of the experimenter: Psychological online-experiments in different settings. International Journal of Internet Science, 1(1), 71–81.
Olteţeanu, A. M., & Zunjani, F. H. (2020). A visual remote associates test and its validation. Frontiers in Psychology, 11, 26. https://doi.org/10.3389/fpsyg.2020.00026. (PMID: 10.3389/fpsyg.2020.00026320474606997336)
Olteţeanu, A. M., Schöttner, M., & Schuberth, S. (2019). Computationally resurrecting the functional remote associates test using cognitive word associates and principles from a computational solver. Knowledge-Based Systems, 168, 1–9. https://doi.org/10.1016/j.knosys.2018.12.023. (PMID: 10.1016/j.knosys.2018.12.023)
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872. https://doi.org/10.1016/j.jesp.2009.03.009. (PMID: 10.1016/j.jesp.2009.03.009)
Palmer, M. G., & Johnson, C. M. (2019). Experimenter presence in human behavior analytic laboratory studies: Confound it? Behavior Analysis: Research and Practice, 19(4), 303–314. https://doi.org/10.1037/bar0000144. (PMID: 10.1037/bar0000144)
Paolacci, G., Chandler, J., & Ipeirotis, P. (2010). Running experiments on Amazon mechanical Turk. Judgment and Decision Making, 5(5), 411–419. (PMID: 10.1017/S1930297500002205)
Patterson, J. (2018). Controlled oral word association test. In J. S. Kreutzer, J. DeLuca, & B. Caplan (Eds.), Encyclopedia of Clinical Neuropsychology (pp. 958–961). Springer International Publishing. https://doi.org/10.1007/978-3-319-57111-9_876.
Patton, J., Stanford, M., & Barratt, E. (1995). Factor structure of the Barratt impulsiveness scale. Journal of Clinical Psychology, 51(6), 768–774. (PMID: 10.1002/1097-4679(199511)51:6<768::AID-JCLP2270510607>3.0.CO;2-18778124)
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021). Data quality of platforms and panels for online behavioral research. Behavior Research Methods, 54(4), 1643–1662. https://doi.org/10.3758/s13428-021-01694-3. (PMID: 10.3758/s13428-021-01694-3345902898480459)
Pietrzak, R. H., Sprague, A., & Snyder, P. J. (2008). Trait impulsiveness and executive function in healthy young adults. Journal of Research in Personality, 42(5), 1347–1351. https://doi.org/10.1016/j.jrp.2008.03.004. (PMID: 10.1016/j.jrp.2008.03.004)
Rand, D. G. (2012). The promise of mechanical turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172–179. https://doi.org/10.1016/j.jtbi.2011.03.004. (PMID: 10.1016/j.jtbi.2011.03.00421402081)
R Core Team (2021). R: A language environment for statistical computing. [Computer Software]. R Foundation for Statistical Computing. https://www.R-project.org/.
R Core Team. (2021). R: A language environment for statistical computing (Version 4.5.2). [Computer Software]. R Foundation for Statistical Computing. https://www.R-project.org/.
Redifer, J. L., Bae, C. L., & Zhao, Q. (2021). Self-efficacy and performance feedback: Impacts on cognitive load during creative thinking. Learning and Instruction, 71, 101395. https://doi.org/10.1016/j.learninstruc.2020.101395. (PMID: 10.1016/j.learninstruc.2020.101395)
Rodd, J. M. (2024). Moving experimental psychology online: How to obtain high quality data when we can’t see our participants. Journal of Memory and Language, 134, 104472. https://doi.org/10.1016/j.jml.2023.104472. (PMID: 10.1016/j.jml.2023.104472)
Ruano, L., Sousa, A., Severo, M., Alves, I., Colunas, M., Barreto, R., Mateus, C., Moreira, S., Conde, E., Bento, V., Lunet, N., Pais, J., & Cruz, T., V (2016). Development of a self-administered web-based test for longitudinal cognitive assessment. Scientific Reports, 6(1), 19114. https://doi.org/10.1038/srep19114. (PMID: 10.1038/srep19114267433294705487)
Sævland, W., & Norman, E. (2016). Studying different tasks of implicit learning across multiple test sessions conducted on the web. Frontiers in Psychology, 7. https://doi.org/10.3389/fpsyg.2016.00808.
Sauter, M., Draschkow, D., & Mack, W. (2020). Building, hosting and recruiting: A brief introduction to running behavioral experiments online. Brain Sciences, 10(4), 251. https://doi.org/10.3390/brainsci10040251. (PMID: 10.3390/brainsci10040251323446717226161)
Scharfen, J., Peters, J. M., & Holling, H. (2018). Retest effects in cognitive ability tests: A meta-analysis. Intelligence, 67, 44–66. https://doi.org/10.1016/j.intell.2018.01.003. (PMID: 10.1016/j.intell.2018.01.003)
Schmalenberger, K. M., Tauseef, H. A., Barone, J. C., Owens, S. A., Lieberman, L., Jarczok, M. N., Girdler, S. S., Kiesner, J., Ditzen, B., & Eisenlohr-Moul, T. A. (2021). How to study the menstrual cycle: Practical tools and recommendations. Psychoneuroendocrinology, 123, 104895. https://doi.org/10.1016/j.psyneuen.2020.104895. (PMID: 10.1016/j.psyneuen.2020.10489533113391)
Schonpflug, W. (2001). Experimental laboratories: Biobehavioral. In N. J. Smelser & P. B. Baltes (Eds.). International encyclopedia of the social and behavioral sciences (pp. 5109–5113). Elsevier.
Schult, J., Stadler, M., Becker, N., Greiff, S., & Sparfeldt, J. R. (2017). Home alone: Complex problem solving performance benefits from individual online assessment. Computers in Human Behavior, 68, 513–519. https://doi.org/10.1016/j.chb.2016.11.054. (PMID: 10.1016/j.chb.2016.11.054)
Semmelmann, K., & Weigelt, S. (2017). Online psychophysics: Reaction time effects in cognitive experiments. Behavior Research Methods, 49(4), 1241–1260. https://doi.org/10.3758/s13428-016-0783-4. (PMID: 10.3758/s13428-016-0783-427496171)
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using mechanical turk to study clinical populations. Clinical Psychological Science, 1(2), 213–220. https://doi.org/10.1177/2167702612469015. (PMID: 10.1177/2167702612469015)
Shields, G. S., Sazma, M. A., & Yonelinas, A. P. (2016). The effects of acute stress on core executive functions: A meta-analysis and comparison with cortisol. Neuroscience and Biobehavioral Reviews, 68, 651–668. https://doi.org/10.1016/j.neubiorev.2016.06.038. (PMID: 10.1016/j.neubiorev.2016.06.038273711615003767)
Strickland, J. C., Hill, J. C., Stoops, W. W., & Rush, C. R. (2019). Feasibility, acceptability, and initial efficacy of delivering alcohol use cognitive interventions via crowdsourcing. Alcoholism, Clinical and Experimental Research, 43(5), 888–899. https://doi.org/10.1111/acer.13987. (PMID: 10.1111/acer.1398730888705)
Thomas, K. A., & Clifford, S. (2017). Validity and mechanical turk: An assessment of exclusion methods and interactive experiments. Computers in Human Behavior, 77, 184–197. https://doi.org/10.1016/j.chb.2017.08.038. (PMID: 10.1016/j.chb.2017.08.038)
Tomczak, J., Gordon, A., Adams, J., Pickering, J. S., Hodges, N., & Evershed, J. K. (2023). What over 1,000,000 participants tell Us about online research protocols. Frontiers in Human Neuroscience, 17, 1228365. https://doi.org/10.3389/fnhum.2023.1228365. (PMID: 10.3389/fnhum.2023.12283653748491910357382)
Torrentira, M. C. Jr. (2020). Online data collection as adaptation in conducting quantitative and qualitative research during the COVID-19 pandemic. European Journal of Education Studies, 7(11). https://doi.org/10.46827/ejes.v7i11.3336.
Tremblay, A., & Ransijn, J. (2020). LMERConvenienceFunctions: Model selection and post-hoc analysis for (G)LMER Models (R package version 3.0) [Computer software]. https://CRAN.R-project.org/package=LMERConvenienceFunctions.
Tricoche, L., Reynaud, A., Pélisson, D., Prado, J., & Meunier, M. (2025). From social inhibition in childhood to social facilitation in adulthood: How social presence systematically enhances the predominant response strategy in cognitive tasks. Journal of Experimental Child Psychology, 260, 106340. https://doi.org/10.1016/j.jecp.2025.106340. (PMID: 10.1016/j.jecp.2025.10634040614570)
Uittenhove, K., Jeanneret, S., & Vergauwe, E. (2022). From lab-testing to web-testing in cognitive research: Who you test is more important than how you test. https://doi.org/10.31234/osf.io/uy4kb.
van der Wee, N., Ramsey, N., Jansma, J., Denys, D., Vanmegen, H., Westenberg, H., & Kahn, R. (2003). Spatial working memory deficits in obsessive compulsive disorder are associated with excessive engagement of the medial frontal cortex. NeuroImage, 20(4), 2271–2280. https://doi.org/10.1016/j.neuroimage.2003.05.001. (PMID: 10.1016/j.neuroimage.2003.05.00114683728)
Vytal, K. E., Cornwell, B. R., Letkiewicz, A. M., Arkin, N. E., & Grillon, C. (2013). The complex interaction between anxiety and cognition: Insight from spatial and verbal working memory. Frontiers in Human Neuroscience. https://doi.org/10.3389/fnhum.2013.00093. (PMID: 10.3389/fnhum.2013.00093237302793656338)
Weydmann, G., Palmieri, I., Simões, R. A. G., Cabral, C., Eckhardt, J. C., Tavares, J., Moro, P., Alves, C., Buchmann, P., Schmidt, S., Friedman, E., R., & Bizarro, L. (2022). Switching to online: Testing the validity of supervised remote testing for online reinforcement learning experiments. Behavior Research Methods, 55(7), 3645–3657. https://doi.org/10.3758/s13428-022-01982-6. (PMID: 10.3758/s13428-022-01982-636220950)
Weyman, K. M., Shake, M., & Redifer, J. L. (2020). Extensive experience with multiple languages May not buffer age-related declines in executive function. Experimental Aging Research, 46(4), 291–310. https://doi.org/10.1080/0361073X.2020.1753402. (PMID: 10.1080/0361073X.2020.175340232314669)
Wickham, H. (2011). The Split-Apply-Combine strategy for data analysis. Journal of Statistical Software, 40(1). https://doi.org/10.18637/jss.v040.i01.
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015). Conducting perception research over the internet: A tutorial review. PeerJ, 3, e1058. https://doi.org/10.7717/peerj.1058. (PMID: 10.7717/peerj.1058262441074517966)
Zajonc, R. B. (1965). Social facilitation: A solution is suggested for an old unresolved social psychological problem. Science, 149(3681), 269–274. https://doi.org/10.1126/science.149.3681.269. (PMID: 10.1126/science.149.3681.26914300526)
Zmigrod, L., Rentfrow, P. J., & Robbins, T. W. (2020). The partisan mind: Is extreme political partisanship related to cognitive inflexibility? Journal of Experimental Psychology: General, 149(3), 407–418. https://doi.org/10.1037/xge0000661. (PMID: 10.1037/xge000066131380695)
Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. K. (2020). Gorilla in our midst: An online behavioral experiment builder. Behavior Research Methods, 52(1), 388–407. https://doi.org/10.3758/s13428-019-01237-x. (PMID: 10.3758/s13428-019-01237-x31016684)
Anwyl-Irvine, A., Dalmaijer, E. S., Hodges, N., & Evershed, J. K. (2021). Realistic precision and accuracy of online experiment platforms, web browsers, and devices. Behavior Research Methods, 53(4), 1407–1425. https://doi.org/10.3758/s13428-020-01501-5. (PMID: 10.3758/s13428-020-01501-533140376)
Archibald, M. M., Ambagtsheer, R. C., Casey, M. G., & Lawless, M. (2019). Using zoom video conferencing for qualitative data collection: Perceptions and experiences of researchers and participants. International Journal of Qualitative Methods, 18, 160940691987459. https://doi.org/10.1177/1609406919874596. (PMID: 10.1177/1609406919874596)
Arechar, A. A., & Rand, D. G. (2021). Turking in the time of COVID. Behavior Research Methods, 53(6), 2591–2595. https://doi.org/10.3758/s13428-021-01588-4. (PMID: 10.3758/s13428-021-01588-4339634958103881)
Backx, R., Skirrow, C., Dente, P., Barnett, J. H., & Cormack, F. K. (2020). Comparing web-based and lab-based cognitive assessment using the Cambridge neuropsychological test automated battery: A within-subjects counterbalanced study. Journal of Medical Internet Research, 22(8), e16792. https://doi.org/10.2196/16792. (PMID: 10.2196/16792327499997435628)
Bar-Hillel, M., Noah, T., & Frederick, S. (2019). Solving stumpers, CRT and CRAT: Are the abilities related? Judgment and Decision Making, 14(5), 620–623. https://doi.org/10.1017/S1930297500004927. (PMID: 10.1017/S1930297500004927)
Barda, G., Mizrachi, Y., Borokchovich, I., Yair, L., Kertesz, D. P., & Dabby, R. (2021). The effect of pregnancy on maternal cognition. Scientific Reports, 11(1), 12187. https://doi.org/10.1038/s41598-021-91504-9. (PMID: 10.1038/s41598-021-91504-9341085328190150)
Baron, R. S. (1986). Distraction-Conflict Theory: Progress and problems. In L. Berkowitz (Ed.), Advances in Experimental Social Psychology (pp. 1–40). Academic Press. https://doi.org/10.1016/S0065-2601(08)60211-7.
Bartels, C., Wegrzyn, M., Wiedl, A., Ackermann, V., & Ehrenreich, H. (2010). Practice effects in healthy adults: A longitudinal study on frequent repetitive cognitive testing. BMC Neuroscience, 11(1), 118. https://doi.org/10.1186/1471-2202-11-118. (PMID: 10.1186/1471-2202-11-118208464442955045)
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear Mixed-Effects models using lme4. Journal of Statistical Software, 67(1). https://doi.org/10.18637/jss.v067.i01.
Belletier, C., & Camos, V. (2018). Does the experimenter presence affect working memory? Annals of the New York Academy of Sciences, 1424(1), 212–220. https://doi.org/10.1111/nyas.13627. (PMID: 10.1111/nyas.1362729524358)
Belletier, C., Davranche, K., Tellier, I. S., Dumas, F., Vidal, F., Hasbroucq, T., & Huguet, P. (2015). Choking under monitoring pressure: Being watched by the experimenter reduces executive attention. Psychonomic Bulletin & Review, 22(5), 1410–1416. https://doi.org/10.3758/s13423-015-0804-9. (PMID: 10.3758/s13423-015-0804-9)
Belletier, C., Normand, A., & Huguet, P. (2019). Social-facilitation-and-impairment effects: From motivation to cognition and the social brain. Current Directions in Psychological Science, 28(3), 260–265. https://doi.org/10.1177/0963721419829699.
Belleville, S., LaPlume, A. A., & Purkart, R. (2023). Web-based cognitive assessment in older adults: Where do we stand? Current Opinion in Neurology, 36(5), 491–497. https://doi.org/10.1097/WCO.0000000000001192. (PMID: 10.1097/WCO.00000000000011923759130310487375)
Benton, A. L., Hamsher, D. S. K., & Sivan, A. B. (1983). Controlled Oral Word Association Test [dataset]. https://doi.org/10.1037/t10132-000.
Bond, C. F., & Titus, L. J. (1983). Social facilitation: A meta-analysis of 241 studies. Psychological Bulletin, 94(2), 265–292. https://doi.org/10.1037/0033-2909.94.2.265. (PMID: 10.1037/0033-2909.94.2.2656356198)
Bowden, E. M., & Jung-Beeman, M. (2003). Normative data for 144 compound remote associate problems. Behavior Research Methods Instruments & Computers, 35(4), 634–639. https://doi.org/10.3758/BF03195543. (PMID: 10.3758/BF03195543)
Bridges, D., Pitiot, A., MacAskill, M. R., & Peirce, J. W. (2020). The timing mega-study: Comparing a range of experiment generators, both lab-based and online. PeerJ, 8, e9414. https://doi.org/10.7717/peerj.9414. (PMID: 10.7717/peerj.9414330054827512138)
Buso, I. M., Di Cagno, D., Ferrari, L., Larocca, V., Lorè, L., Marazzi, F., Panaccione, L., & Spadoni, L. (2021). Lab-like findings from online experiments. Journal of the Economic Science Association, 7(2), 184–193. https://doi.org/10.1007/s40881-021-00114-8. (PMID: 10.1007/s40881-021-00114-88683827)
Calamia, M., Markon, K., & Tranel, D. (2012). Scoring higher the second time around: Meta-analyses of practice effects in neuropsychological assessment. The Clinical Neuropsychologist, 26(4), 543–570. https://doi.org/10.1080/13854046.2012.680913. (PMID: 10.1080/13854046.2012.68091322540222)
Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29(6), 2156–2160. https://doi.org/10.1016/j.chb.2013.05.009. (PMID: 10.1016/j.chb.2013.05.009)
Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1), 112–130. https://doi.org/10.3758/s13428-013-0365-7. (PMID: 10.3758/s13428-013-0365-723835650)
Clifford, S., & Jerit, J. (2014). Is there a cost to convenience? An experimental comparison of data quality in laboratory and online studies. Journal of Experimental Political Science, 1(2), 120–131. https://doi.org/10.1017/xps.2014.5. (PMID: 10.1017/xps.2014.5)
Collins, C. L., Pina, A., Carrillo, A., Ghil, E., Smith-Peirce, R. N., Gomez, M., Okolo, P., Chen, Y., Pahor, A., Jaeggi, S. M., & Seitz, A. R. (2022). Video-based remote administration of cognitive assessments and interventions: A comparison with in-lab administration. Journal of Cognitive Enhancement, 6(3), 316–326. https://doi.org/10.1007/s41465-022-00240-z. (PMID: 10.1007/s41465-022-00240-z352619618891736)
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS ONE, 8(3), Article e57410. https://doi.org/10.1371/journal.pone.0057410. (PMID: 10.1371/journal.pone.0057410235164063596391)
Cuttler, C., LeFrance, E. M., & Stueber, A. (2021). Acute effects of high-potency cannabis flower and cannabis concentrates on everyday life memory and decision making. Scientific Reports, 11(1), Article 13784. https://doi.org/10.1038/s41598-021-93198-5. (PMID: 10.1038/s41598-021-93198-5342157848253757)
Dahm, S. F., Ort, E., Büsel, C., Sachse, P., & Mathot, S. (2023). Implementing multi-session learning studies out of the lab: Tips and tricks using OpenSesame. The Quantitative Methods for Psychology, 19(2), 156–164. https://doi.org/10.20982/tqmp.19.2.p156. (PMID: 10.20982/tqmp.19.2.p156)
de Gregorio, F., & Windels, K. (2021). Are advertising agency creatives more creative than anyone else? An exploratory test of competing predictions. Journal of Advertising, 50(2), 207–216. https://doi.org/10.1080/00913367.2020.1799268. (PMID: 10.1080/00913367.2020.1799268)
Diamond, A. (2013). Executive functions. Annual Review of Psychology, 64(1), 135–168. https://doi.org/10.1146/annurev-psych-113011-143750. (PMID: 10.1146/annurev-psych-113011-14375023020641)
Eagle, D. E., Rash, J. A., Tice, L., & Proeschold-Bell, R. J. (2021). Evaluation of a remote, internet-delivered version of the Trier social stress test. International Journal of Psychophysiology, 165, 137–144. https://doi.org/10.1016/j.ijpsycho.2021.03.009. (PMID: 10.1016/j.ijpsycho.2021.03.00933865901)
Feenstra, H. E. M., Vermeulen, I. E., Murre, J. M. J., & Schagen, S. B. (2017). Online cognition: Factors facilitating reliable online neuropsychological test results. The Clinical Neuropsychologist, 31(1), 59–84. https://doi.org/10.1080/13854046.2016.1190405. (PMID: 10.1080/13854046.2016.119040527266677)
Friedman, N. P., & Robbins, T. W. (2022). The role of prefrontal cortex in cognitive control and executive function. Neuropsychopharmacology, 47(1), 72–89. https://doi.org/10.1038/s41386-021-01132-0. (PMID: 10.1038/s41386-021-01132-034408280)
Fyfe, E. R., DeCaro, M. S., & Rittle-Johnson, B. (2015). When feedback is cognitively-demanding: The importance of working memory capacity. Instructional Science, 43(1), 73–91. https://doi.org/10.1007/s11251-014-9323-8. (PMID: 10.1007/s11251-014-9323-8)
Gabrys, R. L., Tabri, N., Anisman, H., & Matheson, K. (2018). Cognitive control and flexibility in the context of stress and depressive symptoms: The cognitive control and flexibility questionnaire. Frontiers in Psychology, 9, 2219. https://doi.org/10.3389/fpsyg.2018.02219. (PMID: 10.3389/fpsyg.2018.02219305105306252356)
Gagné, N., & Franzen, L. (2023). How to run behavioural experiments online: Best practice suggestions for cognitive psychology and neuroscience. Swiss Psychology Open, 3(1), 1. https://doi.org/10.5334/spo.34. (PMID: 10.5334/spo.34)
Garcia-Marques, T., & Fernandex, A. C. (2024). Meta-analysis of social presence effects on Stroop task performance. Psychological Reports. https://doi.org/10.1177/00332941241227150. (PMID: 10.1177/003329412412271503829160712717298)
Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012). Is the web as good as the lab? Comparable performance from web and lab in cognitive/perceptual experiments. Psychonomic Bulletin & Review, 19(5), 847–857. https://doi.org/10.3758/s13423-012-0296-9. (PMID: 10.3758/s13423-012-0296-9)
Gill, J. (1973). Current status of multiple comparisons of means in designed experiments. Journal of Dairy Science, 56(8), 973–977. (PMID: 10.3168/jds.S0022-0302(73)85291-9)
Gimmig, D., Huguet, P., Caverni, J. P., & Cury, F. (2006). Choking under pressure and working memory capacity: When performance pressure reduces fluid intelligence. Psychonomic Bulletin & Review, 13(6), 1005–1010. https://doi.org/10.3758/BF03213916. (PMID: 10.3758/BF03213916)
Grootswagers, T. (2020). A primer on running human behavioural experiments online. Behavior Research Methods, 52(6), 2283–2286. https://doi.org/10.3758/s13428-020-01395-3. (PMID: 10.3758/s13428-020-01395-332291730)
Gunnar, M. R., Reid, B. M., Donzella, B., Miller, Z. R., Gardow, S., Tsakonas, N. C., Thomas, K. M., DeJoseph, M., & Bendezú, J. J. (2021). Validation of an online version of the Trier social stress test in a study of adolescents. Psychoneuroendocrinology, 125, 105111. https://doi.org/10.1016/j.psyneuen.2020.105111. (PMID: 10.1016/j.psyneuen.2020.10511133341502)
Guo, X., Wang, Y., Kan, Y., Zhang, J., Ball, L. J., & Duan, H. (2024). How does stress shape creativity? The mediating effect of stress hormones and cognitive flexibility. Thinking Skills and Creativity, 52, 101521. https://doi.org/10.1016/j.tsc.2024.101521. (PMID: 10.1016/j.tsc.2024.101521)
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487. (PMID: 10.3102/003465430298487)
Hausknecht, J. P., Halpert, J. A., Di Paolo, N. T., & Moriarty Gerrard, M. O. (2007). Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology, 92(2), 373–385. https://doi.org/10.1037/0021-9010.92.2.373. (PMID: 10.1037/0021-9010.92.2.37317371085)
Heinzen, E., Sinwell, J., Atkinson, E., Gunderson, T., & Dougherty, G. (2021). arsenal: An Arsenal of R functions for large-scale statistical summaries (R package version 3.6.3) [Computer software]. https://CRAN.R-project.org/package=arsenal.
Hicks, K. L., Foster, J. L., & Engle, R. W. (2016). Measuring working memory capacity on the web with the online working memory lab (the OWL). Journal of Applied Research in Memory and Cognition, 5(4), 478–489. https://doi.org/10.1016/j.jarmac.2016.07.010. (PMID: 10.1016/j.jarmac.2016.07.010)
Hilbig, B. E. (2016). Reaction time effects in lab- versus web-based research: Experimental evidence. Behavior Research Methods, 48(4), 1718–1724. https://doi.org/10.3758/s13428-015-0678-9. (PMID: 10.3758/s13428-015-0678-926542972)
Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425. https://doi.org/10.1007/s10683-011-9273-9. (PMID: 10.1007/s10683-011-9273-9)
Howlett, M. (2022). Looking at the ‘field’ through a zoom lens: Methodological reflections on conducting online research during a global pandemic. Qualitative Research, 22(3), 387–402. https://doi.org/10.1177/1468794120985691. (PMID: 10.1177/1468794120985691356630979095994)
James, E., Gaskell, M. G., Pearce, R., Korell, C., Dean, C., & Henderson, L. M. (2021). The role of prior lexical knowledge in children’s and adults’ incidental word learning from illustrated stories. Journal of Experimental Psychology: Learning, Memory, and Cognition, 47(11), 1856–1869. https://doi.org/10.1037/xlm0001080. (PMID: 10.1037/xlm000108034570545)
Keilp, J. G., Sackeim, H. A., & Mann, J. J. (2005). Correlates of trait impulsiveness in performance measures and neuropsychological tests. Psychiatry Research, 135(3), 191–201. https://doi.org/10.1016/j.psychres.2005.03.006. (PMID: 10.1016/j.psychres.2005.03.00615996748)
Kelley, C. M., & McLaughlin, A. C. (2012). Individual differences in the benefits of feedback for learning. Human Factors: The Journal of the Human Factors and Ergonomics Society, 54(1), 26–35. https://doi.org/10.1177/0018720811423919. (PMID: 10.1177/0018720811423919)
Kirschner, P., Kirschner, F., & Paas, F. (2009). Cognitive load theory. In E. M. Anderman & L. H. Anderman (Eds.). Psychology of classroom learning: An encyclopedia (Vol. 1, pp. 205–209). Macmillan Reference.
Kulikowski, K., & Potasz-Kulikowska, K. (2016). Can we measure working memory via the internet? The reliability and factorial validity of an online n-back task. Polish Psychological Bulletin, 47(1), 51–61. https://doi.org/10.1515/ppb-2016-0006. (PMID: 10.1515/ppb-2016-0006)
Kumle, L., Võ, M., & Draschkow, D. (2021). Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R. Behavior Research Methods, 53, 2528–2543. https://doi.org/10.3758/s13428-021-01546-0. (PMID: 10.3758/s13428-021-01546-0339549148613146)
Lenth, R. (2023). emmeans: Estimated Marginal Means, aka Least-Squares Means (R package version 1.8.8) [Computer software]. https://CRAN.R-project.org/package=emmeans.
Leong, V., Raheel, K., Sim, J. Y., Kacker, K., Karlaftis, V. M., Vassiliu, C., Kalaivanan, K., Chen, S. H. A., Robbins, T. W., Sahakian, B. J., & Kourtzi, Z. (2022). A new remote guided method for supervised web-based cognitive testing to ensure high-quality data: Development and usability study. Journal of Medical Internet Research, 24(1), e28368. https://doi.org/10.2196/28368. (PMID: 10.2196/28368349896918778570)
Lourenco, S. F., & Tasimi, A. (2020). No participant left behind: Conducting science during COVID-19. Trends in Cognitive Sciences, 24(8), 583–584. https://doi.org/10.1016/j.tics.2020.05.003. (PMID: 10.1016/j.tics.2020.05.003324512397211671)
Lukasik, K. M., Waris, O., Soveri, A., Lehtonen, M., & Laine, M. (2019). The relationship of anxiety and stress with working memory performance in a large non-depressed sample. Frontiers in Psychology, 10, 4. https://doi.org/10.3389/fpsyg.2019.00004. (PMID: 10.3389/fpsyg.2019.00004307287906351483)
Maresh, E. L., Teachman, B. A., & Coan, J. A. (2017). Are you watching me? Interacting effects of fear of negative evaluation and social context on cognitive performance. Journal of Experimental Psychopathology, 8(3), 303–319. https://doi.org/10.5127/jep.059516. (PMID: 10.5127/jep.059516)
McLaughlin, A. C., Rogers, W. A., & Fisk, A. D. (2008). Feedback support for training: Accounting for learner and task. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 52(26), 2057–2061. https://doi.org/10.1177/154193120805202605. (PMID: 10.1177/15419312080520260525574118)
Ollesch, H., Heineken, E., & Schulte, F. (2006). Physical or virtual presence of the experimenter: Psychological online-experiments in different settings. International Journal of Internet Science, 1(1), 71–81.
Olteţeanu, A. M., & Zunjani, F. H. (2020). A visual remote associates test and its validation. Frontiers in Psychology, 11, 26. https://doi.org/10.3389/fpsyg.2020.00026. (PMID: 10.3389/fpsyg.2020.00026320474606997336)
Olteţeanu, A. M., Schöttner, M., & Schuberth, S. (2019). Computationally resurrecting the functional remote associates test using cognitive word associates and principles from a computational solver. Knowledge-Based Systems, 168, 1–9. https://doi.org/10.1016/j.knosys.2018.12.023. (PMID: 10.1016/j.knosys.2018.12.023)
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872. https://doi.org/10.1016/j.jesp.2009.03.009. (PMID: 10.1016/j.jesp.2009.03.009)
Palmer, M. G., & Johnson, C. M. (2019). Experimenter presence in human behavior analytic laboratory studies: Confound it? Behavior Analysis: Research and Practice, 19(4), 303–314. https://doi.org/10.1037/bar0000144. (PMID: 10.1037/bar0000144)
Paolacci, G., Chandler, J., & Ipeirotis, P. (2010). Running experiments on Amazon mechanical Turk. Judgment and Decision Making, 5(5), 411–419. (PMID: 10.1017/S1930297500002205)
Patterson, J. (2018). Controlled oral word association test. In J. S. Kreutzer, J. DeLuca, & B. Caplan (Eds.), Encyclopedia of Clinical Neuropsychology (pp. 958–961). Springer International Publishing. https://doi.org/10.1007/978-3-319-57111-9_876.
Patton, J., Stanford, M., & Barratt, E. (1995). Factor structure of the Barratt impulsiveness scale. Journal of Clinical Psychology, 51(6), 768–774. (PMID: 10.1002/1097-4679(199511)51:6<768::AID-JCLP2270510607>3.0.CO;2-18778124)
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021). Data quality of platforms and panels for online behavioral research. Behavior Research Methods, 54(4), 1643–1662. https://doi.org/10.3758/s13428-021-01694-3. (PMID: 10.3758/s13428-021-01694-3345902898480459)
Pietrzak, R. H., Sprague, A., & Snyder, P. J. (2008). Trait impulsiveness and executive function in healthy young adults. Journal of Research in Personality, 42(5), 1347–1351. https://doi.org/10.1016/j.jrp.2008.03.004. (PMID: 10.1016/j.jrp.2008.03.004)
Rand, D. G. (2012). The promise of mechanical turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172–179. https://doi.org/10.1016/j.jtbi.2011.03.004. (PMID: 10.1016/j.jtbi.2011.03.00421402081)
R Core Team (2021). R: A language environment for statistical computing. [Computer Software]. R Foundation for Statistical Computing. https://www.R-project.org/.
R Core Team. (2021). R: A language environment for statistical computing (Version 4.5.2). [Computer Software]. R Foundation for Statistical Computing. https://www.R-project.org/.
Redifer, J. L., Bae, C. L., & Zhao, Q. (2021). Self-efficacy and performance feedback: Impacts on cognitive load during creative thinking. Learning and Instruction, 71, 101395. https://doi.org/10.1016/j.learninstruc.2020.101395. (PMID: 10.1016/j.learninstruc.2020.101395)
Rodd, J. M. (2024). Moving experimental psychology online: How to obtain high quality data when we can’t see our participants. Journal of Memory and Language, 134, 104472. https://doi.org/10.1016/j.jml.2023.104472. (PMID: 10.1016/j.jml.2023.104472)
Ruano, L., Sousa, A., Severo, M., Alves, I., Colunas, M., Barreto, R., Mateus, C., Moreira, S., Conde, E., Bento, V., Lunet, N., Pais, J., & Cruz, T., V (2016). Development of a self-administered web-based test for longitudinal cognitive assessment. Scientific Reports, 6(1), 19114. https://doi.org/10.1038/srep19114. (PMID: 10.1038/srep19114267433294705487)
Sævland, W., & Norman, E. (2016). Studying different tasks of implicit learning across multiple test sessions conducted on the web. Frontiers in Psychology, 7. https://doi.org/10.3389/fpsyg.2016.00808.
Sauter, M., Draschkow, D., & Mack, W. (2020). Building, hosting and recruiting: A brief introduction to running behavioral experiments online. Brain Sciences, 10(4), 251. https://doi.org/10.3390/brainsci10040251. (PMID: 10.3390/brainsci10040251323446717226161)
Scharfen, J., Peters, J. M., & Holling, H. (2018). Retest effects in cognitive ability tests: A meta-analysis. Intelligence, 67, 44–66. https://doi.org/10.1016/j.intell.2018.01.003. (PMID: 10.1016/j.intell.2018.01.003)
Schmalenberger, K. M., Tauseef, H. A., Barone, J. C., Owens, S. A., Lieberman, L., Jarczok, M. N., Girdler, S. S., Kiesner, J., Ditzen, B., & Eisenlohr-Moul, T. A. (2021). How to study the menstrual cycle: Practical tools and recommendations. Psychoneuroendocrinology, 123, 104895. https://doi.org/10.1016/j.psyneuen.2020.104895. (PMID: 10.1016/j.psyneuen.2020.10489533113391)
Schonpflug, W. (2001). Experimental laboratories: Biobehavioral. In N. J. Smelser & P. B. Baltes (Eds.). International encyclopedia of the social and behavioral sciences (pp. 5109–5113). Elsevier.
Schult, J., Stadler, M., Becker, N., Greiff, S., & Sparfeldt, J. R. (2017). Home alone: Complex problem solving performance benefits from individual online assessment. Computers in Human Behavior, 68, 513–519. https://doi.org/10.1016/j.chb.2016.11.054. (PMID: 10.1016/j.chb.2016.11.054)
Semmelmann, K., & Weigelt, S. (2017). Online psychophysics: Reaction time effects in cognitive experiments. Behavior Research Methods, 49(4), 1241–1260. https://doi.org/10.3758/s13428-016-0783-4. (PMID: 10.3758/s13428-016-0783-427496171)
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using mechanical turk to study clinical populations. Clinical Psychological Science, 1(2), 213–220. https://doi.org/10.1177/2167702612469015. (PMID: 10.1177/2167702612469015)
Shields, G. S., Sazma, M. A., & Yonelinas, A. P. (2016). The effects of acute stress on core executive functions: A meta-analysis and comparison with cortisol. Neuroscience and Biobehavioral Reviews, 68, 651–668. https://doi.org/10.1016/j.neubiorev.2016.06.038. (PMID: 10.1016/j.neubiorev.2016.06.038273711615003767)
Strickland, J. C., Hill, J. C., Stoops, W. W., & Rush, C. R. (2019). Feasibility, acceptability, and initial efficacy of delivering alcohol use cognitive interventions via crowdsourcing. Alcoholism, Clinical and Experimental Research, 43(5), 888–899. https://doi.org/10.1111/acer.13987. (PMID: 10.1111/acer.1398730888705)
Thomas, K. A., & Clifford, S. (2017). Validity and mechanical turk: An assessment of exclusion methods and interactive experiments. Computers in Human Behavior, 77, 184–197. https://doi.org/10.1016/j.chb.2017.08.038. (PMID: 10.1016/j.chb.2017.08.038)
Tomczak, J., Gordon, A., Adams, J., Pickering, J. S., Hodges, N., & Evershed, J. K. (2023). What over 1,000,000 participants tell Us about online research protocols. Frontiers in Human Neuroscience, 17, 1228365. https://doi.org/10.3389/fnhum.2023.1228365. (PMID: 10.3389/fnhum.2023.12283653748491910357382)
Torrentira, M. C. Jr. (2020). Online data collection as adaptation in conducting quantitative and qualitative research during the COVID-19 pandemic. European Journal of Education Studies, 7(11). https://doi.org/10.46827/ejes.v7i11.3336.
Tremblay, A., & Ransijn, J. (2020). LMERConvenienceFunctions: Model selection and post-hoc analysis for (G)LMER Models (R package version 3.0) [Computer software]. https://CRAN.R-project.org/package=LMERConvenienceFunctions.
Tricoche, L., Reynaud, A., Pélisson, D., Prado, J., & Meunier, M. (2025). From social inhibition in childhood to social facilitation in adulthood: How social presence systematically enhances the predominant response strategy in cognitive tasks. Journal of Experimental Child Psychology, 260, 106340. https://doi.org/10.1016/j.jecp.2025.106340. (PMID: 10.1016/j.jecp.2025.10634040614570)
Uittenhove, K., Jeanneret, S., & Vergauwe, E. (2022). From lab-testing to web-testing in cognitive research: Who you test is more important than how you test. https://doi.org/10.31234/osf.io/uy4kb.
van der Wee, N., Ramsey, N., Jansma, J., Denys, D., Vanmegen, H., Westenberg, H., & Kahn, R. (2003). Spatial working memory deficits in obsessive compulsive disorder are associated with excessive engagement of the medial frontal cortex. NeuroImage, 20(4), 2271–2280. https://doi.org/10.1016/j.neuroimage.2003.05.001. (PMID: 10.1016/j.neuroimage.2003.05.00114683728)
Vytal, K. E., Cornwell, B. R., Letkiewicz, A. M., Arkin, N. E., & Grillon, C. (2013). The complex interaction between anxiety and cognition: Insight from spatial and verbal working memory. Frontiers in Human Neuroscience. https://doi.org/10.3389/fnhum.2013.00093. (PMID: 10.3389/fnhum.2013.00093237302793656338)
Weydmann, G., Palmieri, I., Simões, R. A. G., Cabral, C., Eckhardt, J. C., Tavares, J., Moro, P., Alves, C., Buchmann, P., Schmidt, S., Friedman, E., R., & Bizarro, L. (2022). Switching to online: Testing the validity of supervised remote testing for online reinforcement learning experiments. Behavior Research Methods, 55(7), 3645–3657. https://doi.org/10.3758/s13428-022-01982-6. (PMID: 10.3758/s13428-022-01982-636220950)
Weyman, K. M., Shake, M., & Redifer, J. L. (2020). Extensive experience with multiple languages May not buffer age-related declines in executive function. Experimental Aging Research, 46(4), 291–310. https://doi.org/10.1080/0361073X.2020.1753402. (PMID: 10.1080/0361073X.2020.175340232314669)
Wickham, H. (2011). The Split-Apply-Combine strategy for data analysis. Journal of Statistical Software, 40(1). https://doi.org/10.18637/jss.v040.i01.
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015). Conducting perception research over the internet: A tutorial review. PeerJ, 3, e1058. https://doi.org/10.7717/peerj.1058. (PMID: 10.7717/peerj.1058262441074517966)
Zajonc, R. B. (1965). Social facilitation: A solution is suggested for an old unresolved social psychological problem. Science, 149(3681), 269–274. https://doi.org/10.1126/science.149.3681.269. (PMID: 10.1126/science.149.3681.26914300526)
Zmigrod, L., Rentfrow, P. J., & Robbins, T. W. (2020). The partisan mind: Is extreme political partisanship related to cognitive inflexibility? Journal of Experimental Psychology: General, 149(3), 407–418. https://doi.org/10.1037/xge0000661. (PMID: 10.1037/xge000066131380695)
Grant Information:
DGECR-2019-00103 Natural Sciences and Engineering Research Council of Canada
Contributed Indexing:
Keywords: Executive functions; Experimenter presence; Feedback; Multi session experiments; Online cognitive testing; Remote association test; Spatial working memory
Entry Date(s):
Date Created: 20251226 Date Completed: 20251226 Latest Revision: 20251226
Update Code:
20251226
DOI:
10.1007/s00426-025-02217-x
PMID:
41452391
Database:
MEDLINE
Weitere Informationen
Online cognitive research presents numerous advantages in terms of accessibility and flexibility, often facilitating recruitment and testing. Despite the growing use of online cognitive testing, concerns remain regarding how the unsupervised and uncontrolled environment of this context may be impacting task performance. While various mitigating strategies have been proposed to improve data quality in online testing, their effects have not been consistently evaluated for online cognitive experiments and tend to be assessed in isolation and in single-session studies. To address these limitations, the present study investigated the effects of experimenter presence and instruction feedback on task performance, instruction comprehension, and user experience in an online multi-session study. A total of 109 participants completed one of four conditions where experimenter presence and instruction feedback were manipulated. Each participant was tested over two sessions occurring seven days apart. Participants completed a spatial working memory task in one session and a convergent thinking task in the other, counterbalanced across sessions. Results demonstrated similar instruction comprehension and user experiences across conditions, but significant effects of both experimenter presence and instruction feedback on task performance which varied according to the testing session order, the type of task, and the level of difficulty of the task. The current study adds to the growing literature on the relevance of testing parameters in online cognitive testing by demonstrating how characteristics of the experimental design (type of task, number of sessions) moderate the effects that online parameters have on cognitive performance.
(© 2025. The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.)
Declarations. Ethics approval: Research ethics approval was obtained from the University of Northern British Columbia’s Research Ethics Board (Ethics approval number: E.2020.1116.053.03). The procedures used in this study adhere to the tenets of the Declaration of Helsinki. Consent to participate and consent for publication: Informed consent was obtained from all participants included in the study. Competing interests: The authors declare no competing interests.