class: title-slide, center, middle, inverse background-image: url(img/PurpleSky.jpg) background-position: bottom background-size: cover # Running studies online ## Tips, tools and lessons learnt ## Deborah Apthorp ## University of New England ### April 11th, 2021 Photo by [Vincentiu Solomon](https://unsplash.com/@vincentiu) ??? This slide uses: - a custom `title-slide` class that removes the slide number from the title slide - a background image - background-image: url(img/PurpleSky.jpg) - background-position: bottom - background-size: cover background-image: url(https://media.giphy.com/media/26gs78HRO8sOuhTkQ/giphy.gif) background-size: contain class: center, middle, inverse # Moving to an online university ## UNE 2018 ??? Image credit: [Schitt's Creek](https://www.poptv.com/schittscreek) --- background-image: url("images/Sea.png") class: center, top # Running studies online ### Stage 1 (2018) --- class: inverse, center, middle background-image: url(https://media.giphy.com/media/1msv9p84BukVq63l4R/giphy.gif) background-size: contain # Getting Started --- # Structure of the projects I worked with group projects from 2018-2020 -- 5-10 students per group - group data collection, individual questions -- - Allowed for the possibility of collecting more data than I'd ever had in one place! -- - Catch - all data collection had to be online -- - Plan: Combine survey data with online behavioural experiments -- - Overall theme: Sleep, Memory and Attention -- - Individual questions (examples): -- - Do people with higher visual working memory remember their dreams more? -- - Is "chronotype" related to executive function? -- - Are autism spectrum characteristics related to motor control in the general population? --- background-image: url("images/Farm.png") background-size: cover class: center, bottom, inverse # It's a new world! --- # Online data collection tools considered .pull-left[ - [Qualtrics](https://www.qualtrics.com/au/) - Good for surveys - Requires licence - [Inquisit](https://www.millisecond.com/) - Designed for online behavioural data collection - Requires licence - Easy to program - [Testable](https://www.testable.org/) - Free to use for basic licence - Optional participant pool - Slick interface - [PsychoPy](https://www.psychopy.org/) - Free, open source - Good timing - GUI only for online testing - Links to Pavlovia for testing ] .pull-right[     ] --- # What we chose (2018-2019) - Qualtrics & Inquisit -- - Qualtrics is better for surveys (complex skip logic etc.) -- - Can link through to Inquisit with hyperlink, taking ID code with it -- - Drawbacks: -- - Inquisit requires participants to download a program -- - We lost a lot of participants at this point -- - 2019: lower dropout (warned them better?) -- - Advantages: -- - Inquisit has *huge* test library -- - Easy to program where necessary -- - Relatively good timing due to everything running on Inquisit Player --- # Measures - survey & behavioural -- - Survey measures: -- - DASS (Depression, Anxiety, Stress) -- - AQ-10 (Autism spectrum characteristics) -- - MEQ (Morningness-eveningness Questionnaire) -- - AUDIT alcohol consumption -- - PSQI (Pittsburgh Sleep Quality Inventory) -- - Behavioural Measures: -- - Visual N-Back Task (Working memory) -- - PVT (Psychomotor Vigilance Task) -- - Task-switching (Number-Letter Task) -- - Finger Tapping Task (Motor control) --- # Recruitment -- - Recruitment was not as easy as you'd think -- - Probably about the same pace as in-person recruitment -- - Students who had special populations (shift workers, computer gamers, etc.) struggled to gain enough power -- - Having a large group helped (each student was supposed to recruit at least 50 participants) -- - Platforms/sources used: Facebook, Instagram, Twitter, Reddit, workplace emails, first-year Psychology pool (last resort) -- - Best recruitment source: Reddit! -- - This year: thinking of using Prolific Academic and paying participants (better quality?) --- background-image: url("images/River.png") background-size: cover class: center, middle, inverse # Results --- # Sample Characteristics (2018) <table> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:left;"> Overall </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> n </td> <td style="text-align:left;"> 484 </td> </tr> <tr> <td style="text-align:left;"> Age (mean (SD)) </td> <td style="text-align:left;"> 35.15 (13.87) </td> </tr> <tr> <td style="text-align:left;"> Gender (%) </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> Female </td> <td style="text-align:left;"> 338 (70.3) </td> </tr> <tr> <td style="text-align:left;"> Male </td> <td style="text-align:left;"> 137 (28.5) </td> </tr> <tr> <td style="text-align:left;"> Other </td> <td style="text-align:left;"> 6 (1.2) </td> </tr> <tr> <td style="text-align:left;"> completed_FT = Yes (%) </td> <td style="text-align:left;"> 231 (47.7) </td> </tr> <tr> <td style="text-align:left;"> completed_PVT = Yes (%) </td> <td style="text-align:left;"> 237 (49.0) </td> </tr> <tr> <td style="text-align:left;"> completed_TS = Yes (%) </td> <td style="text-align:left;"> 216 (44.6) </td> </tr> <tr> <td style="text-align:left;"> completed_Nback = Yes (%) </td> <td style="text-align:left;"> 197 (40.7) </td> </tr> </tbody> </table> --- # Descriptives (survey variables) Stratified by whether subjects completed behavioural tasks <table> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:left;"> No </th> <th style="text-align:left;"> Yes </th> <th style="text-align:left;"> p </th> <th style="text-align:left;"> test </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> n </td> <td style="text-align:left;"> 253 </td> <td style="text-align:left;"> 231 </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> AQ10_score (mean (SD)) </td> <td style="text-align:left;"> 2.52 (2.03) </td> <td style="text-align:left;"> 2.91 (1.84) </td> <td style="text-align:left;"> 0.028 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> ME_score (mean (SD)) </td> <td style="text-align:left;"> 53.57 (12.29) </td> <td style="text-align:left;"> 55.09 (11.53) </td> <td style="text-align:left;"> 0.191 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> AUDIT_ALC (mean (SD)) </td> <td style="text-align:left;"> 2.83 (2.53) </td> <td style="text-align:left;"> 3.15 (2.42) </td> <td style="text-align:left;"> 0.172 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> DASS_Depression (mean (SD)) </td> <td style="text-align:left;"> 11.06 (11.14) </td> <td style="text-align:left;"> 9.48 (9.82) </td> <td style="text-align:left;"> 0.125 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> DASS_Anxiety (mean (SD)) </td> <td style="text-align:left;"> 8.20 (8.16) </td> <td style="text-align:left;"> 6.93 (7.80) </td> <td style="text-align:left;"> 0.106 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> DASS_Stress (mean (SD)) </td> <td style="text-align:left;"> 13.19 (9.53) </td> <td style="text-align:left;"> 11.83 (9.42) </td> <td style="text-align:left;"> 0.146 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> PSQI_Global (mean (SD)) </td> <td style="text-align:left;"> 6.52 (3.52) </td> <td style="text-align:left;"> 6.39 (3.56) </td> <td style="text-align:left;"> 0.707 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> Dream_recall (mean (SD)) </td> <td style="text-align:left;"> 3.03 (1.76) </td> <td style="text-align:left;"> 3.06 (1.80) </td> <td style="text-align:left;"> 0.871 </td> <td style="text-align:left;"> </td> </tr> </tbody> </table> --- # Descriptives (behavioural variables) Stratified by computer platform <table> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:left;"> ios </th> <th style="text-align:left;"> mac </th> <th style="text-align:left;"> win </th> <th style="text-align:left;"> p </th> <th style="text-align:left;"> test </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> n </td> <td style="text-align:left;"> 27 </td> <td style="text-align:left;"> 59 </td> <td style="text-align:left;"> 151 </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> PVT reaction time (mean (SD)) </td> <td style="text-align:left;"> 342.94 (31.27) </td> <td style="text-align:left;"> 315.63 (36.63) </td> <td style="text-align:left;"> 305.69 (36.61) </td> <td style="text-align:left;"> &lt;0.001 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> Task Switching RT cost (mean (SD)) </td> <td style="text-align:left;"> 644.17 (291.77) </td> <td style="text-align:left;"> 588.79 (347.30) </td> <td style="text-align:left;"> 577.78 (384.83) </td> <td style="text-align:left;"> 0.742 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> n-Back d prime (mean (SD)) </td> <td style="text-align:left;"> 1.13 (0.97) </td> <td style="text-align:left;"> 1.30 (0.71) </td> <td style="text-align:left;"> 1.19 (0.70) </td> <td style="text-align:left;"> 0.594 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> FTT score (dom) (mean (SD)) </td> <td style="text-align:left;"> 63.15 (12.37) </td> <td style="text-align:left;"> 65.52 (8.16) </td> <td style="text-align:left;"> 67.43 (9.80) </td> <td style="text-align:left;"> 0.153 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> FTT score (non-dom) (mean (SD)) </td> <td style="text-align:left;"> 54.74 (9.57) </td> <td style="text-align:left;"> 60.49 (7.02) </td> <td style="text-align:left;"> 61.98 (8.69) </td> <td style="text-align:left;"> 0.004 </td> <td style="text-align:left;"> </td> </tr> </tbody> </table> --- class: center, top # Survey variables correlate with each other <!-- --> --- class: center, top # Behavioural variables correlate with each other (a bit) <!-- --> --- class: center, top # Survey measures are poor predictors of behaviour <!-- --> --- # Findings of individual projects -- - **Dream recall** significantly predicted **working memory performance** on the N-back working memory task (after controlling for age and gender), but sleep quality did not -- - **Sleep quality** did NOT predict performance on the **PVT** (after controlling for age and gender) -- - **Autism spectrum characteristics** predicted **finger tapping** performance, and interacted with **age** (stronger association in younger participants) - Alycia Messing's talk yesterday -- - **Chronotype** (morningness-eveningness) did NOT predict performance on **task switching** (and neither did sleep quality) -- - **Alcohol use** did NOT moderate the relationship between **anxiety** and **sleep quality** -- - **Shift work** did NOT affect **sleep quality** or **depression** (but underpowered) --- background-image: url("images/Farm2.png") background-size: cover class: center, bottom, inverse # Running studies online ### Stage 2 (2019) --- # Measures - survey & behavioural (2019) -- - Survey measures: -- - DASS (Depression, Anxiety, Stress) -- - **AQ-50** (Autism spectrum characteristics) -- - MEQ (Morningness-eveningness Questionnaire) -- - AUDIT alcohol consumption -- - PSQI (Pittsburgh Sleep Quality Inventory) -- - Behavioural Measures: -- - PVT (Psychomotor Vigilance Task) -- - **Iowa Gambling Task (Decision-making)** -- - Finger Tapping Task (Motor control) --- # Sample Characteristics (2019) <table> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:left;"> Overall </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> n </td> <td style="text-align:left;"> 368 </td> </tr> <tr> <td style="text-align:left;"> Age (mean (SD)) </td> <td style="text-align:left;"> 33.54 (11.63) </td> </tr> <tr> <td style="text-align:left;"> Gender (%) </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> Female </td> <td style="text-align:left;"> 253 (72.5) </td> </tr> <tr> <td style="text-align:left;"> Male </td> <td style="text-align:left;"> 94 (26.9) </td> </tr> <tr> <td style="text-align:left;"> Other </td> <td style="text-align:left;"> 2 (0.6) </td> </tr> <tr> <td style="text-align:left;"> completed_FT = TRUE (%) </td> <td style="text-align:left;"> 248 (67.4) </td> </tr> <tr> <td style="text-align:left;"> completed_PVT = TRUE (%) </td> <td style="text-align:left;"> 249 (67.7) </td> </tr> <tr> <td style="text-align:left;"> completed_IGT = TRUE (%) </td> <td style="text-align:left;"> 248 (67.4) </td> </tr> </tbody> </table> --- # Descriptives (survey variables) - 2019 sample Stratified by whether subjects completed behavioural tasks <table> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:left;"> FALSE </th> <th style="text-align:left;"> TRUE </th> <th style="text-align:left;"> p </th> <th style="text-align:left;"> test </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> n </td> <td style="text-align:left;"> 120 </td> <td style="text-align:left;"> 248 </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> AQ_total (mean (SD)) </td> <td style="text-align:left;"> 20.20 (6.82) </td> <td style="text-align:left;"> 18.20 (6.54) </td> <td style="text-align:left;"> 0.017 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> ME_score (mean (SD)) </td> <td style="text-align:left;"> 50.18 (10.77) </td> <td style="text-align:left;"> 54.35 (10.79) </td> <td style="text-align:left;"> 0.004 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> AUDIT_ALC (mean (SD)) </td> <td style="text-align:left;"> 2.69 (2.40) </td> <td style="text-align:left;"> 3.02 (2.45) </td> <td style="text-align:left;"> 0.275 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> DASS_Depression (mean (SD)) </td> <td style="text-align:left;"> 12.73 (10.24) </td> <td style="text-align:left;"> 9.00 (10.12) </td> <td style="text-align:left;"> 0.010 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> DASS_Anxiety (mean (SD)) </td> <td style="text-align:left;"> 9.30 (7.68) </td> <td style="text-align:left;"> 6.66 (8.09) </td> <td style="text-align:left;"> 0.016 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> DASS_Stress (mean (SD)) </td> <td style="text-align:left;"> 14.66 (9.13) </td> <td style="text-align:left;"> 12.06 (8.89) </td> <td style="text-align:left;"> 0.036 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> PSQI_Global (mean (SD)) </td> <td style="text-align:left;"> 7.76 (3.45) </td> <td style="text-align:left;"> 6.47 (3.22) </td> <td style="text-align:left;"> 0.003 </td> <td style="text-align:left;"> </td> </tr> </tbody> </table> --- # Descriptives (behavioural variables) - 2019 sample Stratified by computer platform <table> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:left;"> ios </th> <th style="text-align:left;"> mac </th> <th style="text-align:left;"> win </th> <th style="text-align:left;"> p </th> <th style="text-align:left;"> test </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> n </td> <td style="text-align:left;"> 20 </td> <td style="text-align:left;"> 90 </td> <td style="text-align:left;"> 138 </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> PVT reaction time (mean (SD)) </td> <td style="text-align:left;"> 332.43 (24.34) </td> <td style="text-align:left;"> 312.42 (32.18) </td> <td style="text-align:left;"> 300.28 (34.85) </td> <td style="text-align:left;"> &lt;0.001 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> FTT score (dom) (mean (SD)) </td> <td style="text-align:left;"> 64.44 (9.72) </td> <td style="text-align:left;"> 68.92 (8.75) </td> <td style="text-align:left;"> 67.91 (8.99) </td> <td style="text-align:left;"> 0.130 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> FTT score (non-dom) (mean (SD)) </td> <td style="text-align:left;"> 54.69 (6.50) </td> <td style="text-align:left;"> 60.75 (9.04) </td> <td style="text-align:left;"> 60.89 (7.90) </td> <td style="text-align:left;"> 0.007 </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> IGT net score (mean (SD)) </td> <td style="text-align:left;"> -7.50 (31.47) </td> <td style="text-align:left;"> -1.91 (32.60) </td> <td style="text-align:left;"> -2.23 (34.86) </td> <td style="text-align:left;"> 0.789 </td> <td style="text-align:left;"> </td> </tr> </tbody> </table> --- class: center, top ## Survey variables correlate with each other (as before) <!-- --> --- class: center, top ## Behavioural variables correlate with each other (a bit more) <!-- --> --- class: center, top ## Survey measures are (again) poor predictors of behaviour <img src="index_files/figure-html/correlate behavioural and survey 2019-1.png" style="display: block; margin: auto;" /> --- # Take home messages -- - Are survey measures measuring what we think they are measuring? (Sleep quality? Alcohol use? Mood? Chronontype? Autism-spectrum characteristics?) -- - Is response style driving the correlations between survey measures? -- - E.g. are some subjects more likely to choose the extremes of scales? -- - Are behavioural measures measuring what we think they are measuring? (Vigilance? Working memory capacity? Motor control? Executive functioning?) -- - We tend to think of these as more objective, but are they? Or is it all just reaction time? (And what is that measuring?) -- - Also need to think more about how to recruit representative samples and motivate participants -- - I would welcome feedback and thoughts! --- background-image: url("images/RainbowCloud2.png") background-size: cover class: center, middle, inverse # Thanks! Thanks so much to all my students from UNE Slides created via the R package [**xaringan**](https://github.com/yihui/xaringan). The chakra comes from [remark.js](https://remarkjs.com), [**knitr**](https://yihui.org/knitr), and [R Markdown](https://rmarkdown.rstudio.com). Talk (+ data & code) will be available on my website shortly