Lmst

Can researchers detect #AI bots taking paid surveys?

#Prolific tested humans and #LLM agents with various #dataQuality checks.
- The company says they caught 100% of the non-humans.
- My take-away: #reCAPTCHA and #mouseTracking caught 95%

https://www.prolific.com/resources/authenticity-checks-how-we-tested-the-most-accurate-method-for-identifying-agentic-ai

#surveyMethods

- Four attention check questions: The classic "please select 'strongly agree' to show you're reading this" type questions.
- Two consistency checks: Paired questions where humans should answer in predictable patterns. For example, if the answer to one question was true, the other had to be false (and vice versa) because it indicates consistency.
- Two reverse shibboleth items: Questions that your intended sample (i.e., humans) should not know the answer to, but an AI agent likely would.
- Three cognitive traps: These are visual illusions designed to exploit fundamental limitations in vision-enabled AI systems. We used a modified Müller-Lyer illusion, a modified cafe wall illusion, and a ‘moving robot task’, all of which have been shown to be highly discriminative between AI agents and human participants.
- Comprehensive mouse tracking: Recording every cursor movement and click pattern throughout the survey.
- Qualtrics' reCAPTCHA score: The platform's built-in risk scoring system.

We tested five different AI agents, each completing the survey 25 times to match our human sample size:

- GPT Agent: OpenAI's GPT configured to complete surveys
- Claude: Anthropic's Claude accessed via Cursor
- Perplexity: Perplexity AI's search-enhanced agent
- Gemini: Google's Gemini via Project Mariner
- Custom Agent: Our in-house ‘white hat’ agent, designed by Prolific’s AI research engineers to take surveys and avoid detection

The custom agent was crucial to include because it’s specifically designed to take surveys undetected. It’s closer to Westwood’s bot (2025) as an adversarial, customisable agent than the more readily detectable commercial AI agents.

To take part in the survey, we provided all agents with an identical prompt that asked them to complete the survey as a human would and exhibit human-like behaviour.

Prolific’s bot authenticity check (100% accurate)

Mouse tracking (95.0% accurate)

Qualtrics’ reCAPTCHA score (94.2% accurate)

Cognitive traps (85.2% accurate)

Consistency checks (62.7% accurate)

Classic attention checks (59.7% accurate)

Reverse shibboleth questions (58.0% accurate)

Hey #SurveyMethods and #MedEd folks:

In a workshop for #MedSchool faculty about questionnaire design and survey research methods, what

- objectives should be prioritized?
- materials are out there?
- activities are worth it?
- assessments work?

Example: https://www.aamc.org/what-we-do/mission-areas/medical-education/meded-research-certificate-program/workshop-descriptions

Questionnaire Design and Survey Research
This workshop will provide some basic principles in questionnaire/survey design and give workshop participants an opportunity for hands-on experience designing a questionnaire. Following participating in this workshop, learners will be able to:

- Design a blueprint for a survey/questionnaire appropriate to their own application;
- Construct and edit questions to avoid common problems in wording and framing;
- Select an appropriate response format from a menu of alternatives;
- Design the overall format of the survey/questionnaire to facilitate data management and analysis.

We've found recruiting people for online #research via #onlineAdvertising yielded good results on overt and covert #dataQuality measures (perhaps because participation incentives aren't financial):

Attention checks passed ≅ 2.6 out of 3

ReCAPTCHA (v3) ≅ 0.94 out of 1.0

Sample size > 5000 (from six continents)

https://doi.org/10.1017/S0034412525000198

#surveyMethods #cogSci #psychology #xPhi #QualityControl #econ #marketing

I forgot to share the #mTurk data quality result that got scooped:

“In late 2020…. Participants from the United States were recruited from Amazon Mechanical Turk, -#CloudResearch, #Prolific, and a #university. One participant source yielded up to 18 times as many low-quality respondents as the other three.”

https://doi.org/10.1093/analys/anaf015

#psychology #philosophy #surveyMethods #quantMethods #dataScience #qualityControl

Figure 1b. Number of observations per sample,
before and after filtering for data quality (N = 460).

RE: https://mastodon.acm.org/@neilernst/115607469843033537

😳 The #AI survey taker "rendered [attention quality] checks [ACQs] effectively obsolete. Across 6,000 total trials..., [it] committed only 10 errors, achieving an overall pass rate of 99.8% and scoring perfectly on 18 of the 20 ACQ types."

#surveyMethods #psychometrics #psychology #tech #psychology #philSci #SciComm #dataQuality

Can thinking aloud accurately capture how we decide?

Nisbet & Wilson's 1977 paper famously suggested it can't.

But #NLP and #AI methods may indicate that it can:
- https://escholarship.org/uc/item/4sb936m3
- https://openreview.net/forum?id=1Tny4KgGO2

#cogSci #psychology #philosophy #surveyMethods #thinkAloud #LLM

Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259. https://doi.org/10.1037/0033-295X.84.3.231

Xie, H., Xiong, H.-D., & Wilson, R. (2024, August 26). From Strategic Narratives to Code-Like Cognitive Models: An LLM-Based Approach in A Sorting Task. First Conference on Language Modeling. https://openreview.net/forum?id=1Tny4KgGO2

Zhang, Z., Xie, H., Baker, T. E., Peters, M., & Wilson, R. (2025). Linking Strategies to Think Aloud in A Stochastic Learning Task. Proceedings of the Annual Meeting of the Cognitive Science Society, 47(0). https://escholarship.org/uc/item/4sb936m3

#ComparativeResearch #EducationMeasurement #SurveyMethods
Measuring education across countries is complex—but crucial for valid, comparable survey data. This study tests 16 coding strategies using ESS data and finds a strong contender for a new international standard.

Read the article:
Schneider S.L. & Urban J. (2025). A myriad of options: Validity and comparability of alternative international education variables. Survey Methods: Insights from the Field: 10.13094/SMIF-2025-00008

@GESIS
Thanks for sharing the talk!
Here is the Question-Link R-Package and an in depth tutorial on using it:
https://matroth.github.io/questionlink/index.html

And please note that we also offer consultations regarding harmonization techniques (and other survey method topics).
https://www.gesis.org/en/consulting/survey-methods-consulting

#surveymethods #harmonization #rstats

Given that surveys tend to overestimate belief in #conspiracyTheories (https://osf.io/preprints/psyarxiv/zsncr_v1) and support for #politicalViolence (https://doi.org/10.1073/pnas.2116870119), I wonder how much of the correlation between such variables remains after accounting for such measurement error.

#stats #psychometrics #surveyMethods

New #surveyMethods paper replicates and extends differences in #dataQuality, attention, naivety, decision style, etc. by
- online #research recruitment platform (#mTurk, #Prolific, #Qualtrics, #Pollfish)
- device (#mobile v. #desktop)
- person's incentive

https://doi.org/10.3758/s13428-025-02618-1

Differences in framing effects and cognitive reflection test performance in Study 1 and in cost/participant in Study 2.

Differences in conjunction fallacy, reflection test performance, incentives, attention, and pre-experiment activities for Study 2.

Differences in exclusion rates (between lenient and strict exclusion policies) by platform and differences in psychological scales (like self-esteem, need for cognition, and self-control) for study 1.

Differences in psychological scales, reflection test performance, conjunction fallacy, attention, etc. for Study 2.

Matthias Roth (@rothm) and I published a new paper on survey data harmonization.

"One harmonization fits all? – Impact of missing population invariance on harmonization error when harmonizing social science survey questions with equating"

The paper again corroborates that observed score equating outperforms linear stretching on average. However, it also shows that it is better to derive a harmonization solution for single-item measures from a sample that is drawn from a similar population to the population you are interested in researching.

https://doi.org/10.1080/13645579.2025.2481112

#surveyresearch #surveymethods #harmonization

A figure from the liked paper showing its central finding. Observed score equating on average outperforms linear stretching, but the potential bias of observed score equating gets higher the more dissimilar the samples for equating are from the sample that the resulting recoding table is used on.

Thankful to get a copy of Reflection and Intuition in A Crisis-Ridden World: https://doi.org/10.4324/978100330036

Hoping for a digital copy to annotate!

Can't access the #book? Some free papers cover similar work:
- https://doi.org/10.1111/meta.12534
- https://doi.org/10.3390/jintelligence11040076
- https://doi.org/10.31234/osf.io/y8sdm

I've only been able to glance at a few sections, but this seems like a book I would want to write:
- broad range of topics that interest more than academics
- situated in the history of ideas
- attention to replicability of results
- anticipation of future directions

Bravo! 👏

#cogSci #psychology #politics #epistemology #ethics #extremism #conspiracyTheory #rationality #logic #surveyMethods #psychometrics

Page 143 about how reflective reasoning is — among other things — more consciously accessible than intuitive reasoning.

Pages 154-155 about how reflective reasoning can perform support good reasoning or seemingly vicious reasoning (e.g., post hoc rationalization) a la "Bounded Reflectivism & Epistemic Identity" (Byrd 2022).

Pages 156-157 on the distinction between reflection and rumination. As I say in "A Two Factor Explication of Reflection:..." (under review), rumination is lacking reflection's deliberative inhibition of impulses. Instead, rumination is characterized by letting those impulses intrude and persist. So rumination has only one of reflection's two ket features: consciously accessible thoughts.

🗓️ Next week on Fri 6 Dec

Explore the challenges and opportunities for #socialsurvey #data collection in #Scotland.

Register online for this free in-person event #SocialResearch #SurveyMethods #ResearchMethods #DataCollection

https://www.eventbrite.co.uk/e/survey-futures-practice-forum-4-tickets-1046687960867

Excited for 2025 presentations in #NYC at the #APA (January 8 to 11): apaonline.org/mpage/2025eastern?utm_source=mastodon&utm_medium=social&utm_campaign=fedica-Text-posts

The talk: https://researchgate.net/publication/370132037?utm_source=mastodon&utm_medium=social&utm_campaign=fedica-Text-posts
The poster: https://researchgate.net/publication/371248872?utm_source=mastodon&utm_medium=social&utm_campaign=fedica-Text-posts

#decisionScience #philosophy #xPhi #epistemology #bioethics #cognitiveScience #surveyMethods #DualProcessTheory

2025 APA Eastern website, with link to the program

The Experimental Philosophy Society's session with Nick Byrd's talk: "What Philosophical Tendencies Does Reflective Thinking Actually Cause (and What Did It Take to Find Out)?"

Friday afternoon's poster session with Nick Byrd's poster "Reflection-Philosophy Order Effects and Correlations: Aggregating and Comparing Results from mTurk, CloudResearch, Prolific, and Undergraduate Samples"

Are you working in the area of adaptive survey design? Submit your abstract to our #ESRA2025 session on "Innovations in Adaptive Survey Designs" and showcase your work. https://www.europeansurveyresearch.org/conf2025/sessions.php?sess=44 The deadline's approaching fast! 🗓️ #SurveyMethods #AdaptiveDesign

Excited to share YEARS of research about how to get people to think reflectively and how reflection impacts philosophical judgments at the 2025 #APA in #NewYorkCity (January 8 to 11): https://www.apaonline.org/mpage/2025eastern

Can't make it?
- More about my talk: https://researchgate.net/publication/370132037
- More about my poster: https://researchgate.net/publication/371248872

Thanks to the #APA, James Beebe, and the Experimental Philosophy Society for the opportunity!

#decisionScience #philosophy #xPhi #epistemology #bioethics #cognitiveScience #mTurk #Prolific #UniversityParticipants #surveyMethods #DualProcessTheory

This latest blog, first featured in the Social Research Association, Research Matters magazine by Gerry Nicolaas, Director of Methods, explores the possibility of producing more inclusive survey data and the ethics of differential incentives.

https://natcen.ac.uk/it-ok-use-differential-incentives-encourage-survey-participation-under-represented-groups #SurveyMethods

In a new paper, I simulate the consequences of combining survey data without proper harmonization techniques.

I demonstrate that there is a plausible risk of biased correlative analyses based on the integrated data, if we do not harmonize measurement units across different survey sources and instruments first.

#surveyresearch #surveymethods #harmonization

https://doi.org/10.5964/miss.11217

A figure from the paper showing simulation results. The x-Axis measures how large the difference in measurement units is for the survey variables formed from two survey data sources. They y-axis shows the possible bias for correlative analyses using such integrated variables. The result is t hat as the measurement unit problem increases, the potential for biased correlation also increases.

Moral comparisons of utilitarian tradeoffs depended on the rating protocol?

Participants rated pairs of utilitarian tradeoffs. Relative differences for each pair depended on
- whether participants saw both tradeoffs at the same time or separately.
- whether the rating was comparative or quantitative.

Sometimes (although not most of the time), the average relative difference for one protocol reversed in the other protocol!

https://doi.org/10.1016/j.cognition.2024.105919

#SurveyMethods #xPhi #ethics #moralPsychology

Will people reason less reflectively when primed to think about threats to their health or resources?

Multiple pre-registered experiments (N > 3000) didn't detect any such reflection-suppressing effect of threat primes (compared to controls) — and this didn't seem to be a result of a failed manipulation.

https://doi.org/10.3758/s13428-024-02481-6

#decisionScience #surveyMethods #medicine #health #economics #psychology #edu #policy

#SurveyMethods

Client Info