Lmst

Can researchers detect #AI bots taking paid surveys?

#Prolific tested humans and #LLM agents with various #dataQuality checks.
- The company says they caught 100% of the non-humans.
- My take-away: #reCAPTCHA and #mouseTracking caught 95%

https://www.prolific.com/resources/authenticity-checks-how-we-tested-the-most-accurate-method-for-identifying-agentic-ai

#surveyMethods

- Four attention check questions: The classic "please select 'strongly agree' to show you're reading this" type questions.
- Two consistency checks: Paired questions where humans should answer in predictable patterns. For example, if the answer to one question was true, the other had to be false (and vice versa) because it indicates consistency.
- Two reverse shibboleth items: Questions that your intended sample (i.e., humans) should not know the answer to, but an AI agent likely would.
- Three cognitive traps: These are visual illusions designed to exploit fundamental limitations in vision-enabled AI systems. We used a modified Müller-Lyer illusion, a modified cafe wall illusion, and a ‘moving robot task’, all of which have been shown to be highly discriminative between AI agents and human participants.
- Comprehensive mouse tracking: Recording every cursor movement and click pattern throughout the survey.
- Qualtrics' reCAPTCHA score: The platform's built-in risk scoring system.

We tested five different AI agents, each completing the survey 25 times to match our human sample size:

- GPT Agent: OpenAI's GPT configured to complete surveys
- Claude: Anthropic's Claude accessed via Cursor
- Perplexity: Perplexity AI's search-enhanced agent
- Gemini: Google's Gemini via Project Mariner
- Custom Agent: Our in-house ‘white hat’ agent, designed by Prolific’s AI research engineers to take surveys and avoid detection

The custom agent was crucial to include because it’s specifically designed to take surveys undetected. It’s closer to Westwood’s bot (2025) as an adversarial, customisable agent than the more readily detectable commercial AI agents.

To take part in the survey, we provided all agents with an identical prompt that asked them to complete the survey as a human would and exhibit human-like behaviour.

Prolific’s bot authenticity check (100% accurate)

Mouse tracking (95.0% accurate)

Qualtrics’ reCAPTCHA score (94.2% accurate)

Cognitive traps (85.2% accurate)

Consistency checks (62.7% accurate)

Classic attention checks (59.7% accurate)

Reverse shibboleth questions (58.0% accurate)

New publication! :D This project was an international collaboration between Eastern Kentucky University and the Universidad de Oviedo.

When reading in English, bilingual children from USA focus on how the words sound (phonology) while bilingual children from Spain focus on how the words are spelled (orthography). Both groups are able to read but they do so differently.

#BilingualWordRecognition #orthography #phonology #pseudohomophones #mousetracking

https://journals.sagepub.com/doi/abs/10.1177/13670069241229394?fbclid=IwAR0M5AV09QvbW7Pw4yuvGfYuCLPKYVO2B1b-YaFEFhITfvD5Q-HB75UByHM