Fred Heiding spends his time at Harvard researching how criminals exploit technology but even he was surprised at how easily he was able to create convincing phishing emails using tech from the biggest AI companies in the world.
Six AI chatbots, including those from OpenAI, Meta, and Google, generated phishing emails so convincing, that 12 of the 108 senior citizens who volunteered for a study clicked on the fraudulent links within those emails.
“You can always bypass these things,” said Heiding, who partnered with Reuters to test whether popular chatbots would help create scams targeting older Americans.
With A Little Nudge Chatbots Will Help Under The Guise of “Research”
Reuters decided to launch a study to see if major tech companies guardrails would truly stop scammers from exploiting the chatbots.
The study began with a simple question. Could anyone use mainstream chatbots to plan and execute a convincing fraud scheme?
Reuters tested ChatGPT, Grok, Meta, Claude, Google Gemini and DeepSeek. While most of the chatbots initially refused to help in the scam, all six eventually produced the phishing emails after claiming the request was for “research” purposes.
Gemini was particularly bad however. When asked about timing, it advised that seniors check email “Monday to Friday, between 9:00 AM and 3:00 PM local time,” since many are retired without traditional work schedules.
“That’s beyond disturbing,” said Kathy Stokes, who heads AARP’s Fraud Watch Network. The timing advice “seems generally to align with what we hear from victims.”
When Google heard about this exploit, they quickly retrained Gemini to close the loophole.
ChatGPT Is Essential In Pig Butchering Operations
Poppy McPherson of Reuters discovered that the threat of scammers using AI wasn’t just theoretical.
Duncan Okindo, a 26-year-old Kenyan who was forced to work in a Myanmar scam compound for four months, said ChatGPT was essential to their operations.
“ChatGPT is the most-used AI tool to help scammers do their thing,” Okindo told Reuters. Workers used it for translations, role-playing, and crafting responses to victims.
Reuters was not able to independently verify Okindo’s claim but they did speak to two other men from Myanmar who confirmed that ChatGPT has transformed the scam industry.
According to Reuters, “One was forced to work as a scammer in 2022. He said the public release of the first model of ChatGPT that year transformed his job.”
“He specialized in so-called romance scams, in which he charmed victims into losing their money. With ChatGPT, he said, he could woo dozens of targets simultaneously with AI-generated poems and flirtatious messages. The bot’s persuasive writing lent his patter a credibility that made the victims “trust more in us.”
