How one phrase breaks AI scammers and makes bots loop forever

Phone calls from AI voice bots posing as postal workers, bank employees, or delivery services have stopped being anything unusual. The technology has become cheap and widespread, and telling a synthesized voice apart from a real person keeps getting harder without special training. The heaviest burden falls on elderly people and those who rarely deal with new technology.

One telling case that made the rounds online involved a bot posing as a worker at the UK's Royal Mail.

The scheme is a familiar one – a fake notification about a package, a demand to pay an extra duty, and then a push for card details. The only difference is that the voice on the other end of the line now belongs to a language model with voice synthesis instead of an actual person.

As it turns out, these automated scammers have a pretty amusing weakness. YouTuber Kitboga, who has spent years trolling scam call centers, showed that a simple prompt injection can knock a bot completely off track.

After the right command, the AI on the other end loses track of its task and starts endlessly repeating a nonsense phrase about Albuquerque, New Mexico.

It's the same type of vulnerability that makes AI voiceovers stumble over acronyms like WWE. The language model doesn't separate a system instruction from user input, so a phrase framed as a new command easily overrides the original scam script. The whole thing breaks down, and the call turns into a loop of gibberish syllables.

The other direction scammers have moved into is real-time deepfake video. YouTuber Jim Browning recently broke down a scheme where a scammer hops on a video call wearing someone else's face to convince the victim he's legit. A dead simple check breaks that illusion too.

In the clip, the victim asks the person on the call to hold three fingers right up against his face. The model rendering the fake face over the real one can't handle the occlusion, the image starts warping, and the scammer panics while trying to explain why he won't do it. Similar tricks work with sharp head turns to the side or quickly waving a hand in front of the camera.

The scale of the problem looks more serious than a handful of funny clips. According to the AI Incident Database, fakes are now being produced on essentially an industrial scale, according to a study covered by The Guardian.

Simon Mylius, an MIT researcher and AI Incident Database collaborator, put it this way:

Capabilities have suddenly reached that level where fake content can be produced by pretty much anybody[…]It's become very accessible to a point where there is really effectively no barrier to entry.

Bobby Ford, chief strategy and experience officer at social engineering defense company Doppel, gave a similar assessment. In an interview with CNET, he noted that the speed of deepfake production has multiplied, and with it the overall volume of attacks.

Whereas before, it would take a threat actor a certain amount of time, as well as resources, in order to create that deepfake…it doesn't take that same amount of time anymore – it's done at much faster clips, so the volume increases.

Practical ways to check for fakes remain simple and don't require any technical knowledge:

• ask the person on a video call to hold their palm or fingers right up against their face and turn to the side

• hang up and call back yourself using a number from the organization's official website

• ask a question only the real family member would know the answer to, without hints available on social media

It's worth keeping in mind that the audience targeted by these schemes rarely reads articles about prompt injections. Videos of bots breaking down work as a clear demonstration for people who don't respond to text warnings. Forwarding a couple of clips to the family group chat costs a lot less than dealing with the aftermath of a scam call that actually works.