AI Identity Fraud: Deepfakes, Synthetic IDs, Voice Cloning

AI identity fraud is the use of generative AI to impersonate people at scale — fabricating identity documents, cloning voices, and producing deepfake video convincing enough to authorise payments. What changed is not the crime but its economics. Techniques that once needed skill, time and money are now commodity tools, which means the volume of sophisticated fraud rose without any corresponding rise in attacker skill.

What actually changed

Identity fraud used to have a skill ceiling. A convincing forged document, a persuasive pretext call, a plausible synthetic identity — each required craft, and that craft limited how many attacks any one actor could run.

Generative AI removed the ceiling. Purpose-built criminal tools such as FraudGPT and WormGPT are sold on dark web marketplaces and need almost no expertise to operate. Research by Transmit Security into that ecosystem found a second effect worth noting: fraudsters share technique openly, so a method that works propagates through the community quickly. Defenders face a population that learns collectively, while most organisations still learn alone.

The toolkit

Synthetic identities and fabricated documents

A synthetic identity blends real and invented attributes into a person who does not exist but passes checks. Generative AI supplies the missing artefacts — plausible identity documents and supporting imagery — at negligible cost. The uncomfortable part is that these are increasingly built to defeat automated identity verification specifically, which means the AI-driven KYC many institutions adopted to handle volume is now the surface under attack.

Deepfake video

The clearest demonstration remains the Hong Kong case reported in 2024, in which a company lost US$25 million after an employee joined a video call with what appeared to be several colleagues, including a senior finance figure. Every participant was synthetic. The employee followed a payment instruction that looked, sounded and behaved exactly like a legitimate one.

The control that failed there was not technical. It was the assumption that seeing and hearing a colleague constitutes authentication. That assumption is embedded in most payment approval processes and almost never written down. See deepfake videos for the underlying technique.

Voice cloning

Voice is the higher-volume problem, because it needs only seconds of source audio, most people’s voices are public somewhere, and phone channels are lower-fidelity and so more forgiving of artefacts. It also attacks a control many banks and contact centres deliberately invested in: voice biometrics as an authentication factor. Where a cloned voice is accepted as proof of identity, the control has become a liability rather than a defence.

Credential attacks and reconnaissance

Machine learning has also been turned on passwords — tools such as PassGAN learn the patterns of how humans actually construct them rather than brute-forcing blindly. The same capability accelerates reconnaissance, letting an attacker profile a target organisation and identify weak points far faster than manual research allows. See AI password cracking.

The numbers, and how to read them

Figures in this area age fast and are often quoted without their period, so each is dated here.

Australia: fraudulent payment card activity rose 35.6% to AUD 677.5 million in the year to June 2023 (Australian Payment Fraud Report).
New Zealand: unauthorised payment scams exceeded NZD 200 million annually, per the banking ombudsman.
United States: data breaches rose roughly 15% between 2022 and 2023, expanding the pool of stolen credentials that synthetic identities are built from.
Breach cost: the average reached US$4.45 million in 2023.
Scale of attempts: JPMorgan Chase has reported repelling around 45 billion attack attempts a day.

Two cautions. The Australian and New Zealand figures come from a study focused on those markets — the mechanisms travel, but the numbers should not be quoted as global. And none of these figures isolates AI-enabled fraud from fraud generally; they describe a rising tide in which AI is one contributing current. Anyone citing them as “the cost of AI fraud” is overstating what the data supports.

Why the usual controls fail

Most identity verification was designed against a threat model where forgery was expensive. Document checks assume a forged document costs real effort. Voice biometrics assume a voice is hard to reproduce. Callback procedures assume the person on the other end is who they sound like. Video presence assumes a face on a call is a person.

Each of those assumptions has now been falsified independently, and the controls built on them degrade quietly — they keep producing a passing result, so nothing looks broken until a loss occurs. That is the specific danger: an authentication control that has been defeated still returns “verified”.

What still works

Move authentication off perception. Anything a human judges by looking or listening is now weak. Cryptographic factors, device binding and out-of-band confirmation on a separately established channel are not.
Make high-value payments procedurally slow. The Hong Kong loss required a single approval inside one call. Mandatory second approval on an independent channel would have broken it, and costs nothing to implement.
Retire voice as a sole factor. As one signal among several it retains some value; as proof of identity it does not.
Correlate signals rather than trusting one. Device, behaviour, network and history together are far harder to fake in combination than any single artefact.
Break the silos. Fraud prevention, identity verification and customer identity management usually sit in separate systems that do not share signal — which is exactly the gap coordinated fraud exploits.
Train for the specific scenario. Staff who authorise payments should know that a familiar face and voice on a call is not authentication. That is a policy statement, not an awareness poster.

Frequently asked questions

What is synthetic identity fraud?

Combining real stolen attributes — often a genuine national insurance or social security number — with fabricated ones to create a person who does not exist. Because no real victim notices the misuse, synthetic identities can build credit history over months before being cashed out, which makes them harder to detect than conventional identity theft.

Can deepfakes really defeat identity verification?

Yes, and the Hong Kong case showed it against a live video call rather than a static check. Liveness detection raises the cost but is not decisive, since detection and generation improve against each other continuously. Any control that depends on distinguishing real from synthetic media is in an arms race it cannot permanently win.

Is voice authentication still safe to use?

Not as a sole factor. Cloning needs very little source audio and works well over telephone-quality channels. Voice can still contribute as one signal within a risk score, but organisations using it as proof of identity for account access or payment authorisation should treat that as an open finding.

What are FraudGPT and WormGPT?

Language models sold on criminal marketplaces with safety alignment removed or absent, marketed for writing phishing and business email compromise content and supporting fraud workflows. Their importance is less technical than economic: they lower the skill floor, so unskilled actors can run attacks that previously required expertise.