How a Mental Health AI Tool Accidentally Discovered Accurate Deepfake Detection

As tech giant Open AI launched its flagship Sora 2 video and audio generational model in September 2025, deepfake videos have flooded social media platforms, making audiences increasingly familiar with potentially dangerous hyper-realistic content.

Although Open AI deemed the responsible launch of Sora 2 as a top priority, claiming it would give users “the tools and optionality to be in control of what they see in their feed” and control over their likeness end-to-end, an October 2025 study found the model produced false claim videos 80% of the time.

From videos that mimicked news reports of a Moldovan electoral official destroying ballots to fabricated scenes of a toddler being detained by immigration officers or a Coca-Cola spokesperson announcing the company would not sponsor the Super Bowl, the stakes for producing misinformation in an interconnected world could not be higher.

Beyond Sora: Vishing

Even before Open AI’s tool was launched, the creation and online dissemination of deepfake files was on the rise. According to a September, 2025 report by cybersecurity firm DeepStrike, deepfake content surged from 500,000 in 2023 to a staggering 8 million in 2025, much of which was used for fraudulent purposes.

The trend shows no signs of stopping; AI fraud in the U.S. alone is expected to reach $40 billion USD by 2027.

Such a surge is not limited to quantity. With tools like Sora 2 and Google’s Veo 3, content of AI-generated faces, voices, and full-body performances are now more realistic than ever. As signaled by computer scientist and deepfake researcher Siwei Luy, contemporary models are capable of producing stable faces without warping or distortions, while voice cloning has crossed an “indistinguishable threshold.”

The truth is, deepfakes are outpacing detection. What technology companies sell as fun tools to generate everything from Olympic gymnastics routines to sophisticated background soundscapes, has also been leveraged by criminals to target businesses and individuals alike. Just in the first half of 2025, deepfake incidents prompted losses of $356 million USD for companies, and $541 million USD for individuals.

Traditional deepfake detection – including identifying watermarks, airbrushed faces and metadata checks – is failing. And, as voice deepfakes remain the second-most common form of AI-enabled fraud and voice phishing (vishing) surged 442% in 2025, the consequences are already being felt.

“A few seconds of audio now suffice to generate a convincing clone – complete with natural intonation, rhythm, emphasis, emotion, pauses and breathing noise,” Lyu wrote.

The Science of Listening to Humans

Kintsugi, a healthtech startup developing AI voice biomarker technology to detect signs of clinical depression and anxiety. Their work started from a seemingly simple premise: we must listen to humans.

“I started Kintsugi because of a problem I experienced personally. I spent nearly five months calling my provider just to schedule an initial therapy appointment, and no one ever returned my calls. I kept trying – but I remember thinking very clearly that if this were my dad or my brother, they would have stopped long before I did,” said CEO Grace Chang while in conversation with Unite.AI.

The California-based company was founded in 2019 as a solution to what Chang described as a “triage bottleneck”. The founder believed detecting severity earlier and passively could help get people to the right level of care faster. And, through Kintsugi Voice, voice biomarkers identify clinical depression and anxiety.

Research abounds proving the successful use of AI-driven speech and voice analysis as a biomarker for mental health conditions. A May 2025 paper, for example, found that acoustic biomarkers can detect early signs of mental health and neurodivergence, and argued for the integration of singing analyses in clinical settings to assess patients’ potential cognitive decline.

Voice measures, in fact, have an accuracy rate of 78% to 96% in identifying people with depression versus those without it, according to the American Psychiatric Association. Another study used a one-minute verbal fluency test in which an individual named as many words as possible within a given category – finding 70% to 83% accuracy in detecting when a subject had both depression and anxiety.

To assess their users’ mental health, Kintsugi requests a short speech clip, after which its vocal biomarker technology analyzes pitch, intonation, tone and pauses – markers found to be associated with conditions like depression, anxiety, bipolar disorder and dementia.

What Chang did not initially realize, however, was that the technology had unlocked one of the security industry’s most pressing contemporary challenges: identifying what makes human voices human.

From Mental Health Care to Cybersecurity

While attending a summit in New York in late 2025, Chang mentioned to a friend in the cybersecurity field that her team’s experimentation with synthetic voices had been disappointing.

“We were exploring synthetic data to augment training for our mental-health models, but the generated voices were so different from authentic human speech that we could tell nearly 100% of the time,” she said.

“He stopped me and said, ‘Grace – that’s not a solved problem in security.’ That was the moment everything clicked. Since then, conversations with security, financial services, and telco companies have confirmed just how quickly deepfake voice attacks are rising – and how real the need is to distinguish human from synthetic voices in live calls,” the CEO added.

In April last year, the FBI warned of a malicious text and voice messaging campaign that posed as communications from senior U.S. officials and targeted former government workers and their contacts. Large national banks in the U.S. were also targeted with 5.5 average daily voice manipulation fraud attempts, and hospital staff at Vanderbilt University Medical Center reported vishing attacks from impersonators posing as friends, supervisors, and co-workers.

Regardless, deepfakes did not initially factor into Kintsugi’s work. While the company’s team had been using off-the-shelf models like Cartesia, Sesame and ElevenLabs to experiment with synthetic voices for administrative call center agents and outbound workflows, deepfake fraud was not their focus amid a crowded and accessible market featuring models like Sora.

Human-level signals that indicate voice authenticity, however, are the same biomarkers that make someone human in the first place. Irrespective of language or semantics, Kintsugi Voice operates with signal processing and the physical latency of speech, capturing subtle timing, prosodic variability, cognitive load, and physiological markers that reflect how speech is produced… not what is said.

“Synthetic voices can sound fluent, but they don’t carry the same biological and cognitive artifacts,” said Chang. The company’s model is consistently a top-decile performer in detection accuracy, using as little as 3 to 5 seconds of audio.

Kintsugi may be revolutionary for those who struggle with mental health, especially in areas where getting treatment with professionals takes time and resources. By the same token, its technology poses a revolution for deepfake detection and cybersecurity more generally: authenticity detection rather than deepfake recognition.

The Future Lies on Human-Centered Tech

Cybersecurity has long been focused on malignant use of technologies or perpetrators themselves. Kintsugi’s accidental discovery, however, bets on humanity itself.

“We’re operating on a completely different surface area: human authenticity itself. LLMs can’t reliably detect LLM-generated content, and artifact-based methods are fragile. Capturing large, clinically labeled datasets that encode real human variability is expensive, slow, and outside the core expertise of most security companies — which makes this approach difficult to replicate,” Chang noted.

The startup’s approach also suggests a broader shift: cross-domain innovation. Those front runners in health care might just lead the charge in AI-backed vishing detection, just as those innovators in space tech could support newfound emergency response mechanisms, or gamers architecture and urban planning.

As for Chang, she plans to become a standard for verifying real humans and, eventually, real intent through voice interactions.

“Just as HTTPS became a default trust layer for the web, we believe “proof of human” will become a foundational layer for voice-based systems. Signal is the beginning of that infrastructure,” she said.

As generative AI continues to accelerate, the most effective safeguards might come from understanding what makes humans… well, human.