In This Story
An unlikely connection between financial market algorithms and human speech is the focus of a paper by George Mason University researcher Anomadarshi Barua that was accepted by Empirical Methods in Natural Language Processing (EMNLP), one of the world’s most prestigious conferences for natural language processing (NLP).
The paper, which could open a new direction in NLP research, was among the 14 percent accepted at a conference that received nearly 18,000 paper submissions.
Barua, an assistant professor of cyber security engineering, and students Tarikul Islam Tamiti, Sajid Fardin Dipto, David Vergano, and Luke Baja-Ricketts, borrowed ideas used in finance and economics to tackle a stubborn problem in speech processing: reconstructing important pieces of audio that have been lost. “This is the first paper to show that we can actually pick up important algorithms from those fields and merge them into the speech domain,” Barua said.
The paper addresses something called speech bandwidth extension, the process of reconstructing missing content in limited or degraded audio by artificially recreating missing high-frequency sounds. Those missing high-frequency components can be particularly important for sounds such as fricatives (a consonant sound made by forcing air through a narrow gap in the mouth or throat), sibilants (high-pitched, noisy consonant sounds made by forcing air through a narrow channel in the mouth against the teeth), and sharp consonant sounds. This missing audio affects how natural and intelligible speech sounds.
Because limited-quality audio contains few clues about these missing sounds, NLP AI must determine what sound is missing, when it should occur, and how strong it should be. Traditional machine-learning approaches struggle with those pieces of information because they evaluate the speech spectrum more broadly. That can cause brief high-frequency details to be smoothed over.
“So that was the missing piece,” Barua said. “Existing discriminators focus equally on the whole spectrum and can't focus specifically on the high frequencies. We took a detour from the machine learning domain and to try to explore some new knowledge.”
The connection works because finance and speech processing share a similar challenge: important information can be hidden in unusual or uncertain events. Financial models are often designed to pay special attention to rare but potentially costly outcomes instead of simply averaging everything together. Barua’s team applied that same idea to speech. The new approach directs more attention to brief, high-frequency sounds that are easy to miss but important to understanding speech. Other techniques borrowed from decision science help the system recognize when it is uncertain about what sound is missing and prevent it from adding too much high-frequency information.
The resulting system, Uncertainty- and Decision-Science Inspired Swin BandWidth Extension (UDSS-BWE), incorporates five specialized “discriminators” inspired by decision science and approaches to uncertainty. Among them are concepts based on “conditional value at risk,” used in finance to focus attention on potentially severe outcomes, and chance constraints, used in decision-making under uncertainty. Instead of treating every part of a speech signal equally, the system can pay greater attention to rare-but-important errors while guarding against artificially boosting high frequencies.
In testing, UDSS-BWE achieved the best perceptual score on an English speech dataset. It also proved robust under noisy conditions. Perhaps most strikingly, it accomplished its gains with only 18.5 million parameters compared with 72 million for a leading comparison model.
“Our model is actually outperforming the current state-of-the-art,” Barua said. “So this is a strong indication that the five algorithms we took from finance are actually preventing the smoothing out of high-frequency components.”
Broader significance, according to Barua, goes beyond improving speech. The underlying problem is reconstructing signals from sparse information, meaning the same approach could potentially extend to electrical signals, sonar, and satellite data.
Tamiti said, “I am hopeful that this work can move beyond theoretical development into real-world implementation and eventually contribute to cutting-edge defense and military applications.”
Tamiti will represent the team at EMNLP. “Conferences like this provide a unique opportunity to understand not only the latest technical developments but also how researchers are approaching emerging challenges from different perspectives,” he said. “I am particularly interested in learning how current technologies and security research are evolving as we transition into the post-quantum era and what new research directions may emerge from that transition.”