In This Story
Associate Professor ThanhVu Nguyen and Assistant Professor Ziyu Yao, both in the College of Engineering and Computing's Department of Computer Science, received awards through Amazon's Build on Trainium program. George Mason is one of only four universities with multiple recipients in the latest round of awards, which includes researchers from 30 universities.
Nguyen's project, "Leveraging AWS Trainium for Verifiable AI and ML-Assisted Mathematical Reasoning," addresses a fundamental problem accompanying AI's rapid growth: How can we know whether the answers AI produces are actually correct? Yao's project is "Representation Engineering of LLMs for Secure Code Generation." Her research focuses on natural language processing and the internal workings of large language models (LLMs), including how those models can generate computer code more securely.
Build on Trainium is a $110 million credit program designed to support AI research and university education using AWS Trainium, Amazon Web Services' purpose-built AI chips. Recipients receive AWS credits, good for use on Trainium, and access to AI and machine learning services, public datasets, technical resources, and an Amazon research contact.
Nguyen said having two researchers selected in the same round is particularly noteworthy given the international competition for the awards. "The chance of a university getting this is low," he said. "The chance that a university has two people getting it at the same time is exponentially low."
For Nguyen, the award provides an opportunity to explore a relatively new computing platform while advancing his research. "It's not just about getting credits to run AI," Nguyen said. "It's a new platform. If we can make discoveries or help Amazon realize what the platform is capable of, that’s a great scientific contribution."
Nguyen works on neural network verification, including research supported by a National Science Foundation CAREER award and a previous Amazon Research Award. He describes the problem through applications such as autonomous vehicles, where recognizing a pedestrian, traffic light, or other object incorrectly could have serious consequences.
His latest project extends that interest into mathematical reasoning. AI systems can now generate increasingly sophisticated software, mathematical proofs, and other solutions, but those outputs can still contain errors.
"AI will give you a program or software or theorems or solutions that look good, but we don't know if they're correct," Nguyen said. "It hallucinates. When it writes a program for you, it does a great job, but there are bugs in there." Mathematical reasoning, he said, offers a way to verify those results using Lean, an emerging formal theorem-proving platform, rather than simply trusting an AI system's output.
Two PhD students will work with Nguyen on the project, with one concentrating on neural network verification and the other on machine learning-assisted mathematical reasoning. Nguyen sees the latter as an increasingly important direction for research as AI systems become more capable and their outputs become more difficult to evaluate.
LLMs are powerful backbones for automated code generation and are widely used to assist programmers in software engineering (SE) tasks. Yao said, “Our prior work (with PhD student Hao Yan and computer science faculty member Xiaokuan Zhang, and funded by the Virginia Commonwealth Cyber Initiative) found that LLMs still produce a non-negligible amount of insecure code that contains vulnerabilities—weaknesses or flaws in the code that attackers can exploit to compromise a system.”
Yao explained that detecting and preventing LLMs from generating vulnerable source code is critical. In this project, she and her students will explore a new perspective of secure code generation, or “representation engineering,” as an approach to directly monitor and control an LLM’s safety behavior in code generation.
“Representation engineering is an emerging research topic aimed at attributing a neural model’s behavior to its internal neural representations, such that one can monitor the model’s behavioral intent by tracking the activation of its representations and control its behavior by directly steering these representations, like doing a ‘surgery’ on an LLM without changing its model parameters,” said Yao. Yao will be the first to explore how this methodology could be applied to monitor and control an LLM’s secure code generation, producing new knowledge about why LLMs generate secure or vulnerable source code and novel approaches for direct mitigation of vulnerable code generation.
Yao said, “Trainium allows us to perform model interpretation and representation engineering for very large LLMs. We expect the project to result in not only paper publications but also open-sourced code LLMs steered to have better code security for researchers and SE practitioners to reuse in the future.”
For Nguyen, the value of the program goes beyond access to computing resources. It also creates an opportunity to deepen connections between George Mason researchers and Amazon while exploring what a new generation of AI hardware can enable.