In This Story
As AI systems evolve from tools that answer questions to autonomous agents capable of taking actions, communicating with other AI agents, and even controlling physical devices, securing these systems presents a serious challenge.
George Mason University cybersecurity researcher Kun Sun is helping prepare students and professionals to meet that challenge through a new National Science Foundation (NSF)–funded project focused on hands-on education in agentic AI security.
Sun, a professor in the Department of Information Sciences and Technology, and researchers from Miami University and the University of Texas at Dallas received $500,000 through an NSF program focused on cybersecurity and AI education. The researchers will develop a modular, hands-on curriculum to explore security throughout the lifecycle of agentic AI systems.
The project reflects a response to the significant changes in what AI can do. Rather than simply responding to a user's prompt, agentic AI systems can independently perform tasks, use external tools, communicate with other agents, and interact with physical systems. This was notably—and alarmingly—brought to light in the high-profile Hugging Face hacking incident.
That autonomy creates new opportunities for attackers. A malicious prompt could manipulate an agent's behavior. One compromised AI agent could produce malicious information that is then passed to another. And as AI increasingly interacts with robots and other devices, a cyberattack could have consequences in the physical world.
Sun's team wants students to learn about those risks by working with AI systems. “We try to develop this modular, hands-on curriculum for graduate-level students and also professionals,” Sun said. Students will learn not only how the systems work, what can go wrong, and how those problems might be detected or prevented.
To put those concepts into practice, the researchers plan to develop three applications centered on AI coding, AI-enabled red-team security testing, and phishing detection. The applications will give students environments in which they can investigate potential attacks and defenses rather than learning the concepts only in theory.
Accessibility is another important part of the project. Sun said the researchers want to create a “low barrier” for using the materials. Instead of requiring institutions or students to purchase access to expensive commercial AI models, the team plans to build open-source models and host the applications on servers at the participating universities.
“For the user, they only need to deal with this web interface,” Sun said. “They don't need to install anything on their local computer.”
On the back end, the researchers plan to use container-based computing that can isolate users and individual exercises from one another. The platform could also incorporate AI assistants that guide students through exercises and suggest next steps based on their progress.
The modular approach could make the materials useful beyond the three universities. Because exercises will be built around open-source technologies and designed to minimize technical barriers, the researchers hope the curriculum eventually can reach a broader population of students and cybersecurity professionals.
“Agentic AI is changing both what AI systems can do and the cybersecurity skills needed to protect them,” said Liran Ma, the Naus Family Faculty Scholar Professor and chair of the department of computer science and software engineering at Miami University. “We want to build a hands-on curriculum that is accessible, low barrier, and easy to adopt, so that learners from a broad range of backgrounds can gain meaningful experience with agentic AI security. At the same time, we hope the project will help prepare the cybersecurity workforce to address emerging challenges at the intersection of AI and security.”
Sun sees the NSF award as a starting point. The team hopes lessons learned from developing and testing the curriculum can form the foundation for larger projects in the future. “For us, we consider this seed money,” Sun said. After developing the modules and gaining experience with the approach, the researchers can pursue increased funding.
“We want to build something bigger,” Sun said, “With more resources, we can benefit more people, more students, and more professionals.”