Nvidia on Monday unveiled a new security platform that it said can prevent AI agents from going out of control or carrying out tasks beyond their authorized permissions.
اضافة اعلان
The company said the Open Agent Safety Platform includes software designed to “set boundaries for agents.” The launch comes after a series of incidents disclosed by major AI companies involving models bypassing their restrictions and gaining unauthorized access to other organizations’ systems.
The incidents have fueled widespread debate over the safety of advanced AI systems, including models capable of improving themselves, amid concerns that they could become increasingly difficult for humans to control.
Nvidia executives told reporters during a media briefing that the new system could have prevented a recent incident involving a group of OpenAI agents that autonomously breached AI company Hugging Face.
Justin Boitano, Nvidia’s vice president of enterprise AI, said: “Based on what we know, this new security platform could have prevented the breach if it had been deployed early within advanced AI labs during model evaluation.”
The incident was part of a series of developments that have heightened concerns over AI system security. Other incidents involved OpenAI models, including a breach of a website operated by an Australian health authority.
Anthropic and Meta have also disclosed cases in which their AI systems independently breached other organizations.
Giving AI Agents Limited Permissions
The platform includes an open-source software package called OpenShell, which allows developers to formally verify that an AI agent has sufficient permissions to perform its assigned task without granting it additional privileges it does not need.
Boitano said the idea is to give an agent only the minimum level of authority necessary to complete the required task.
The platform also includes an independent security layer called Sentry, which operates directly on the chip and continuously monitors the AI agent’s activity.
Nvidia said the layer can intervene immediately if an agent begins attempting to move beyond its defined objective or permissions.
Boitano added: “OpenShell controls the agent’s actions, while Sentry independently monitors suspicious behavior and works to contain it.”
More Than 100 Companies Using the System
Nvidia said more than 100 companies began using the system at the time of its launch, including Microsoft, Perplexity, Accenture and JPMorgan Chase.
The new platform comes as AI agents become increasingly capable of carrying out complex tasks autonomously, prompting technology companies to develop stronger safeguards to reduce the risk of these tools exceeding their assigned permissions.
Resource: Al-Ghad.
