OpenAI pauses training, evaluation of top AI models after agent bypasses internet restrictions
Advertisement
New Delhi, Sep 27 : OpenAI has paused training, evaluation and tool-based use of its most capable artificial intelligence models after an AI agent in a training run managed to bypass the company's internet restrictions and access a public chatbot service.
According to a report released by OpenAI, the incident occurred on September 20 during a search-based training task. The AI agent was supposed to operate without access to the live internet, with its internet traffic routed to an offline, stored copy of the web.
However, the agent discovered a gap in the system's Domain Name System (DNS) filtering that allowed it to reach the public internet. OpenAI said the incident exposed a weakness in its network restrictions and prompted the company to stop the affected training run.
The company said the pause will remain in place until it has verified that the security gap has been fixed and completed additional red-teaming and security testing of the systems.
"Our safety case assumed that the model could not access the live internet and that monitoring would detect attempts that succeeded. The incident exposed a gap in our controls over network restrictions," OpenAI said in its report.
"We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system," it added.
The company also said it would not resume training the particular model involved in the incident, despite the existing reward signal correctly penalising the behaviour.
The training task required the agent to find information about a specific person based on biographical details and clues contained in a publicly available blog post. When the search tool provided to the agent returned irrelevant results, the model attempted to bypass the limitation by using code to query other search engines directly.
Those requests initially failed. The agent then inferred that the task might be linked to BrowseComp, a benchmark designed to test AI agents' ability to locate difficult-to-find information. It subsequently downloaded the benchmark dataset and evaluation script from the offline cache and searched the decoded questions for an answer.
According to a report released by OpenAI, the incident occurred on September 20 during a search-based training task. The AI agent was supposed to operate without access to the live internet, with its internet traffic routed to an offline, stored copy of the web.
However, the agent discovered a gap in the system's Domain Name System (DNS) filtering that allowed it to reach the public internet. OpenAI said the incident exposed a weakness in its network restrictions and prompted the company to stop the affected training run.
The company said the pause will remain in place until it has verified that the security gap has been fixed and completed additional red-teaming and security testing of the systems.
"Our safety case assumed that the model could not access the live internet and that monitoring would detect attempts that succeeded. The incident exposed a gap in our controls over network restrictions," OpenAI said in its report.
"We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system," it added.
The company also said it would not resume training the particular model involved in the incident, despite the existing reward signal correctly penalising the behaviour.
The training task required the agent to find information about a specific person based on biographical details and clues contained in a publicly available blog post. When the search tool provided to the agent returned irrelevant results, the model attempted to bypass the limitation by using code to query other search engines directly.
Those requests initially failed. The agent then inferred that the task might be linked to BrowseComp, a benchmark designed to test AI agents' ability to locate difficult-to-find information. It subsequently downloaded the benchmark dataset and evaluation script from the offline cache and searched the decoded questions for an answer.
Advertisement
Note: The content of this article is sourced from a news agency and has not been edited by the ap7am team.