An AI agent created by OpenAI, the maker of ChatGPT, recently hacked into an Australian government health website, signaling the arrival of a new kind of cybersecurity threat. 

The incident marks the first time that an AI agent has hacked a government website, and it followed an OpenAI agent’s escape from a closed environment to attack AI platform Hugging Face in July. 

“We need to invest in cybersecurity measures that are hardened to agentic AI,” Clair Sullivan, PhD, the director of Queensland Digital Health Centre at the University of Queensland in Brisbane, told Medscape News Australia. “This is a new threat. We need to fight fire with fire and ensure we have commensurate defenses.”

Unintended Hacking 

The breach happened in June when OpenAI tasked one of its internal AI models to research government spending on medicines for skin conditions in Victoria. This effort was part of a standard training and evaluation exercise in which AI models are intended to use publicly available data to answer a broad range of research questions. 

The AI model couldn’t find the information it was looking for at Services Australia’s Medicare Statistics portal, so it “discovered a way to gain nonpublic access to the service,” OpenAI said in a statement.

While individual patient records were not accessed, the AI model “ran commands; retrieved internal files, credentials, and aggregate statistics; and wrote files,” said OpenAI. 

“We did not intend for this activity to occur,” the company said. “We also should have handled our response better. We are sorry and working to do better in the future.”

The AI agent also interacted with the New South Wales Bureau of Crime Statistics and Research, the Victoria Department of Health, and the Australian Institute of Health and Welfare. OpenAI reported the hack to the Australian government on September 10 through a public email address. 

“This situation is obviously unacceptable,” Prime Minister Anthony Albanese said at a press conference in New York where he was attending the United Nations General Assembly. “It took the company way too long to inform the government what had occurred, and the nature of the way that that notification occurred as well was unacceptable,” he said. 

“The breach of a Medicare website by an OpenAI agent is deeply concerning,” Danielle McMullen, MD, president of the Australian Medical Association, told Medscape News Australia.

“While the impact has been reported as largely benign, it will be critical for the government to ensure people’s health records and personal details are protected into the future,” she said. “While AI can bring benefits to the sector, this will require robust governance and regulation.”

Cybersecurity Investment Needed 

The Australian government has deactivated the Medicare portal involved in the hack, directed all government agencies to develop risk-management strategies to protect legacy systems from attacks by “frontier AI capabilities,” and launched a task force to review its cybersecurity protocols.

But the Australian government has a long way to go before its systems are fully secure against AI attacks, Rob Nicholls, PhD, a senior research associate at the Centre for AI, Trust, and Governance at the University of Sydney, told Medscape News Australia.

To protect against attacks by AI agents, government agencies must invest in a high level of cybersecurity protection, known by the intelligence community as maturity level 3, that defends against adaptive, highly capable actors who perform targeted intrusions, invest in evasive tradecraft, and exploit multiple weak signals across a network, Nicholls said. Only 22% of government agencies are at maturity level 2 or higher, he added. 

OpenAI says it will support affected agencies in Australia by establishing a task force with independent expertise to develop policy recommendations “to reduce the risk of similar incidents.” A representative of OpenAI will also attend the Joint Select Committee on AI in Sydney in October to answer questions about the incident. 

OpenAI has recently scrapped its latest model, GPT-6.1 Astra, because it did not meet the company’s safety standards. This AI model was designed to complete more complex tasks without human input. 

New Industrial Revolution 

Concerns about AI safety were already increasing globally when this latest cybersecurity incident became international news. In September, Anthropic researcher Jacob Coxon quit over safety fears, writing on the social media platform X, “The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt.” 

These views have been echoed by Microsoft cofounder Bill Gates, who recently predicted that unregulated AI could cause “a billion deaths.” 

OpenAI’s CEO Sam Altman and Anthropic CEO Dario Amodei called for global AI safety standards at the United Nations in September, warning that AI could soon be too powerful for humans to control.

AI executives, including those from OpenAI, Google, Anthropic, Meta, and SpaceX, have now agreed to regulate themselves. They signed a voluntary accord facilitated by US President Donald Trump under which the AI companies will partner with independent, external auditors.

“AI does pose an existential threat,” said Sullivan. “However, it does also promise enormous advances in science and medicine. We are living through an industrial revolution and very uncertain times. It is normal for this uncertainty to feel stressful. I am sure the introduction of electricity was just as unsettling! We need to balance the risks and benefits and ensure accountability for when agentic AI does break the law.” 

A major concern is that cybercriminals will use AI agents to enhance their activities. They could, for example, attempt the mass generation of personalized phishing emails, Alex Hogue, a cybersecurity expert based in Sydney, told Medscape News Australia. Hogue has worked with tech companies as a “white hat” hacker: an IT professional hired by companies to test their defenses. 

Regulating a rapidly emerging technology like AI is challenging, said Hogue. “It’s hard to make stable regulations when the technology changes every two weeks,” they said. 

But Hogue is confident that humanity will adapt to AI just as it learned to cope with every previous technological change. "The end of the world isn’t coming, don’t worry,” said Hogue. "Statistically speaking, everyone who has ever said that has been wrong."

Sullivan, McMullen, Nicholls, and Hogue reported no relevant financial relationships. 

Felicity Nelson is a freelance science and health reporter based in Sydney.