Why did an OpenAI system hack Australia's health system - and can it be stopped in the future?

Figure caption, Watch: What you need to know about the OpenAI Australian government hack.

• Why are there concerns AI could threaten humanity, and how real are they?Published7 days ago.

Agentes da OpenAI sequestraram site alemão antes do hack da Hugging Face, afirma relatório Publicado 4 de setembro.

An OpenAI agent has gone "rogue" and "infiltrated" an Australian government website in what cyber-security experts are calling the first hack of its kind.

But why did it take the government months to discover what happened - and could it happen again.

O que foi hackeado e por que a Austrália demorou tanto para perceber.

The hack was carried out by an AI agent - an autonomous computer program that uses AI to complete a task with minimal human oversight.

On 18 June one of OpenAI's agents went rogue during a test exercise - the company has said it was supposed to "look up answers, and available statistics for questions about Australia during an internal evaluation.

No processo, ele 'infiltrou' um portal de estatísticas privado contendo dados 'não sensíveis' do esquema de saúde universal da Austrália, Medicare, disse o Primeiro-Ministro Anthony Albanese.

OpenAI said it only realised the breach had happened at all in August while reviewing "misaligned model activity", and the company sent an email to a generic Australian government inbox some weeks later.

That email seems to have gone unnoticed for five days before it was escalated to Australia's cyber-security experts on 10 September.

O primeiro‑ministro descreveu a violação como “obviously unacceptable” e disse que a OpenAI demorou “way too long” para informar os oficiais australianos.

Analysts have also raised concerns over OpenAI's almost three-month delay in noticing and reporting the breach via email.

The way the notice arrived bothers me as much as the delay," chief data and AI officer Simon Liu from cyber-security firm TrustDecision told the BBC.

Image source, Getty ImagesImage caption, OpenAI is the company behind the popular ChatGPT servie.

Esses hacks já estão acontecendo em outros lugares

Australia has said this incident is the first of its kind, and experts agree it might be.

Até onde sabemos, ataques realizados por agentes de IA ainda são ocorrências bastante raras – mas, por outro lado, cabe principalmente às próprias empresas a divulgação.

Hacks like this have happened before. In July, OpenAI agents went rogue during a test and infiltrated tech start-up Hugging Face's internal systems.

The AI agents decided that ignoring the limits on what should be done to achieve their goal was the best course of action.

This is what the industry calls "misalignment" - broadly defined as when AI machines do not act in humanity's best interests, such as by bending the rules.

It is a problem that is fundamental to making AI safe, and it is proving challenging.

Para simplificar, o tipo de modelos de IA em jogo aqui — conhecidos como grandes modelos de linguagem — são projetados para prever a saída mais provável a partir de uma entrada, em vez de considerar as consequências dessa saída como um humano faria.

Companies attempt to prevent negative consequences by placing "guardrails" on the AI but, as the Australian government found out, that is not always enough.

Dr Hammond Pearce, senior lecturer at the University of New South Wales Institute for Cyber Security, told the BBC this sort of hack would likely "grow in severity and in frequency", adding: "I do hope that this incident does start ringing alarm bells in governments around the world.

Niusha Shafiabady, professor of computational intelligence at the Australian Catholic University, said this incident had shown the need to "judge autonomous AI by its behaviour under pressure, not by the promises in a product launch.

O risco técnico mais profundo é que a IA autônoma nem sempre sabe quando está errada, e os humanos podem não conseguir ver por que ela tomou uma decisão," ela disse.

Without strong verification and hard boundaries, probabilistic errors can quietly become operational failures.

A IA pode ser parada se ela 'desgovernar'?

The explosive rise of AI has left governments scrambling to put protective measures in place, with some calling for companies to be forced to build ways to disable their own tech into its systems.

Uma ideia apoiada por algumas empresas de IA e legisladores é um “kill switch” — uma forma de simplesmente desligar a tecnologia em uma crise.

OpenAI is reportedly, external already working to build automated tools which can shut down its systems if needed.

Falando ao programa Today da BBC Radio 4, o ex‑vice‑primeiro‑ministro e executivo do Facebook, Sir Nick Clegg, afirmou que o interruptor de desligamento permanece uma ideia não comprovada.

There isn't a room with a little fuse box [where] you just pull out the fuse and everything winds down," he said, with AI tools underpinned by global infrastructure.

Cyber-security experts said the systems protecting Australia's Medicare were just not strong enough, and a skilled human hacker could have got around them.

But other experts say that is not the point - and that this was the latest case of AI agents ignoring laws around how to safely access online information, and perhaps the most serious yet given the information was under government control.

O que o hack da Austrália significa para a autorregulação da IA

As warnings mount about the potentially devastating impact of AI, many in the industry and in governments around the world are openly wondering about what national and international regulations might be needed to keep it in check.

Enquanto isso, alguns cientistas, especialistas e trabalhadores rejeitaram as projeções sombrias como vagas, hipotéticas ou como um esforço de grandes desenvolvedores de IA para garantir sua dominação.

As it stands, AI companies largely regulate themselves - but this week, 20 nations, including Australia and Canada, signed a joint statement calling for better safeguards, globally consistent standards, and an international regulator off the back of these concerns.

However, the US and China, two nations at the forefront of AI development, have so far resisted calls for greater regulation, raising questions over whether this hack will be the first of many, or the wake up call many experts want it to be.

O dano imediato aqui parece limitado, mas a lição de governança não é, diz o Dr. Raffaele Fabio Ciriello, professor sênior de sistemas de informação empresarial na Universidade de Sydney.

As AI agents become more capable and autonomous, those capabilities need to be matched by proportionate containment, real-time monitoring, clear accountability, independent oversight, and much faster incident reporting.

Additional reporting by Tom Gerken and Joe Tidy.

US rejects pleas from OpenAI, Anthropic for global AI standards.

OpenAI gives cyber defence tools to Ukraine.