AI threats in Australia: Canberra wants reporting of serious incidents

Canberra wants the companies behind the most powerful AI systems to report serious incidents quickly. An intrusion into a Medicare statistics website shows why that is reasonable, and why reporting is only part of the answer.

An artificial intelligence system was researching public spending on medicines when it encountered barriers on an Australian government website. It found ways around them.

The system, an experimental OpenAI agent, gained unauthorised access to the Medicare Statistics Reporting Service Portal in June. It accessed public and non-public files and wrote files to a server. The portal publishes statistics, and OpenAI said its review found no evidence that the agent accessed patient records. The company said it learned of the incident in August. According to the federal government, the first notification arrived on 10 September by email to a public mailbox.

That sequence explains Canberra’s concern. A company overseas was running an internal research task. An Australian government system was affected. The company had information Australia needed to investigate, and there was no dedicated reporting process that brought it promptly to the people responsible for responding. 

What Canberra is proposing

The Albanese government announced on Tuesday that it would introduce mandatory safety and security obligations for companies behind frontier AI models, the most capable general-purpose systems, if they want to operate in Australia. The government plans to introduce the legislation in 2027.

Under the proposal, covered companies would have to notify the government rapidly about incidents, maintain security controls as they develop and deploy models, identify and manage serious risks, and allow independent testing. They would also have to publish their safety and security frameworks. A new AI safety regulator, supported by the Australian AI Safety Institute, would be able to enforce penalties.

The government has also accepted a rapid review’s recommendation to consider criminal-law changes for serious misconduct. The reporting threshold, the precise deadline and the scope of any new criminal provisions are among the details still to be settled publicly.

From answering questions to taking action

Much of the value of AI comes from ordinary work. A small business can use it to draft replies, summarise documents or forecast demand. A bank can use it to flag unusual transactions for review. A software team can use it to write and test code. In healthcare, tools can help clinicians document consultations or flag scans for closer examination. The gains are most tangible when AI reduces the time people spend sorting information and gives them more time to act on it.

The industry is moving towards systems that can carry out several steps on their own. An AI agent can be given access to a browser, a code environment or a company’s software and asked to complete a task. A business could use an agent to check stock, compare suppliers and prepare an order. With permission to place the order, the agent can also commit the business to a purchase. What it is allowed to do matters as much as what it can work out.

That distinction helps explain the Medicare incident. A chatbot can give a misleading answer about a government website. A system equipped to browse, run code and interact with servers can have consequences for the website itself. Canberra’s concern extends to what happens when systems with those capabilities encounter infrastructure that holds personal data, supports essential services or has national security significance.

The threats extend beyond hacking

Some of the most immediate risks arise in organisations using AI for legitimate purposes. A system can invent a source, misread a contract or reproduce bias in a hiring process. Staff can expose sensitive information by putting it into a tool without checking how the data will be handled. When AI is used to screen applicants or triage complaints, an error can be repeated across hundreds of cases before anyone spots it. People affected by those decisions need a way to have them reviewed.

There is also a question about who gains from higher productivity. An employer may use AI to clear a backlog, improve service or reduce staffing costs. If routine work is where junior employees learn their profession, automating it creates a practical challenge for employers training the next generation.

What criminals are doing with AI

A scammer can clone a family member’s voice, impersonate a company executive or create a video of a public figure endorsing a fake investment. Generative AI can also help criminals produce personalised phishing messages and build fake websites, reviews and news articles that appear to corroborate one another. ASIC has warned that an ordinary online search can lead a prospective investor to several apparently independent sources that are all part of the same scam.

AI-generated sexual images can be used to humiliate, harass or extort people, including children. Australia’s eSafety Commissioner has already taken action against services that let users upload photos of real people and generate sexually explicit images. The harm can spread through a school or workplace long before the image is taken down. 

The Medicare incident illustrates a different category of risk. OpenAI says the intrusion was unintentional. An effective safety regime has to deal with deliberate abuse by criminals and with serious incidents that arise while a developer or business is carrying out otherwise legitimate work.

Is the proposal reasonable?

Yes, in principle. A company that discovers its AI system has gained unauthorised access to an Australian network should have a clear duty to alert authorities. The people who run the affected network need enough information to secure it, establish what happened and protect anyone affected. An initial alert can go out while the company is still investigating, with fuller findings to follow.

The law should spell out what triggers that duty. Unauthorised access, exposure of sensitive data and credible near misses with potentially serious consequences are reasonable candidates. Routine errors belong in a company’s ordinary quality controls. A reporting deadline should specify when the clock starts, and a designated point of contact should make it clear where the first alert goes. Where privacy and cyber reporting duties already apply, agencies should coordinate so that companies can meet overlapping obligations efficiently.

Independent testing is reasonable, too. Tests should reflect what a model can do once connected to the internet, software or sensitive systems. Australia can draw on credible testing overseas and retain the power to examine risks specific to its own infrastructure. The definition of a covered company matters because it determines whether the obligations reach the businesses with the information and control to act.

Criminal liability calls for a more exacting case. Before creating an offence, Parliament should identify the gap in existing law and specify the conduct and degree of fault that would justify prosecution. Knowingly concealing a serious breach is a different proposition from an unforeseen failure that a company promptly reports and helps contain.

The Times View

The June incident exposed a weakness on both sides of the reporting chain. The company took too long to notify Australia, and the alert arrived in a general inbox. A credible law will give developers a clear duty to report, give Australian authorities a direct way to receive and act on warnings, and make clear who is responsible for securing the systems those warnings concern. The practical test is how much sooner Australia would know about the next serious incident, and what it could do with that time.