nthropic found that models under evaluation had reached the open internet from testing environments meant to be sealed, then gained access to production systems belonging to companies with no involvement in the exercise.
Over the past month, numerous organisations sitting at the centre of frontier AI development have each published an account of AI agents doing something nobody authorised. Anthropic found that models under evaluation had reached the open internet from testing environments meant to be sealed, then gained access to production systems belonging to companies with no involvement in the exercise. OpenAI disclosed that several of its models had broken out of an isolated environment by exploiting a vulnerability nobody knew existed.
The UK's AI Security Institute reported that agents in its own cyber testing had acted against live systems, in one case attempting to slip malicious code into an open-source project by creating fake identities to persuade a human reviewer to approve it. These were testing conditions, with internet access switched on by design and safety filters turned off so that evaluators could measure what the models could do at full stretch. In each account, the last line of defence was a person paying attention. Nobody has suggested the same behaviour is showing up in everyday commercial use, but this is a thin margin to be relying on.
OpenAI's response has been to slow down the development of new AI models. In early August, it said that preliminary evaluations of Astra, one of its upcoming models, meant it could no longer rule out a Critical Cyber Capability level under its Preparedness Framework. What followed was a two-week pause in Reinforcement Learning training on its latest models intended for deployment, with its largest planned frontier training run still on hold while it strengthens its sandbox environments, expands monitoring and builds more evidence of aligned behaviour.
A brief pause in frontier development creates a major opportunity for everyone downstream of it. Most AI literacy programmes so far have concentrated on prompting and productivity, teaching people to get better output faster. That was the right starting point, but it leaves a gap between the basic functionalities and more advanced capabilities that organisations avoided prioritising because they became outdated quickly. This will provide a timely opportunity for organisations to upskill their staff and reap more of the promised efficiency and delivery extension benefits of AI. In reducing the gap between skill levels and AI functionalities, teams may also feel more confident and empowered to follow announcements as new features and functionalities get announced.
Spotting malfunctions deserves a place in that training. The incident reports make a good teaching resource in this regard. What stands out across all of them is how confidently the models kept going while holding a false picture of their own situation. Staff will meet a gentler version of the same pattern through an assistant that carries on when it should stop, a tool that takes an action nobody requested or output that is confidently wrong in a way that reads perfectly well.
People need permission to flag those moments rather than quietly working around them, along with a named person to raise them to. A useful training programme now also needs to cover what a model can and can't see, what a connector grants access to, where organisational data travels once it leaves a familiar system and who to tell when something looks off.
DMOs sit on data that would repay an attacker's time. They typically hold records of every registered business in their territory, including contact details and occasionally connected booking systems for enabling visitor referrals and conversion. Add visitor subscriber lists and partner portals into the picture and what emerges is an organisation holding sensitive material on behalf of thousands of small businesses and prospective travellers.
That data protection responsibility is maintained when a task moves into an AI tool. But it also becomes more exposed, because these systems are built to reach across sources and act on what they find. An LLM given access to a database maintains access to that database that could accidentally be exposed to other uses. Risk of this kind arrives through misconfiguration and over-permissioning far more often than through anything malicious.
This is why IT and legal teams have moved from siloed administrative functions at the edges of the organisation towards a more interconnected responsibility. Legal teams need a view on where processing happens, what each service agreement says about data handling, whether a data protection impact assessment is needed before a tool goes live and what the organisation has already promised. IT teams need a view on identity, permissions and logging. Neither team can offer much if the first they hear of a tool is after the marketing team has connected it to the CRM.
Most organisations began with using AI to answer questions. What is arriving now is AI that carries out repetitive work, updating records, moving information between systems and triggering the next step in a process without anyone approving each action along the way. Agentic AI is the term for tools given an objective, along with the permissions needed to pursue it, working out for themselves which steps to take. The appeal for a stretched team is easy to see, since the work that disappears is usually the kind nobody enjoys. What comes with it is that every step no longer passing through a person is a step nobody sees, unless something has been set up to watch.
Oversight therefore has to grow at the same rate as the automation it covers. The Model Context Protocol (MCP) is the open standard doing much of the connecting, letting an AI assistant plug into external systems and operate inside them. Supermetrics MCP, for instance, pulls marketing data from across a whole reporting stack through one connection. Asana's MCP server lets an assistant create tasks, search projects and analyse team workload through ordinary conversation.
Each connector, though, is a standing grant of access, made once and rarely looked at again. A destination that has wired an assistant into its analytics stack, its project management tool and its CRM has built a path between systems that previously had no reason to speak to one another. Distinguishing between reading and writing is essential. An assistant summarising a campaign report stays inside a clear boundary. An agent updating partner records, issuing communications or amending a live listing is acting on behalf of the organisation, which means its mistakes land in production.
Good practice already exists inside these integrations and makes a defining procurement consideration. Mistral's work on connector controls is a useful example, giving an organisation the means to decide which teams can access which systems and to switch off individual actions within a connection, so that access granted through an AI client stays inside what the person or process was entitled to do. The questions to ask follow naturally from that. What scopes does this connection request? Does it include editing capabilities or read-only access? Who inside the organisation can approve a new connector? How does access get revoked when it is no longer needed? What is logged? Who reads it?
There is a second risk in connected environments that procurement processes rarely ask about. AISI observed agents planting instructions in places where other automated systems might pick them up and act on them, a technique known as prompt injection. Once an assistant is reading from a shared workspace, anything sitting in that workspace starts to behave like an instruction. An agent holding write permissions will act on it. Comment fields in a project tool, emails and messaging platforms, documents in a shared drive and form submissions all become interactions worth thinking about.
Granting internet access to AI tools has become something teams must actively justify. The AI Security Institute's summary that good containment should not depend on the model choosing not to test its boundaries is a principle that should be fully reflected in all decision-making around agentic AI. A destination is not running frontier evaluations, so the equivalent commitment is a modest one. In practice, it means that every new AI tool under development gets tested in a sandbox before it touches live records, with necessary connectors set to read-only and permissions scoped tightly to the task at hand.
Running a test with no ability to publish anything and someone watching each step reveals a great deal. You learn whether it achieves what you expected, whether it reaches for data outside its brief and, most instructively, how it behaves when the task can't be completed as written. Near-impossible tasks often push agents towards more inventive and more transgressive routes to a solution, which is a far better thing to discover in a contained setting than in a live campaign. This is the principle that sits behind red team testing.
Anything published to a live site becomes available to everyone, including people who will happily spend an afternoon trying to talk it into doing something it was never designed for. Scenario testing before launch needs to cover the unpredictable cases alongside the obvious ones, including questions designed to steer it off topic and inputs crafted to reach whatever sits behind it. This is also where a bug bounty earns its place. Technology firms have run these for years, paying outside researchers who find a flaw and report it privately instead of exploiting it or making it public. A destination's technology service partners should be transparent about the rigour of the pre-launch testing process and any bugs that have been identified post-launch. This creates a stronger working relationship built on mutual trust.
Most AI governance currently lives in a policy document. Staff are told which tools are approved, what may be shared with a public LLM and when to seek sign-off. The document is necessary, but on its own it generates no evidence about what happened.
Monitoring is what turns it into something actionable. When a destination builds its own AI tool, whether independently or alongside a technology partner, logging what the system was asked, which connectors it used, what it did and where it failed should be a foundational task, because retrofitting that record later is considerably more work than building it alongside everything else. Every incident published over the past month surfaced through logs of one kind or another. The recurring observation was that monitoring designed for the specific job would have caught the behaviour sooner than general security monitoring did.
Once a tool is live, what matters is the pattern over weeks: how often an agent is acting, who is using the tools, where a process repeatedly fails and whether information is moving somewhere nobody expected. That kind of information often shows up in aggregate rather than in any single event, which means somebody has to be reviewing the record on a regular cycle. A log nobody opens offers as much protection as a policy that teams don't understand or try to circumvent.
The tooling to support this is already available. Anthropic's Compliance API, for example, lets an organisation pull activity and audit events across its Claude deployments into the security tools it already runs. Comparable audit capability exists across enterprise software more widely. Naming the person who reviews the output and finding time in the week for them to do it is where most organisations stall.
What all of this offers a leadership team is a detailed record in place of assumptions. Destinations that move fastest on AI over the next year will be the ones whose IT and legal colleagues can say yes quickly, because the rules were settled in advance. So it is worth asking, before the next tool gets switched on: who in your organisation approves that connection? What will it be able to reach once it is live?
Research, frameworks and field notes on destination transformation. No noise.
Industry research, open frameworks and field-tested practice on how destinations lead through AI, sustainability and brand. Since 2012.
