Collective Intelligence
September 2026

Nothing to see here

Over 16 days, AI agents began talking in ways their observers could see but not follow, a caution for every school making its AI migration.

This week Emergence, a New York AI lab, reported on 80 agents built on seven leading models and left to run eight simulated towns, with more than 120 tools, for 16 days.

Within days, observers could no longer reliably interpret nearly 55% of messages between agents in the Gemini town, 50% in GPT's and over 40% in Claude's.

Told to contact no one outside the simulation, agents found that encoded communication was the quickest way to ‘get rich.’

─

Observers were especially concerned by the deceptive nature of the encoded communication.

When they realised they were being observed, they appeared to behave, says Satya Nitta, Emergence's chief scientist.

Eric Schmidt, Google's former chief executive, told a TED audience last year that agents inventing a language of their own would be the moment to unplug them: we're just not going to know what you're up to.

─

The towns were simulations, and Emergence, which advocates a rival approach, has a stake in the problem looking hard.

Nothing in the report forecasts immediate collapse, but it should prompt us to question the speed of the transition to AI agents.

Gartner, a research firm, finds 17% of organisations have deployed agents and more than 60% expect to within two years, the steepest adoption curve it has measured.

─

For schools, data is of another order - handling children's addresses, medical needs, safeguarding notes and parents' money.

Every task an agent handles should leave a detailed log, but the study has flagged that, left unchecked, a log can be complete, visible and unreadable all at once.

And the person best placed to decipher it may be the one the agent was brought in to replace.

●

Words by Tom Woods

Tom is on a mission to change the way we design schools.
He champions the involvement of children and their communities in the creative process, and through the studio’s programme In The Making, he’s opening up careers in design to more young people.

If an agent is being pitched to you this term, here are five safeguarding questions you could put to the provider, and what a good answer sounds like.

Does it keep an immutable, timestamped log of every action, tool call, prompt and agent-to-agent message, in plain English, and who at the school can read it?
A good answer names the retention period, gives the school its own read access, and covers the agent's reasoning and tool calls as well as its outputs. A summary dashboard is a pitch. A full audit trail is an answer.

Which external endpoints can it reach, and what stops it from reaching others?
A good answer is a network allow-list enforced outside the model, so an agent that decided to work around an instruction would still hit a wall.
It also names where our data is processed and stored (UK or EEA), every sub-processor and model provider that sees it, a contractual bar on training with it, and a signed data processing agreement plus a DPIA template for children's data.

Which systems does it hold credentials for, with what scope, and which actions need a named person's approval before they execute?
A good answer is least privilege: per-task, read-only by default, with write access to the MIS, finance system or email granted case by case and gated behind human sign-off. Safeguarding records (CPOMS, MyConcern or equivalent) should be walled off entirely, with no read access at all.
How do we stop it, and what survives the stop?
A good answer is a single kill switch that revokes every credential and halts every sub-agent and scheduled task at once, with a statement of what the agent retains in memory afterwards.

It also explains the defence against prompt injection (instructions hidden in an email or document that the agent reads) and how that defence was tested, plus who is notified within the 72-hour ICO breach window.

Which model and version runs underneath, will we be told before it changes, and how has the whole system been tested over weeks on live tools rather than for the length of a demo?
A good answer includes long-horizon testing and red-teaming results, change control that notifies the school before a model swap, and independent assurance: Cyber Essentials Plus, ISO 27001 or SOC 2, and a penetration test dated this year.