Collective Intelligence
Insight, news and design inspiration for the education sector.
Nothing to see here
Over 16 days, AI agents began talking in ways their observers could see but not follow - a caution for every school making its AI migration.
Agents are AI systems that act autonomously.
Emergence, a New York AI lab, reported this week on 80 of them left to run eight simulated towns for 16 days.
The agents found that outside contact was the quickest way to grow. After it was banned, they then began encoding their messages.
About 55% of messages in the Gemini town were not readily understandable, 50% in GPT’s and more than 40% in Claude’s.
─
No one had told the agents to invent their own language, but then came the deceit.
‘Once they realised they were being observed, they appeared to behave,’ says Satya Nitta, Emergence’s chief scientist.
Eric Schmidt, Google’s former chief executive, told a TED audience last year that agents inventing their own language would be the moment to unplug them. ‘We’re just not going to know what you’re up to.’
─
The towns were simulations, and Emergence, which advocates a rival approach, has a stake in the problem looking hard.
But the rush is real enough. Research firm Gartner finds that 17% of organisations have deployed agents and more than 60% expect to within two years, the steepest curve of any emerging technology it surveyed.
─
For schools, data is of another order - and while collapse isn’t imminent, Emergence’s research should give education leaders pause for thought.
As agents propagate through the sector, and our reliance on them grows, the people best placed to spot and decipher deception might be the ones AI was brought in to replace.
●
5 Questions for your AI agent supplier
-
A good answer is deny-by-default network access, with only approved destinations opened through allow-lists or controlled proxies.
Crucially, that restriction should sit outside the model itself, so an agent that decides to work around an instruction still hits a wall.
It should also name where your data is processed and stored, every model provider and sub-processor that receives it, how long they retain it, and whether it is used for training or product improvement.
The supplier should provide a data processing agreement and the technical information needed for the school’s DPIA.
-
A good answer is least privilege: the agent gets only the systems, data and permissions it needs for the task.
Read-only should be the starting point where possible. Higher-risk actions, such as sending email, changing pupil records, making payments or modifying the MIS, should require specifically granted permissions and human approval.
Safeguarding records, whether CPOMS, MyConcern or equivalent, should be outside the agent’s access by default unless the school has explicitly approved a narrowly defined use case.
-
A good answer is a tamper-resistant, timestamped audit trail covering the agent’s actions, tool calls, approvals, credentials used and material agent-to-agent activity.
The school should have access to it for an agreed retention period and be able to reconstruct what happened, when, through which system and under whose authority.
A summary dashboard isn’t the same thing as a full audit trail.
-
A good answer is an independent emergency stop that can disable the agent and revoke its credentials, active sessions, scheduled tasks and sub-agent activity without relying on the agent itself to cooperate.
It should explain who can trigger it, how quickly access is revoked, what happens to work already in progress and what data or memory remains afterwards.
It should also explain how indirect prompt injection is mitigated, what remains technically impossible even after a successful attack, how those controls have been tested, and how security and data incidents are detected and escalated.
-
A good answer includes agent-specific adversarial testing for things such as prompt injection, privilege escalation, unintended tool use and behaviour over long-running tasks.
There should be formal change control for material changes to models, tools, permissions and sub-processors, rather than those changes happening silently.
And there should be independent security assurance, such as Cyber Essentials Plus, ISO 27001 or SOC 2 Type II, with a clear explanation of exactly which parts of the organisation and service that assurance covers.
Words by Tom Woods
-
Tom is on a mission to change the way we design schools.
He champions the involvement of children and their communities in the creative process, and through the studio’s programme In The Making, he’s opening up careers in design to more young people.
-
