At a European distribution centre, a critical incident occurs. Deliveries are at risk of being delayed. Within seconds, around 10 agents reorganise the entire logistics chain: reallocating stock, renegotiating delivery slots, readjusting priorities. The
twist? No human intervention was required: the agents in question are software powered by artificial intelligence. This scenario, already operational at several major industrial companies, illustrates the power of agentic AI. But it also raises a
legitimate concern: what happens if these agents make decisions with irreversible consequences? How can we ensure that a system capable of self-organisation remains under control? Michel Poujol, AI Expert Lead and CTO of the rAIse programme at Sopra
Steria, examines the orchestration mechanisms that prevent autonomy from degenerating into chaos.
A technological breakthrough?
Multi-agent systems are nothing new. As early as the 1990s, researchers and industrial players explored architectures in which multiple software entities collaborate to solve complex problems. But today's agentic AI represents far more than an incremental
evolution of these pioneering systems. "What distinguishes current multi-agent systems powered by generative AI from those of the 1990s is above all their ability to autonomously perform tasks sometimes very complex ones that were previously impossible
or impossible without human assistance," explains Michel Poujol.
This breakthrough is based on the unprecedented reasoning capabilities of language models embedded within agents. Unlike traditional expert systems that follow predefined rules, these language models can adapt to far less formal situations, potentially
including erratic ones. According to Michel Poujol, "they behave somewhat like a human brain, able to question itself when things do not go as planned." But this comparison with human intelligence has its limits: "This brain is often far less reliable
and powerful than that of a human especially an expert and safeguards are often necessary."
So what exactly are these agents used for? "Agentic AI considerably extends the range of possible uses of AI in general, and generative AI in particular, to use cases that would have been unimaginable just a few months ago," the expert emphasizes. Tasks
that previously required several weeks of work by specialized teams can now be accomplished in just a few hours by agentic AI applications driven by natural language, as if one were interacting with humans.
The Orchestrator at the heart
At the heart of any agentic system lies an agent that plays the crucial role of conductor. In most current applications, this orchestrator uses a language model to reason about the tasks to be performed and then select the appropriate tools to execute
them.
"Each task is specified in a prompt that defines its context. The selection of the tool to perform this task is generally carried out based on the verbal definition provided in its 'docstring' (typically a few sentences describing the tool, located in
the header of its code)," explains Michel Poujol.
"It is easy to understand that this selection process, based on the interpretation of definitions expressed in natural language, may be more or less reliable and robust," the expert acknowledges. "Fortunately, there are many ways to address this, for
example by using a rule-based AI or a hybrid AI combining generative AI and symbolic AI." Where generative models reason through pattern recognition in texts like prompts, symbolic AI operates through formal reasoning on predefined concepts and relationships.
This combination leverages the flexibility of language models while adding the reliability of deterministic logic.
Failure management is another major challenge. When an agent fails in its task or returns an incoherent response, the system must be able to react appropriately. This is where the concept of the Operational Design Domain (ODD) also sometimes referred
to, depending on standards, as OD (Operational Design) or OE (Operational Environment) comes into play, gaining importance in AI-based system engineering.
"This is a concept that will certainly become increasingly important in the engineering of AI-based systems or applications," says Michel Poujol.
The ODD is used to precisely define the functional scope of each component in a system. For agentic AI systems, it must therefore define the functional scope of each agent and each AI model. "Any behavior of an agent or an AI model outside its functional
scope must lead the system to execute another component, within another scope specifically defined to handle such situations," he explains.
This approach makes it possible to handle unforeseen or unpredictable situations, such as the well-known hallucinations and other erratic behaviors, which can therefore be avoided in most AI applications through good design. However, there remain AI applications
such as general-purpose generative or agentic AI for which it is impossible to fully delimit their functional scope, making these issues much harder to avoid.
Between autonomy and control: A delicate balance
Once this framework is established, the question remains of where to place human control points in a system where autonomous agents reason in parallel and exchange information. For Michel Poujol, the answer must once again be provided during the specification
and design phases: "These questions must be defined during the engineering process of this system, and more specifically during its specification and design phases."
Agent autonomy should never be synonymous with a lack of control: "We can and must include, in any system whether AI-based or not as many human interactions as necessary to meet the requirements of its specifications or regulatory constraints," insists
the expert. "Just because it is now possible to integrate models into these systems that can reason or make decisions in place of humans does not mean we are obliged to give them complete control."
Since 2020, as part of the confiance.ai program and now its follow-up initiatives, Sopra Steria has been contributing to the development of an AI-based systems engineering method that includes an approach ensuring that issues of human supervision are
systematically addressed and resolved in a structured manner.
A controlled deployment
Poor system specification inevitably leads to deployment failure. But for AI-based systems in general and agentic AI systems in particular another major cause of failure must be added: the adoption of such solutions. "Without adoption by the subject
matter experts responsible for expressing requirements, it is practically impossible to succeed in an AI project," our expert states.
Agentic AI promises spectacular productivity gains and opens up incredible prospects for automating complex tasks. As seen above, this promise cannot be fulfilled without a rigorous engineering methodology that integrates the necessary safeguards from
the design phase onward but that alone will not be enough to fully benefit from this revolution.
"In principle, no serious system should be deployed with a risk of erratic or even chaotic behavior above an acceptable threshold, which is practically always close to zero," Michel Poujol reminds us. But he adds: "however, continuing to tolerate absolutely
no risk for systems based on these new AI technologies would amount to depriving ourselves of their extraordinary potential."
He therefore advocates for the introduction of benefit–risk analyses into the engineering methods for such systems, as well as into the associated regulations and/or standards, while warning: "it will always be our responsibility to ensure that
this incredible potential is used for the best, and not for the worst."